Segmentation model training method, image processing method, device, equipment and medium

Abstract

The present application discloses a training method, image processing method, device, equipment and medium for a segmentation model, which belongs to the field of artificial intelligence technology. The training method of the segmentation model includes: obtaining a sample image and a point label; obtaining a first channel image; calling a first initial segmentation model to segment the sample image to obtain a first segmentation result; calling a second initial segmentation model to segment the first channel image to obtain a second segmentation result; based on the first segmentation result, the second segmentation result and the point label, obtaining a target loss function; using the target loss function to train the first initial segmentation model and the second initial segmentation model to obtain a first target segmentation model and a second target segmentation model. In this way, the two target segmentation models obtained by training can comprehensively consider the texture features and boundary features of the sub-image, the information considered is more comprehensive, the model training effect is higher, and it is conducive to improving the accuracy of the obtained target segmentation results.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a training method, image processing method, apparatus, device, and medium for a segmentation model. Background Art

[0002] With the development of artificial intelligence technology, image processing applications are becoming increasingly common. One such application involves processing an image containing numerous, densely packed sub-images of reference objects. A segmentation model is then used to segment these sub-images and obtain the target segmentation result. Because the reference objects are small, numerous, and densely packed, annotation costs are typically reduced by obtaining the corresponding point labels for sample images. In other words, the segmentation model used for image processing is trained based on the point labels of the sample images.

[0003] In related art, an initial segmentation model is directly called to segment the sample image to obtain a segmentation result. This initial segmentation model is then trained based on the segmentation result and point labels to obtain a target segmentation model for obtaining the target segmentation result. Because the sample image is used to provide the texture features of the sub-image, the segmentation model trained using this method focuses on texture features, which is relatively limited in the information considered. This results in poor training results, and the target segmentation results obtained using the trained target segmentation model are less accurate. Summary of the Invention

[0004] The present invention provides a segmentation model training method, image processing method, apparatus, device, and medium, which can be used to improve the training effect of the model and thereby improve the accuracy of the target segmentation results obtained using the trained model. The technical solution is as follows:

[0005] In one aspect, an embodiment of the present application provides a method for training a segmentation model, the method comprising:

[0006] Acquiring a sample image and acquiring point labels corresponding to the sample image, wherein the sample image includes a sub-image of a reference object dyed by a first dye component, the point labels corresponding to the sample image being determined based on reference points within a region within the sample image where the sub-image is located, and the sample image is used to provide texture features of the sub-image;

[0007] Based on the sample image, obtaining a first channel image corresponding to the first staining component, wherein the first channel image is used to provide a boundary feature of the sub-image;

[0008] Calling a first initial segmentation model to segment the sample image to obtain a first segmentation result; calling a second initial segmentation model to segment the first channel image to obtain a second segmentation result;

[0009] Based on the first segmentation result, the second segmentation result and the point label, a target loss function is obtained; the first initial segmentation model and the second initial segmentation model are trained using the target loss function to obtain a first target segmentation model and a second target segmentation model, and the first target segmentation model and the second target segmentation model are used to obtain the target segmentation result of the image to be processed.

[0010] An image processing method is also provided, the method comprising:

[0011] Obtaining an image to be processed, a first object segmentation model, and a second object segmentation model, wherein the image to be processed includes a subimage of a reference object stained by a first staining component, and the first object segmentation model and the second object segmentation model are trained based on a sample image, a first channel image corresponding to the first staining component obtained based on the sample image, and point labels corresponding to the sample image;

[0012] Based on the image to be processed, obtaining a second channel image corresponding to the first staining component;

[0013] Calling the first target segmentation model to segment the image to be processed to obtain a first segmentation result; calling the second target segmentation model to segment the second channel image to obtain a second segmentation result;

[0014] Based on the first segmentation result and the second segmentation result, a target segmentation result of the image to be processed is obtained.

[0015] In another aspect, a training apparatus for a segmentation model is provided, the apparatus comprising:

[0016] a first acquisition unit, configured to acquire a sample image and a point label corresponding to the sample image, wherein the sample image includes a sub-image of a reference object dyed by a first dye component, the point label corresponding to the sample image being determined based on a reference point within a region where the sub-image is located in the sample image, and the sample image is used to provide a texture feature of the sub-image;

[0017] a second acquiring unit, configured to acquire, based on the sample image, a first channel image corresponding to the first staining component, wherein the first channel image is used to provide a boundary feature of the sub-image;

[0018] A calling unit, configured to call a first initial segmentation model to segment the sample image to obtain a first segmentation result; and call a second initial segmentation model to segment the first channel image to obtain a second segmentation result;

[0019] A third acquisition unit is configured to acquire a target loss function based on the first segmentation result, the second segmentation result, and the point label;

[0020] A training unit is used to train the first initial segmentation model and the second initial segmentation model using the target loss function to obtain a first target segmentation model and a second target segmentation model, wherein the first target segmentation model and the second target segmentation model are used to obtain the target segmentation result of the image to be processed.

[0021] In one possible implementation, the third acquisition unit is used to obtain an auxiliary label corresponding to the sample image based on the point label, and the auxiliary label provides more supervisory information than the point label; and obtain the target loss function based on the first segmentation result, the second segmentation result and the auxiliary label.

[0022] In one possible implementation, the auxiliary label includes a first label; the third acquisition unit is further used to determine a reference point in the area where the sub-image is located in the sample image based on the point label; generate a Thiessen polygon corresponding to the reference point in the sample image; and acquire the first label based on the reference point and the Thiessen polygon, the first label including a sub-label for indicating that the pixel point located at the reference point belongs to the sub-image, a sub-label for indicating that the pixel point located on the Thiessen polygon does not belong to the sub-image, and a sub-label for indicating that the pixel point located at the reference point and outside the Thiessen polygon belongs to an uncertain pixel point.

[0023] In one possible implementation, the auxiliary label includes a second label; the third acquisition unit is further used to determine the reference point of the sub-image in the area where the sub-image is located in the sample image based on the point label; based on the reference point, obtain the reference features corresponding to each pixel point in the sample image; based on the reference features corresponding to each pixel point, cluster the each pixel point to obtain a clustering result, and the clustering result includes a first cluster cluster and a second cluster cluster; based on the clustering result, obtain the second label, and the second label includes a sub-label for indicating that the pixel points in the first cluster cluster belong to the sub-image, a sub-label for indicating that the pixel points in the second cluster cluster do not belong to the sub-image, and a sub-label for indicating that the pixel points other than the pixel points in the first cluster cluster and the second cluster cluster belong to uncertain pixel points.

[0024] In one possible implementation, the third acquisition unit is also used to, for a first pixel point, use the distance between the first pixel point and a target reference point as a distance feature corresponding to the first pixel point, where the target reference point is the reference point closest to the first pixel point, and the first pixel point is any one of the pixel points; based on the distance feature corresponding to the first pixel point and the color feature of the first pixel point, obtain the reference feature corresponding to the first pixel point.

[0025] In one possible implementation, the third acquisition unit is further used to obtain a cross-entropy loss function between the first segmentation result and the auxiliary label, and a cross-entropy loss function between the second segmentation result and the auxiliary label; based on the first segmentation result and the second segmentation result, obtain a first loss function; based on the cross-entropy loss function between the first segmentation result and the auxiliary label, the cross-entropy loss function between the second segmentation result and the auxiliary label, and the first loss function, obtain the target loss function.

[0026] In one possible implementation, the third acquisition unit is further used to obtain a cross entropy loss function between the first segmentation result and the second segmentation result, and a divergence loss function between the first segmentation result and the second segmentation result; and obtain the first loss function based on the cross entropy loss function and the divergence loss function.

[0027] In one possible implementation, the sample image has a reference object label, and the first acquisition unit is used to determine the area where the sub-image is located in the sample image based on the reference object label; determine the area center of the area where the sub-image is located in the sample image; determine the reference point based on the area center; and determine the point label corresponding to the sample image based on the reference point.

[0028] An image processing device is also provided, comprising:

[0029] A first acquisition unit is configured to acquire an image to be processed, a first object segmentation model, and a second object segmentation model, wherein the image to be processed includes a sub-image of a reference object stained by a first staining component, and the first object segmentation model and the second object segmentation model are trained based on a sample image, a first channel image corresponding to the first staining component obtained based on the sample image, and point labels corresponding to the sample image;

[0030] A second acquisition unit, configured to acquire a second channel image corresponding to the first staining component based on the image to be processed;

[0031] A calling unit, configured to call the first target segmentation model to segment the image to be processed to obtain a first segmentation result; and call the second target segmentation model to segment the second channel image to obtain a second segmentation result;

[0032] The third acquisition unit is configured to acquire a target segmentation result of the image to be processed based on the first segmentation result and the second segmentation result.

[0033] In one possible implementation, the image to be processed is an image presented in a first color space, and the second acquisition unit is used to map the image to be processed from the first color space to the color space corresponding to the dye in which the first dye component is located, and obtain the channel value corresponding to the pixel point in the image to be processed under the first dye component; based on the channel value corresponding to the pixel point in the image to be processed under the first dye component, obtain the second channel image.

[0034] In a possible implementation, the third acquisition unit is configured to acquire an average result of the first segmentation result and the second segmentation result, and use the average result as a target segmentation result of the image to be processed.

[0035] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the computer device implements any of the above-mentioned segmentation model training methods or image processing methods.

[0036] On the other hand, a computer-readable storage medium is also provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor so that the computer implements any of the above-mentioned segmentation model training methods or image processing methods.

[0037] In another aspect, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the above-described segmentation model training methods or image processing methods.

[0038] The technical solutions provided by the embodiments of the present application bring at least the following beneficial effects:

[0039] In an embodiment of the present application, during the model training process, in addition to paying attention to the sample image used to provide the texture features of the sub-image of the reference object, attention is also paid to the first channel image used to provide the boundary features of the sub-image. The two target segmentation models trained in this way can comprehensively consider the texture features and boundary features of the sub-image, the information considered is more comprehensive, and the model training effect is better, which is conducive to improving the accuracy of the target segmentation results obtained using the trained target segmentation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0041] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0042] Figure 2 This is a flow chart of a training method for a segmentation model provided in an embodiment of the present application;

[0043] Figure 3 is a schematic diagram of reference object labels and point labels corresponding to a sample image provided in an embodiment of the present application;

[0044] Figure 4 is a schematic diagram of a sample image and a first channel image provided in an embodiment of the present application;

[0045] Figure 5 is a schematic diagram of a first label provided in an embodiment of the present application;

[0046] Figure 6 is a schematic diagram of a second label provided in an embodiment of the present application;

[0047] Figure 7 is a schematic diagram of a training process of a segmentation model provided in an embodiment of the present application;

[0048] Figure 8 This is a flowchart of an image processing method provided by an embodiment of the present application;

[0049] Figure 9 This is a flow chart of a process for segmenting a sub-image of a cell nucleus in a tissue pathology image provided by an embodiment of the present application;

[0050] Figure 10 is a schematic diagram of a training device for a segmentation model provided in an embodiment of the present application;

[0051] Figure 11 is a schematic diagram of an image processing device provided in an embodiment of the present application;

[0052] Figure 12 This is a schematic diagram of the structure of a server provided in an embodiment of the present application;

[0053] Figure 13 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0055] In exemplary embodiments, the segmentation model training method and image processing method provided in the embodiments of the present application can be applied to the field of artificial intelligence technology. Next, artificial intelligence technology is introduced.

[0056] Artificial Intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and create new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0057] Artificial intelligence technology is a comprehensive discipline covering a wide range of fields, encompassing both hardware- and software-level technologies. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technologies, operating / interactive systems, and mechatronics. Artificial intelligence software technologies primarily encompass computer vision, speech processing, natural language processing, machine learning / deep learning, autonomous driving, and smart transportation. The segmentation model training method and image processing method provided in the embodiments of this application involve computer vision and machine learning technologies.

[0058] Computer vision (CV) technology is the study of how machines can "see." Specifically, it refers to the use of cameras and computers to replace the human eye in identifying and measuring objects, and further image processing to transform the computer-generated images into images more suitable for human observation or transmission to instrumentation. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / action recognition, three-dimensional object reconstruction, 3D (three-dimensional) technology, virtual reality, augmented reality, map construction, autonomous driving, and smart transportation. It also includes common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0059] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0060] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0061] In an exemplary embodiment, the training method of the segmentation model and the image processing method provided in the embodiments of the present application are implemented in a blockchain system. The sample images, point labels corresponding to the sample images, the first initial segmentation model, the second initial segmentation model, the first target segmentation model, the second target segmentation model, etc. involved in the training method of the segmentation model provided in the embodiments of the present application, as well as the images to be processed and the target segmentation results involved in the image processing method are all stored on the blockchain in the blockchain system for application by various node devices in the blockchain system to ensure the security and reliability of the data.

[0062] Figure 1 The schematic diagram of the implementation environment provided by the embodiment of the present application is shown. The implementation environment may include: a terminal 11 and a server 12.

[0063] The training method for the segmentation model provided in the embodiment of the present application can be executed by the terminal 11, can be executed by the server 12, or can be jointly executed by the terminal 11 and the server 12, and the embodiment of the present application does not limit this. In the case where the training method for the segmentation model provided in the embodiment of the present application is jointly executed by the terminal 11 and the server 12, the server 12 undertakes the primary computing work and the terminal 11 undertakes the secondary computing work; or, the server 12 undertakes the secondary computing work and the terminal 11 undertakes the primary computing work; or, the server 12 and the terminal 11 adopt a distributed computing architecture to perform collaborative computing.

[0064] The image processing method provided in the embodiment of the present application can be executed by the terminal 11, can be executed by the server 12, or can be executed jointly by the terminal 11 and the server 12, and the embodiment of the present application does not limit this. In the case where the image processing method provided in the embodiment of the present application is jointly executed by the terminal 11 and the server 12, the server 12 performs the primary computing work and the terminal 11 performs the secondary computing work; or, the server 12 performs the secondary computing work and the terminal 11 performs the primary computing work; or, the server 12 and the terminal 11 use a distributed computing architecture to perform collaborative computing.

[0065] The training method of the segmentation model and the image processing method provided in the embodiment of the present application can be executed by the same device or by different devices, and the embodiment of the present application is not limited to this.

[0066] In one possible implementation, the terminal 11 may be any electronic product that can interact with a user through one or more methods such as a keyboard, touchpad, touch screen, remote control, voice interaction, or handwriting device, such as a PC (Personal Computer), a mobile phone, a smart phone, a PDA (Personal Digital Assistant), a wearable device, a PPC (Pocket PC), a tablet computer, a smart car computer, a smart TV, a smart speaker, etc. The server 12 may be a single server, a server cluster consisting of multiple servers, or a cloud computing service center. The terminal 11 establishes a communication connection with the server 12 via a wired or wireless network.

[0067] Those skilled in the art should understand that the above-mentioned terminal 11 and server 12 are only examples. Other existing or future terminals or servers that are applicable to this application should also be included in the scope of protection of this application and are included here by reference.

[0068] Based on the above Figure 1 In the implementation environment shown, the present application embodiment provides a training method for a segmentation model, which is executed by a computer device, which can be a server 12 or a terminal 11, and the present application embodiment does not limit this. Figure 2 As shown, the training method of the segmentation model provided in the embodiment of the present application includes the following steps 201 to 204:

[0069] In step 201, a sample image and a point label corresponding to the sample image are obtained. The sample image includes a sub-image of a reference object dyed by a first dye component. The point label corresponding to the sample image is determined based on a reference point within an area where the sub-image is located in the sample image. The sample image is used to provide texture features of the sub-image.

[0070] A sample image refers to an image required for training a segmentation model. The sample image includes a sub-image of a reference object stained with a first staining component. The sub-image of the reference object refers to an image region of interest within the sample image. In an exemplary embodiment, the reference objects are small, numerous, and closely arranged, making it difficult to manually accurately label the sub-images of the reference objects. For example, the sample image is a histopathology image obtained by capturing a pathological tissue on a pathology slide, and the reference object is a specific tissue (such as a cell nucleus, cell, or blood vessel) within the histopathology image.

[0071] A pathology slide may have specific characteristics. For example, the pathology slide may be a pathology slide of biological tissue with a certain pathological condition, such as a pathology slide of animal or plant tissue with a specific pathological condition, or a pathology slide of tumor tissue in a certain part of the human body. Exemplarily, the pathology slide is a stained slide. Exemplarily, the pathology slide is a slide stained with HE (Hematoxylin-Eosin). By capturing an image of a certain area of ​​the pathology slide stained with HE, a histopathology image may be obtained, and the histopathology image may be used as a sample image.

[0072] In an embodiment of the present application, the sample image includes a sub-image of a reference object colored by a first coloring component. The sub-image of the reference object colored by the first coloring component is visually distinguishable from sub-images of uncolored objects and sub-images of objects colored by other coloring components. Exemplarily, in addition to the sub-image of the reference object colored by the first coloring component, the sample image may also include sub-images of other objects colored by a second coloring component, although this embodiment of the present application is not limited thereto. Exemplarily, the sample image is used to provide texture features of the sub-image of the reference object. The texture features of the sub-image of the reference object refer to the texture features of the sub-image of the reference object.

[0073] For example, the reference object is a cell nucleus, and the sample image is obtained by capturing an image of a pathology slide stained with HE stain. The HE stain includes an H staining component and an E staining component. The H staining component is used to stain the cell nucleus on the pathology slide blue, and the E staining component is used to stain the extracellular matrix and cytoplasm on the pathology slide pink. In other words, the H staining component is the first staining component in the HE stain, and the E staining component is the second staining component in the HE stain. The sample image includes a sub-image of the cell nucleus stained blue by the H staining component, and also includes a sub-image of the extracellular matrix and cytoplasm stained pink by the E staining component.

[0074] It should be noted that the sample images mentioned in the embodiments of this application refer to the sample images used to train the first initial segmentation model and the second initial segmentation model once. The number of sample images can be one or more, and this embodiment of the application is not limited to this. For example, the number of sample images is multiple to ensure the effectiveness of model training.

[0075] In an exemplary embodiment, the computer device obtains the sample image in the following manner: the computer device extracts the sample image from an image library.

[0076] In an exemplary embodiment, the computer device acquires sample images in the following manner: the computer device uses training images from a public dataset as sample images. For example, the computer device uses training images from the MoNuSeg (Multi-organ Nucleus Segmentation) dataset as sample images. The MoNuSeg dataset was obtained by accurately annotating histopathological images of different tumor organs from multiple patients at multiple hospitals and consists of 40x magnified HE-stained images downloaded from the TCGA (The Cancer Genome Atlas) archive. The MoNuSeg dataset contains 30 training images and 14 test images, each with a size of 1000×1000 (pixels). The training data covers seven different organs, including breast, liver, kidney, prostate, bladder, colon, and stomach, and contains approximately 22,000 complete nuclear boundary annotations. The test data covers seven different organs, including kidney, lung, colon, breast, bladder, prostate, and brain, and contains approximately 7,000 complete nuclear boundary annotations.

[0077] In an exemplary embodiment, the computer device obtains a sample image in the following manner: the computer device processes the original image captured by the image acquisition device (such as a microscope, an imported scanner, a domestic scanner, etc.) to obtain a sample image. In this case, the original image is extracted from an image library or manually uploaded, etc., which is not limited in this embodiment of the present application. The method of processing the original image includes but is not limited to cropping, data enhancement, etc., which is not limited in this embodiment of the present application. Exemplarily, the method of processing the original image is related to the computing power of the computer device and the input size required by the segmentation model.

[0078] The point label corresponding to the sample image is determined based on the reference point in the area where the sub-image of the reference object is located in the sample image. Since the reference point is located in the area where the sub-image is located in the sample image, the reference point can provide partial position information of the sub-image. Exemplarily, a sample image includes one or more sub-images of a reference object, and a sub-image of a reference object has a reference point in the area where the sample image is located, so as to roughly represent the position of the sub-image of the reference object using the reference point. Which point in the area where the sub-image of a reference object is located in the sample image is the reference point is set based on experience, or is manually specified, or is flexibly adjusted according to the actual application scenario, and the embodiment of the present application does not limit this. Exemplarily, the embodiment of the present application does not limit the size and shape of the reference point, for example, the reference point is a 1×1 (pixel) square point; or, the reference point is a 2×1 (pixel) rectangular point.

[0079] The point labels corresponding to the sample images include a sub-label for indicating that pixels located at the reference point belong to the sub-image of the reference object, and a sub-label for indicating that pixels not located at the reference point do not belong to the sub-image of the reference object. During the model training process of the embodiment of the present application, it is not necessary to obtain the pixel-level labels corresponding to the sample images; only the point labels corresponding to the sample images are required, which can effectively reduce the burden of manual annotation.

[0080] The embodiment of the present application does not limit the form of the point labels corresponding to the sample images. Exemplarily, the form of the point labels corresponding to the sample images is a numerical pair, which consists of the pixel position coordinates and the label value corresponding to the pixel. For example, the label value corresponding to the pixel located on the reference point (that is, the pixel belonging to the sub-image of the reference object) is 1, and the label value corresponding to the pixel not located on the reference point (that is, the pixel not belonging to the sub-image of the reference object) is 0. In this case, the form of the sub-label used to indicate that the pixel located on the reference point belongs to the sub-image of the reference object is a numerical pair including a label value of 1, and the form of the sub-label used to indicate that the pixel not located on the reference point does not belong to the sub-image of the reference object is a numerical pair including a label value of 0.

[0081] Exemplarily, the point label corresponding to the sample image is in the form of an image, which has the same size as the sample image. In the image, pixels located at the reference point (i.e., pixels belonging to the sub-image of the reference object) and pixels not located at the reference point (i.e., pixels not belonging to the sub-image of the reference object) are presented in different presentation methods. For example, pixels located at the reference point and pixels not located at the reference point are presented in different colors. For example, pixels located at the reference point are presented in white, and pixels not located at the reference point are presented in black. In this case, the sub-label used to indicate that the pixel located at the reference point belongs to the sub-image of the reference object is in the form of an image presented in white; and the sub-label used to indicate that the pixel not located at the reference point does not belong to the sub-image of the reference object is in the form of an image presented in black.

[0082] In one possible implementation, the point labels corresponding to the sample image are generated by a computer device based on reference points manually annotated in the sample image. In another possible implementation, the point labels corresponding to the sample image are stored with the sample image, so that the point labels corresponding to the sample image can be extracted simultaneously with the sample image.

[0083] In another possible implementation, if the sample image is obtained by processing the original image, the point labels corresponding to the sample image may be point labels obtained by correspondingly processing the point labels corresponding to the original image. The point labels corresponding to the original image are determined based on reference points within the region of the original image where the sub-image of the reference object is located. For example, if the sample image is obtained by processing the original image in a manner that includes cropping, the point labels corresponding to the sample image are obtained by extracting, from the point labels corresponding to the original image, those point labels that match the image retained after cropping.

[0084] In one possible implementation, the sample image has a reference object label. That is, the sample image and the reference object label are stored in correspondence, and the reference object label can be extracted simultaneously with the sample image. The reference object label indicates the region within the sample image where the sub-image of the reference object is located. Exemplarily, the reference object label is derived based on the boundaries of the sub-image of the reference object manually marked in the sample image. The reference object label takes the form of a numerical value pair or an image, which is not limited in this embodiment of the present application.

[0085] In the case where the sample image has a reference object label, the process of obtaining the point label corresponding to the sample image includes the following steps 2011 to 2014:

[0086] Step 2011: Based on the reference object label, determine the region where the sub-image is located in the sample image.

[0087] Since the reference object label is used to indicate the region where the sub-image of the reference object is located in the sample image, the region where the sub-image of the reference object is located in the sample image can be determined based on the reference object label.

[0088] Step 2012: Determine the center of the region where the sub-image is located in the sample image.

[0089] After determining the region in the sample image where the sub-image of the reference object is located, the region center of the region in the sample image where the sub-image of the reference object is located is determined. Exemplarily, the region center of the region in the sample image where the sub-image of the reference object is located refers to the centroid of the region in the sample image where the sub-image of the reference object is located. In an exemplary embodiment, if the sample image includes multiple sub-images of the reference object, the regions in the sample image where the sub-images of different reference objects are located are different, and the region centers of different regions are also different.

[0090] Step 2013: Determine the reference point based on the region center.

[0091] The center of the region can represent the center of the sub-image of the reference object. After determining the center of the region, the reference point is determined based on the center of the region. In an exemplary embodiment, the center of the region is directly used as the reference point within the region where the sub-image of the reference object is located in the sample image. This method is more efficient in determining the reference point.

[0092] In an exemplary embodiment, the center of the region is expanded to obtain a reference point. For example, the center of the region is expanded by 3 pixels; or, the center of the region is expanded by 5 pixels, etc. The reference point obtained by expanding the center of the region is clearly visible.

[0093] Step 2014: Based on the reference points, determine the point labels corresponding to the sample images.

[0094] Exemplarily, in the case where the point label corresponding to the sample image is in the form of an image, based on the reference point, the method for determining the point label corresponding to the sample image is: using different presentation methods to present the pixel points located at the reference point and other pixel points, to obtain the point label in the form of an image corresponding to the sample image. The presentation method is set based on experience or flexibly adjusted according to the application scenario, and the embodiments of the present application are not limited to this. Exemplarily, the presentation method is to present color, such as using white to present the pixel points located at the reference point and using black to present other pixel points. Exemplarily, the presentation method is to present stripes, such as using horizontal stripes to present the pixel points located at the reference point and using vertical stripes to present other pixel points.

[0095] Exemplarily, for the case where the point label corresponding to the sample image is a numerical pair, based on the reference point, the method for determining the point label corresponding to the sample image is: assign a first label value to the pixel point whose position coordinates are on the reference point, assign a second label value to the pixel point whose position coordinates are not on the reference point, and use the position coordinate-label value pairs of all pixels as point labels in the form of numerical pairs corresponding to the sample image. The first label value and the second label value are set based on experience, or flexibly adjusted according to the application scenario, and this is not limited in the embodiments of the present application. For example, the first label value is 1 and the second label value is 0; or, the first label value is 0 and the second label value is 1.

[0096] The point labels corresponding to the sample images can provide partial position information of the sub-image of the reference object. For example, the point labels corresponding to the sample images are called weak labels of the sample images, and the process of model training based on point labels is the process of model training based on weakly supervised learning.

[0097] For example, the reference object label in image form corresponding to the sample image is as follows: Figure 3 As shown in (a) in Figure 3In (a), the pixels within the region where the sub-image of the reference object is located are presented in white, and the pixels outside the region where the sub-image of the reference object is located are presented in black. Figure 3 The reference object label shown in (a) can be obtained as Figure 3 The sample image shown in (b) corresponds to the point label in image form. Figure 3 In (b), the pixel points located on the reference point are presented in white, and the pixel points not located on the reference point are presented in black.

[0098] In step 202, a first channel image corresponding to a first staining component is acquired based on a sample image, and the first channel image is used to provide a boundary feature of a sub-image.

[0099] Because the first coloring component is used to color the reference object, the first channel image corresponding to the first coloring component obtained based on the sample image can more prominently highlight the sub-image of the reference object and weaken the sub-images of other objects. Therefore, the first channel image is used to provide boundary features of the sub-image of the reference object. The boundary features of the sub-image of the reference object refer to the boundary features of the sub-image of the reference object. Boundary features and texture features are two different aspects of the sub-image of the reference object, and these two different aspects of features can complement each other.

[0100] In one possible implementation, the sample image is an image presented in a first color space. Based on the sample image, a method for obtaining a first channel image corresponding to the first color component is as follows: mapping the sample image from the first color space to the color space corresponding to the color component where the first color component is located, and obtaining the channel values ​​corresponding to each pixel point in the sample image under the first color component; and obtaining the first channel image based on the channel values ​​corresponding to each pixel point in the sample image under the first color component.

[0101] The first color space is the default color space used by the sample image. Exemplarily, the first color space refers to the RGB (Red, Green, Blue) color space. The dye containing the first coloring component may include one or more other coloring components in addition to the first coloring component, which is not limited in this embodiment of the present application. Exemplarily, when the reference object is a cell nucleus, the first coloring component is the H coloring component, and the dye containing the first coloring component is the HE dye. The HE dye includes the H coloring component and the E coloring component.

[0102] Different color spaces can be converted to each other to represent the same image using different color spaces. In one possible implementation, pixels in a sample image have corresponding channel values ​​under each color component in a first color space. The sample image is mapped from the first color space to the color space corresponding to the dye in which the first dye component is located. The method for obtaining the channel values ​​corresponding to each pixel in the sample image under the first dye component is as follows: determining the conversion benchmarks corresponding to each dye component in the dye in the first color space; based on the conversion benchmarks corresponding to each dye component in the dye in the first color space, converting the channel values ​​corresponding to each color component in the sample image under the first color space into the channel values ​​corresponding to each dye component in the dye in the sample image; and extracting the channel values ​​corresponding to each pixel in the sample image under the first dye component from the channel values ​​corresponding to each dye component in the dye in the sample image.

[0103] The conversion benchmarks corresponding to the various color components in the colorant in the first color space are pre-stored data and can be directly retrieved and utilized. Exemplarily, the conversion benchmarks corresponding to the color components in the first color space include sub-conversion benchmarks that match the various color components in the first color space.

[0104] Exemplarily, the first color space is an RGB color space, the colorant is an HE colorant, the color components in the RGB color space are respectively an R component, a G component, and a B component, and the color components in the HE colorant are respectively an H color component and an E color component. The conversion benchmark corresponding to the H color component in the HE colorant in the first color space includes sub-conversion benchmarks that match the R component, the G component, and the B component in the RGB color space, respectively. Exemplarily, the conversion benchmark corresponding to the H color component in the first color space is [0.644, 0.717, 0.267], where 0.644 is the sub-conversion benchmark that matches the R component, 0.717 is the sub-conversion benchmark that matches the G component, and 0.267 is the sub-conversion benchmark that matches the B component.

[0105] Similarly, the conversion benchmark corresponding to the E coloring component in the HE coloring agent in the first color space includes sub-conversion benchmarks that match the R component, G component, and B component in the RGB color space, respectively. Exemplarily, the conversion benchmark corresponding to the E coloring component in the first color space is [0.093, 0.954, 0.283], where 0.093 is the sub-conversion benchmark matching the R component, 0.954 is the sub-conversion benchmark matching the G component, and 0.283 is the sub-conversion benchmark matching the B component.

[0106] In one possible implementation, taking the first color space as an RGB color space and the dye as an HE dye as an example, based on the conversion benchmarks corresponding to the respective dye components in the dye in the first color space, a process of converting the channel values ​​corresponding to the respective color components of the pixel points in the sample image in the first color space into the channel values ​​corresponding to the respective dye components in the dye in the sample image is implemented based on Formula 1:

[0107]

[0108] Among them, HE i,j Including the channel value corresponding to the H dye component in the HE dye and the channel value corresponding to the E dye component of the pixel point with the position coordinate (i, j) in the sample image; RGB i,j It includes the channel value corresponding to the R component, the channel value corresponding to the G component, and the channel value corresponding to the B component of the pixel point with the position coordinate (i, j) in the sample image in the RGB color space; the symbol "+" indicates the pseudo-inverse of the matrix.

[0109] Formula 1 is described using a pixel with coordinates (i, j) in a sample image as an example. Based on Formula 1, the channel values ​​corresponding to all pixels in the sample image under each dye component in the dye can be determined.

[0110] After obtaining the channel values ​​corresponding to the pixel points in the sample image under each color component in the dye, the channel value corresponding to the pixel points in the sample image under the first color component is extracted from the channel values ​​corresponding to the pixel points in the sample image under each color component in the dye. Thus, the channel value corresponding to the pixel points in the sample image under the first color component is obtained.

[0111] After obtaining the channel values ​​corresponding to the pixels in the sample image under the first color component, a first channel image is obtained based on the channel values ​​corresponding to the pixels in the sample image under the first color component. The pixel value of a pixel at a certain position in the first channel image is the channel value corresponding to the pixel at the same position in the sample image under the first color component. Exemplarily, the first channel image is a grayscale image obtained based on the channel values ​​corresponding to the pixels in the sample image under the first color component.

[0112] The process of obtaining the first channel image can be regarded as a process of applying color decomposition technology, constructing a color representation model based on the Beer-Lambert law, mapping the RGB color space to the HE staining color space, extracting the hematoxylin component from the HE staining sample image, and obtaining a component map with a clearer boundary of the sub-image of the reference object.

[0113] For example, taking the reference object as a cell nucleus, the sample image and the first channel image are respectively as follows: Figure 4 It should be noted that the sample image is a color image, the sub-image of the cell nucleus in the sample image is stained blue by the H staining component, and the sub-image of the extracellular matrix and cytoplasm in the sample image is stained pink by the E staining component. Figure 4 (a) in the figure shows the image after the sample image is converted into a grayscale image. Figure 4 As can be seen from (a) and (b) in the figure, the sample image presents the texture features of the sub-image of the cell nucleus more fully than the first channel image, and the first channel image presents the boundary features of the sub-image of the cell nucleus more fully than the sample image. Figure 4 The sub-image of the cell nucleus (black area) in (b) has a clearer boundary.

[0114] In one possible implementation, if the sample image is obtained by processing the original image, the method for obtaining the first channel image corresponding to the first color component based on the sample image is: based on the original image, obtain the third channel image corresponding to the first color component; process the third channel image using a target processing method, and use the resulting image as the first channel image corresponding to the first color component. The target processing method refers to the processing method used to process the original image to obtain the sample image. For example, the target processing method is to crop the upper left quarter area and add noise to the cropped image.

[0115] In step 203, the first initial segmentation model is called to segment the sample image to obtain a first segmentation result; and the second initial segmentation model is called to segment the first channel image to obtain a second segmentation result.

[0116] The first initial segmentation model and the second initial segmentation model are segmentation models to be trained, and are both used to segment the image to segment out a sub-image of a reference object in the image. The model structure of the first initial segmentation model may be the same as or different from the model structure of the second initial segmentation model, and the embodiments of the present application do not limit this. Exemplarily, the model structure of the first initial segmentation model and the model structure of the second initial segmentation model are both Link Net (connection network); or, the model structure of the first initial segmentation model and the model structure of the second initial segmentation model are both FC-Dense Net (Fully Convolutional Dense Net); or, the model structure of the first initial segmentation model is FC-Dense Net, and the model structure of the second initial segmentation model is Mobile Net (mobile network), etc.

[0117] The sample image is input into a first initial segmentation model, which segments the sample image and outputs a first segmentation result. The first segmentation result indicates the region in the sample image where the sub-image of the reference object predicted by the first initial segmentation model is located.

[0118] The embodiment of the present application does not limit the form of the first segmentation result. Exemplarily, the first segmentation result is in the form of a two-channel probability map, where one channel's probability map is used to display the probability that each pixel point belongs to the sub-image of the reference object, and the other channel's probability map is used to display the probability that each pixel point does not belong to the sub-image of the reference object. Exemplarily, the first segmentation result is in the form of a value pair, where one pixel point corresponds to one value pair, and the value pair corresponding to one pixel point includes the position coordinates of the pixel point, the probability that the pixel point belongs to the sub-image of the reference object, and the probability that the pixel point does not belong to the sub-image of the reference object.

[0119] For example, the goal of image segmentation in the embodiment of the present application is to determine whether each pixel in the image belongs to a sub-image of a reference object or does not belong to a sub-image of the reference object through a segmentation model. For each pixel, this is a binary classification problem.

[0120] The first channel image is input into the second initial segmentation model, and the second initial segmentation model segments the first channel image and outputs a second segmentation result. The second segmentation result is used to indicate the area in the first channel image where the sub-image of the reference object in the first channel image is located, as predicted by the second initial segmentation model. Since the first channel image and the sample image have the same size and the first channel image is based on the channel image obtained from the sample image, the area in the first channel image where the sub-image of the reference object in the first channel image is located is the same as the area in the sample image where the sub-image of the reference object in the sample image is located. In other words, the second segmentation result can be used to indicate the area in the sample image where the sub-image of the reference object in the sample image is located, as predicted by the second initial segmentation model.

[0121] In an exemplary embodiment, the form of the second segmentation result is the same as that of the first segmentation result, so as to facilitate rapid calculation of the loss function required for training the model.

[0122] It should be noted that the number of sample images can be one or more, and this is not limited in the embodiments of the present application. A first channel image can be obtained based on each sample image, that is, the number of first channel images is the same as the number of sample images. In the case where there are multiple sample images, in step 203, it is necessary to call the first initial segmentation model to segment each sample image separately to obtain multiple first segmentation results; and call the second initial segmentation model to segment each first channel image separately to obtain multiple second segmentation results. The multiple first segmentation results correspond one to one to the multiple second segmentation results.

[0123] In step 204, a target loss function is obtained based on the first segmentation result, the second segmentation result and the point label; the first initial segmentation model and the second initial segmentation model are trained using the target loss function to obtain a first target segmentation model and a second target segmentation model, and the first target segmentation model and the second target segmentation model are used to obtain the target segmentation result of the image to be processed.

[0124] After obtaining the first segmentation result and the second segmentation result, the first initial segmentation model and the second initial segmentation model are trained based on the first segmentation result, the second segmentation result, and the point labels. The process of training the first initial segmentation model and the second initial segmentation model based on the first segmentation result, the second segmentation result, and the point labels is as follows: based on the first segmentation result, the second segmentation result, and the point labels, a target loss function is obtained; and the first initial segmentation model and the second initial segmentation model are trained using the target loss function.

[0125] The number of sample images is one or more. For the case where the number of sample images is one, the number of the first segmentation result and the second segmentation result are both one. Then, the target loss function is directly obtained based on the first segmentation result, the second segmentation result and the point label. For the case where the number of sample images is multiple, the number of the first segmentation result and the second segmentation result are also multiple, and the first segmentation result and the second segmentation result correspond one to one. In this case, the method of obtaining the target loss function is: based on each first segmentation result, each second segmentation result corresponding to the first segmentation result and the point label, a sub-loss function is obtained; and the average value of all the obtained sub-loss functions is used as the target loss function. The method of obtaining a sub-loss function when the number of sample images is multiple is the same as the method of obtaining the target loss function when the number of sample images is one. The embodiment of the present application is described by taking the number of sample images as one as an example.

[0126] In an exemplary embodiment, the point labels contain partial position information of the sub-image of the reference object. Compared to the entire sample image, the point labels can provide less supervisory information, which is not conducive to model training. Therefore, in one possible implementation, based on the first segmentation result, the second segmentation result, and the point labels, the process of obtaining the target loss function includes the following steps 2041 and 2042:

[0127] Step 2041: Based on the point labels, obtain auxiliary labels corresponding to the sample images. The auxiliary labels provide more supervisory information than the point labels.

[0128] Obtaining auxiliary labels that can provide more supervision information based on point labels is conducive to using auxiliary labels to obtain loss functions that can provide more accurate guidance. Auxiliary labels can include one or more labels, which is not limited in the embodiment of the present application. Each label in the auxiliary labels is obtained based on point labels. For example, auxiliary labels include but are not limited to the following three situations:

[0129] Case 1: The auxiliary tag includes the first tag.

[0130] In this case 1, the process of obtaining the auxiliary tag based on the point tag is the process of obtaining the first tag based on the point tag. In one possible implementation, the process of obtaining the first tag based on the point tag includes the following steps A to C:

[0131] Step A: Based on the point labels, determine the reference points in the region where the sub-image is located in the sample image.

[0132] Because the point label corresponding to the sample image is determined based on the reference point within the region where the sub-image of the reference object is located in the sample image, the reference point within the region where the sub-image is located in the sample image can be determined based on the point label. For example, if the point label is in the form of an image, the image may display pixels located at the reference point in white and other pixels in black. In this case, the reference point can be determined based on the white region displayed in the image.

[0133] Step B: Generate Thiessen polygons corresponding to reference points in the sample image.

[0134] Thiessen polygons are continuous polygons composed of the perpendicular bisectors of the lines connecting two adjacent points. After determining the reference point, the adjacent reference points can be connected, and then the Thiessen polygons corresponding to the reference points can be obtained based on the perpendicular bisectors of the lines connecting the adjacent reference points. Thiessen polygons divide the plane where the sample image is located into ideal polygonal blocks. In the embodiment of the present application, the pixel points located on the Thiessen polygons are considered to be pixel points of the sub-image that do not belong to the reference object. In other words, Thiessen polygons can provide reliable negative samples, which can help highly clustered sub-images of the reference object not overlap during segmentation.

[0135] Step C: Based on the reference point and the Thiessen polygon, obtain a first label, the first label including a sub-label for indicating that the pixel point located on the reference point belongs to the sub-image, a sub-label for indicating that the pixel point located on the Thiessen polygon does not belong to the sub-image, and a sub-label for indicating that the pixel point located outside the reference point and the Thiessen polygon belongs to an uncertain pixel point.

[0136] After determining the reference point and generating the corresponding Thiessen polygon, a first label is obtained based on the reference point and the Thiessen polygon. This first label can be used to determine which pixels in the sample image belong to the sub-image of the reference object, which pixels do not belong to the sub-image of the reference object, and which pixels are uncertain pixels, thereby providing powerful supervision information for the model training process.

[0137] Based on the reference point and the Thiessen polygon, the method of obtaining the first label is related to the form of the first label. For example, if the first label is in the form of an image, the pixels located on the reference point, the pixels located on the Thiessen polygon, and other pixels with the same size as the sample image will be presented in different presentation methods as the first label. For example, the first label in the form of an image is as follows: Figure 5 As shown, in Figure 5 In the image shown, pixels at the reference point are rendered in white, pixels on Thiessen polygons are rendered in gray, and pixels outside the reference point and Thiessen polygons are rendered in black. In this case, the sub-label indicating that a pixel at the reference point belongs to a sub-image is rendered in white; the sub-label indicating that a pixel on a Thiessen polygon does not belong to a sub-image is rendered in gray; and the sub-label indicating that a pixel at the reference point and outside the Thiessen polygons is an undefined pixel is rendered in black.

[0138] Exemplarily, the first label is in the form of a numerical pair. Based on the reference point and the Thiessen polygon, the first label is obtained by assigning a first value to the pixel located at the reference point, a second value to the pixel located on the Thiessen polygon, and a third value to the pixel located outside the reference point and the Thiessen polygon, and using the position coordinate-label value pair of each pixel as the first label in the form of a numerical pair. In this case, the sub-label used to indicate that the pixel located at the reference point belongs to the sub-image is in the form of a numerical pair including a label value of the first value; the sub-label used to indicate that the pixel located on the Thiessen polygon does not belong to the sub-image is in the form of a numerical pair including a label value of the second value; the sub-label used to indicate that the pixel located at the reference point and outside the Thiessen polygon belongs to an uncertain pixel is in the form of a numerical pair including a label value of the third value. The first value, the second value, and the third value are set based on experience or flexibly adjusted according to the application scenario, and the embodiments of the present application do not limit this. For example, the first value is 1, the second value is 0, and the third value is -1.

[0139] In an exemplary embodiment, the first label obtained based on the reference point and the Thiessen polygon may also be referred to as a point-edge label.

[0140] Case 2: The auxiliary tag includes a second tag.

[0141] In this case 2, the process of obtaining the auxiliary tag based on the point tag is the process of obtaining the second tag based on the point tag. In one possible implementation, the process of obtaining the second tag based on the point tag includes the following steps a to d:

[0142] Step a: Based on the point labels, determine the reference points within the region where the sub-image is located in the sample image.

[0143] The implementation method of step a refers to step A under the above situation 1 and will not be repeated here.

[0144] Step b: Based on the reference points, obtain the reference features corresponding to each pixel point in the sample image.

[0145] The reference features corresponding to pixels are the features used to cluster pixels. These features are determined based on and are related to the reference points. The principle for obtaining the reference features corresponding to each pixel in a sample image is the same. Taking any one pixel (called the first pixel) as an example, this article describes how to obtain the reference features corresponding to that first pixel based on the reference point.

[0146] In one possible implementation, based on the reference point, the method for obtaining the reference feature corresponding to the first pixel point is: taking the distance between the first pixel point and the target reference point as the distance feature corresponding to the first pixel point, and the target reference point is the reference point closest to the first pixel point; based on the distance feature corresponding to the first pixel point and the color feature of the first pixel point, obtain the reference feature corresponding to the first pixel point.

[0147] By calculating the distance between the first pixel and each reference point, the target reference point closest to the first pixel can be determined. The embodiment of the present application does not limit the method for calculating the distance between two points. For example, the Euclidean distance between the two points is calculated. After determining the target reference point, the distance between the first pixel and the target reference point is used as the distance feature corresponding to the first pixel.

[0148] Since different pixels in the sample image may be dyed into different colors, for example, when the reference object is a cell nucleus and the dye is an HE dye, the pixels of the sub-image belonging to the cell nucleus in the sample image are dyed into blue, and the pixels of the sub-image belonging to the extracellular matrix and cytoplasm are dyed into pink. Therefore, in the process of obtaining the reference feature corresponding to the first pixel, in addition to considering the distance feature, the color feature of the first pixel is also considered. Exemplarily, the color feature of the first pixel refers to the color value of the first pixel under each color component of the first color space. Exemplarily, if the first color space is an RBG color space, the color feature of the first pixel can be expressed as (r i ,g i ,b i ).

[0149] The reference feature corresponding to the first pixel is obtained based on the distance feature corresponding to the first pixel and the color feature of the first pixel. For example, the first pixel x i The corresponding distance feature is recorded as d i , the first pixel x i The color feature is recorded as (r i ,g i ,b i ), then the first pixel x i The corresponding reference feature can be expressed as f xi =(d i ,r i ,g i ,b i ).

[0150] The color difference between the pixels of the sub-image belonging to the reference object and the pixels of the sub-image not belonging to the reference object is large. Therefore, in the process of obtaining the reference features corresponding to the pixels, the color features of the pixels are taken into account so that the color features can be used to divide the pixels of different categories into different clusters to a certain extent. In addition, in addition to considering the color features, the distance features are also considered to avoid the adverse effects of uneven coloring on the clustering results. Under the premise of similar colors, the pixels of the sub-image belonging to the same reference object should be close enough to the reference points corresponding to the sub-image of the reference object. The pixels of the sub-image not belonging to the reference object not only have a large color difference with the pixels of the sub-image belonging to the reference object, but also have a sufficiently large distance from the reference points. Therefore, using the distance between the pixel and the nearest reference point as the distance feature can use the distance feature to divide the pixels of different categories into different clusters to a certain extent.

[0151] By referring to the method of obtaining the reference feature corresponding to the first pixel point, the reference features corresponding to each pixel point in the sample image can be obtained.

[0152] Step c: clustering each pixel point based on the reference features corresponding to each pixel point to obtain a clustering result, which includes a first cluster and a second cluster.

[0153] After obtaining the reference features corresponding to the respective pixel points, the respective pixel points are clustered based on the reference features corresponding to the respective pixel points.

[0154] In an exemplary embodiment, before clustering each pixel based on the reference features corresponding to each pixel, the number of clusters K (K is an integer not less than 1) to be ultimately obtained is first specified, and then the clustering of each pixel is implemented based on the K-Means clustering method. In an exemplary embodiment, the process of clustering each pixel based on the K-Means clustering method is as follows: for a given set of N (N is an integer not less than 1) pixel points (x1, x2, ..., x N ) of the sample image x, according to the reference feature f corresponding to the pixel point xi (i=1,2,…,N) divides N pixels into K clusters (C=C1,C2,…C K ), so that the difference in reference features between pixels of the same class is as small as possible, and the difference in reference features between pixels of different classes is as large as possible. The goal of the K-Means clustering method is to minimize the squared error E, which is calculated based on Formula 2:

[0155]

[0156] Among them, μ jRepresents cluster C j The average reference feature of the pixels in μ j The calculation formula is:

[0157] In an embodiment of the present application, K is an integer not less than 2, so that the pixels are divided into at least two categories of pixels, pixels of the sub-image belonging to the reference object and pixels of the sub-image not belonging to the reference object, through clustering, thereby obtaining a clustering result including a first cluster and a second cluster. The first cluster is the cluster in which the pixels contained in each cluster in the clustering result are closest to the reference point, and the second cluster is the cluster farthest from the reference point in each cluster in the clustering result. Based on this approach, it is considered that the first cluster is a cluster composed of pixels of the sub-image belonging to the reference object, and the second cluster is a cluster composed of pixels of the sub-image not belonging to the reference object. Exemplarily, the way to calculate the distance between the pixels contained in the cluster and the reference point is: the average value of the distance features corresponding to each pixel in the cluster is used as the distance between the pixels contained in the cluster and the reference point.

[0158] Step d: Based on the clustering results, obtain a second label, which includes a sub-label for indicating that the pixel points in the first cluster belong to the sub-image, a sub-label for indicating that the pixel points in the second cluster do not belong to the sub-image, and a sub-label for indicating that the pixel points other than the pixel points in the first cluster and the second cluster belong to uncertain pixel points.

[0159] The second label is obtained based on the clustering results. This second label can be used to determine which pixels in the sample image belong to the sub-image of the reference object, which pixels do not belong to the sub-image of the reference object, and which pixels are uncertain pixels, thereby providing powerful supervisory information for the model training process. For example, the area containing pixels of the sub-image that do not belong to the reference object is referred to as the background, and the area containing pixels of uncertain pixels is referred to as the uncertain area at the junction of the sub-image of the reference object and the background.

[0160] Based on the clustering results, the method for obtaining the second label is related to the form of the second label. For example, if the second label is in the form of an image, an image of the same size as the sample image, which is presented using different presentation methods and includes pixels in the first cluster, pixels in the second cluster, and pixels other than pixels in the first and second clusters, is used as the second label.

[0161] In an exemplary embodiment, in the process of presenting the pixel points in the first cluster, the pixel points in the second cluster, and the pixel points other than the pixel points in the first cluster and the second cluster using different presentation methods, the pixel points in the first cluster, the pixel points in the second cluster, and the pixel points other than the pixel points in the first cluster and the second cluster can be directly presented using different presentation methods, or the pixel points obtained by performing a morphological opening operation on the pixel points in the first cluster, the pixel points obtained by performing a morphological opening operation on the pixel points in the second cluster, and the pixel points other than the pixel points in the first cluster and the second cluster can be presented using different presentation methods.

[0162] For example, a second label in the form of an image such as Figure 6 As shown. Figure 6 In the example, white is used to represent the pixels in the first cluster, gray is used to represent the pixels in the second cluster, and black is used to represent the pixels other than the pixels in the first and second clusters. Figure 6 Pixels in the white area belong to the sub-image of the reference object, pixels in the gray area do not belong to the sub-image of the reference object, and pixels in the black area are undefined pixels. In this case, the sub-label indicating that the pixels in the first cluster belong to the sub-image is in the form of an image presented in white; the sub-label indicating that the pixels in the second cluster do not belong to the sub-image is in the form of an image presented in gray; and the sub-label indicating that the pixels other than those in the first and second clusters belong to undefined pixels is in the form of an image presented in black.

[0163] Exemplarily, the second label is in the form of a numerical pair. Based on the clustering results, the second label is obtained by assigning a fourth value to the pixels in the first cluster, a fifth value to the pixels in the second cluster, and a sixth value to the pixels other than the pixels in the first cluster and the second cluster, and using the position coordinate-label value pair of each pixel as the second label in the form of a numerical pair. In this case, the sub-label used to indicate that the pixel in the first cluster belongs to the sub-image is in the form of a numerical pair including a label value of the fourth value; the sub-label used to indicate that the pixel in the second cluster does not belong to the sub-image is in the form of a numerical pair including a label value of the fifth value; and the sub-label used to indicate that the pixel other than the pixel in the first cluster and the second cluster belongs to an uncertain pixel is in the form of a numerical pair including a label value of the sixth value.

[0164] The fourth value, the fifth value, and the sixth value are set based on experience or flexibly adjusted according to the application scenario, and are not limited in this embodiment of the present application. For example, the fourth value is the same as the first value (e.g., 1), the fifth value is the same as the second value (e.g., 0), and the sixth value is the same as the third value (e.g., -1).

[0165] Clustering based on the K-Means clustering method is unsupervised clustering. This embodiment of the present application obtains a second label based on the color and distance feature differences between pixels in the sub-image belonging to the reference object and pixels in the sub-image not belonging to the reference object. For example, the second label can also be referred to as a cluster label.

[0166] Case 3: The auxiliary tag includes a first tag and a second tag.

[0167] In this case 3, the process of obtaining auxiliary labels based on point labels is the same as the process of obtaining the first label and the second label based on the point labels. The first label is obtained based on the reference point and Thiessen polygons and cannot effectively provide the shape information of the sub-image of the reference object. The second label is obtained through clustering and can provide some shape information. The target loss function obtained by using the auxiliary labels including the first label and the second label can provide more reliable constraint information.

[0168] In a possible implementation, the process of obtaining the first label and the second label based on the point label refers to steps A to C in case 1 and steps a to d in case 2, which are not repeated here.

[0169] Step 2042: Obtain a target loss function based on the first segmentation result, the second segmentation result, and the auxiliary label.

[0170] After obtaining auxiliary labels based on point labels, the target loss function is obtained based on the first segmentation result, the second segmentation result, and the auxiliary labels. Because the auxiliary labels provide more supervisory information than the point labels, using the auxiliary labels as weak supervisory information to obtain the target loss function can provide more effective guidance for the model training process.

[0171] In one possible implementation, the implementation process of step 2042 includes the following steps 20421 to 20423:

[0172] Step 20421: Obtain a cross entropy loss function between the first segmentation result and the auxiliary label, and a cross entropy loss function between the second segmentation result and the auxiliary label.

[0173] The cross-entropy loss function between the first segmentation result and the auxiliary label is used to reflect the difference between the first segmentation result and the auxiliary label. If there are multiple first sample images, a cross-entropy loss function needs to be obtained based on each first segmentation result and auxiliary label. This embodiment of the application is described as an example with one sample image.

[0174] It should be noted that the auxiliary label includes at least one of the first label and the second label. If the auxiliary label only includes the first label or only includes the second label, the cross entropy loss function between the first segmentation result and the auxiliary label is directly obtained.

[0175] In an exemplary embodiment, in the process of obtaining the cross-entropy loss function between the first segmentation result and the auxiliary label, the cross-entropy loss function is calculated based on the valid pixels in the sample image, and the invalid pixels in the sample image are ignored. Exemplarily, the valid pixels in the sample image refer to the pixels of the sub-image belonging to the reference object indicated by the sub-label in the auxiliary label (also referred to as the pixels representing the positive sample) and the pixels of the sub-image not belonging to the reference object (also referred to as the pixels representing the negative sample), and the invalid pixels in the sample image refer to the pixels that are indicated by the sub-label in the auxiliary label as being uncertain pixels.

[0176] For example, the auxiliary tag is Figure 5 The first label shown, or Figure 6 As an example of the second label shown, the valid pixel points in the sample image are Figure 5 and Figure 6 The white area in the image and the pixels in the gray area, the invalid pixels in the sample image are Figure 5 and Figure 6 Pixels in the black area.

[0177] For example, the cross entropy loss function between the first segmentation result and a certain label is calculated based on Formula 3:

[0178]

[0179] Among them, L ce represents the cross entropy loss function between the first segmentation result and a certain label; y represents a certain label; Indicates the first segmentation result.

[0180] For the case where the auxiliary label includes the first label or the second label, the cross entropy loss function between the first segmentation result and the auxiliary label refers to the sum or weighted sum of the cross entropy loss function between the first segmentation result and the first label and the cross entropy loss function between the first segmentation result and the second label. For example, for the case where the auxiliary label includes the first label and the second label, the cross entropy loss function between the first segmentation result and the auxiliary label can be calculated based on Formula 4:

[0181]

[0182] in, represents the cross entropy loss function between the first segmentation result and the auxiliary label; represents the cross entropy loss function between the first segmentation result and the first label; Represents the cross entropy loss function between the first segmentation result and the second label.

[0183] The cross entropy loss function between the second segmentation result and the auxiliary label is used to reflect the difference between the second segmentation result and the auxiliary label. The implementation method of obtaining the cross entropy loss function between the second segmentation result and the auxiliary label refers to the implementation method of obtaining the cross entropy loss function between the first segmentation result and the auxiliary label, which will not be repeated here.

[0184] For example, when the auxiliary label includes a first label and a second label, the cross entropy loss function between the second segmentation result and the auxiliary label can be calculated based on Formula 5:

[0185]

[0186] in, represents the cross entropy loss function between the second segmentation result and the auxiliary label; represents the cross entropy loss function between the second segmentation result and the first label; Represents the cross entropy loss function between the second segmentation result and the second label.

[0187] Step 20422: Based on the first segmentation result and the second segmentation result, obtain a first loss function.

[0188] The first segmentation result is predicted by the first initial segmentation model based on the input sample image, and the second segmentation result is predicted by the second initial segmentation model based on the input first channel image. The first loss function is a loss function obtained based on the segmentation results output by the two models, and is used to collaboratively train the first initial segmentation model and the second initial segmentation model. The embodiment of the present application uses two models for collaborative training in order to effectively combine the complementary information between the sample image and the first channel image. Therefore, the predicted outputs of the two segmentation models during the training process must be as consistent as possible, while preventing the two segmentation models from deceiving each other.

[0189] In one possible implementation, the process of obtaining the first loss function based on the first segmentation result and the second segmentation result is: obtaining a cross entropy loss function between the first segmentation result and the second segmentation result, and a divergence loss function between the first segmentation result and the second segmentation result; and obtaining the first loss function based on the cross entropy loss function and the divergence loss function.

[0190] The cross entropy loss function between the first segmentation result and the second segmentation result is used to reflect the difference between the first segmentation result and the second segmentation result, so that the two segmentation models have consistent feature distributions. For example, the cross entropy loss function between the first segmentation result and the second segmentation result is calculated based on Formula 6:

[0191]

[0192] in, represents the cross entropy loss function between the first segmentation result and the second segmentation result; y′2 represents the label obtained after the second segmentation result is transformed; y1 represents the first segmentation result. Of course, in an exemplary embodiment, y′2 can also represent the label obtained after the first segmentation result is transformed, and y1 can represent the second segmentation result.

[0193] Exemplarily, the segmentation result is transformed to obtain a label in the following manner: based on the probability value of each pixel point indicated in the segmentation result belonging to the sub-image of the reference object and the probability value of the sub-image not belonging to the reference object, determine whether each pixel point belongs to the sub-image of the reference object; and use the label used to indicate whether each pixel point belongs to the sub-image of the reference object as the label obtained after transforming the segmentation result.

[0194] The embodiments of the present application do not limit the method for determining whether each pixel belongs to a sub-image of the reference object based on the probability value of each pixel indicated in the segmentation result belonging to the sub-image of the reference object and the probability value of each pixel not belonging to the sub-image of the reference object. For example, if the probability value of a certain pixel belonging to the sub-image of the reference object is greater than the probability value of not belonging to the sub-image of the reference object, then the pixel is considered to belong to the sub-image of the reference object; if the probability value of a certain pixel belonging to the sub-image of the reference object is less than the probability value of not belonging to the sub-image of the reference object, then the pixel is considered not to belong to the sub-image of the reference object; if the probability value of a certain pixel belonging to the sub-image of the reference object is equal to the probability value of not belonging to the sub-image of the reference object, then the pixel is considered to be an uncertain pixel.

[0195] For example, if the probability value of a certain pixel belonging to the sub-image of the reference object is greater than the probability value of not belonging to the sub-image of the reference object and the probability value of the pixel belonging to the sub-image of the reference object is greater than a first threshold, then the pixel is considered to belong to the sub-image of the reference object; if the probability value of a certain pixel belonging to the sub-image of the reference object is less than the probability value of not belonging to the sub-image of the reference object and the probability value of the pixel belonging to the sub-image of the reference object is less than a second threshold, then the pixel is considered not to belong to the sub-image of the reference object; if the relationship between the probability value of a certain pixel belonging to the sub-image of the reference object and the probability value of not belonging to the sub-image of the reference object is other, then the pixel is considered to be an uncertain pixel. The first threshold and the second threshold are set based on experience or flexibly adjusted according to the application scenario. For example, the first threshold is 0.7 and the second threshold is 0.3.

[0196] The divergence loss function between the first segmentation result and the second segmentation result is a loss function set to prevent the two segmentation models from deceiving each other. For example, the divergence loss function between the first segmentation result and the second segmentation result is calculated based on Formula 7:

[0197]

[0198] in, represents the divergence loss function between the first segmentation result and the second segmentation result; y1 represents the first segmentation result; y2 represents the second segmentation result; ε represents a threshold, which is set empirically; ||·||1 represents the L1 norm of the segmentation result; |·| represents the absolute value.

[0199] After obtaining the cross entropy loss function between the first segmentation result and the second segmentation result, and the divergence loss function between the first segmentation result and the second segmentation result, a first loss function is obtained based on the cross entropy loss function and the divergence loss function. Exemplarily, the process of obtaining the first loss function based on the cross entropy loss function and the divergence loss function is implemented based on Formula 8:

[0200]

[0201] Among them, L CO represents the first loss function; represents the cross entropy loss function between the first segmentation result and the second segmentation result; represents the divergence loss function between the first segmentation result and the second segmentation result; η is a hyperparameter that balances the cross entropy loss function and the divergence loss function. η is set based on experience or flexibly adjusted according to the application scenario.

[0202] Step 20423: Obtain a target loss function based on the cross entropy loss function between the first segmentation result and the auxiliary label, the cross entropy loss function between the second segmentation result and the auxiliary label, and the first loss function.

[0203] The target loss function is obtained based on the cross-entropy loss function between the first segmentation result and the auxiliary label, the cross-entropy loss function between the second segmentation result and the auxiliary label, and the first loss function. The cross-entropy loss function between the first segmentation result and the auxiliary label is used to optimize the segmentation effect of the first initial segmentation model, the cross-entropy loss function between the second segmentation result and the auxiliary label is used to optimize the segmentation effect of the second initial segmentation model, and the first loss function is used to collaboratively optimize the segmentation effect of the first and second initial segmentation models. The target loss function covers multiple aspects of loss functions, which is conducive to improving the training effect of the model.

[0204] The embodiment of the present application does not limit the specific implementation method of obtaining the target loss function based on the cross entropy loss function between the first segmentation result and the auxiliary label, the cross entropy loss function between the second segmentation result and the auxiliary label, and the first loss function. Exemplarily, the weighted sum of the cross entropy loss function between the first segmentation result and the auxiliary label, the cross entropy loss function between the second segmentation result and the auxiliary label, and the first loss function is used as the target loss function. In the process of calculating the weighted sum, the weights corresponding to the cross entropy loss function between the first segmentation result and the auxiliary label, the cross entropy loss function between the second segmentation result and the auxiliary label, and the first loss function are set according to experience or flexibly adjusted according to the application scenario. The embodiment of the present application does not limit this. For example, if the weights corresponding to the cross entropy loss function between the first segmentation result and the auxiliary label, the cross entropy loss function between the second segmentation result and the auxiliary label, and the first loss function are all 1, then the target loss function is the sum of the cross entropy loss function between the first segmentation result and the auxiliary label, the cross entropy loss function between the second segmentation result and the auxiliary label, and the first loss function.

[0205] It should be noted that the above-described method of obtaining the target loss function is only an illustrative example, and the embodiments of the present application are not limited to this. In an exemplary embodiment, it is also possible to obtain the target loss function directly based on the first segmentation result, the second segmentation result and the point label without obtaining the auxiliary label. The process is: obtaining the cross entropy loss function between the first segmentation result and the point label, and the cross entropy loss function between the second segmentation result and the point label; obtaining the second loss function based on the first segmentation result and the second segmentation result; obtaining the target loss function based on the cross entropy loss function between the first segmentation result and the point label, the cross entropy loss function between the second segmentation result and the point label, and the second loss function. The implementation principle of this process is the same as that of steps 20421 to 20423 above, and will not be repeated here.

[0206] After obtaining the target loss function, the target loss function is used to train the first initial segmentation model and the second initial segmentation model to obtain the first target segmentation model and the second target segmentation model. In an exemplary embodiment, the process of training the first initial segmentation model and the second initial segmentation model using the target loss function is an iterative process: the target loss function is used to reversely update the parameters of the first initial segmentation model and the parameters of the second initial segmentation model; each time the parameters of the two segmentation models are updated, it is determined whether the training process meets the training termination condition; if the training process meets the training termination condition, the iterative process is stopped, and the model obtained by training the first initial segmentation model is used as the first target segmentation model, and the model obtained by training the second initial segmentation model is used as the second target segmentation model.

[0207] If the training process does not meet the training termination condition, a new target loss function is obtained according to the method of steps 201 to 204, and the parameters of the two segmentation models are reversely updated using the new target loss function. And so on, until the training process meets the training termination condition, a first target segmentation model and a second target segmentation model are obtained. It should be noted that, in the process of obtaining a new target loss function according to the method of steps 201 to 204, the sample image based on it may change or remain unchanged, and this is not limited in the embodiment of the present application. The segmentation model used to segment the sample image is a segmentation model obtained by updating the parameters of the first initial segmentation model, and the segmentation model used to segment the first channel image is a segmentation model obtained by updating the parameters of the second initial segmentation model.

[0208] For example, the training process of the segmentation model is as follows Figure 7As shown. The sample image is input into the first initial segmentation model, and the sample image is segmented by the first initial segmentation model to obtain a first segmentation result; the first channel image is input into the second initial segmentation model, and the first channel image is segmented by the second initial segmentation model to obtain a second segmentation result. According to the cross entropy loss function between the first segmentation result and the point edge label (i.e., the first label) and the cross entropy loss function between the first segmentation result and the cluster label (i.e., the second label), the cross entropy loss function between the first segmentation result and the auxiliary label is obtained; according to the cross entropy loss function between the second segmentation result and the point edge label (i.e., the first label) and the cross entropy loss function between the second segmentation result and the cluster label (i.e., the second label), the cross entropy loss function between the first segmentation result and the auxiliary label is obtained; based on the first segmentation result and the second segmentation result, a first loss function is obtained. Exemplarily, the first loss function can also be called a collaborative training loss function. The first initial segmentation model and the second initial segmentation model are trained using the target loss function obtained based on the cross entropy between the first segmentation result and the auxiliary label, the cross entropy between the second segmentation result and the auxiliary label, and the first loss function to obtain a first target segmentation model and a second target segmentation model.

[0209] The first target segmentation model and the second target segmentation model are trained segmentation models. After the first target segmentation model and the second target segmentation model are obtained through training, the target segmentation results of the image to be processed are obtained using the first target segmentation model and the second target segmentation model.

[0210] In one possible implementation, the process of obtaining the target segmentation result using the first target segmentation model and the second target segmentation model is as follows: based on the image to be processed, obtain the second channel image corresponding to the first dye component; call the first target segmentation model to segment the image to be processed to obtain the first segmentation result; call the second target segmentation model to segment the second channel image to obtain the second segmentation result; based on the first segmentation result and the second segmentation result, obtain the target segmentation result. The implementation of this process can be found in Figure 8 The embodiments shown are not described in detail here.

[0211] In an exemplary embodiment, the segmentation model training method provided in the embodiments of the present application can be used in a variety of operating system environments of a computer device, such as Linux (an operating system), Windows (an operating system), and MacOS (an operating system). Exemplarily, the computer device is equipped with a GPU (Graphics Processing Unit) to significantly increase the computing speed.

[0212] In an embodiment of the present application, during the model training process, in addition to paying attention to the sample image used to provide the texture features of the sub-image of the reference object, attention is also paid to the first channel image used to provide the boundary features of the sub-image. The two target segmentation models trained in this way can comprehensively consider the texture features and boundary features of the sub-image, the information considered is more comprehensive, and the model training effect is better, which is conducive to improving the accuracy of the target segmentation results obtained using the trained target segmentation model.

[0213] Based on the above Figure 1 In the implementation environment shown, the embodiment of the present application provides an image processing method, which is executed by a computer device. The computer device can be a server 12 or a terminal 11, and the embodiment of the present application does not limit this. Figure 8 As shown, the image processing method provided in the embodiment of the present application includes the following steps 801 to 804:

[0214] In step 801 , an image to be processed, a first object segmentation model, and a second object segmentation model are acquired, where the image to be processed includes a sub-image of a reference object stained by a first staining component.

[0215] The first target segmentation model and the second target segmentation model are trained based on the sample image, the first channel image corresponding to the first dye component obtained based on the sample image, and the point labels corresponding to the sample image. The first target segmentation model and the second target segmentation model are used to process the image to be processed. The method for training the first target segmentation model and the second target segmentation model can be found in Figure 2 The embodiments shown are not described in detail here. Acquiring the first target segmentation model and the second target segmentation model in the embodiments of the present application may refer to training the first target segmentation model and the second target segmentation model in real time, or may refer to extracting the first target segmentation model and the second target segmentation model that have been pre-trained and stored, and the embodiments of the present application are not limited to this.

[0216] The image to be processed is an image that needs to be processed to obtain an area in the image where the sub-image of the reference object is located. Figure 2 The sample images in the illustrated embodiment are of the same type and size to ensure that the first target segmentation model and the second target segmentation model can process the image to be processed effectively. For example, if the sample image is a histopathology image, the image to be processed is also a histopathology image.

[0217] The embodiments of this application do not limit the method for acquiring the image to be processed. In exemplary embodiments, the computer device may acquire the image to be processed in ways that include, but are not limited to: the computer device extracting the image to be processed from an image library; an image acquisition device that has established a communication connection with the computer device sending the acquired image to be processed to the computer device; the computer device acquiring an image to be processed that has been manually uploaded, etc.

[0218] It should be noted that the number of images to be processed is one or more. In the case where there are multiple images to be processed, one target segmentation result is obtained for each image to be processed according to the method of steps 802 to 804. The embodiment of the present application is described by taking the case where there is one image to be processed as an example.

[0219] In step 802, based on the image to be processed, a second channel image corresponding to the first staining component is obtained.

[0220] In one possible implementation, the image to be processed is an image presented in a first color space. Based on the image to be processed, the second channel image corresponding to the first color component is obtained by mapping the image to be processed from the first color space to the color space corresponding to the dye in which the first color component is located, obtaining the channel value corresponding to the pixel point in the image to be processed under the first color component; and obtaining the second channel image based on the channel value corresponding to the pixel point in the image to be processed under the first color component. The implementation of this process can be found in Figure 2 Step 202 in the illustrated embodiment will not be described in detail here.

[0221] In step 803, the first target segmentation model is called to segment the image to be processed to obtain a first segmentation result; the second target segmentation model is called to segment the second channel image to obtain a second segmentation result.

[0222] The implementation of step 803 is shown in Figure 2 Step 203 in the illustrated embodiment will not be described in detail here.

[0223] In step 804 , a target segmentation result of the image to be processed is obtained based on the first segmentation result and the second segmentation result.

[0224] The first segmentation result is obtained by segmenting the image to be processed using the first target segmentation model, focusing more on texture features. The second segmentation result is obtained by segmenting the channel image decomposed from the image to be processed, which more fully reflects the boundary features, focusing more on boundary features. Obtaining the target segmentation result based on the first and second segmentation results helps improve the reliability of the target segmentation result. The target segmentation result indicates the area in the image to be processed where the sub-image of the reference object is located. In other words, based on the target segmentation result, it is possible to determine which areas in the image to be processed contain the sub-image of the reference object.

[0225] The embodiment of the present application does not limit the method of obtaining the target segmentation result based on the first segmentation result and the second segmentation result, as long as the target segmentation result is obtained by combining the first segmentation result and the second segmentation result.

[0226] In one possible implementation, based on the first segmentation result and the second segmentation result, a target segmentation result of the image to be processed is obtained by obtaining an average result of the first segmentation result and the second segmentation result, and using the average result as the target segmentation result of the image to be processed.

[0227] Exemplarily, both the first segmentation result and the second segmentation result are used to indicate the probability that a pixel belongs to a sub-image of a reference object and the probability that it does not belong to a sub-image of the reference object. The method for obtaining the average result of the first segmentation result and the second segmentation result is as follows: the average of the probability that a certain pixel belongs to a sub-image of the reference object indicated by the first segmentation result and the probability that the same pixel belongs to a sub-image of the reference object indicated by the second segmentation result is used as the average probability that the pixel belongs to the sub-image of the reference object; the average of the probability that the pixel does not belong to a sub-image of the reference object indicated by the first segmentation result and the probability that the pixel does not belong to a sub-image of the reference object indicated by the second segmentation result is used as the average probability that the pixel does not belong to the sub-image of the reference object; the result indicating the average probability of each pixel belonging to a sub-image of the reference object and the average probability of not belonging to a sub-image of the reference object is used as the average result. This average result is the target segmentation result of the image to be processed.

[0228] In one possible implementation, the first segmentation result includes a first segmentation sub-result corresponding to each pixel, where the first segmentation sub-result corresponding to any pixel indicates a probability, determined by segmenting the image to be processed, that the pixel belongs to a sub-image of the reference object, and a probability, determined by segmenting the image to be processed, that the pixel does not belong to the sub-image of the reference object. The second segmentation result includes a second segmentation sub-result corresponding to each pixel, where the second segmentation sub-result corresponding to any pixel indicates a probability, determined by segmenting the second channel image, that the pixel belongs to a sub-image of the reference object, and a probability, determined by segmenting the second channel image, that the pixel does not belong to the sub-image of the reference object.

[0229] Based on the first segmentation result and the second segmentation result, a method for obtaining a target segmentation result of the image to be processed is: for a certain pixel point, the segmentation sub-result that meets the selection conditions among the first segmentation sub-result corresponding to the pixel point included in the first segmentation result and the second segmentation sub-result corresponding to the pixel point included in the second segmentation result is used as the target segmentation sub-result corresponding to the pixel point; and the segmentation result including the target segmentation sub-results corresponding to each pixel point is used as the target segmentation result of the image to be processed.

[0230] The segmentation sub-results that meet the selection criteria are set based on experience or flexibly adjusted according to the application scenario, and are not limited in this embodiment of the present application. For example, the segmentation sub-results that meet the selection criteria are the segmentation sub-results that indicate a higher probability of belonging to the sub-image of the reference object between the first segmentation sub-result and the second segmentation sub-result; or the segmentation sub-results that meet the selection criteria are the segmentation sub-results that indicate a lower probability of belonging to the sub-image of the reference object between the first segmentation sub-result and the second segmentation sub-result, etc.

[0231] In one possible implementation, after obtaining the target segmentation result of the image to be processed, the method further includes: transforming the target segmentation result to obtain an image processing result. The target segmentation result is used to indicate the probability that each pixel point belongs to the sub-image of the reference object and the probability that each pixel point does not belong to the sub-image of the reference object. The process of transforming the target segmentation result is the process of determining whether each pixel point belongs to the sub-image of the reference object based on the target segmentation result. The result used to indicate whether each pixel point belongs to the sub-image of the reference object is used as the image processing result. Exemplarily, the image processing result can be in the form of an image or a value pair. Exemplarily, in the case where the image processing result is in the form of a value pair, the value pair can be visualized as an image for intuitive observation.

[0232] In an embodiment of the present application, the target segmentation result of the image to be processed is obtained by calling the first target segmentation model and the second target segmentation model. The first target segmentation model and the second target segmentation model can comprehensively consider the texture features and boundary features of the sub-image of the reference object. The information considered is richer and the training effect of the model is better, so that the accuracy of the obtained target segmentation result is higher.

[0233] Next, an exemplary application of the embodiment of the present application in a practical application scenario is introduced.

[0234] In an exemplary embodiment, the segmentation model training method and image processing method provided in the embodiments of the present application can be applied to an application scenario for segmenting sub-images of cell nuclei in a histopathology image. In this application scenario, both the sample image and the image to be processed are histopathology images, which are images obtained by capturing an image of a certain area of ​​a pathology slide stained with HE stain. The sub-image of the reference object stained with the first staining component included in the sample image and the image to be processed refers to the sub-image of the cell nucleus stained with the H staining component in the histopathology image.

[0235] In histopathological image analysis, indicators such as the shape, size, and density of cell nuclei are related to the diagnosis and treatment of cancer. Successful segmentation results, especially accurate segmentation boundaries of cell nuclei sub-images, are of great significance for clinical predictions such as pathological diagnosis and prognosis. Therefore, the segmentation of cell nuclei sub-images is an important step in the analysis. However, in histopathological images, cell nuclei are small in size and numerous in number. It is time-consuming and labor-intensive to obtain a large number of complete cell nuclei annotations. It is more time-saving and labor-saving to use point labels to mark cell nuclei sub-images. Therefore, the present application provides a weakly supervised cell nuclei sub-image segmentation method based on collaborative training. This method can accurately segment cell nuclei sub-images in HE-stained histopathological images with only point labels, thereby reducing the annotation burden of pathologists while meeting the segmentation accuracy.

[0236] See also Figure 9 Based on the training method and image processing method of the segmentation model provided in the embodiment of the present application, the process of segmenting the sub-image of the cell nucleus in the tissue pathology image includes the following steps 901 to 909:

[0237] In step 901 , a sample tissue pathology image and point labels corresponding to the sample tissue pathology image are acquired, where the sample tissue pathology image includes a sub-image of a cell nucleus stained with a hematoxylin staining component.

[0238] In step 902, a first channel image corresponding to the hematoxylin staining component is acquired based on the sample tissue pathology image.

[0239] In step 903, the first initial segmentation model is called to segment the sample tissue pathology image to obtain a first sample segmentation result; the second initial segmentation model is called to segment the first channel image to obtain a second sample segmentation result.

[0240] In step 904 , a first label and a second label are obtained based on the point label.

[0241] In step 905, based on the first sample segmentation result, the first label and the second label, a cross entropy loss function between the first sample segmentation result and the auxiliary label is obtained; based on the second sample segmentation result, the first label and the second label, a cross entropy loss function between the second sample segmentation result and the auxiliary label is obtained; based on the first sample segmentation result and the second sample segmentation result, a first loss function is obtained; based on the cross entropy loss function between the first sample segmentation result and the auxiliary label, the cross entropy loss function between the second sample segmentation result and the auxiliary label, and the first loss function, a target loss function is obtained; and the first initial segmentation model and the second initial segmentation model are trained using the target loss function to obtain a first target segmentation model and a second target segmentation model.

[0242] In step 906 , a tissue pathology image to be processed, a first target segmentation model, and a second target segmentation model are acquired, wherein the tissue pathology image to be processed includes a sub-image of cell nuclei stained with a hematoxylin staining component.

[0243] In step 907, a second channel image corresponding to the hematoxylin staining component is acquired based on the tissue pathology image to be processed.

[0244] In step 908, the first target segmentation model is called to segment the tissue pathology image to be processed to obtain a first segmentation result; the second target segmentation model is called to segment the second channel image to obtain a second segmentation result.

[0245] In step 909 , an average result of the first segmentation result and the second segmentation result is obtained, and the average result is used as a target segmentation result of the tissue pathology image to be processed.

[0246] For the implementation of the above steps 901 to 905, see Figure 2 For the embodiment shown, the implementation of steps 906 to 909 is described in detail. Figure 8 The embodiment shown is not described in detail here. In an exemplary embodiment, the process of steps 901 to 905 is performed offline, and the process of steps 906 to 909 is performed online. After performing the process of steps 901 to 905 once to obtain the first target segmentation model and the second target segmentation model, the process of steps 906 to 909 can be performed multiple times online.

[0247] Exemplarily, the channel image can also be referred to as a component map. In an embodiment of the present application, the hematoxylin component is first extracted from the color space of the HE-stained sample tissue pathology image to obtain a component map with a clearer sub-image of the cell nucleus. The sample tissue pathology image has more sufficient texture features, and the component map has a clearer boundary of the sub-image of the cell nucleus. Then, the point labels are used to generate auxiliary labels (first labels and second labels), and the two segmentation models are trained for weakly supervised learning based on the sample tissue pathology image and the component map, respectively. At the same time, the two segmentation models are collaboratively trained to learn complementary cell nucleus information.

[0248] A weakly supervised learning approach reduces the annotation burden on pathologists, effectively utilizing point labels representing the precise location of cell nucleus sub-images on histopathology images to generate auxiliary labels, providing rough learning information for the model. A collaborative training approach effectively combines the complementary information of the original HE-stained color histopathology images and the compositional map of the extracted hematoxylin components, achieving relatively accurate segmentation of cell nucleus sub-images and thus laying the foundation for further automated analysis of histopathology images.

[0249] The method provided in the embodiment of the present application can be applied to the automatic analysis of tissue pathology images. By subsequent calculation of the segmentation results of the sub-image of the cell nucleus, the average size, density, arrangement and other characteristics of the cell nucleus can be obtained, thereby realizing the clinical diagnosis and treatment of cancer such as different types of cancer grading and patient risk stratification. In addition, in addition to the segmentation of the sub-image of the cell nucleus, the method provided in the embodiment of the present application can also be applied to the segmentation of the sub-image of the cell, or the segmentation of the sub-image of the tissue that is small in size, large in number and densely arranged. The principle of the segmentation of the sub-image of the cell and the segmentation of the sub-image of other tissues is the same as the principle of the segmentation of the sub-image of the cell nucleus, and will not be repeated here.

[0250] See also Figure 10 , an embodiment of the present application provides a training device for a segmentation model, the device comprising:

[0251] A first acquisition unit 1001 is configured to acquire a sample image and obtain point labels corresponding to the sample image, wherein the sample image includes a sub-image of a reference object dyed with a first dye component, and the point labels corresponding to the sample image are determined based on reference points within a region where the sub-image is located in the sample image, and the sample image is used to provide texture features of the sub-image;

[0252] A second acquiring unit 1002 is configured to acquire a first channel image corresponding to the first staining component based on the sample image, where the first channel image is used to provide a boundary feature of the sub-image;

[0253] The calling unit 1003 is configured to call the first initial segmentation model to segment the sample image to obtain a first segmentation result; and call the second initial segmentation model to segment the first channel image to obtain a second segmentation result;

[0254] A third acquisition unit 1004 is configured to acquire a target loss function based on the first segmentation result, the second segmentation result, and the point labels;

[0255] The training unit 1005 is used to train the first initial segmentation model and the second initial segmentation model using the target loss function to obtain the first target segmentation model and the second target segmentation model. The first target segmentation model and the second target segmentation model are used to obtain the target segmentation result of the image to be processed.

[0256] In one possible implementation, the third acquisition unit 1004 is used to obtain auxiliary labels corresponding to the sample image based on the point labels, where the auxiliary labels provide more supervisory information than the point labels; and obtain the target loss function based on the first segmentation result, the second segmentation result and the auxiliary labels.

[0257] In one possible implementation, the auxiliary label includes a first label; the third acquisition unit 1004 is further used to determine a reference point in the area where the sub-image is located in the sample image based on the point label; generate a Thiessen polygon corresponding to the reference point in the sample image; based on the reference point and the Thiessen polygon, obtain the first label, the first label including a sub-label for indicating that the pixel point located at the reference point belongs to the sub-image, a sub-label for indicating that the pixel point located on the Thiessen polygon does not belong to the sub-image, and a sub-label for indicating that the pixel point located at the reference point and outside the Thiessen polygon belongs to an uncertain pixel point.

[0258] In one possible implementation, the auxiliary label includes a second label; the third acquisition unit 1004 is further used to determine a reference point in the area where the sub-image is located in the sample image based on the point label; based on the reference point, obtain reference features corresponding to each pixel point in the sample image; based on the reference features corresponding to each pixel point, cluster each pixel point to obtain a clustering result, and the clustering result includes a first cluster cluster and a second cluster cluster; based on the clustering result, obtain a second label, and the second label includes a sub-label for indicating that the pixel points in the first cluster cluster belong to the sub-image, a sub-label for indicating that the pixel points in the second cluster cluster do not belong to the sub-image, and a sub-label for indicating that the pixel points other than the pixel points in the first cluster cluster and the second cluster cluster belong to uncertain pixel points.

[0259] In one possible implementation, the third acquisition unit 1004 is also used to, for the first pixel point, use the distance between the first pixel point and the target reference point as the distance feature corresponding to the first pixel point, where the target reference point is the reference point closest to the first pixel point, and the first pixel point is any one of the pixel points; based on the distance feature corresponding to the first pixel point and the color feature of the first pixel point, obtain the reference feature corresponding to the first pixel point.

[0260] In one possible implementation, the third acquisition unit 1004 is also used to obtain the cross entropy loss function between the first segmentation result and the auxiliary label, and the cross entropy loss function between the second segmentation result and the auxiliary label; based on the first segmentation result and the second segmentation result, obtain the first loss function; based on the cross entropy loss function between the first segmentation result and the auxiliary label, the cross entropy loss function between the second segmentation result and the auxiliary label, and the first loss function, obtain the target loss function.

[0261] In one possible implementation, the third acquisition unit 1004 is further used to obtain a cross entropy loss function between the first segmentation result and the second segmentation result, and a divergence loss function between the first segmentation result and the second segmentation result; and obtain the first loss function based on the cross entropy loss function and the divergence loss function.

[0262] In one possible implementation, the sample image has a reference object label, and the first acquisition unit 1001 is used to determine the area where the sub-image is located in the sample image based on the reference object label; determine the center of the area where the sub-image is located in the sample image; determine a reference point based on the center of the area; and determine a point label corresponding to the sample image based on the reference point.

[0263] In an embodiment of the present application, during the model training process, in addition to paying attention to the sample image used to provide the texture features of the sub-image of the reference object, attention is also paid to the first channel image used to provide the boundary features of the sub-image. The two target segmentation models trained in this way can comprehensively consider the texture features and boundary features of the sub-image, the information considered is more comprehensive, and the model training effect is better, which is conducive to improving the accuracy of the target segmentation results obtained using the trained target segmentation model.

[0264] See also Figure 11 , an embodiment of the present application provides an image processing device, the device comprising:

[0265] A first acquisition unit 1101 is configured to acquire an image to be processed, a first object segmentation model, and a second object segmentation model, wherein the image to be processed includes a sub-image of a reference object stained by a first staining component, and the first object segmentation model and the second object segmentation model are trained based on a sample image, a first channel image corresponding to the first staining component obtained based on the sample image, and point labels corresponding to the sample image;

[0266] A second acquisition unit 1102 is configured to acquire a second channel image corresponding to the first dye component based on the image to be processed;

[0267] The calling unit 1103 is configured to call the first target segmentation model to segment the image to be processed to obtain a first segmentation result; and call the second target segmentation model to segment the second channel image to obtain a second segmentation result;

[0268] The third acquiring unit 1104 is configured to acquire a target segmentation result of the image to be processed based on the first segmentation result and the second segmentation result.

[0269] In one possible implementation, the image to be processed is an image presented in a first color space, and the second acquisition unit 1102 is used to map the image to be processed from the first color space to the color space corresponding to the dye where the first dye component is located, to obtain the channel value corresponding to the pixel point in the image to be processed under the first dye component; based on the channel value corresponding to the pixel point in the image to be processed under the first dye component, obtain the second channel image.

[0270] In a possible implementation, the third acquisition unit 1104 is configured to acquire an average result of the first segmentation result and the second segmentation result, and use the average result as a target segmentation result of the image to be processed.

[0271] In an embodiment of the present application, the target segmentation result of the processed image is obtained by calling the first target segmentation model and the second target segmentation model. The first target segmentation model and the second target segmentation model can comprehensively consider the texture features and boundary features of the sub-image of the reference object. The information considered is richer and the training effect of the model is better, so that the accuracy of the obtained target segmentation result is higher.

[0272] It should be noted that the apparatus provided in the above embodiments is merely illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0273] In an exemplary embodiment, a computer device is also provided, comprising a processor and a memory, wherein the memory stores at least one computer program. The at least one computer program is loaded and executed by one or more processors, so that the computer device implements any of the above-described segmentation model training methods or image processing methods. The computer device may be a server or a terminal, which is not limited in this embodiment of the present application. The structures of the server and terminal are described below.

[0274] Figure 12 This is a structural diagram of a server provided in an embodiment of the present application. The server may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1201 and one or more memories 1202, wherein the one or more memories 1202 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 1201, so that the server implements the training method or image processing method of the segmentation model provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server may also include other components for implementing device functions, which will not be described here.

[0275] Figure 13 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. For example, the terminal may be a smartphone, tablet computer, laptop computer, or desktop computer. The terminal may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0276] Typically, the terminal includes: a processor 1301 and a memory 1302 .

[0277] The processor 1301 may include one or more processing cores. The processor 1301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1301 may be integrated with a GPU, which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1301 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0278] The memory 1302 may include one or more computer-readable storage media, which may be non-transitory. The memory 1302 may also include a high-speed random access memory and a non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1302 is used to store at least one instruction, which is used to be executed by the processor 1301 so that the terminal implements the training method or image processing method of the segmentation model provided in the method embodiment of the present application.

[0279] In some embodiments, the terminal may optionally include a peripheral device interface 1303 and at least one peripheral device. The processor 1301, memory 1302, and peripheral device interface 1303 may be connected via a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 1303 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 1304, a display screen 1305, a camera assembly 1306, an audio circuit 1307, and a power supply 1309.

[0280] The peripheral device interface 1303 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 1301 and the memory 1302. The radio frequency circuit 1304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1304 communicates with a communication network and other communication devices via electromagnetic signals. The display screen 1305 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. The camera assembly 1306 is used to capture images or videos.

[0281] Audio circuit 1307 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting them into electrical signals that are then input into processor 1301 for processing or into RF circuit 1304 for voice communication. The speaker is used to convert electrical signals from processor 1301 or RF circuit 1304 into sound waves. Power supply 1309 is used to power various components in the terminal. Power supply 1309 can be AC, DC, disposable batteries, or rechargeable batteries.

[0282] In some embodiments, the terminal further includes one or more sensors 1310 , including but not limited to: an acceleration sensor 1311 , a gyroscope sensor 1312 , a pressure sensor 1313 , an optical sensor 1315 , and a proximity sensor 1316 .

[0283] The acceleration sensor 1311 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal. The gyroscope sensor 1312 can detect the body direction and rotation angle of the terminal. The gyroscope sensor 1312 can cooperate with the acceleration sensor 1311 to collect the user's 3D actions on the terminal. The pressure sensor 1313 can be set on the side frame of the terminal and / or the lower layer of the display screen 1305. When the pressure sensor 1313 is set on the side frame of the terminal, it can detect the user's holding signal of the terminal, and the processor 1301 performs left and right hand recognition or quick operation based on the holding signal collected by the pressure sensor 1313. When the pressure sensor 1313 is set on the lower layer of the display screen 1305, the processor 1301 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 1305.

[0284] The optical sensor 1315 is used to collect ambient light intensity. The proximity sensor 1316, also known as a distance sensor, is typically provided on the front panel of the terminal. The proximity sensor 1316 is used to collect the distance between the user and the front of the terminal.

[0285] Those skilled in the art will understand that Figure 13 The structure shown in the figure does not constitute a limitation on the terminal, and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0286] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor of a computer device so that the computer implements any of the above-mentioned segmentation model training methods or image processing methods.

[0287] In one possible implementation, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0288] In an exemplary embodiment, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the above-described segmentation model training methods or image processing methods.

[0289] In some embodiments, the computer programs involved in the embodiments of the present application may be deployed and executed on a single computer device, or on multiple computer devices located in a single location, or on multiple computer devices distributed across multiple locations and interconnected via a communication network. Multiple computer devices distributed across multiple locations and interconnected via a communication network may constitute a blockchain system. In other words, the aforementioned terminals and servers may serve as node devices in the blockchain system.

[0290] It should be noted that the terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. The implementations described in the above exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with certain aspects of the present application as detailed in the appended claims.

[0291] It should be understood that the term "plurality" used herein refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0292] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A training method for a segmentation model, characterized in that: The method comprises: Acquiring a sample image and obtaining point labels corresponding to the sample image, wherein the sample image includes a sub-image of a reference object dyed by a first dye component, the point labels corresponding to the sample image being determined based on reference points within a region within the sample image where the sub-image is located, the sample image being used to provide texture features of the sub-image, and the sample image being an image presented in a first color space; Mapping the sample image from the first color space to the color space corresponding to the dye containing the first dye component, obtaining the channel values ​​corresponding to each pixel in the sample image under the first dye component; obtaining a first channel image corresponding to the first dye component based on the channel values ​​corresponding to each pixel under the first dye component, wherein the first channel image is used to provide boundary features of the sub-image; Calling a first initial segmentation model to segment the sample image to obtain a first segmentation result; calling a second initial segmentation model to segment the first channel image to obtain a second segmentation result; Based on the first segmentation result, the second segmentation result and the point label, a target loss function is obtained; the first initial segmentation model and the second initial segmentation model are trained using the target loss function to obtain a first target segmentation model and a second target segmentation model, and the first target segmentation model and the second target segmentation model are used to obtain the target segmentation result of the image to be processed.

2. The method according to claim 1, characterized in that The acquiring a target loss function based on the first segmentation result, the second segmentation result, and the point label includes: Based on the point labels, obtaining auxiliary labels corresponding to the sample images, wherein the auxiliary labels provide more supervisory information than the point labels; The target loss function is obtained based on the first segmentation result, the second segmentation result, and the auxiliary label.

3. The method according to claim 2, characterized in that The auxiliary label includes a first label; and obtaining the auxiliary label corresponding to the sample image based on the point label includes: Determining, based on the point labels, a reference point within the region where the sub-image is located in the sample image; generating Thiessen polygons corresponding to the reference points in the sample image; Based on the reference point and the Thiessen polygon, the first label is obtained, wherein the first label includes a sub-label for indicating that the pixel point located on the reference point belongs to the sub-image, a sub-label for indicating that the pixel point located on the Thiessen polygon does not belong to the sub-image, and a sub-label for indicating that the pixel point located outside the reference point and the Thiessen polygon is an uncertain pixel point.

4. The method according to claim 2, characterized in that The auxiliary label includes a second label; and obtaining the auxiliary label corresponding to the sample image based on the point label includes: Determining, based on the point labels, a reference point within the region where the sub-image is located in the sample image; Based on the reference points, obtaining reference features corresponding to each pixel point in the sample image; Clustering the pixels based on the reference features corresponding to the pixels to obtain a clustering result, where the clustering result includes a first cluster and a second cluster; Based on the clustering result, the second label is obtained, which includes a sub-label for indicating that the pixel points in the first cluster belong to the sub-image, a sub-label for indicating that the pixel points in the second cluster do not belong to the sub-image, and a sub-label for indicating that the pixel points other than the pixel points in the first cluster and the second cluster are uncertain pixels.

5. The method according to claim 4, characterized in that The acquiring, based on the reference point, reference features corresponding to respective pixel points in the sample image includes: For a first pixel point, the distance between the first pixel point and a target reference point is used as a distance feature corresponding to the first pixel point, the target reference point is a reference point closest to the first pixel point, and the first pixel point is any one of the pixel points; Based on the distance feature corresponding to the first pixel and the color feature of the first pixel, a reference feature corresponding to the first pixel is obtained.

6. The method according to any one of claims 2 to 5, characterized in that: The acquiring the target loss function based on the first segmentation result, the second segmentation result, and the auxiliary label includes: Obtaining a cross entropy loss function between the first segmentation result and the auxiliary label, and a cross entropy loss function between the second segmentation result and the auxiliary label; Obtaining a first loss function based on the first segmentation result and the second segmentation result; The target loss function is obtained based on a cross entropy loss function between the first segmentation result and the auxiliary label, a cross entropy loss function between the second segmentation result and the auxiliary label, and the first loss function.

7. The method according to claim 6, characterized in that The obtaining a first loss function based on the first segmentation result and the second segmentation result includes: Obtaining a cross entropy loss function between the first segmentation result and the second segmentation result, and a divergence loss function between the first segmentation result and the second segmentation result; Based on the cross entropy loss function and the divergence loss function, the first loss function is obtained.

8. The method according to any one of claims 1 to 5 and 7, characterized in that: The sample image has a reference object label, and obtaining a point label corresponding to the sample image includes: Based on the reference object label, determining the region where the sub-image is located in the sample image; Determining a region center of a region where the sub-image is located in the sample image; Determining the reference point based on the center of the region; Based on the reference point, a point label corresponding to the sample image is determined.

9. An image processing method, characterized in that: The method comprises: Obtaining an image to be processed, a first object segmentation model, and a second object segmentation model, wherein the image to be processed includes a subimage of a reference object stained by a first coloring component, the first object segmentation model and the second object segmentation model are trained based on a sample image, a first channel image corresponding to the first coloring component obtained based on the sample image, and point labels corresponding to the sample image, and the image to be processed is an image presented in a first color space; Mapping the image to be processed from the first color space to the color space corresponding to the dye containing the first dye component, obtaining the channel value corresponding to each pixel in the image to be processed under the first dye component; and obtaining a second channel image corresponding to the first dye component based on the channel value corresponding to each pixel under the first dye component; Calling the first target segmentation model to segment the image to be processed to obtain a first segmentation result; calling the second target segmentation model to segment the second channel image to obtain a second segmentation result; Obtaining a target segmentation result of the image to be processed based on the first segmentation result and the second segmentation result; The target segmentation result is the average result between the first segmentation result and the second segmentation result; or, the target segmentation result includes the target segmentation sub-results corresponding to each pixel point respectively, and the target segmentation sub-result corresponding to any pixel point is the first segmentation sub-result corresponding to any pixel point included in the first segmentation result and the second segmentation sub-result corresponding to any pixel point included in the second segmentation result, whichever segmentation sub-result meets the selection conditions.

10. A training device for a segmentation model, characterized in that: The device comprises: a first acquisition unit, configured to acquire a sample image and a point label corresponding to the sample image, wherein the sample image includes a sub-image of a reference object dyed by a first dye component, and the point label corresponding to the sample image is determined based on a reference point within a region where the sub-image is located in the sample image, and the sample image is used to provide texture features of the sub-image, and the sample image is an image presented in a first color space; a second acquisition unit, configured to map the sample image from the first color space to a color space corresponding to the dye in which the first dye component is located, and obtain a channel value corresponding to each pixel point in the sample image under the first dye component; and based on the channel value corresponding to each pixel point under the first dye component, obtain a first channel image corresponding to the first dye component, wherein the first channel image is used to provide a boundary feature of the sub-image; A calling unit, configured to call a first initial segmentation model to segment the sample image to obtain a first segmentation result; and call a second initial segmentation model to segment the first channel image to obtain a second segmentation result; A third acquisition unit is configured to acquire a target loss function based on the first segmentation result, the second segmentation result, and the point label; A training unit is used to train the first initial segmentation model and the second initial segmentation model using the target loss function to obtain a first target segmentation model and a second target segmentation model, wherein the first target segmentation model and the second target segmentation model are used to obtain the target segmentation result of the image to be processed.

11. The device according to claim 10, characterized in that The third acquiring unit is configured to acquire, based on the point label, an auxiliary label corresponding to the sample image, wherein the auxiliary label provides more supervisory information than the point label; The target loss function is obtained based on the first segmentation result, the second segmentation result, and the auxiliary label.

12. The device according to claim 11, characterized in that The auxiliary label includes a first label; the third acquisition unit is used to determine a reference point in the area where the sub-image is located in the sample image based on the point label; A Thiessen polygon corresponding to the reference point is generated in the sample image; and the first label is obtained based on the reference point and the Thiessen polygon, the first label including a sub-label for indicating that the pixel point located on the reference point belongs to the sub-image, a sub-label for indicating that the pixel point located on the Thiessen polygon does not belong to the sub-image, and a sub-label for indicating that the pixel point located at the reference point and outside the Thiessen polygon is an uncertain pixel point.

13. The device according to claim 11, characterized in that The auxiliary label includes a second label; the third acquisition unit is used to determine a reference point within the area where the sub-image is located in the sample image based on the point label; Based on the reference points, reference features corresponding to each pixel point in the sample image are obtained; based on the reference features corresponding to each pixel point, the pixel points are clustered to obtain a clustering result, and the clustering result includes a first cluster cluster and a second cluster cluster; based on the clustering result, the second label is obtained, and the second label includes a sub-label for indicating that the pixel points in the first cluster cluster belong to the sub-image, a sub-label for indicating that the pixel points in the second cluster cluster do not belong to the sub-image, and a sub-label for indicating that the pixel points other than the pixel points in the first cluster cluster and the second cluster cluster belong to uncertain pixel points.

14. The device according to claim 13, characterized in that The third acquisition unit is used to use the distance between the first pixel point and the target reference point as the distance feature corresponding to the first pixel point, where the target reference point is the reference point closest to the first pixel point, and the first pixel point is any one of the pixel points; based on the distance feature corresponding to the first pixel point and the color feature of the first pixel point, obtain the reference feature corresponding to the first pixel point.

15. The device according to any one of claims 11 to 14, characterized in that: The third acquisition unit is configured to acquire a cross entropy loss function between the first segmentation result and the auxiliary label, and a cross entropy loss function between the second segmentation result and the auxiliary label; and acquire a first loss function based on the first segmentation result and the second segmentation result; The target loss function is obtained based on a cross entropy loss function between the first segmentation result and the auxiliary label, a cross entropy loss function between the second segmentation result and the auxiliary label, and the first loss function.

16. The device according to claim 15, characterized in that The third acquisition unit is used to obtain a cross entropy loss function between the first segmentation result and the second segmentation result, and a divergence loss function between the first segmentation result and the second segmentation result; and obtain the first loss function based on the cross entropy loss function and the divergence loss function.

17. The device according to any one of claims 10-14 and 16, characterized in that: The sample image has a reference object label, and the first acquisition unit is configured to determine a region in the sample image where the sub-image is located based on the reference object label; Determine the center of the region where the sub-image is located in the sample image; determine the reference point based on the region center; and determine the point label corresponding to the sample image based on the reference point.

18. An image processing device, characterized in that: The device comprises: a first acquisition unit, configured to acquire an image to be processed, a first object segmentation model, and a second object segmentation model, wherein the image to be processed includes a sub-image of a reference object dyed by a first dye component, the first object segmentation model and the second object segmentation model are trained based on a sample image, a first channel image corresponding to the first dye component obtained based on the sample image, and point labels corresponding to the sample image, and the image to be processed is an image presented in a first color space; a second acquisition unit configured to map the image to be processed from the first color space to a color space corresponding to the dye containing the first dye component, obtain a channel value corresponding to each pixel in the image to be processed under the first dye component, and obtain a second channel image corresponding to the first dye component based on the channel value corresponding to each pixel under the first dye component; A calling unit, configured to call the first target segmentation model to segment the image to be processed to obtain a first segmentation result; and call the second target segmentation model to segment the second channel image to obtain a second segmentation result; a third acquiring unit, configured to acquire a target segmentation result of the image to be processed based on the first segmentation result and the second segmentation result; The target segmentation result is the average result between the first segmentation result and the second segmentation result; or, the target segmentation result includes the target segmentation sub-results corresponding to each pixel point respectively, and the target segmentation sub-result corresponding to any pixel point is the first segmentation sub-result corresponding to any pixel point included in the first segmentation result and the second segmentation sub-result corresponding to any pixel point included in the second segmentation result, whichever segmentation sub-result meets the selection conditions.

19. A computer device, characterized in that: The computer device includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the computer device implements the training method of the segmentation model as described in any one of claims 1 to 8, or the image processing method as described in claim 9.

20. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor so that the computer implements the training method of the segmentation model as described in any one of claims 1 to 8, or the image processing method as described in claim 9.

21. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the training method of the segmentation model as described in any one of claims 1 to 8, or the image processing method as described in claim 9.

Images