Pathological image lesion region determination method and model training method and device

Abstract

The application discloses a pathological image lesion region determination method and device and a model training method, and relates to the technical field of computer vision. The method comprises the following steps: at least two first instance images are sampled from a pathological image by using a first sampling mode; candidate lesion regions in the pathological image are determined based on feature information extracted from the at least two first instance images; at least two second instance images are sampled from the candidate lesion regions by using a second sampling mode, and the overlapping degree between the second instance images is greater than the overlapping degree between the first instance images; and lesion indication information of the pathological image is determined based on feature information extracted from the at least two second instance images, wherein the lesion indication information is used for indicating lesion regions in the pathological image. In the application, the consumption of human resources is reduced, and the cost required for determining the lesion regions is saved.

Description

Technical Field

[0001] This application relates to the field of computer vision, and in particular to a method, model training method and apparatus for determining lesion areas in pathological images. Background Technology

[0002] Currently, the specific location of lesions can be determined through pathological images.

[0003] In related technologies, histopathological examination begins with a biopsy, where doctors obtain tissue sections from the patient's body and prepare them into slides through steps such as embedding and staining. Pathologists then place the slides under a microscope to observe them and locate specific lesions within the pathological images.

[0004] However, in the aforementioned related technologies, the process of determining the location of the lesion relies on manual diagnosis, which consumes a lot of human resources. Summary of the Invention

[0005] This application provides a method, model training method, and apparatus for determining lesion regions in pathological images, which can reduce the consumption of human resources and save the cost required to determine lesion regions. The technical solution is as follows.

[0006] According to one aspect of the embodiments of this application, a method for determining lesion areas in pathological images is provided, the method comprising the following steps:

[0007] At least two first instance images are obtained by sampling from pathological images using a first sampling method;

[0008] Based on the feature information extracted from the at least two first instance images, candidate lesion regions in the pathological images are determined;

[0009] At least two second instance images are obtained from the candidate lesion region using a second sampling method, wherein the overlap between the second instance images is greater than the overlap between the first instance images;

[0010] Based on feature information extracted from the at least two second instance images, lesion indication information of the pathological image is determined, the lesion indication information being used to indicate the lesion area in the pathological image.

[0011] According to one aspect of the embodiments of this application, a method for training a lesion region determination model is provided, the method comprising the following steps:

[0012] Obtain a training sample set, wherein the training sample set includes at least one sample pathological image;

[0013] The pathological images of the samples are sampled using a first sampling method and a second sampling method, respectively, to obtain at least two first sample instances and at least two second sample instances corresponding to the pathological images of the samples, wherein the overlap between the second sample instances is greater than the overlap between the first sample instances;

[0014] Based on the feature information extracted from the at least two first sample instances and the feature information extracted from the at least two second sample instances, the lesion probability distribution information of the sample pathological image is determined; wherein, the lesion probability distribution information is used to indicate the probability distribution of lesion regions in the sample pathological image;

[0015] The model for determining the lesion region is trained based on the lesion probability distribution information.

[0016] According to one aspect of the embodiments of this application, a device for determining lesion areas in pathological images is provided, the device comprising the following modules:

[0017] The first image acquisition module is used to sample at least two first instance images from a pathological image using a first sampling method;

[0018] The lesion region determination module is used to determine candidate lesion regions in the pathological image based on feature information extracted from the at least two first instance images;

[0019] The second image acquisition module is used to sample at least two second instance images from the candidate lesion region using a second sampling method, wherein the overlap between the second instance images is greater than the overlap between the first instance images;

[0020] The lesion information determination module is used to determine lesion indication information of the pathological image based on feature information extracted from the at least two second instance images, wherein the lesion indication information is used to indicate the lesion area in the pathological image.

[0021] According to one aspect of the embodiments of this application, a training apparatus for a lesion area determination model is provided, the apparatus comprising the following modules:

[0022] A sample acquisition module is used to acquire a training sample set, wherein the training sample set includes at least one sample pathological image;

[0023] The instance acquisition module is used to sample the sample pathological image using a first sampling method and a second sampling method respectively, to obtain at least two first sample instances and at least two second sample instances corresponding to the sample pathological image, wherein the overlap between the second sample instances is greater than the overlap between the first sample instances;

[0024] The information acquisition module is used to determine the lesion probability distribution information of the sample pathological image based on feature information extracted from the at least two first sample instances and feature information extracted from the at least two second sample instances; wherein, the lesion probability distribution information is used to indicate the probability distribution of lesion regions in the sample pathological image;

[0025] The model training module is used to train a model for the lesion region based on the lesion probability distribution information.

[0026] According to one aspect of the embodiments of this application, the embodiments of this application provide a computer device, the computer device including a processor and a memory, the memory storing at least one program, the at least one program being loaded and executed by the processor to implement the above-described method for determining lesion areas in pathological images, or the training method for the above-described lesion area determination model.

[0027] According to one aspect of the embodiments of this application, the embodiments of this application provide a computer-readable storage medium storing at least one program, which is loaded and executed by a processor to implement the above-described method for determining lesion areas in pathological images, or the training method for the above-described lesion area determination model.

[0028] According to one aspect of the embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned method for determining lesion regions in pathological images, or the aforementioned method for training a lesion region determination model.

[0029] The technical solution provided in this application can bring the following beneficial effects:

[0030] By sampling pathological images to obtain instance images, extracting feature information from instance images, and automatically determining the lesion area of ​​the pathological image based on the feature information of the instance images, the consumption of human resources is reduced and the cost required to determine the lesion area is saved.

[0031] In addition, in this embodiment, candidate lesion regions in the pathological image are first determined based on the first sampling method, and then a second instance image is obtained based on the second sampling method with a greater sampling overlap than the first sampling method. The lesion region of the pathological image is determined from the candidate lesion region based on the second instance image. In this way, the area of ​​the pathological image that needs to be sampled using the second sampling method is reduced, thereby reducing the number of second instance images that need to be collected and feature information extracted, thereby reducing the computational resources required to determine the lesion region and improving the efficiency required to determine the lesion region.

[0032] Furthermore, candidate lesion areas are more likely to contain lesion areas than other areas, and have a higher signal-to-noise ratio. By performing a second sampling method only on candidate lesion areas, information loss can be reduced and the ability to perceive smaller lesion areas can be enhanced, thereby enabling more accurate identification of lesion areas in pathological images. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the model architecture of a lesion area determination model provided in one embodiment of this application;

[0034] Figure 2 This is a schematic diagram of a lesion area determination system provided in one embodiment of this application;

[0035] Figure 3 This is a flowchart of a method for determining lesion areas in pathological images provided in one embodiment of this application;

[0036] Figure 4 This is a schematic diagram of a lesion area determination method provided in one embodiment of this application;

[0037] Figure 5 This is a schematic diagram of a lesion area determination method provided in another embodiment of this application;

[0038] Figure 6 This is a schematic diagram of a lesion area determination method provided in another embodiment of this application;

[0039] Figure 7 This is a schematic diagram of a lesion area determination method provided in another embodiment of this application;

[0040] Figure 8 This is a flowchart of a training method for a lesion area determination model provided in one embodiment of this application;

[0041] Figure 9 This is a schematic diagram of a training method for a lesion area determination model provided in one embodiment of this application;

[0042] Figure 10This is a schematic diagram of a lesion area determination method provided in another embodiment of this application;

[0043] Figure 11 This is a block diagram of a lesion area determination device for pathological images provided in one embodiment of this application;

[0044] Figure 12 This is a block diagram of a lesion area determination device for pathological images provided in another embodiment of this application;

[0045] Figure 13 This is a block diagram of a training device for a lesion area determination model provided in one embodiment of this application;

[0046] Figure 14 This is a block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0048] The training method for the sustainable learning model in this application involves the following techniques:

[0049] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0050] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0051] Computer vision (CV) is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, and map building.

[0052] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learn-by-doing.

[0053] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0054] The solution provided in this application involves artificial intelligence technologies such as machine learning and computer vision, and uses a trained lesion area determination model to determine the lesion area in a pathological image.

[0055] The technical solution of this application will be described below with reference to several embodiments.

[0056] Please refer to Figure 1 The diagram illustrates a schematic representation of the lesion region determination model provided in one embodiment of this application. The lesion region determination model may include: a coding network 10, a first classification network 20, a second classification network 30, and a third classification network 40.

[0057] The encoding network 10 is used to perform feature encoding on the instance images to obtain feature information corresponding to the instance images. For example, in this embodiment, the instance images include a first instance image and a second instance image. The first instance image is obtained by sampling a pathological image using a first sampling method, and the second instance image is obtained by sampling a candidate lesion region using a second sampling method. Furthermore, in this embodiment, the overlap between the second instance images is greater than the overlap between the first instance images.

[0058] The first classification network 20 is used to determine the first predicted probability corresponding to the first instance image and the global feature information of the pathological image based on the feature information corresponding to the first instance image, and then determine the lesion area in the pathological image based on the first predicted probability. The first predicted probability refers to the probability that a lesion area exists in the first instance image.

[0059] The second classification network 30 is used to determine the local feature information of the pathological image for the candidate lesion area based on the feature information corresponding to the second instance image.

[0060] The third classification network 40 is used to determine the lesion indication information of the pathological image based on the aforementioned global feature information and the aforementioned local feature information. This lesion indication information is used to indicate the lesion region in the pathological image.

[0061] In some embodiments, the lesion region determination model described above can be applied to a lesion region determination system. For example, as shown... Figure 2 The lesion area determination system includes a terminal device 50 and a server 60.

[0062] Terminal device 50 can be an electronic device such as a mobile phone, tablet computer, game console, e-book reader, multimedia playback device, wearable device, PC (Personal Computer), intelligent voice interaction device, smart home appliance, vehicle terminal, and aircraft, etc., and this application embodiment does not limit this. In some embodiments, terminal device 50 includes a client application. This application can be any application with pathological image acquisition capabilities. Exemplarily, the above application can be an application that requires downloading and installation, or it can be an application that can be used immediately upon clicking, and this application embodiment does not limit this.

[0063] Server 60 provides backend services to terminal devices 50. Server 60 can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center. Optionally, server 60 can be a backend server for the client of the aforementioned application. In an exemplary embodiment, server 60 provides backend services to multiple terminal devices 50.

[0064] The terminal device 50 and the server 60 transmit data via a network. In some embodiments, the server 60 includes a lesion area determination model. The terminal device 50 acquires a pathological image and sends the image to the server 60. The server 60 then processes the image according to the lesion area determination model to determine the lesion area.

[0065] One point that needs to be clarified is that the above Figure 2 The descriptions provided are merely exemplary and explanatory. In exemplary embodiments, the functions of the terminal device 50 and the server 60 can be flexibly configured and adjusted, and this application embodiment does not limit them in this regard. Exemplarily, the aforementioned lesion area determination model can also be set in the terminal device 50, and after acquiring a pathological image, the terminal device 50 processes the pathological image based on the lesion area determination model to determine the lesion area of ​​the pathological image.

[0066] It should also be noted that the training device for the above-mentioned lesion area determination model can be the server 60 mentioned above, or other computer devices. This application embodiment does not limit this.

[0067] Please refer to Figure 3 This document illustrates a flowchart of a method for determining lesion regions in pathological images according to an embodiment of this application. The steps in this method can be performed by the methods described above. Figure 2 The method is executed by the terminal device 50 and / or server 60 (hereinafter collectively referred to as "computer device"). The method may include the following steps (310-340):

[0068] Step 310: At least two first instance images are obtained by sampling from the pathological images using the first sampling method.

[0069] Pathological images, also known as whole-slide images (WSI), are a type of medical image. In some embodiments, pathological images are generated by processing tissues or organs in a patient's body through methods such as sectioning, embedding, staining, and scanning. In this application embodiment, after acquiring the aforementioned pathological image, the computer device samples at least two first instance images from the pathological image using a first sampling method. In some embodiments, the aforementioned pathological image may be a pathological image of the teeth, arm, heart, liver, kidney, lung, prostate, stomach, etc. In some embodiments, the pathological image may be a CT (Computed Tomography) image, an MRI (Magnetic Resonance Imaging) image, a B-scan ultrasonography image, or other types of pathological images; this application embodiment does not specifically limit the types of pathological images.

[0070] In some embodiments, the pathological image includes a background image and a foreground image. The foreground image refers to the image region corresponding to the tissue or organ in the patient's body, while the background image refers to the image region unrelated to the tissue or organ in the patient's body; alternatively, the foreground image refers to the image region of the body part requiring pathological analysis, and the background image refers to the remaining image regions, i.e., image regions other than the image regions of the body parts requiring pathological analysis.

[0071] For example, the first sampling method described above is uniform sampling. Uniform sampling can refer to a sampling method that divides a two-dimensional continuous image plane into equal intervals in both the horizontal and vertical directions. In the embodiments of this application, uniform sampling refers to dividing the pathological image into multiple grids of the same shape and size. The grids can be squares, rectangles, triangles, parallelograms, etc., and the embodiments of this application do not specifically limit this, and these grids do not overlap. In some embodiments, the first instance image obtained by uniform sampling is an instance image cropped to 224x224 pixels at a scaling factor of 10x.

[0072] In some embodiments, step 310 further includes the following sub-steps:

[0073] 1. Divide the pathological images into background and foreground images.

[0074] In some embodiments, the case image is converted into a binary image, using two different colors to represent the background image and the foreground image, respectively. For example, the pathological image may only contain black and white, with the black image portion representing the background image and the white image portion representing the foreground image; or, the white image portion representing the background image and the black image portion representing the foreground image.

[0075] 2. Perform uniform segmentation on the pathological image to obtain at least two first candidate instance images.

[0076] In some embodiments, after determining the background and foreground images in the pathological image, the pathological image is uniformly sampled to uniformly segment it into multiple candidate instance images. In some embodiments, each segmented candidate instance image has the same shape and size.

[0077] 3. From at least two first candidate instance images, determine the first candidate instance image containing the foreground image as the first instance image.

[0078] In some embodiments, a first candidate instance image containing a foreground image is determined from at least two first candidate instance images, and the first candidate instance image containing the foreground image is determined as the first instance image.

[0079] In some embodiments, for each first candidate instance image obtained after segmentation, if the proportion of the foreground image contained in the first candidate instance image is greater than or equal to a first threshold, then the first candidate instance image is determined to be a first instance image; if the proportion of the foreground image contained in the first candidate instance image is less than the first threshold, then the first candidate instance image is determined not to be a first instance image. The first threshold is greater than or equal to 0% and less than or equal to 100%, and the first threshold can be 0%, 5%, 8%, 10%, 15%, 24%, 30%, 45%, 50%, 62%, 70%, 100%, etc.; of course, the first threshold can also be other values, and the specific value of the first threshold can be set by relevant technicians according to actual conditions. This application embodiment does not specifically limit this.

[0080] In some embodiments, there may be multiple first instance images determined from at least two first candidate instance images, or there may be only one; the embodiments of this application do not specifically limit this.

[0081] In some embodiments, prior to step 310, the method further includes: acquiring an initial pathological image; and scaling the initial pathological image to a fixed size to obtain a pathological image. Since the size of the initial pathological image may not be fixed, after acquiring the initial pathological image, it can be scaled to a set fixed size to obtain the pathological image required in step 310. For example, if the initial pathological image is smaller than the fixed size, it is enlarged to the fixed size; if the initial pathological image is larger than the fixed size, it is reduced to the fixed size. The fixed size can be set by those skilled in the art according to actual conditions, and this application embodiment does not specifically limit it.

[0082] Step 320: Based on the feature information extracted from at least two first instance images, determine the candidate lesion regions in the pathological image.

[0083] In some embodiments, after acquiring the first instance images, the computer device performs feature extraction on each first instance image to obtain feature information corresponding to each first instance image, and then determines the candidate lesion region in the pathological image based on the feature information corresponding to each first instance image. The feature information extracted from at least two first instance images includes first feature information corresponding to each first instance image and global feature information of the pathological image. For example, the feature vectors (i.e., feature information of the first instance images) corresponding to each first instance image are extracted from the Swin Transformer backbone network.

[0084] The feature information extracted from the first instance image using the first sampling method yields the global aggregated features of the pathological image and the coarse distribution of the lesion area.

[0085] Candidate lesion areas refer to areas where lesions may exist. A lesion area can refer to an area containing a tumor, an area with cancerous changes, an area with perforation, an area that is ulcerated or shows signs of ulceration, or an area with a significantly abnormal color. A lesion area can also refer to an area containing other types of lesions; the specific definition can be made by relevant technical personnel, and this application embodiment does not specifically limit this.

[0086] Step 330: At least two second instance images are obtained from the candidate lesion region using a second sampling method.

[0087] In some embodiments, after determining the candidate lesion region, the computer device uses a second sampling method to sample at least two second instance images from the candidate lesion region. Optionally, multiple samplings are performed around a given coordinate (such as the center point of the candidate lesion region) to obtain at least two second instance images. The overlap between the second instance images is greater than the overlap between the first instance images. For example, there may be overlap between the second instance images but no overlap between the first instance images; or, for example, there may be overlap between both the second and first instance images, but the overlap between the second instance images is greater than the overlap between the first instance images. In some embodiments, the second instance images are images obtained through dense sampling, and the sizes of different second instance images may be the same or different.

[0088] In this embodiment, overlap is used to indicate the degree of overlap between images, such as the overlap between instance images in a candidate lesion region. In some embodiments, overlap can be represented by the overlap rate corresponding to each instance image. For example, the average of the overlap rates corresponding to each instance image is determined as the overlap of these instance images in the candidate region.

[0089] For each instance image, the overlap rate can refer to the ratio between the total area of ​​the overlapping region between the instance image and other instance images and the area of ​​the instance image itself. In some embodiments, for a second target instance image among at least two second instance images, there exists at least one second instance image whose overlap rate with the second target instance image is greater than or equal to an overlap threshold. Of course, the specific value of the overlap threshold can also be set by those skilled in the art according to the actual situation, and this application embodiment does not specifically limit it in this regard.

[0090] In some embodiments, overlap can also be expressed as the ratio of the sum of the areas of all instance images in the candidate lesion region to the candidate lesion region; overlap can also be expressed as the ratio of the difference between the sum of the areas of all instance images in the candidate lesion region and the candidate lesion region to the candidate lesion region; overlap can also be expressed as the ratio of the total area of ​​the overlapping regions between each pair of instance images in the candidate lesion region to the candidate lesion region. It should be noted that if a region is both an overlapping region between instance images A and B, and also an overlapping region between instance images A and C, then this region is also an overlapping region between instance images B and C. Therefore, when calculating the total area of ​​the overlapping regions between each pair of instance images, this region should be calculated three times, that is, the area of ​​this region should be multiplied by 3 before being included in the total area of ​​the overlapping regions between each pair of instance images. Thus, the calculated total area of ​​the overlapping regions between each pair of instance images can be greater than the area of ​​the candidate region. In some embodiments, overlap is a value greater than or equal to 0; of course, the value of overlap can also be greater than 1. The degree of overlap can also be defined and calculated in other ways, which can be set by relevant technical personnel according to the actual situation. This application embodiment does not make specific limitations on this.

[0091] In some embodiments, step 330 may further include the following sub-steps:

[0092] 1. Based on the candidate lesion area, extract the candidate lesion image from the pathological image.

[0093] In some embodiments, after identifying the candidate lesion region in the pathological image, the image of the candidate lesion region can be directly determined as the candidate lesion image; alternatively, the image of the candidate lesion region plus the image of the surrounding area can be determined as the candidate lesion image. In some embodiments, after identifying the candidate lesion region, a region with the same shape as the pathological image and containing the candidate lesion region is determined, with the center of the candidate lesion region as the center of the candidate lesion image, and the image within this region is determined as the candidate lesion image.

[0094] 2. Based on the size of the pathological image, the candidate lesion image is scaled to obtain the target lesion image, and the size of the target lesion image is consistent with the size of the pathological image.

[0095] In some embodiments, the size of the candidate lesion image is generally smaller than the size of the pathological image (i.e., the fixed size mentioned above). In such cases, the candidate lesion image can be magnified to obtain a target lesion image with the same size as the pathological image.

[0096] 3. At least two second instance images are obtained by sampling from the target lesion image using a second sampling method.

[0097] In some embodiments, the target lesion image is sampled using the second sampling method described above (such as dense sampling) to obtain at least two second instance images. In some embodiments, the number of second instance images sampled from the target lesion image is greater than or equal to a second threshold, thereby ensuring a certain number of samples.

[0098] In the above embodiments, when the shape of the candidate lesion image is the same as that of the pathological image, it is only necessary to enlarge the size of the candidate lesion image in all directions according to the same ratio to obtain the target lesion image with the same size data as the pathological image, without needing to stretch or shorten the size of the candidate lesion image in a certain direction.

[0099] Step 340: Determine the lesion indication information of the pathological image based on the feature information extracted from at least two second instance images.

[0100] In some embodiments, after acquiring the second instance image, the computer device determines lesion indication information of the pathological image based on feature information extracted from the at least two second instance images. The lesion indication information is used to indicate the lesion area in the pathological image.

[0101] In summary, the technical solution provided in this application provides that by sampling a pathological image to obtain an instance image, extracting feature information from the instance image, and automatically determining the lesion area of ​​the pathological image based on the feature information of the instance image, the consumption of human resources is reduced and the cost required to determine the lesion area is saved.

[0102] In addition, in this embodiment, candidate lesion regions in the pathological image are first determined based on a first sampling method, and then a second instance image is obtained based on a second sampling method with a greater sampling overlap than the first sampling method. The lesion region of the pathological image is determined from the candidate lesion region based on the second instance image. That is, the pathological image is processed globally using the first instance image and then locally using the second instance image. From global to local, this reduces the area of ​​the pathological image that needs to be sampled using the second sampling method, thereby reducing the number of second instance images that need to be collected and feature information extracted. This reduces the computational resources required to determine the lesion region and improves the efficiency required to determine the lesion region.

[0103] Furthermore, candidate lesion areas are more likely to contain lesion areas than other areas, and have a higher signal-to-noise ratio. By performing a second sampling method only on candidate lesion areas, information loss can be reduced and the ability to perceive smaller lesion areas can be enhanced, thereby enabling more accurate identification of lesion areas in pathological images.

[0104] The method for determining the above-mentioned candidate lesion areas is described below.

[0105] In some possible implementations, step 320 above may also include the following steps (1-4):

[0106] 1. Perform feature encoding on each first instance image to obtain the first feature information corresponding to each first instance image.

[0107] In some embodiments, feature encoding is performed on each first instance image to obtain a feature vector corresponding to each first instance image. The feature vector corresponding to the first instance image can be a 768-dimensional feature or a feature of other dimensions. This application embodiment does not specifically limit this.

[0108] In some embodiments, a first feature information is extracted from a first instance image via a backbone network. The process of the backbone network extracting the first feature information from the first instance image can be described as follows:

[0109] h k =Encoder(x k )

[0110] Among them, h k It can be a 768-dimensional feature extracted from an instance.

[0111] 2. Perform feature fusion on the first feature information corresponding to each first instance image to obtain the global feature information of the pathological image.

[0112] In some embodiments, the first feature information corresponding to each first instance image is aggregated (i.e., feature fusion) through a first classification network to obtain the global feature information of the pathological image.

[0113] In some embodiments, an attention mechanism is used to process the first feature information corresponding to each first instance image to obtain the weights corresponding to each first feature information; based on the weights corresponding to each first feature information, a weighted summation is performed on each first feature information to obtain the global feature information of the pathological image.

[0114] In some embodiments, the process of obtaining global feature information of a pathological image can refer to the following formula:

[0115] Logits(B) = c{g(h0,h1,…,h...} k )}

[0116]

[0117] in:

[0118]

[0119] Wherein, Logits(B) represents the global features of the pathological image, c represents the first classification network, and c represents the aggregation network based on the attention mechanism. V∈R L×M .

[0120] In some embodiments, the attention module corresponding to the attention mechanism assigns weights to each first instance image and sums them with weights to obtain the feature vector representing the bag (i.e., global feature information). The fused global feature information can be fed into a first classification network to predict the classification label corresponding to each first instance image. The attention module can also be viewed as a binary classification network.

[0121] The classification networks (such as the first classification network, the second classification network, and the third classification network) involved in the embodiments of this application can be represented as follows:

[0122]

[0123] Among them, c(h) i ) n This represents the nth classification network.

[0124] 3. Based on the global feature information and the first feature information corresponding to each first instance image, determine the first prediction probability corresponding to each first instance image. The first prediction probability refers to the probability that the first instance image contains a lesion area.

[0125] In some embodiments, after obtaining the first feature information corresponding to each first instance image and the global feature information of the pathological image, the probability that each first instance image contains a lesion region can be determined by a first classification network based on the global feature information and the first feature information corresponding to each first instance image.

[0126] In some embodiments, determining the first prediction probability corresponding to each first instance image based on global feature information and the first feature information corresponding to each first instance image may include the following sub-steps (3.1 to 3.3):

[0127] 3.1 Generate the first probability coefficient based on the global feature information mapping. The first probability coefficient refers to the probability that the pathological image contains a lesion area.

[0128] In some embodiments, global feature information is mapped to a probability distribution between 0 and 1 using Softmax. For example, if the set (also called the bag) of all first instance images is defined as B, then B = {(x0,y0),(x1,y1),…,(x...} k ,yk )}, where x k ,y k Let Y and Y represent the labels of the k-th first instance image and the k-th first instance image of the package, respectively. Then the label Y(B) of B can be defined as:

[0129]

[0130] Where Y(B) = 0 indicates that there is no lesion area in the pathological image, and Y(B) = 1 indicates that there is a lesion area in the pathological image.

[0131] 3.2 For each first instance image, generate a second probability coefficient based on the first feature information corresponding to the first instance image.

[0132] In some embodiments, the first feature information corresponding to each first instance image is mapped by Sigmoid to obtain the second probability coefficient corresponding to each first instance image. The second probability coefficient refers to the initial probability that the first instance image contains a lesion area.

[0133] 3.3. Determine the first predicted probability corresponding to the first instance image based on the first probability coefficient and the second probability coefficient.

[0134] In some embodiments, the first probability coefficient and the second probability are multiplied to obtain the first prediction probability corresponding to each first instance image. This process can be expressed as:

[0135]

[0136] Among them, p(h i ) represents the first predicted probability of the i-th first instance image, softmax{c(h i )} represents the first probability coefficient. This represents the second probability coefficient.

[0137] In some embodiments, the first predicted probability corresponding to the first instance image can also be expressed as:

[0138]

[0139] 4. Based on the position of the first instance image corresponding to the first predicted probability that satisfies the first condition in the pathological image, determine the candidate lesion area.

[0140] In some embodiments, after determining the first predicted probability corresponding to each first instance image, a first instance image that meets the first condition is selected, and a candidate lesion region is determined based on this.

[0141] In some embodiments, the first condition may be that the first predicted probability corresponding to the first instance image is greater than or equal to a third threshold.

[0142] In some embodiments, the first condition may also be the sorting of first instance images by their first predicted probability from largest to smallest, with the first instance image corresponding to a first instance image having a first predicted probability of not less than x%, such as the first instance image corresponding to a first instance image having a first predicted probability of not less than 2%. That is, the first instance images that satisfy the first condition refer to the top x% of first instance images with the highest first predicted probability of having a lesion area. Here, x can be 1, 2, 3, etc., and the specific value of x can be set by those skilled in the art according to the actual situation. This application embodiment does not specifically limit this value.

[0143] In some embodiments, the image region composed of the first instance images that meet the first condition can be directly determined as the candidate lesion region, or the first instance image that meets the first condition and its surrounding region can be determined as the candidate lesion region.

[0144] The following describes how the above-mentioned lesion indication information is obtained.

[0145] In one possible implementation, step 340 above further includes the following steps (1-3):

[0146] 1. Perform feature encoding on each second instance image to obtain the second feature information corresponding to each second instance image;

[0147] 2. Perform feature fusion on the second feature information corresponding to each second instance image to obtain the local feature information of the pathological image for the candidate lesion area;

[0148] 3. Based on local feature information and global feature information of the pathological image, determine the lesion probability distribution information in the pathological image. The lesion probability distribution information is used to indicate the probability distribution of lesion areas in the pathological image; among them, the lesion indication information includes the lesion probability distribution information.

[0149] In some embodiments, feature fusion is performed on the second feature information corresponding to each second instance image to obtain local feature information of the pathological image for the candidate lesion region, including: processing the second feature information corresponding to each second instance image using an attention mechanism to obtain the weights corresponding to each second feature information; and performing weighted summation on each second feature information according to the weights corresponding to each second feature information to obtain local feature information.

[0150] In some embodiments, lesion indication information is obtained from a lesion region determination model, which includes a coding network, a first classification network, a second classification network, and a third classification network; wherein:

[0151] An encoding network is used to encode features of a first instance image and a second instance image to obtain first feature information corresponding to the first instance image and second feature information corresponding to the second instance image.

[0152] The first classification network is used to determine the first prediction probability and global feature information of the pathological image corresponding to each first instance image based on the first feature information corresponding to each first instance image.

[0153] The second classification network is used to determine the second prediction probability corresponding to each second instance image and the local feature information of the pathological image for the candidate lesion area based on the second feature information corresponding to each second instance image.

[0154] The third classification network is used to determine the probability distribution of lesions in pathological images based on global and local feature information.

[0155] Some of the steps in this implementation method can be found in the above embodiment, and will not be repeated here.

[0156] In some embodiments, the second instance image obtained by sampling using the second sampling method can be represented as:

[0157]

[0158] Where, p u This represents the first prediction probability corresponding to the first instance image. This represents the second instance image obtained by sampling using the second sampling method.

[0159] In some embodiments, local feature information and global feature information of the pathological image are concatenated and input into a third classification network to obtain the lesion probability distribution information in the pathological image. This process can be represented as:

[0160] Logits(B) = c3{concat(z1,z2)}

[0161] Wherein, Logits(B) represents the probability distribution information of lesions in the pathological image, z1 represents global feature information, and z2 represents local feature information.

[0162] In another possible implementation, step 340 above may include the following steps (1-4):

[0163] 1. Perform feature encoding on each second instance image to obtain the second feature information corresponding to each second instance image;

[0164] 2. Perform feature fusion on the second feature information corresponding to each second instance image to obtain the local feature information of the pathological image for the candidate lesion area;

[0165] 3. Based on the local feature information and the second feature information corresponding to each second instance image, determine the second prediction probability corresponding to each second instance image. The second prediction probability refers to the probability that the second instance image contains a lesion area.

[0166] 4. Based on the position of the second instance image corresponding to the second predicted probability that satisfies the second condition in the pathological image, determine the lesion indication information of the pathological image.

[0167] In some embodiments, the second condition may be that the second predicted probability corresponding to the second instance image is greater than or equal to the fourth threshold; or it may be that the second instance image is sorted from largest to smallest according to the second predicted probability, and the second predicted probability corresponding to the second instance image is not lower than the second proportion threshold, such as the second instance image with a second predicted probability of not less than 15%.

[0168] In some embodiments, the image region composed of the second instance images that meet the second condition can be directly determined as the lesion region, or the second instance image that meets the second condition and its surrounding area can be determined as the lesion region.

[0169] In some embodiments, determining the second prediction probability corresponding to each second instance image based on local feature information and the second feature information corresponding to each second instance image further includes the following steps (3.1 to 3.3):

[0170] 3.1 Generate a third probability coefficient based on the mapping of local feature information. The third probability coefficient refers to the probability that the candidate lesion region contains the lesion region.

[0171] 3.2 For each second instance image, a fourth probability coefficient is generated based on the second feature information corresponding to the second instance image. The fourth probability coefficient refers to the initial probability that the second instance image contains a lesion area.

[0172] 3.3. Determine the second prediction probability corresponding to the second instance image based on the third and fourth probability coefficients.

[0173] In some embodiments, referring to the above embodiments, local feature information is mapped using Softmax to generate a third probability coefficient; and second feature information corresponding to the second instance image is mapped using Sigmoid to generate a fourth probability coefficient.

[0174] Some of the steps in this implementation method can be found in the above embodiment, and will not be repeated here.

[0175] In this implementation, after the candidate lesion region is determined, the lesion indication information can be predicted directly from the second instance image without needing global feature information or other information related to the first instance image. This simplifies the determination of lesion indication information, saves the processing resources and time required to determine lesion indication information, and improves the efficiency of lesion indication information determination.

[0176] like Figures 4-7 As shown, the method may include the following steps (1-4):

[0177] 1. For example Figure 4 As shown, the background image 41 and the foreground image 42 in the pathological image are separated, and then the foreground image 42 is uniformly cropped into multiple first instance images 43 through uniform sampling (i.e., the first sampling method);

[0178] 2. For example Figure 5 As shown, multiple first instance images 43 are input into the encoding network 10 and encoded into 768-dimensional vectors, thus obtaining the feature information of the first instance images 43; the feature information of the first instance images 43 is input into the first classification network to obtain the first predicted probability corresponding to each first instance image and the global feature information of the pathological image, thereby obtaining the probability distribution of each local region; optionally, the encoding network 10 can be composed of convolutional blocks and Swing transform blocks;

[0179] 3. For example Figure 6 As shown, based on the probability distribution of each local region obtained in the previous step, the lesion region determination model can perform dense sampling on the candidate lesion region (i.e. key region) 44, and use the second classification network 30 to encode and infer it, so as to obtain the second predicted probability corresponding to each second instance image 45 and the local feature information of the pathological image for the candidate lesion region 44.

[0180] 4. For example Figure 7 As shown, the results obtained from the second and third steps above are concatenated and input into another classification network (i.e., the third classification network) to obtain the final lesion indication information.

[0181] Please refer to Figure 8 This document illustrates a flowchart of a method for determining lesion regions in a pathological image according to another embodiment of this application. Each step of this method can be performed by the aforementioned computer device. The method may include the following steps (810-840):

[0182] Step 810: Obtain a training sample set, which includes at least one sample pathological image.

[0183] In some embodiments, the sample pathological image is a pathological image with the lesion area already labeled.

[0184] Step 820: Sample the pathological images of the samples using the first sampling method and the second sampling method respectively, to obtain at least two first sample instances and at least two second sample instances corresponding to the pathological images of the samples, wherein the overlap between the second sample instances is greater than the overlap between the first sample instances.

[0185] Step 830: Based on the feature information extracted from at least two first sample instances and the feature information extracted from at least two second sample instances, determine the lesion probability distribution information of the sample pathological image; wherein, the lesion probability distribution information is used to indicate the probability distribution of lesion areas in the sample pathological image.

[0186] In some embodiments, such as Figure 9 As shown, the lesion area determination model includes a coding network 10, a first classification network 20, a second classification network 30, and a third classification network. Step 430 may include the following steps:

[0187] 1. The coding network 10 is used to encode the features of each first sample instance 46 and each second sample instance 47 respectively, so as to obtain the feature information 48 corresponding to each first sample instance 46 and the feature information 49 corresponding to each second sample instance 47 respectively.

[0188] 2. The first classification network 20 is used to process the feature information 48 corresponding to each first sample instance 46 to obtain the pseudo label and first prediction probability corresponding to each first sample instance 46, as well as the global feature information of the sample pathological image; wherein, the first prediction probability refers to the probability that the first sample instance 46 contains a lesion area.

[0189] 3. The feature information corresponding to each second sample instance 47 is processed by the second classification network to obtain the second prediction probability and local feature information of the pathological image corresponding to each second sample instance 47; wherein, the second prediction probability refers to the probability that the second sample instance 47 contains a lesion area.

[0190] 4. A third classification network is used to process global and local feature information to obtain the lesion probability distribution information in the sample pathological images.

[0191] Step 840: Train the lesion area determination model based on the lesion probability distribution information.

[0192] In some embodiments, such as Figure 9 As shown, step 840 may include the following steps:

[0193] 1. Based on the pseudo-labels corresponding to each first sample instance 46, generate a first loss 52. The first loss 52 is used to measure the ability of the first classification network 20 to distinguish between positive sample instances and negative sample instances. A positive sample instance refers to a first sample instance 46 that contains a lesion area, and a negative sample instance refers to a first sample instance 46 that does not contain a lesion area.

[0194] 2. Based on the first prediction probability corresponding to each first sample instance 46, a second loss 53 is generated. The second loss 53 is used to measure the accuracy of the prediction result of the first classification network 20 on whether the first sample instance 46 contains a lesion area.

[0195] 3. Based on the second prediction probability corresponding to each second sample instance 47, a third loss 50 is generated. The third loss 50 is used to measure the accuracy of the second classification network 30 in predicting whether the second sample instance 47 contains a lesion area.

[0196] 4. A fourth loss 51 is generated based on the lesion probability distribution information. The fourth loss 51 is used to measure the accuracy of the prediction results of the third classification network 40 on the probability distribution information of the lesion area in the sample pathological image.

[0197] 5. The lesion area determination model is trained based on the first loss 48, the second loss 49, and the third loss 50.

[0198] In some embodiments, a self-supervised model training method, such as the Moco V3 method, is used to train the lesion region determination model. Theoretically, both branches of the lesion region determination model (i.e., the branch processing the first example image and the branch processing the second example image) can obtain sufficient information to determine whether a lesion region exists in the pathological image. Therefore, the label of the pathological image (i.e., the label indicating whether a lesion region exists in the pathological image) can serve as the final predicted label or as an auxiliary signal during the supervised model training process.

[0199] In some embodiments, during the training of the lesion region determination model, the labels of the sample pathological images can serve as both training labels and auxiliary signals to supervise the training process. In some embodiments, the k sample instances with the highest probability of containing a lesion region in the first or second sample instance are designated as positive sample instances (also called positivity sample instances), and the k sample instances with the lowest probability of containing a lesion region are designated as negative sample instances (also called negativity sample instances). Pseudo-labels are then generated and constrained using cross-entropy loss. The loss of the lesion region determination model can be represented as follows:

[0200]

[0201] in, This represents the loss of the i-th classification network. These are the losses corresponding to the first classification network, the second classification network, the third classification network, and the encoding network, respectively.

[0202] The total loss of the lesion area determination model can be expressed as follows:

[0203]

[0204] in, This represents the total loss of the lesion region determination model, where λ1 to λ4 are... The coefficients corresponding to these four losses.

[0205] In summary, the technical solution provided in this application involves processing the entire pathological image using a first sample instance to quickly identify candidate lesion regions. Then, a second sample instance is used to process key areas (i.e., candidate lesion regions) of the pathological image in a more targeted manner. Since the probability of a candidate lesion region containing a lesion region is greater than that of other regions, the signal-to-noise ratio is higher. In the second stage, by processing only the candidate lesion regions, information loss can be reduced and the ability to perceive smaller lesion regions can be enhanced. This allows the trained lesion region determination model to more accurately determine the lesion regions in the pathological image, thereby improving the accuracy and precision of the lesion region determination model.

[0206] Taking the prostate as an example, comparative experiments and ablation experiments on the prostate dataset and its difficult sample subset demonstrate the superiority of the technical solution provided in this application and its superior visualization effects. Figure 10 As shown, for each pathological image, the global pathological image 55 is first processed using the first instance image. Feature information from the first instance image is extracted using the first sampling method, resulting in the global aggregated features of the pathological image and the coarse distribution information 56 of the lesion region, thus identifying candidate lesion regions 57. Then, the local area of ​​the pathological image (i.e., the candidate lesion region 57) is processed using the second instance image to obtain the probability distribution heatmap 58 of the candidate lesion region 57. Figure 10 It can be seen that even for relatively small lesion areas (such as lesion area 59), the technical solution provided by the embodiments of this application can be identified relatively accurately. It is evident that the embodiments of this application have a relatively good recall rate for small lesion areas (such as small tumors), that is, the solution provided by the embodiments of this application can also have a good recall rate for difficult sample images.

[0207] As shown in Table 1 below, through comparative experiments, the test set using the technical solution provided in the embodiments of this application showed significantly better performance in all aspects compared to the comparative cases. It is evident that, compared to the comparative cases, the technical solution provided in the embodiments of this application has higher accuracy and precision in determining lesion areas in pathological images.

[0208] Table 1. Results of comparative and ablation experiments

[0209]

[0210] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0211] Please refer to Figure 11 This diagram illustrates a block diagram of a lesion region determination device for pathological images according to an embodiment of this application. The device has the function of implementing the aforementioned lesion region determination method for pathological images; this function can be implemented in hardware or by hardware executing corresponding software. The device can be a computer device or can be installed within a computer device. The device 1100 may include: a first image acquisition module 1110, a lesion region determination module 1120, a second image acquisition module 1130, and a lesion information determination module 1140.

[0212] The first image acquisition module 1110 is used to sample at least two first instance images from a pathological image using a first sampling method.

[0213] The lesion area determination module 1120 is used to determine candidate lesion areas in the pathological image based on feature information extracted from the at least two first instance images.

[0214] The second image acquisition module 1130 is used to sample at least two second instance images from the candidate lesion region using a second sampling method, wherein the overlap between the second instance images is greater than the overlap between the first instance images.

[0215] The lesion information determination module 1140 is used to determine lesion indication information of the pathological image based on feature information extracted from the at least two second instance images, wherein the lesion indication information is used to indicate the lesion area in the pathological image.

[0216] In some embodiments, such as Figure 12 As shown, the lesion area determination module 1120 includes: a feature encoding submodule 1121, a feature fusion submodule 1122, a probability determination submodule 1123, and a region determination submodule 1124.

[0217] The feature encoding submodule 1121 is used to perform feature encoding on each of the first instance images to obtain the first feature information corresponding to each of the first instance images.

[0218] The feature fusion submodule 1122 is used to perform feature fusion on the first feature information corresponding to each of the first instance images to obtain the global feature information of the pathological image.

[0219] The probability determination submodule 1123 is used to determine the first prediction probability corresponding to each of the first instance images based on the global feature information and the first feature information corresponding to each of the first instance images. The first prediction probability refers to the probability that the first instance image contains a lesion area.

[0220] The region determination submodule 1124 is used to determine the candidate lesion region based on the position of the first instance image in the pathological image corresponding to the first predicted probability that satisfies the first condition.

[0221] In some embodiments, such as Figure 12 As shown, the feature fusion submodule 1122 is used for:

[0222] An attention mechanism is used to process the first feature information corresponding to each of the first instance images to obtain the weights corresponding to each of the first feature information.

[0223] Based on the weights corresponding to each of the first feature information, a weighted summation is performed on each of the first feature information to obtain the global feature information of the pathological image.

[0224] In some embodiments, such as Figure 12 As shown, the probability determination submodule 1123 is used for:

[0225] A first probability coefficient is generated based on the global feature information mapping, where the first probability coefficient refers to the probability that the pathological image contains a lesion area.

[0226] For each of the first instance images, a second probability coefficient is generated based on the first feature information corresponding to the first instance image. The second probability coefficient refers to the initial probability that the first instance image contains a lesion area.

[0227] Based on the first probability coefficient and the second probability coefficient, the first prediction probability corresponding to the first instance image is determined.

[0228] In some embodiments, such as Figure 12 As shown, the lesion information determination module 1140 includes: a probability determination submodule 1141.

[0229] The feature encoding submodule 1121 is further configured to perform feature encoding on each of the second instance images to obtain the second feature information corresponding to each of the second instance images.

[0230] The feature fusion submodule 1122 is further configured to perform feature fusion on the second feature information corresponding to each of the second instance images to obtain the local feature information of the pathological image for the candidate lesion area.

[0231] The probability determination submodule 1141 is used to determine the lesion probability distribution information in the pathological image based on the local feature information and the global feature information of the pathological image. The lesion probability distribution information is used to indicate the probability distribution of lesion areas in the pathological image. The lesion indication information includes the lesion probability distribution information.

[0232] In some embodiments, such as Figure 12 As shown, the feature fusion submodule 1122 is used for:

[0233] An attention mechanism is used to process the second feature information corresponding to each of the second instance images to obtain the weights corresponding to each of the second feature information.

[0234] Based on the weights corresponding to each of the second feature information, a weighted summation is performed on each of the second feature information to obtain the local feature information.

[0235] In some embodiments, such as Figure 12 As shown, the lesion information determination module 1140 includes: lesion information determination submodule 1142.

[0236] The feature encoding submodule 1121 is further configured to perform feature encoding on each of the second instance images to obtain the second feature information corresponding to each of the second instance images.

[0237] The feature fusion submodule 1122 is further configured to perform feature fusion on the second feature information corresponding to each of the second instance images to obtain the local feature information of the pathological image for the candidate lesion area.

[0238] The probability determination submodule 1141 is used to determine the second prediction probability corresponding to each of the second instance images based on the local feature information and the second feature information corresponding to each of the second instance images. The second prediction probability refers to the probability that the second instance image contains the lesion area.

[0239] The lesion information determination submodule 1142 is used to determine the lesion indication information of the pathological image based on the position of the second instance image corresponding to the second predicted probability that satisfies the second condition in the pathological image.

[0240] In some embodiments, such as Figure 12 As shown, the probability determination submodule 1142 is used for:

[0241] A third probability coefficient is generated based on the local feature information mapping, whereby the third probability coefficient refers to the probability that the candidate lesion region contains the lesion region.

[0242] For each second instance image, a fourth probability coefficient is generated based on the second feature information corresponding to the second instance image. The fourth probability coefficient refers to the initial probability that the second instance image contains a lesion area.

[0243] The second predicted probability corresponding to the second instance image is determined based on the third probability coefficient and the fourth probability coefficient.

[0244] In some embodiments, the first image acquisition module 1110 is configured to:

[0245] The pathological image is segmented to determine the background image and foreground image in the pathological image;

[0246] The pathological image is segmented using the first sampling method to obtain at least two first candidate instance images;

[0247] From the at least two first candidate instance images, the first candidate instance image containing the foreground image is determined as the first instance image.

[0248] In some embodiments, the second image acquisition module 1130 is configured to:

[0249] Candidate lesion images are extracted from the pathological images based on the candidate lesion regions;

[0250] Based on the size of the pathological image, the candidate lesion image is scaled to obtain the target lesion image, the size of the target lesion image being consistent with the size of the pathological image;

[0251] The at least two second instance images are obtained by sampling from the target lesion image using the second sampling method.

[0252] In some embodiments, the lesion indication information is obtained from a lesion region determination model, which includes a coding network, a first classification network, a second classification network, and a third classification network; wherein,

[0253] The encoding network is used to perform feature encoding on the first instance image and the second instance image to obtain first feature information corresponding to the first instance image and second feature information corresponding to the second instance image.

[0254] The first classification network is used to determine the first prediction probability corresponding to each of the first instance images and the global feature information of the pathological image based on the first feature information corresponding to each of the first instance images.

[0255] The second classification network is used to determine the second prediction probability corresponding to each of the second instance images and the local feature information of the pathological image for the candidate lesion region based on the second feature information corresponding to each of the second instance images.

[0256] The third classification network is used to determine the lesion probability distribution information in the pathological image based on the global feature information and the local feature information.

[0257] In summary, the technical solution provided in this application provides that by sampling a pathological image to obtain an instance image, extracting feature information from the instance image, and automatically determining the lesion area of ​​the pathological image based on the feature information of the instance image, the consumption of human resources is reduced and the cost required to determine the lesion area is saved.

[0258] In addition, in this embodiment, candidate lesion regions in the pathological image are first determined based on the first sampling method, and then a second instance image is obtained based on the second sampling method with a greater sampling overlap than the first sampling method. The lesion region of the pathological image is determined from the candidate lesion region based on the second instance image. In this way, the area of ​​the pathological image that needs to be sampled using the second sampling method is reduced, thereby reducing the number of second instance images that need to be collected and feature information extracted, thereby reducing the computational resources required to determine the lesion region and improving the efficiency required to determine the lesion region.

[0259] Furthermore, candidate lesion areas are more likely to contain lesion areas than other areas, and have a higher signal-to-noise ratio. By performing a second sampling method only on candidate lesion areas, information loss can be reduced and the ability to perceive smaller lesion areas can be enhanced, thereby enabling more accurate identification of lesion areas in pathological images.

[0260] Please refer to Figure 13This diagram illustrates a block diagram of a training apparatus for a lesion region determination model according to an embodiment of this application. The apparatus has the function of implementing the training method for the aforementioned lesion region determination model; this function can be implemented in hardware or by hardware executing corresponding software. The apparatus can be a computer device or can be installed within a computer device. The apparatus 1300 may include: a sample acquisition module 1310, an instance acquisition module 1320, an information acquisition module 1330, and a model training module 1340.

[0261] The sample acquisition module 1310 is used to acquire a training sample set, which includes at least one sample pathological image.

[0262] The instance acquisition module 1320 is used to sample the sample pathological image using a first sampling method and a second sampling method respectively, to obtain at least two first sample instances and at least two second sample instances corresponding to the sample pathological image, wherein the overlap between the second sample instances is greater than the overlap between the first sample instances.

[0263] The information acquisition module 1330 is used to determine the lesion probability distribution information of the sample pathological image based on the feature information extracted from the at least two first sample instances and the feature information extracted from the at least two second sample instances; wherein the lesion probability distribution information is used to indicate the probability distribution of lesion areas in the sample pathological image.

[0264] The model training module 1340 is used to train a model for the lesion region based on the lesion probability distribution information.

[0265] In some embodiments, the lesion area determination model includes a coding network, a first classification network, a second classification network, and a third classification network; the information acquisition module 1330 is used for:

[0266] The encoding network is used to encode the features of each first sample instance and each second sample instance respectively, so as to obtain the feature information corresponding to each first sample instance and the feature information corresponding to each second sample instance respectively;

[0267] The first classification network is used to process the feature information corresponding to each first sample instance to obtain the pseudo label and first prediction probability corresponding to each first sample instance, as well as the global feature information of the sample pathological image; wherein, the first prediction probability refers to the probability that the first sample instance contains a lesion area.

[0268] The second classification network is used to process the feature information corresponding to each second sample instance to obtain the second predicted probability and the local feature information of the pathological image corresponding to each second sample instance; wherein, the second predicted probability refers to the probability that the second sample instance contains a lesion area;

[0269] The third classification network is used to process the global feature information and the local feature information to obtain the lesion probability distribution information in the sample pathological image.

[0270] In some embodiments, the model training module 1340 is configured to:

[0271] A first loss is generated based on the pseudo-labels corresponding to each of the first sample instances. The first loss is used to measure the ability of the first classification network to distinguish between positive sample instances and negative sample instances. The positive sample instance refers to the first sample instance that contains the lesion area, and the negative sample instance refers to the first sample instance that does not contain the lesion area.

[0272] A second loss is generated based on the first prediction probability corresponding to each of the first sample instances. The second loss is used to measure the accuracy of the first classification network's prediction of whether the first sample instance contains the lesion area.

[0273] A third loss is generated based on the second prediction probability corresponding to each of the second sample instances. The third loss is used to measure the accuracy of the second classification network's prediction of whether the second sample instance contains the lesion area.

[0274] A fourth loss function is generated based on the lesion probability distribution information. The fourth loss function is used to measure the accuracy of the third classification network's prediction of the probability distribution information of the lesion region in the sample pathological image.

[0275] The model for determining the lesion region is trained based on the first loss, the second loss, and the third loss.

[0276] In summary, the technical solution provided in this application involves processing the entire pathological image using a first sample instance to quickly identify candidate lesion regions. Then, a second sample instance is used to process key areas (i.e., candidate lesion regions) of the pathological image in a more targeted manner. Since the probability of a candidate lesion region containing a lesion region is greater than that of other regions, the signal-to-noise ratio is higher. In the second stage, by processing only the candidate lesion regions, information loss can be reduced and the ability to perceive smaller lesion regions can be enhanced. This allows the trained lesion region determination model to more accurately determine the lesion regions in the pathological image, thereby improving the accuracy and precision of the lesion region determination model.

[0277] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0278] Please refer to Figure 14 This diagram illustrates the structural block diagram of a computer device provided in one embodiment of this application. This computer device can be used to implement the functions of the above-described method for determining lesion regions in pathological images, or to implement the functions of the above-described method for training lesion region determination models. Specifically:

[0279] Computer device 1400 includes a central processing unit (CPU) 1401, a system memory 1404 including random access memory (RAM) 1402 and read-only memory (ROM) 1403, and a system bus 1405 connecting the system memory 1404 and the CPU 1401. Computer device 1400 also includes a basic input / output system (I / O system) 1406 that facilitates information transfer between various devices within the computer, and a mass storage device 1407 for storing the operating system 1413, application programs 1414, and other program modules 1415.

[0280] The basic input / output system 1406 includes a display 1408 for displaying information and an input device 1409 for user input, such as a mouse or keyboard. Both the display 1408 and the input device 1409 are connected to the central processing unit 1401 via an input / output controller 1410 connected to the system bus 1405. The basic input / output system 1406 may also include the input / output controller 1410 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1410 also provides output to a display screen, printer, or other types of output devices.

[0281] Mass storage device 1407 is connected to central processing unit 1401 via a mass storage controller (not shown) connected to system bus 1405. Mass storage device 1407 and its associated computer-readable media provide non-volatile storage for computer device 1400. That is, mass storage device 1407 may include computer-readable media (not shown) such as hard disk or CD-ROM (Compact Disc Read-Only Memory) drive.

[0282] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage devices, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 1404 and mass storage device 1407 described above can be collectively referred to as memory.

[0283] According to various embodiments of this application, the computer device 1400 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1400 can be connected to the network 1412 via the network interface unit 1411 connected to the system bus 1405, or the network interface unit 1411 can be used to connect to other types of networks or remote computer systems (not shown).

[0284] The memory also includes a computer program stored in the memory and configured to be executed by one or more processors to implement the above-described method for determining lesion regions in pathological images, or to implement the above-described method for training the lesion region determination model.

[0285] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein a computer program is stored therein, which, when executed by a processor, implements the above-described method for determining lesion regions in pathological images, or implements the above-described method for training a lesion region determination model.

[0286] Optionally, the computer-readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0287] In an exemplary embodiment, a computer program product is also provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the aforementioned method for determining lesion regions in pathological images, or to perform the aforementioned method for training a lesion region determination model.

[0288] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0289] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for determining a lesion area of a pathological image, characterized in that, The method comprises: sampling at least two first instance images from the pathological image by using a first sampling manner; respectively performing feature coding on each of the first instance images to obtain first feature information corresponding to each of the first instance images; respectively processing the first feature information corresponding to each of the first instance images by using an attention mechanism to obtain weights corresponding to each of the first feature information; performing weighted summation processing on each of the first feature information according to the weights corresponding to each of the first feature information to obtain global feature information of the pathological image; determining first prediction probabilities corresponding to each of the first instance images according to the global feature information and the first feature information corresponding to each of the first instance images, the first prediction probability being a probability that the first instance image contains a lesion region; determining a candidate lesion region in the pathological image based on a position of a first instance image corresponding to a first prediction probability satisfying a first condition in the pathological image; sampling at least two second instance images from the candidate lesion region by using a second sampling manner, an overlap degree between the second instance images being greater than an overlap degree between the first instance images; determining lesion indication information of the pathological image based on feature information extracted from the at least two second instance images, the lesion indication information being used to indicate a lesion region in the pathological image.

2. The method of claim 1, wherein, The determining of the first prediction probabilities corresponding to each of the first instance images according to the global feature information and the first feature information corresponding to each of the first instance images comprises: generating a first probability coefficient according to the global feature information, the first probability coefficient being a probability that the pathological image contains a lesion region; for each of the first instance images, generating a second probability coefficient according to the first feature information corresponding to the first instance image, the second probability coefficient being an initial probability that the first instance image contains a lesion region; and determining the first prediction probability corresponding to the first instance image according to the first probability coefficient and the second probability coefficient.

3. The method of claim 1, wherein, The determining of the lesion indication information of the pathological image based on the feature information extracted from the at least two second instance images comprises: respectively performing feature coding on each of the second instance images to obtain second feature information corresponding to each of the second instance images; performing feature fusion on the second feature information corresponding to each of the second instance images to obtain local feature information of the pathological image for the candidate lesion region; determining lesion probability distribution information in the pathological image based on the local feature information and global feature information of the pathological image, the lesion probability distribution information being used to indicate a probability distribution of a lesion region in the pathological image; and 4. The method of claim 3, wherein, the lesion indication information comprises the lesion probability distribution information. The performing of the feature fusion on the second feature information corresponding to each of the second instance images to obtain the local feature information of the pathological image for the candidate lesion region comprises: The attention mechanism is used to process the second feature information corresponding to each of the second instance images respectively, to obtain a weight corresponding to each of the second feature information respectively; The second feature information is weighted and summed according to the weight corresponding to each of the second feature information, to obtain the local feature information.

5. The method of claim 1, wherein, The lesion indication information of the pathological image is determined based on the feature information extracted from the at least two second instance images, including: Each of the second instance images is respectively encoded to obtain second feature information corresponding to each of the second instance images respectively; The second feature information corresponding to each of the second instance images is respectively fused to obtain local feature information of the pathological image for the candidate lesion area; According to the local feature information and the second feature information corresponding to each of the second instance images respectively, a second prediction probability corresponding to each of the second instance images is determined, the second prediction probability being a probability that the second instance image contains the lesion area; The lesion indication information of the pathological image is determined based on the position of the second instance image corresponding to the second prediction probability satisfying the second condition in the pathological image.

6. The method of claim 5, wherein, The second prediction probability corresponding to each of the second instance images is determined according to the local feature information and the second feature information corresponding to each of the second instance images, including: A third probability coefficient is generated by mapping the local feature information, the third probability coefficient being a probability that the candidate lesion area contains a lesion area; For each of the second instance images, a fourth probability coefficient is generated by mapping the second feature information corresponding to the second instance image, the fourth probability coefficient being an initial probability that the second instance image contains a lesion area; The second prediction probability corresponding to the second instance image is determined according to the third probability coefficient and the fourth probability coefficient.

7. The method according to any one of claims 1 to 6, characterized in that, The first sampling method is used to sample at least two first instance images from the pathological image, including: The background of the pathological image is divided to determine the background image and the foreground image in the pathological image; The first sampling method is used to segment the pathological image to obtain at least two first candidate instance images; The first candidate instance image containing the foreground image is determined as the first instance image from the at least two first candidate instance images.

8. The method according to any one of claims 1 to 6, characterized in that, The second sampling method is used to sample at least two second instance images from the candidate lesion area, including: The candidate lesion image is extracted from the pathological image according to the candidate lesion area; The candidate lesion image is scaled to obtain a target lesion image, the size of the target lesion image being consistent with the size of the pathological image; The second sampling method is used to sample the at least two second instance images from the target lesion image.

9. The method according to any one of claims 1 to 6, characterized in that, The lesion indication information is obtained by a lesion area determination model, the lesion area determination model including an encoding network, a first classification network, a second classification network and a third classification network; wherein, The encoding network is configured to encode features of the first instance images and the second instance images to obtain first feature information corresponding to the first instance images and second feature information corresponding to the second instance images. The first classification network is configured to determine first prediction probabilities corresponding to the first instance images respectively and global feature information of the pathological image according to the first feature information corresponding to the first instance images respectively. The second classification network is configured to determine second prediction probabilities corresponding to the second instance images respectively and local feature information of the pathological image for the candidate lesion region according to the second feature information corresponding to the second instance images respectively. The third classification network is configured to determine lesion probability distribution information in the pathological image according to the global feature information and the local feature information.

10. A training method for a lesion area determination model, characterized in that, The method comprises: obtaining a training sample set, wherein the training sample set comprises at least one sample pathological image; sampling at least two first sample instances from the sample pathological image by using a first sampling mode; encoding features of the first sample instances respectively to obtain first feature information corresponding to the first sample instances respectively; processing the first feature information corresponding to the first sample instances respectively by using an attention mechanism to obtain weights corresponding to the first feature information respectively; performing weighted summation processing on the first feature information according to the weights corresponding to the first feature information respectively to obtain global feature information of the sample pathological image; determining first prediction probabilities corresponding to the first sample instances respectively according to the global feature information and the first feature information corresponding to the first sample instances respectively, wherein the first prediction probabilities refer to probabilities of the first sample instances containing lesion regions; determining a candidate lesion region in the pathological image based on a position of a first sample instance corresponding to a first prediction probability satisfying a first condition in the pathological image; sampling at least two second sample instances from the candidate lesion region by using a second sampling mode, wherein an overlap degree between the second sample instances is greater than an overlap degree between the first sample instances; determining lesion probability distribution information of the sample pathological image based on feature information extracted from the at least two second sample instances, wherein the lesion probability distribution information is used to indicate a probability distribution of a lesion region in the sample pathological image; training the lesion region determination model according to the lesion probability distribution information.

11. The method of claim 10, wherein, The lesion region determination model comprises an encoding network, a first classification network, a second classification network, and a third classification network. The determining of the lesion probability distribution information of the sample pathological image based on the feature information extracted from the at least two first sample instances and the feature information extracted from the at least two second sample instances comprises: The coding network is used to code features of each first sample instance and each second sample instance, to obtain feature information corresponding to each first sample instance and feature information corresponding to each second sample instance; The first classification network is used to process the feature information corresponding to each first sample instance, to obtain a pseudo label and a first prediction probability corresponding to each first sample instance and global feature information of the sample pathological image; the first prediction probability refers to a probability that the first sample instance contains a lesion region; The second classification network is used to process the feature information corresponding to each second sample instance, to obtain a second prediction probability corresponding to each second sample instance and local feature information of the pathological image; the second prediction probability refers to a probability that the second sample instance contains a lesion region; The third classification network is used to process the global feature information and the local feature information, to obtain lesion probability distribution information in the sample pathological image.

12. The method of claim 11, wherein, The training of the lesion region determination model according to the lesion probability distribution information includes: A first loss is generated according to the pseudo label corresponding to each first sample instance, and the first loss is used to measure a distinguishing ability of the first classification network for positive sample instances and negative sample instances; the positive sample instance refers to a first sample instance containing the lesion region, and the negative sample instance refers to a first sample instance not containing the lesion region; A second loss is generated according to the first prediction probability corresponding to each first sample instance, and the second loss is used to measure an accuracy of a prediction result of the first classification network for whether the first sample instance contains the lesion region; A third loss is generated according to the second prediction probability corresponding to each second sample instance, and the third loss is used to measure an accuracy of a prediction result of the second classification network for whether the second sample instance contains the lesion region; A fourth loss is generated according to the lesion probability distribution information, and the fourth loss is used to measure an accuracy of a prediction result of the third classification network for probability distribution information of the lesion region in the sample pathological image; The lesion region determination model is trained according to the first loss, the second loss, the third loss and the fourth loss.

13. An apparatus for determining a lesion area of a pathological image, characterized by, The device includes: A first image acquisition module is configured to sample at least two first instance images from a pathological image by using a first sampling manner; The device comprises: The sample acquisition module is configured to acquire a training sample set, wherein the training sample set comprises at least one sample pathological image; The instance acquisition module is configured to sample at least two first sample instances from the sample pathological image by using a first sampling manner; 14.A device for training a lesion region determination model, characterized in that, The information acquisition module is configured to encode features of each of the first sample instances respectively to obtain first feature information corresponding to each of the first sample instances respectively; the information acquisition module is further configured to process the first feature information corresponding to each of the first sample instances respectively by using an attention mechanism to obtain weights corresponding to each of the first feature information respectively; the information acquisition module is further configured to perform weighted summation processing on each of the first feature information according to the weights corresponding to each of the first feature information respectively to obtain global feature information of the sample pathological image; the information acquisition module is further configured to determine first prediction probabilities corresponding to each of the first sample instances respectively according to the global feature information and the first feature information corresponding to each of the first sample instances respectively, wherein the first prediction probability refers to a probability that a lesion region is contained in the first sample instance; and the information acquisition module is further configured to determine a candidate lesion region in the sample pathological image based on a position of a first sample instance corresponding to a first prediction probability satisfying a first condition in the sample pathological image; The instance acquisition module is configured to sample at least two second sample instances from the candidate lesion region by using a second sampling manner, wherein an overlap degree between the second sample instances is greater than an overlap degree between the first sample instances; The lesion information determination module is configured to determine lesion indication information of the sample pathological image based on feature information extracted from the at least two second sample instances. The device comprises: The sample acquisition module is configured to acquire a training sample set, wherein the training sample set comprises at least one sample pathological image; The instance acquisition module is configured to sample at least two first sample instances from the sample pathological image by using a first sampling manner; The information acquisition module is configured to encode features of each of the first sample instances respectively to obtain first feature information corresponding to each of the first sample instances respectively; the information acquisition module is further configured to process the first feature information corresponding to each of the first sample instances respectively by using an attention mechanism to obtain weights corresponding to each of the first feature information respectively; the information acquisition module is further configured to perform weighted summation processing on each of the first feature information according to the weights corresponding to each of the first feature information respectively to obtain global feature information of the sample pathological image; the information acquisition module is further configured to determine first prediction probabilities corresponding to each of the first sample instances respectively according to the global feature information and the first feature information corresponding to each of the first sample instances respectively, wherein the first prediction probability refers to a probability that a lesion region is contained in the first sample instance; and the information acquisition module is further configured to determine a candidate lesion region in the sample pathological image based on a position of a first sample instance corresponding to a first prediction probability satisfying a first condition in the sample pathological image; The instance acquisition module is configured to sample at least two second sample instances from the candidate lesion region by using a second sampling manner, wherein an overlap degree between the second sample instances is greater than an overlap degree between the first sample instances; The information obtaining module is configured to determine lesion probability distribution information of the sample pathological image based on the feature information extracted from the at least two second sample instances, wherein the lesion probability distribution information is used to indicate a probability distribution of a lesion region in the sample pathological image. The model training module is configured to train the lesion region determination model according to the lesion probability distribution information.

15. A computer device, comprising: The computer device comprises a processor and a memory, and the memory stores a computer program, which is loaded and executed by the processor to implement the lesion region determination method for a pathological image according to any one of claims 1 to 9 or the training method of the lesion region determination model according to any one of claims 10 to 12.

16. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which is loaded and executed by the processor to implement the lesion region determination method for a pathological image according to any one of claims 1 to 9 or the training method of the lesion region determination model according to any one of claims 10 to 12.

17. A computer program product, characterised in that, The computer program product comprises a computer program stored in a computer readable storage medium, and the processor reads and executes the computer program from the computer readable storage medium to implement the lesion region determination method for a pathological image according to any one of claims 1 to 9 or the training method of the lesion region determination model according to any one of claims 10 to 12.

Images