Device, method and electronic equipment for determining lesion area based on medical images

Abstract

The present application discloses a device, method and electronic device for determining a lesion region based on medical images, and relates to the field of medical technology. The device for determining a lesion region based on medical images includes: a first acquisition unit for acquiring a target medical image; a first determination unit for determining the probability that each pixel in the target medical image belongs to a lesion region through a segmentation model, wherein the segmentation model is a model trained based on a reference image of a known lesion delineation region; and a second determination unit for determining the lesion region of the target medical image based on the probability that each pixel belongs to the lesion region. The present application solves the technical problem of inaccurate lesion region identification in the prior art when identifying areas where lesions may exist through automatic segmentation algorithms.

Description

Technical Field

[0001] The present application relates to the field of medical technology, and more specifically, to a device, method and electronic device for determining a lesion area based on medical images. Background Art

[0002] Pathology slides are considered the gold standard for cancer diagnosis and prognosis. Pathology laboratories scan and digitize pathology slides, which are then used by histopathologists to make diagnoses of biopsy specimens. However, considering the large size of pathology slides and the increasing number of cases in which pathology techniques are used for cancer detection, manual analysis of pathology slides is a time-consuming task, and manual analysis can only make diagnoses based on pathology slides and cannot predict prognostic information. Therefore, automatic processing and segmentation of lesion areas on digitized pathology slides is of great clinical significance. This can not only help histopathologists make accurate and efficient preliminary diagnoses, but also provide important basis for the selection and decision-making of treatment options.

[0003] Among them, in the prior art, the pathological region segmentation algorithm used usually retains the foreground part of the entire digital pathological slice (i.e., the part containing cell tissue), and then cuts it into multiple smaller block images, and then uses a pre-trained neural network to identify whether there is a lesion area in each block image. However, this recognition method is too dependent on the recognition accuracy of the neural network model for the lesion area, and it cannot be quantified and lacks interpretability. The lesion area can only be determined directly based on the output results of the model, resulting in technical problems such as inaccurate recognition of the lesion area.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The present application provides a device, method and electronic device for determining a lesion region based on medical images, so as to at least solve the technical problem of inaccurate lesion region identification in the prior art when identifying areas where lesions may exist through automatic segmentation algorithms.

[0006] According to one aspect of the present application, a device for determining a lesion area based on a medical image is provided, comprising: a first acquisition unit, for acquiring a target medical image; a first determination unit, for determining the probability that each pixel in the target medical image belongs to the lesion area through a segmentation model, wherein the segmentation model is a model trained based on a reference image of a known lesion outline area; and a second determination unit, for determining the lesion area of ​​the target medical image based on the probability that each pixel belongs to the lesion area.

[0007] Optionally, the lesion region determination device based on medical images further includes: a second acquisition unit, used to acquire a reference image and a lesion delineation region corresponding to the reference image; a third acquisition unit, used to acquire N reference block images corresponding to the reference image, wherein N is an integer greater than 1, and each reference block image corresponds to a partial region of the reference image; a setting unit, used to set a lesion label for each reference block image according to a degree of overlap between each reference block image and the lesion delineation region, wherein the lesion label of each reference block image is used to characterize whether a lesion exists in the reference block image; and a training unit, used to train a segmentation model based on the N reference block images and the lesion label of each reference block image.

[0008] Optionally, in the medical image-based lesion area determination device of the present application, the third acquisition unit includes: a conversion subunit, used to convert the reference image into an image to be processed with M resolutions, wherein M is an integer greater than 1; and a processing subunit, used to divide the image to be processed with a first resolution into N reference block images, wherein the first resolution is the minimum resolution among the M resolutions.

[0009] Optionally, in the medical image-based lesion area determination device of the present application, the training unit includes: a first setting subunit, used to set a first encoder and a first decoder, wherein the first encoder is used to extract image features of each reference block image, and the first decoder is used to convert the image features of each reference block image into a feature vector; a second setting subunit, used to set a second decoder, wherein the second decoder is used to predict the probability that a pixel in the reference block image belongs to the lesion area based on the image features of the reference block image extracted by the first encoder; and a training subunit, used to train a segmentation model according to the first encoder, the first decoder and the second decoder.

[0010] Optionally, in the medical image-based lesion area determination device of the present application, the training subunit includes: an update module, used to update the first encoder and the first decoder multiple times according to a loss function, wherein the loss function is used to control the first encoder and the first decoder to learn the feature commonality information between reference block images with the same lesion label and the feature difference information between reference block images with different lesion labels during the update process; a processing module, used to form a feature extractor with the first encoder and the first decoder after the last update, wherein images to be processed with different resolutions are used to train different feature extractors; a training module, used to obtain a segmentation model based on the first encoder and the second decoder that constitute the feature extractor.

[0011] Optionally, in the lesion region determination device based on medical images of the present application, the training module includes: a label labeling submodule, which is used to label the first label or the second label for each pixel in the reference block image according to the probability that the pixel in each reference block image belongs to the lesion region predicted by the second decoder, wherein the pixel of the first label indicates that the pixel belongs to the lesion delineation region, and the pixel of the second label indicates that the pixel does not belong to the lesion delineation region; a calculation submodule, which is used to calculate the ratio of the number of pixels of the first label to the number of pixels of the second label in each reference block image, and obtain the lesion risk value of each reference block image; a first determination submodule, which is used to determine the lesion risk value of each reference block image according to the probability that the pixel in each reference block image belongs to the lesion region predicted by the second decoder The prediction label of the reference block image is determined by the lesion risk value of the reference block image, wherein when the lesion risk value is greater than a first threshold value, the prediction label is a third label; when the lesion risk value is less than or equal to the first threshold value, the prediction label is a fourth label; an updating submodule is used to update the second decoder multiple times according to the prediction label and the lesion label of each reference block image, until the prediction label of each reference block image determined according to the output result of the second decoder is the same as the lesion label of the reference block image, and the updating of the second decoder is stopped; a second determining submodule is used to use the first encoder constituting the feature extractor and the second decoder after the last update as a segmentation model.

[0012] Optionally, in the medical image-based lesion area determination device of the present application, the second determination unit includes: a pixel classification unit, used to determine that each pixel is a first pixel or a second pixel based on the probability that each pixel belongs to the lesion area, wherein the probability that the first pixel belongs to the lesion area is greater than a second threshold, and the probability that the second pixel belongs to the lesion area is less than or equal to the second threshold; a target area determination unit, used to take the area where the first pixel is located as the target area; and an aggregation unit, used to perform aggregation operations on adjacent target areas, and take the aggregated connected area as the lesion area.

[0013] According to another aspect of the present application, a method for determining a lesion area based on medical images is also provided, which includes: acquiring a target medical image; determining the probability that each pixel in the target medical image belongs to the lesion area through a segmentation model, wherein the segmentation model is a model trained based on a reference image of a known lesion outline area; determining the lesion area of ​​the target medical image based on the probability that each pixel belongs to the lesion area.

[0014] According to another aspect of the present application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors control the operation of any one of the above-mentioned medical image-based lesion area determination devices.

[0015] According to another aspect of the present application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, wherein when the computer program is executed, the device where the computer-readable storage medium is located controls the operation of any one of the above-mentioned medical image-based lesion area determination devices.

[0016] In the present application, a device for determining a lesion region based on a medical image is provided, comprising: a first acquisition unit, a first determination unit, and a second determination unit. The first acquisition unit is used to acquire a target medical image; the first determination unit is used to determine the probability that each pixel in the target medical image belongs to the lesion region through a segmentation model, wherein the segmentation model is a model trained based on a reference image of a known lesion delineation region; the second determination unit is used to determine the lesion region of the target medical image based on the probability that each pixel belongs to the lesion region.

[0017] As can be seen from the above content, the present application first determines the probability that each pixel in the target medical image belongs to the lesion area through the segmentation model, thereby achieving the determination of the lesion area at the pixel level. Compared with the prior art that directly uses the neural network model to determine the lesion area in the block image, the recognition accuracy of the lesion area of ​​the present application is higher. In addition, since the present application determines the lesion area in the medical image based on the probability that each pixel belongs to the lesion area, the determination process of each lesion area is interpretable, that is, it can be quantified based on the probability that each pixel belongs to the lesion area, thereby overcoming the technical problem that the process of determining the lesion area in the prior art cannot be quantified and lacks interpretability.

[0018] It can be seen that according to the technical solution provided in the present application, not only can the interpretability of the process of determining the lesion area be improved, but also the technical problem of inaccurate lesion area identification in the prior art when identifying areas where lesions may exist through automatic segmentation algorithms can be solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 is a schematic diagram of an optional device for determining a lesion region based on medical images according to an embodiment of the present application;

[0021] Figure 2 is a schematic diagram of a process of preparing and preprocessing training data for an optional segmentation model according to an embodiment of the present application;

[0022] Figure 3is a schematic diagram of an optional architecture used in a segmentation model training process according to an embodiment of the present application;

[0023] Figure 4 is a schematic diagram of a training process of an optional medium-resolution feature extractor according to an embodiment of the present application;

[0024] Figure 5 is a schematic diagram of a training process of an optional high-resolution feature extractor according to an embodiment of the present application;

[0025] Figure 6 is an optional flow chart of using an image feature extractor to extract image features according to an embodiment of the present application;

[0026] Figure 7 It is a schematic diagram of an optional training process of a segmentation model according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] It should also be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) collected by this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with the relevant laws, regulations and standards of the relevant regions, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation portals are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or institution.

[0030] According to an embodiment of the present application, an embodiment of a lesion region determination device based on medical images (hereinafter referred to as a lesion region determination device) is provided, wherein: Figure 1 is a schematic diagram of an optional device for determining a lesion region based on medical images according to an embodiment of the present application, such as Figure 1 As shown, the device for determining a lesion region based on medical images includes: a first acquiring unit 101 , a first determining unit 102 and a second determining unit 103 .

[0031] Optionally, the first acquisition unit 101 is used to acquire a target medical image, wherein the target medical image may be a medical image taken for a target object, the target object being a patient who is preparing to receive radiotherapy or has received radiotherapy, and the target medical image includes but is not limited to various types of medical images such as CT images, MR images, and PET images.

[0032] Optionally, the first determination unit 102 is used to determine the probability that each pixel in the target medical image belongs to the lesion area through a segmentation model, wherein the segmentation model is a model trained based on a reference image of a known lesion outline area.

[0033] Optionally, the segmentation model can be a neural network model, and the segmentation model can be a model composed of a feature extractor and a decoder, wherein the feature extractor is used to extract image features of each pixel on the target medical image, and the decoder is used to determine the probability that each pixel in the target medical image belongs to the lesion area based on the image features output by the feature extractor.

[0034] It should be noted that since the first determination unit 102 of the present application can determine the probability that each pixel in the target medical image belongs to the lesion area through the segmentation model, thereby achieving the technical effect of quantifying the correlation between each pixel in the target medical image and the lesion area, it is easy to understand that if the probability that a certain pixel belongs to the lesion area is higher, it means that the correlation between this pixel and the lesion area is stronger.

[0035] Optionally, the second determination unit 103 is used to determine the lesion area of ​​the target medical image according to the probability that each pixel belongs to the lesion area.

[0036] Optionally, the second determination unit 103 may set a classification threshold, and then classify each pixel according to the classification threshold. For example, the classification threshold may be set to 0.5. If the probability that a certain pixel belongs to the lesion area is greater than 0.5, the pixel is determined to be the first pixel. If the probability that a certain pixel belongs to the lesion area is less than or equal to 0.5, the pixel is determined to be the second pixel. Finally, the second determination unit 103 determines the lesion area of ​​the target medical image according to the area where the first pixel is located.

[0037] It should be noted that the above steps are used to determine the lesion area of ​​the target medical influence. The entire process can be quantified according to the probability that each pixel belongs to the lesion area, thereby ensuring the interpretability of the lesion area determination process. If the user believes that the determination result of the lesion area is not accurate enough, the classification threshold can be adjusted to adjust the recognition accuracy of the lesion area.

[0038] As can be seen from the above content, the present application first determines the probability that each pixel in the target medical image belongs to the lesion area through the segmentation model, thereby achieving the determination of the lesion area at the pixel level. Compared with the prior art that directly uses the neural network model to determine the lesion area in the block image, the recognition accuracy of the lesion area of ​​the present application is higher. In addition, since the present application determines the lesion area in the medical image based on the probability that each pixel belongs to the lesion area, the determination process of each lesion area is interpretable, that is, it can be quantified based on the probability that each pixel belongs to the lesion area, thereby overcoming the technical problem that the process of determining the lesion area in the prior art cannot be quantified and lacks interpretability.

[0039] It can be seen that according to the technical solution provided in the present application, not only can the interpretability of the process of determining the lesion area be improved, but also the technical problem of inaccurate lesion area identification in the prior art when identifying areas where lesions may exist through automatic segmentation algorithms can be solved.

[0040] In an optional embodiment, the device for determining a lesion area based on medical images further includes: a second acquisition unit, a third acquisition unit, a setting unit, and a training unit.

[0041] Among them, the second acquisition unit is used to acquire a reference image and a lesion delineation area corresponding to the reference image; the third acquisition unit is used to acquire N reference block images corresponding to the reference image, wherein N is an integer greater than 1, and each reference block image corresponds to a partial area of ​​the reference image; the setting unit is used to set a lesion label for each reference block image according to the overlap between each reference block image and the lesion delineation area, wherein the lesion label of each reference block image is used to characterize whether there is a lesion in the reference block image; the training unit is used to train a segmentation model according to the N reference block images and the lesion label of each reference block image.

[0042] Optionally, the above segmentation model can be trained by the following steps:

[0043] First, a reference image and the lesion delineation area corresponding to the reference image are obtained, and then the reference image is divided into N reference block images, where N is an integer greater than 1, and each reference image corresponds to a partial area of ​​the reference image. Subsequently, a lesion label is set for each reference block image according to the degree of overlap between each reference block image and the lesion delineation area, where the lesion label of each reference block image is used to characterize whether there is a lesion in the reference block image. Finally, a segmentation model is trained based on the N reference block images and the lesion label of each reference block image.

[0044] Optionally, Figure 2 is a schematic diagram of the preparation and preprocessing process of training data for an optional segmentation model according to an embodiment of the present application, such as Figure 2 As shown, the reference image can be a digital pathology section of a historical patient, wherein each digital pathology section corresponds to a prediction endpoint. For example, the prediction endpoint can be the actual diagnosis result (negative, positive) of the historical patient, the prediction endpoint can also be the prognosis information of the historical patient (disease relief, disease non-relief), the prediction endpoint can also be the survival time of the historical patient, and so on.

[0045] Alternatively, if Figure 2As shown, on the one hand, the lesion area on the digital pathology section can be delineated by manual delineation or automatic delineation by a delineation model. On the other hand, the digital pathology section can also be stained and standardized, and then the digital pathology section is automatically segmented by a threshold segmentation algorithm (such as the Otsu algorithm) to obtain a threshold segmentation label, and the digital pathology section is processed according to the threshold segmentation label to obtain a foreground area (i.e., an area with cell tissue), and the area without cell tissue is removed, thereby obtaining a cell tissue area of ​​the digital pathology section. Finally, modeling is performed based on the cell tissue area and the lesion delineation area. For example, the cell tissue area is divided into N reference block images, and then a lesion label is set for each reference block image according to the overlap between each reference block image and the lesion delineation area, and then a segmentation model is trained based on the N reference block images and the lesion label of each reference block image.

[0046] From the above content, it can be seen that in the training process of the segmentation model, each training data is a reference block image, and the training label is the lesion label of the reference block image (i.e., whether there is a lesion in the reference block image). It should be noted that the lesion area is determined based on the overlap between the reference block image and the lesion delineation area. Since the lesion delineation area is a pixel-level delineation completed in data preparation, it can be ensured that the training dimension of the segmentation model is also pixel-level. Assuming that the size of the reference block image is 256x256, the reference block image has a total of 256x256 pixels, and each pixel is set with a preset label. For example, the preset label may be a first label or a second label, wherein a pixel of the first label indicates that the pixel belongs to a lesion delineation area, and a pixel of the second label indicates that the pixel does not belong to a lesion delineation area. If more than 60% (a threshold value may be selected by a technician or a doctor based on experience) of the pixels in a reference block image are pixels of the first label, the lesion label of the reference block image is set to "1", wherein "1" indicates that a lesion exists in the reference block image; if the pixels of the first label in a reference image block do not exceed 60% of all pixels in the reference block image, the lesion label of the reference block image is set to "0", wherein "0" indicates that no lesion exists in the reference block image.

[0047] Through the above operations, each reference block image becomes a single lesion label instead of tags.

[0048] In an optional embodiment, the third acquisition unit includes: a conversion subunit and a processing subunit.

[0049] Among them, the conversion subunit is used to convert the reference image into an image to be processed with M resolutions, where M is an integer greater than 1; the processing subunit is used to divide the image to be processed with a first resolution into N reference block images, where the first resolution is the minimum resolution among the M resolutions.

[0050] Optionally, in addition to directly dividing the reference image into N reference block images, the present application also provides another method for dividing the reference block images, namely, first converting the reference image into an image to be processed with M resolutions, and then dividing the image to be processed with a first resolution into N reference block images, wherein the first resolution is the minimum resolution among the M resolutions.

[0051] Of course, the present application also supports the use of images to be processed at other resolutions than the first resolution to be divided into N reference block images. The reason why the present application preferably uses the first resolution of the images to be processed to obtain N reference block images is that images with higher resolutions occupy more computing resources, and the purpose of training the segmentation model in the present application is to determine from a macroscopic perspective whether each area in the target medical image is a lesion area. Therefore, using the first resolution of the images to be processed to complete the training of the segmentation model can achieve the purpose of training the segmentation model while occupying as few computing resources as possible.

[0052] For example, when training a segmentation model, a lesion region determination device based on medical images can first perform resolution processing of reference images at different magnifications. For example, two scales can be set, namely 10x and 20x, where 10x magnification is called medium resolution and 20x magnification is called high resolution. The image to be processed converted to high resolution has a larger size and more detailed information, while the image converted to medium resolution has a smaller size and is convenient for extracting global information.

[0053] In the above process, since there are only two scales, medium resolution and high resolution, the medium resolution is the minimum resolution.

[0054] Optionally, the device for determining the lesion region based on medical images can also perform slicing processing on the medium-resolution image to be processed and the high-resolution image to be processed respectively, and cut them to obtain a reference block image of a smaller size (for example , , ).

[0055] In an optional embodiment, the training unit includes: a first setting subunit, a second setting subunit and a training subunit.

[0056] The first setting subunit is used to set a first encoder and a first decoder, wherein the first encoder is used to extract image features of each reference block image, and the first decoder is used to convert the image features of each reference block image into a feature vector. The second setting subunit is used to set a second decoder, wherein the second decoder is used to predict the probability that a pixel in the reference block image belongs to a lesion area based on the image features of the reference block image extracted by the first encoder. The training subunit is used to train a segmentation model based on the first encoder, the first decoder, and the second decoder.

[0057] Optionally, Figure 3 is a schematic diagram of an optional architecture used in the segmentation model training process according to an embodiment of the present application, such as Figure 3 As shown, the first encoder, the first decoder and the second decoder are required to be used in the training process of the segmentation model, wherein the first encoder is connected to the first decoder and the second decoder respectively.

[0058] Optionally, during the training process of the segmentation model, the first encoder and the first decoder are first pre-trained, and then the second decoder is trained based on the trained first encoder. Finally, the trained first encoder and the second decoder are combined into a segmentation model.

[0059] In an optional embodiment, the training subunit includes: an updating module, a processing module and a training module.

[0060] Among them, the updating module is used to update the first encoder and the first decoder multiple times according to the loss function, wherein the loss function is used to control the first encoder and the first decoder to learn the feature commonality information between reference block images with the same lesion label and the feature difference information between reference block images with different lesion labels during the updating process; the processing module is used to form a feature extractor with the first encoder and the first decoder after the last update, wherein the images to be processed with different resolutions are used to train different feature extractors; the training module is used to obtain a segmentation model according to the first encoder and the second decoder that constitute the feature extractor.

[0061] Optionally, Figure 4 is a schematic diagram of a training process of an optional medium-resolution feature extractor according to an embodiment of the present application, such as Figure 4As shown, supervised contrast learning can be performed based on the medium-resolution reference block image and the corresponding medium-resolution lesion delineation area. Specifically, the first encoder and the first decoder are updated multiple times according to the first loss function, wherein the first loss function is used to control the first encoder and the first decoder to learn the feature commonality information between the reference block images with the same lesion label and the feature difference information between the reference block images with different lesion labels during the update process. Finally, the first encoder and the first decoder after the last update are combined into a medium-resolution image feature extractor.

[0062] Optionally, Figure 5 is a schematic diagram of a training process of an optional high-resolution feature extractor according to an embodiment of the present application, such as Figure 5 As shown, supervised contrast learning can be performed based on the high-resolution reference block image and the corresponding high-resolution lesion delineation area, specifically, the second encoder and the third decoder are updated multiple times according to the second loss function, wherein the second loss function is used to control the second encoder and the third decoder to learn the feature commonality information between the reference block images with the same lesion label and the feature difference information between the reference block images with different lesion labels during the updating process. Finally, the second encoder and the third decoder after the last update form a high-resolution image feature extractor.

[0063] Optionally, Figure 6 is an optional flow chart of using an image feature extractor to extract image features according to an embodiment of the present application, such as Figure 6 As shown, the lesion area determination device first slices the reference image based on the reference image and the lesion delineation area corresponding to the reference image to obtain a reference block image with a medium resolution and a reference block image with a high resolution, and inputs the reference block image with a medium resolution and the reference block image with a high resolution into their respective corresponding image feature extractors (including an encoder and a decoder), and then obtains image features of the reference block image with a medium resolution and image features of the reference block image with a high resolution.

[0064] in addition, Figure 6 It also shows that the label of each pixel on the reference block image with medium resolution is determined according to the lesion delineation area with medium resolution, and then the lesion label of the reference block image with medium resolution is determined according to the label of each pixel; the label of each pixel on the reference block image with high resolution is determined according to the lesion delineation area with high resolution, and then the lesion label of the reference block image with high resolution is determined according to the label of each pixel.

[0065] In an optional embodiment, the training module includes: a labeling submodule, a calculation submodule, a first determination submodule, an update submodule and a second determination submodule.

[0066] Among them, the label labeling submodule is used to label the first label or the second label for each pixel in the reference block image according to the probability that the pixel in each reference block image belongs to the lesion area predicted by the second decoder, wherein the pixel of the first label indicates that the pixel belongs to the lesion delineation area, and the pixel of the second label indicates that the pixel does not belong to the lesion delineation area; the calculation submodule is used to calculate the ratio of the number of pixels of the first label to the number of pixels of the second label in each reference block image to obtain the lesion risk value of each reference block image; the first determination submodule is used to determine the predicted label of each reference block image according to the lesion risk value of the reference block image, wherein when the lesion risk value is greater than the first threshold, the predicted label is the first label; when the lesion risk value is less than or equal to the first threshold, the predicted label is the second label; the updating submodule is used to update the second decoder multiple times according to the predicted label and the lesion label of each reference block image, until the predicted label of each reference block image determined according to the output result of the second decoder is the same as the lesion label of the reference block image, and the updating of the second decoder is stopped; the second determination submodule is used to use the first encoder constituting the feature extractor and the second decoder after the last update as the segmentation model.

[0067] Optionally, in the process of training the segmentation model, the lesion region determination device may randomly extract N reference block images from the data set, and each reference block image may be used Indicates that, where the subscript Then, the lesion area determination device can randomly perform multiple data enhancements on each reference block image. The data enhancement methods include but are not limited to contrast enhancement, horizontal flipping, brightness adjustment, rotation operation, etc. Finally, 2N image data are obtained through data enhancement. These 2N image data can be represented by I, and each image data can be represented by Indicates that .

[0068] Optionally, the lesion region determination device may perform supervised contrast learning training according to the loss function of the following formula (1).

[0069]

[0070] Optionally, in the above formula (1), represents the set of other reference block images that have the same lesion label as the reference block image i, represents the set of other reference block images except the reference block image i, The reference patch image is used by the image feature extractor The feature vector after mapping to the feature space, Represents the feature vectors of other block images with the same lesion label as the reference block image i, Represents the feature vectors of other reference block images except the reference block image i. Represents a picture block The corresponding loss value is is a hyperparameter used to control the concentration of cosine similarity between feature vectors. It can influence the model's training process to make it focus more on those difficult-to-distinguish sample pairs.

[0071] It should be noted that the goal of supervised contrastive learning training is to minimize the calculation result of the loss function of the above formula (1). Therefore, the training process will bring the feature vectors of the reference block images with the same lesion label closer and separate the feature vectors of the reference block images with different lesion labels.

[0072] Optionally, during the above training process, on the one hand, the feature extractor will learn basic pathological image features, and on the other hand, the feature extractor will also learn feature difference information between the reference block image of the area where the lesion may exist and the non-lesion area.

[0073] Optionally, the image feature extractor consists of two parts, an encoder and a decoder. The input reference block image x is first processed by the encoder and then processed by the decoder to obtain a feature vector z.

[0074] Optionally, a medium-resolution reference block image can be used to train the segmentation model. Since the size of medium-resolution digital pathology slices is large, the range of possible lesion areas is also large. Therefore, compared with high-resolution block images, using medium-resolution block images to train the segmentation model can focus on whether lesions may exist in each area at a macro level.

[0075] When training the segmentation model, the input data is the reference block image x, and the label y is whether each pixel in the reference block image may belong to the lesion delineation area (if the size of the reference block image is , then pixels, each pixel has a label indicating whether it belongs to the lesion delineation area), where the label y is the first label or the second label mentioned above.

[0076] In addition, the segmentation model can be any deep learning model structure, wherein the encoder of the model adopts the encoder part of the image feature extractor obtained by supervised contrast learning, and the encoder already includes the basic ability to extract pathological image features and identify possible lesion areas. On this basis, the lesion area determination device based on medical images can further train a pixel-level segmentation model for determining the lesion area based on the encoder. Among them, inheriting the encoder that has been trained instead of training the model from scratch helps to achieve higher segmentation accuracy.

[0077] In an optional embodiment, Figure 7 is a schematic diagram of an optional training process of a segmentation model according to an embodiment of the present application, such as Figure 7 As shown, the reference block image is first input into the trained first encoder, which is used to extract the image features of the reference block image. Then, the image features extracted by the first encoder are input into the second decoder, which predicts the probability that each pixel in the reference block image belongs to the lesion area. According to the probability that each pixel belongs to the lesion area, the second decoder can further annotate the first label or the second label for each pixel. For example, if the probability that a certain pixel belongs to the lesion area is greater than the classification threshold, the first label is annotated for the pixel; if the probability that a certain pixel belongs to the lesion area is less than or equal to the classification threshold, the second label is annotated for the pixel.

[0078] Furthermore, after labeling each pixel, the device for determining a lesion area based on medical images may also calculate a ratio of the number of pixels of the first label to the number of pixels of the second label in the reference block image, thereby obtaining a lesion risk value of the reference block image. For example, if the number of pixels of the first label in the reference block image is 100 and the number of pixels of the second label is 200, the lesion risk value of the reference block image is 0.5.

[0079] Optionally, the lesion region determination device based on medical images can determine the predicted label of each reference block image according to the lesion risk value of the reference block image, wherein when the lesion risk value is greater than the first threshold, the predicted label is a third label; when the lesion risk value is less than or equal to the first threshold, the predicted label is a fourth label. The third label is used to indicate that a lesion exists in the reference block image, and the fourth label is used to indicate that a lesion does not exist in the reference block image.

[0080] It should be noted that if Figure 7As shown, the training goal of the segmentation model is to minimize the difference between the predicted label and the lesion label of the reference block image. Therefore, the lesion area determination device based on medical images can update the second decoder multiple times according to the predicted label and the lesion label of each reference block image, until the predicted label of each reference block image determined according to the output result of the second decoder is the same as the lesion label of the reference block image, and then stop updating the second decoder. Finally, the trained first encoder and the second decoder after the last update are used as the segmentation model.

[0081] In an optional embodiment, the second determination unit includes: a pixel classification unit, a target area determination unit and an aggregation unit.

[0082] Among them, the pixel classification unit is used to determine whether each pixel is the first pixel or the second pixel according to the probability that each pixel belongs to the lesion area, wherein the probability that the first pixel belongs to the lesion area is greater than the second threshold, and the probability that the second pixel belongs to the lesion area is less than or equal to the second threshold; the target area determination unit is used to take the area where the first pixel is located as the target area; the aggregation unit is used to perform aggregation operations on adjacent target areas, and take the aggregated connected area as the lesion area.

[0083] Optionally, the above-mentioned second threshold value may also be referred to as a classification threshold value, wherein the preset threshold value may be custom set, for example, 0.5, 0.6, 0.75, etc. The lesion region determination device classifies each pixel in the medical image by comparing the probability of each pixel belonging to the lesion region with the preset threshold value. If the probability of a certain pixel belonging to the lesion region is greater than the preset threshold value, the lesion region determination device classifies the pixel as a first pixel; if the probability of a certain pixel belonging to the lesion region is less than or equal to the preset threshold value, the lesion region determination device classifies the pixel as a second pixel.

[0084] Optionally, the lesion region determination device may use the region where each first pixel in the medical image is located as the target region. Since there may be multiple first pixels, there may also be multiple target regions, and any two different target regions may be discrete or connected. The target region may be understood as the lesion region determined by the lesion region determination device.

[0085] Optionally, the lesion area determination device can aggregate adjacent target areas and perform appropriate corrosion and expansion processing. In addition, the lesion area determination device can also fill unnecessary holes in the aggregated target area, and finally obtain multiple connected domains based on the entire pathological image (i.e., various connected areas where lesions may exist).

[0086] As can be seen from the above content, the present application first determines the probability that each pixel in the target medical image belongs to the lesion area through the segmentation model, thereby achieving the determination of the lesion area at the pixel level. Compared with the prior art that directly uses the neural network model to determine the lesion area in the block image, the recognition accuracy of the lesion area of ​​the present application is higher. In addition, since the present application determines the lesion area in the medical image based on the probability that each pixel belongs to the lesion area, the determination process of each lesion area is interpretable, that is, it can be quantified based on the probability that each pixel belongs to the lesion area, thereby overcoming the technical problem that the process of determining the lesion area in the prior art cannot be quantified and lacks interpretability.

[0087] It can be seen that according to the technical solution provided in the present application, not only can the interpretability of the process of determining the lesion area be improved, but also the technical problem of inaccurate lesion area identification in the prior art when identifying areas where lesions may exist through automatic segmentation algorithms can be solved.

[0088] According to another aspect of the present application, the present application also provides a method for determining a lesion area based on medical images, including: acquiring a target medical image; determining the probability that each pixel in the target medical image belongs to the lesion area through a segmentation model, wherein the segmentation model is a model trained based on a reference image of a known lesion outline area; determining the lesion area of ​​the target medical image based on the probability that each pixel belongs to the lesion area.

[0089] Optionally, the execution subject of the lesion region determination method based on medical images may be a lesion region determination device, or other software systems, embedded systems and devices. This application does not make any special limitation on this.

[0090] As can be seen from the above content, the present application first determines the probability that each pixel in the target medical image belongs to the lesion area through the segmentation model, thereby achieving the determination of the lesion area at the pixel level. Compared with the prior art that directly uses the neural network model to determine the lesion area in the block image, the recognition accuracy of the lesion area of ​​the present application is higher. In addition, since the present application determines the lesion area in the medical image based on the probability that each pixel belongs to the lesion area, the determination process of each lesion area is interpretable, that is, it can be quantified based on the probability that each pixel belongs to the lesion area, thereby overcoming the technical problem that the process of determining the lesion area in the prior art cannot be quantified and lacks interpretability.

[0091] According to another aspect of the present application, an electronic device is also provided, which includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors control the operation of any one of the above-mentioned medical image-based lesion area determination devices.

[0092] According to another aspect of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program runs, the device where the computer-readable storage medium is located controls the operation of any one of the above-mentioned medical image-based lesion area determination devices.

[0093] In the embodiments provided in the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0094] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0095] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0096] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program code.

[0097] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A device for determining a lesion region based on medical images, characterized in that: include: A first acquisition unit, used for acquiring a target medical image; The first determining unit is used to determine the probability that each pixel in the target medical image belongs to the lesion area through a segmentation model, wherein the segmentation model is a model trained based on a reference image of a known lesion delineation area; the second determining unit is used to determine the lesion area of ​​the target medical image based on the probability that each pixel belongs to the lesion area; Wherein, the lesion region determination device based on medical images further includes: a second acquisition unit, used to acquire a reference image and a lesion delineation region corresponding to the reference image; a third acquisition unit, used to acquire N reference block images corresponding to the reference image, wherein N is an integer greater than 1, and each reference block image corresponds to a partial region of the reference image; wherein the third acquisition unit includes: a conversion subunit, used to convert the reference image into an image to be processed with M resolutions, wherein M is an integer greater than 1; a processing subunit, used to divide the image to be processed with a first resolution into the N reference block images, wherein the first resolution is the minimum resolution among the M resolutions; The device further includes: a label labeling submodule, used to label the first label or the second label for each pixel in the reference block image according to the probability that the pixel in each reference block image belongs to the lesion area predicted by the second decoder, wherein the pixel of the first label indicates that the pixel belongs to the lesion delineation area, and the pixel of the second label indicates that the pixel does not belong to the lesion delineation area; a calculation submodule, used to calculate the ratio of the number of pixels of the first label to the number of pixels of the second label in each reference block image to obtain the lesion risk value of each reference block image; a first determination submodule, used to determine the predicted label of the reference block image according to the lesion risk value of each reference block image, wherein, When the lesion risk value is greater than a first threshold, the predicted label is a third label; when the lesion risk value is less than or equal to the first threshold, the predicted label is a fourth label; an updating submodule, used to update the second decoder multiple times according to the predicted label and the lesion label of each reference block image, until the predicted label of each reference block image determined according to the output result of the second decoder is the same as the lesion label of the reference block image, and the updating of the second decoder is stopped; a second determining submodule, used to use the first encoder constituting the feature extractor and the second decoder after the last update as the segmentation model, and the first encoder is used to extract the image features of each reference block image.

2. The device for determining a lesion region based on medical images according to claim 1, characterized in that: The device for determining the lesion region based on medical images further comprises: A setting unit is used to set a lesion label for each reference block image according to the overlap between each reference block image and the lesion delineation area, wherein the lesion label of each reference block image is used to indicate whether there is a lesion in the reference block image.

3. The device for determining a lesion region based on medical images according to claim 2, characterized in that: The device also includes: A first setting subunit is used to set a first encoder and a first decoder, wherein the first encoder is used to extract the image features of each reference block image, and the first decoder is used to convert the image features of each reference block image into a feature vector; A second setting subunit is used to set a second decoder, wherein the second decoder is used to predict the probability that a pixel in the reference block image belongs to a lesion area according to the image features of the reference block image extracted by the first encoder; A training subunit is used to obtain the feature extractor by training the first encoder and the first decoder.

4. The device for determining a lesion region based on medical images according to claim 3, characterized in that: The training subunit comprises: An updating module, configured to update the first encoder and the first decoder multiple times according to a loss function, wherein the loss function is used to control the first encoder and the first decoder to learn feature commonality information between reference block images with the same lesion label and feature difference information between reference block images with different lesion labels during the updating process; The processing module is used to form a feature extractor with the first encoder and the first decoder after the last update, wherein the images to be processed with different resolutions are used to train different feature extractors.

5. The device for determining a lesion region based on medical images according to claim 1, characterized in that: The second determining unit includes: a pixel classification unit, configured to determine whether each pixel is a first pixel or a second pixel according to a probability that each pixel belongs to the lesion area, wherein the probability that the first pixel belongs to the lesion area is greater than a second threshold, and the probability that the second pixel belongs to the lesion area is less than or equal to the second threshold; a target region determining unit, configured to determine a region where the first pixel is located as a target region; The aggregation unit is used to perform an aggregation operation on the adjacent target areas and use the connected areas after aggregation as the lesion area.

6. A method for determining a lesion region based on medical images, characterized in that: include: Acquire target medical images; Determine the probability that each pixel in the target medical image belongs to the lesion area through a segmentation model, wherein the segmentation model is a model trained based on a reference image with a known lesion outline area; Determining the lesion area of ​​the target medical image according to the probability that each pixel belongs to the lesion area; The method for determining the lesion region based on medical images further includes: obtaining a reference image and a lesion delineation region corresponding to the reference image; obtaining N reference block images corresponding to the reference image, wherein N is an integer greater than 1, and each reference block image corresponds to a partial region of the reference image; Wherein, obtaining N reference block images corresponding to the reference image includes: converting the reference image into an image to be processed with M resolutions, wherein M is an integer greater than 1; dividing the image to be processed with a first resolution into the N reference block images, wherein the first resolution is the minimum resolution among the M resolutions; The training step of the segmentation model includes: labeling the pixels in each reference block image with a first label or a second label according to the probability that the pixels in each reference block image belong to the lesion area predicted by the second decoder, wherein the pixels of the first label represent that the pixels belong to the lesion delineation area, and the pixels of the second label represent that the pixels do not belong to the lesion delineation area; calculating the ratio of the number of pixels of the first label to the number of pixels of the second label in each reference block image to obtain the lesion risk value of each reference block image; determining the predicted label of the reference block image according to the lesion risk value of each reference block image, wherein when the lesion risk value is greater than a first threshold, the predicted label is a third label; when the lesion risk value is less than or equal to the first threshold, the predicted label is a fourth label; updating the second decoder multiple times according to the predicted label and the lesion label of each reference block image until the predicted label of each reference block image determined according to the output result of the second decoder is the same as the lesion label of the reference block image, and stopping updating the second decoder; using the first encoder constituting the feature extractor and the second decoder after the last update as the segmentation model, and the first encoder is used to extract the image features of each reference block image.

7. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors control the operation of the lesion area determination device based on medical images as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located controls the operation of the medical image-based lesion region determination device as described in any one of claims 1 to 5.

Images