Abnormal cell detection method and related product

Abstract

The invention provides an abnormal cell detection method and a related product. According to a specific implementation mode of the method, an image to be recognized is firstly segmented into at least two image blocks to be recognized; and for each to-be-identified image block, generating a corresponding abnormal cell attention heat map. And then, combining the abnormal cell attention heat maps of the to-be-identified image blocks to obtain an overall abnormal cell attention heat map. And finally, generating an abnormal cell detection result of each segmented cell in an overall cell segmentation result based on the overall cell segmentation result of cell segmentation of the to-be-identified image and the overall abnormal cell attention heat map. Therefore, the abnormal cell detection result of each cell can be automatically determined, so that professional technicians only need to carry out artificial review interpretation on the part detected as the abnormal cell and do not need to further carry out artificial review interpretation on the other part, and the labor cost is greatly reduced.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of image recognition technology, and in particular to abnormal cell detection methods and related products. Background Art

[0002] Immunofluorescence technique, also known as fluorescent antibody technique, is based on the principle of antigen-antibody reaction. First, a known antigen or antibody is labeled with a fluorescent group, and then the fluorescent antibody (or antigen) is used as a probe to detect the corresponding antigen (or antibody) in the cell. In recent years, immunofluorescence technique has been widely used in the study of specific immune cell populations in the tumor microenvironment (TME), and it has been found to be helpful in determining the clinical prognosis and predicting the efficacy of tumor patients.

[0003] At present, the main method for identifying tumor cells based on immunofluorescence technology is manual interpretation under a fluorescence microscope, but the workload is enormous to accurately locate tumor cells from a large number of cells. Summary of the invention

[0004] Embodiments of the present disclosure provide an abnormal cell detection method, an apparatus, an electronic device, a storage medium, and a computer program product.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for detecting abnormal cells, the method comprising:

[0006] Acquiring an image to be identified, wherein the image to be identified is an immunofluorescence image obtained by photographing the same field of view including the cells;

[0007] Dividing the image to be identified into at least two image blocks to be identified;

[0008] For each image block to be identified, the following abnormal cell attention heat map generation step is performed: the image block to be identified is input into a pre-trained image block feature extraction model to obtain image block features; based on the distance between the image block features and each normal cell image block feature in the pre-trained normal cell image block feature set, the abnormal cell image block score of the image block to be identified is determined, wherein the abnormal cell image block score is used to characterize the degree of inclusion of abnormal cell objects; according to the abnormal cell image block score of the image block to be identified, it is determined whether the image block to be identified is an abnormal cell image block; in response to determining yes, the image block features are back-propagated in the image block feature extraction model to obtain the abnormal cell attention heat map of the image block to be identified; in response to determining no, a preset low attention heat map is determined as the abnormal cell attention heat map of the image block to be identified;

[0009] Merge the abnormal cell attention heat maps of each image block to be identified to obtain the overall abnormal cell attention heat map;

[0010] Based on the overall cell segmentation result of cell segmentation of the image to be identified and the overall abnormal cell attention heat map, an abnormal cell detection result of each segmented cell in the overall cell segmentation result is generated.

[0011] In some optional embodiments, the overall cell segmentation result of the image to be identified includes at least one cell segmentation mask image for segmenting cells, the cell segmentation mask image includes cell pixels for characterizing cell objects and background pixels for characterizing non-cell objects; and

[0012] The generating of the abnormal cell detection result of each segmented cell in the overall cell segmentation result based on the overall cell segmentation result of the image to be identified and the overall abnormal cell attention heat map comprises:

[0013] For each segmented cell, the abnormal cell attention degree of each cell pixel point in the cell segmentation mask image of the segmented cell corresponds to the abnormal cell attention degree of the corresponding pixel point in the overall abnormal cell attention degree heat map, and is determined as the abnormal cell score of the corresponding cell pixel point. In addition, the abnormal cell detection result of the segmented cell is generated according to the abnormal cell score of the cell pixel point in the cell segmentation mask image of the segmented cell.

[0014] In some optional embodiments, determining the abnormal cell image block score of the image block to be identified based on the distance between the image block feature and each normal cell image block feature in a pre-trained normal cell image block feature set includes:

[0015] Determine the distance between the image block feature and each normal cell image block feature in a normal cell image block feature set;

[0016] Based on the smallest K distances among the calculated distances, the abnormal cell image block score of the image block to be identified is determined, where K is a natural number.

[0017] In some optional embodiments, the pre-trained normal cell image block features and image block feature extraction model are pre-trained through the following feature extraction model training steps:

[0018] Acquire a first training sample set consisting of normal cell image blocks and a first verification sample set consisting of normal cell image blocks and abnormal cell image blocks, wherein the normal cell image blocks and the abnormal cell image blocks are both image blocks in immunofluorescence images, the labels of the normal cell image blocks are that they do not include abnormal cell objects, and the labels of the abnormal cell image blocks are that they include abnormal cell objects;

[0019] Initialize the model structure and model parameters of the image block feature extraction model, and initialize the normal cell image block feature set;

[0020] The optimal area under the curve AUC is set to a preset smaller AUC;

[0021] For the first training sample in the first training sample set, the following model parameter adjustment steps are performed until the preset parameter adjustment end condition is met: data enhancement is performed on the first training sample to obtain at least two enhanced training samples; each enhanced training sample is input into the image block feature extraction model to obtain corresponding image block features; based on the difference between the image block features of each enhanced training sample, the adjusted model parameters of the image block feature extraction model are determined; based on the adjusted parameters of the image block feature extraction model, the adjusted image block features of each first training sample and each first verification sample are respectively extracted; for each first verification sample, the adjusted image block features of the first verification sample are determined to be different from the adjusted image block features of each first training sample; The distance between the image block features after the abnormal cell image block is determined, and based on the smallest K distances among the determined distances, the abnormal cell image block score of the first verification sample is determined; based on the difference between the abnormal cell image block score and the label of each first verification sample in the first verification sample set, the current area under the curve AUC is calculated; it is determined whether the calculated AUC is greater than the optimal AUC; in response to determining that it is greater than, the optimal AUC is updated to the current AUC, the model parameters of the image block feature extraction model are updated to the adjusted model parameters, and the normal cell image block feature set is updated to the image block features after parameter adjustment of each of the first training samples; in response to determining that it is not greater than, continuing to execute the model parameter adjustment step.

[0022] In some optional embodiments, the overall cell segmentation result of performing cell segmentation on the image to be identified is obtained through the following cell segmentation steps:

[0023] Performing cell segmentation on each of the image blocks to be identified based on the cell segmentation model to obtain corresponding cell segmentation results;

[0024] The cell segmentation results of each of the image blocks to be identified are combined to obtain the overall cell segmentation result.

[0025] In some optional embodiments, the cell segmentation model is obtained by the following cell segmentation model training steps:

[0026] Acquire a second training sample set and a second verification sample set, wherein a segmentation result obtained by first segmenting the second training sample or the second verification sample based on a preset cell segmentation model is marked as a correct segmentation result;

[0027] Based on the second training sample set and the second validation sample set, the preset cell segmentation model is trained to obtain the cell segmentation model, wherein the labeled cell segmentation results of the second training sample and the second validation sample are respectively the cell segmentation results obtained by the corresponding first segmentation.

[0028] In some optional embodiments, the method further comprises:

[0029] The overall abnormal cell attention heat map and the image to be identified are superimposed to obtain an abnormal cell attention visualization image.

[0030] In a second aspect, an embodiment of the present disclosure provides an abnormal cell detection device, the device comprising:

[0031] An image acquisition module is configured to acquire an image to be identified, wherein the image to be identified is an immunofluorescence image obtained by photographing a same field of view including cells;

[0032] A segmentation module, configured to segment the image to be identified into at least two image blocks to be identified;

[0033] The heat map generation module is configured to perform the following abnormal cell attention heat map generation steps for each image block to be identified: input the image block to be identified into a pre-trained image block feature extraction model to obtain image block features; determine the abnormal cell image block score of the image block to be identified based on the distance between the image block features and each normal cell image block feature in the pre-trained normal cell image block feature set, wherein the abnormal cell image block score is used to characterize the degree of including abnormal cell objects; determine whether the image block to be identified is an abnormal cell image block according to the abnormal cell image block score of the image block to be identified; in response to determining yes, back-propagating the image block features in the image block feature extraction model to obtain the abnormal cell attention heat map of the image block to be identified; in response to determining no, determining a preset low attention heat map as the abnormal cell attention heat map of the image block to be identified;

[0034] A merging module is configured to merge the abnormal cell attention heat maps of each image block to be identified to obtain an overall abnormal cell attention heat map;

[0035] The detection module is configured to generate an abnormal cell detection result for each segmented cell in the overall cell segmentation result based on the overall cell segmentation result of cell segmentation of the image to be identified and the overall abnormal cell attention heat map.

[0036] In some optional embodiments, the overall cell segmentation result of the image to be identified includes at least one cell segmentation mask image for segmenting cells, the cell segmentation mask image includes cell pixels for characterizing cell objects and background pixels for characterizing non-cell objects; and

[0037] The detection module is further configured to: for each segmented cell, determine the abnormal cell attention degree of each cell pixel point in the cell segmentation mask image of the segmented cell corresponding to the corresponding pixel point in the overall abnormal cell attention heat map as the abnormal cell score of the corresponding cell pixel point, and generate the abnormal cell detection result of the segmented cell according to the abnormal cell score of the cell pixel point in the cell segmentation mask image of the segmented cell.

[0038] In some optional embodiments, determining the abnormal cell image block score of the image block to be identified based on the distance between the image block feature and each normal cell image block feature in a pre-trained normal cell image block feature set includes:

[0039] Determine the distance between the image block feature and each normal cell image block feature in a normal cell image block feature set;

[0040] Based on the smallest K distances among the calculated distances, the abnormal cell image block score of the image block to be identified is determined, where K is a natural number.

[0041] In some optional embodiments, the pre-trained normal cell image block features and image block feature extraction model are pre-trained through the following feature extraction model training steps:

[0042] Acquire a first training sample set consisting of normal cell image blocks and a first verification sample set consisting of normal cell image blocks and abnormal cell image blocks, wherein the normal cell image blocks and the abnormal cell image blocks are both image blocks in immunofluorescence images, the labels of the normal cell image blocks are that they do not include abnormal cell objects, and the labels of the abnormal cell image blocks are that they include abnormal cell objects;

[0043] Initialize the model structure and model parameters of the image block feature extraction model, and initialize the normal cell image block feature set;

[0044] The optimal area under the curve AUC is set to a preset smaller AUC;

[0045] For the first training sample in the first training sample set, the following model parameter adjustment steps are performed until the preset parameter adjustment end condition is met: data enhancement is performed on the first training sample to obtain at least two enhanced training samples; each enhanced training sample is input into the image block feature extraction model to obtain corresponding image block features; based on the difference between the image block features of each enhanced training sample, the adjusted model parameters of the image block feature extraction model are determined; based on the adjusted parameters of the image block feature extraction model, the adjusted image block features of each first training sample and each first verification sample are respectively extracted; for each first verification sample, the adjusted image block features of the first verification sample are determined to be different from the adjusted image block features of each first training sample; The distance between the image block features after the abnormal cell image block is determined, and based on the smallest K distances among the determined distances, the abnormal cell image block score of the first verification sample is determined; based on the difference between the abnormal cell image block score and the label of each first verification sample in the first verification sample set, the current area under the curve AUC is calculated; it is determined whether the calculated AUC is greater than the optimal AUC; in response to determining that it is greater than, the optimal AUC is updated to the current AUC, the model parameters of the image block feature extraction model are updated to the adjusted model parameters, and the normal cell image block feature set is updated to the image block features after parameter adjustment of each of the first training samples; in response to determining that it is not greater than, continuing to execute the model parameter adjustment step.

[0046] In some optional embodiments, the overall cell segmentation result of performing cell segmentation on the image to be identified is obtained through the following cell segmentation steps:

[0047] Performing cell segmentation on each of the image blocks to be identified based on the cell segmentation model to obtain corresponding cell segmentation results;

[0048] The cell segmentation results of each of the image blocks to be identified are combined to obtain the overall cell segmentation result.

[0049] In some optional embodiments, the cell segmentation model is obtained by the following cell segmentation model training steps:

[0050] Acquire a second training sample set and a second verification sample set, wherein a segmentation result obtained by first segmenting the second training sample or the second verification sample based on a preset cell segmentation model is marked as a correct segmentation result;

[0051] Based on the second training sample set and the second validation sample set, the preset cell segmentation model is trained to obtain the cell segmentation model, wherein the labeled cell segmentation results of the second training sample and the second validation sample are respectively the cell segmentation results obtained by the corresponding first segmentation.

[0052] In some optional embodiments, the device further includes a visualization module configured to superimpose the overall abnormal cell attention heat map and the image to be identified to obtain an abnormal cell attention visualization image.

[0053] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0054] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method described in any implementation manner in the first aspect.

[0055] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program / instruction, which implements the method described in any implementation manner in the first aspect when the computer program / instruction is executed by a processor.

[0056] In order to solve the problem of high labor costs required for manual interpretation of abnormal cells, the abnormal cell detection method, device, electronic device, storage medium and computer program product provided by the embodiments of the present disclosure first divide the image to be identified into at least two image blocks to be identified. Then, for each image block to be identified, a corresponding abnormal cell attention heat map is generated. Then, the abnormal cell attention heat maps of each image block to be identified are merged to obtain the overall abnormal cell attention heat map. Finally, based on the overall cell segmentation result of cell segmentation of the image to be identified and the overall abnormal cell attention heat map, the abnormal cell detection result of each segmented cell in the overall cell segmentation result is generated. In this way, the abnormal cell detection result of each cell can be automatically determined, and then professional and technical personnel only need to manually review and interpret the part detected as an abnormal cell, and no further manual review and interpretation is required for the remaining part, which greatly reduces labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Other features, objects and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are only for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the drawings:

[0058] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0059] Figure 2A is a flow chart of an embodiment of an abnormal cell detection method according to the present disclosure;

[0060] Figure 2B is a decomposed flow chart of one embodiment of step 203 according to the present disclosure;

[0061] Figure 2C is a decomposed flow chart of one embodiment of step 205 according to the present disclosure;

[0062] Figure 3A is a flowchart of an embodiment of a feature extraction model training step according to the present disclosure;

[0063] Figure 3B is an exemplary structural diagram of an embodiment of an image block feature extraction model according to the present disclosure;

[0064] Figure 3C is a flow chart of an embodiment of a model parameter adjustment step according to the present disclosure;

[0065] Figure 4 is a flow chart of one embodiment of a cell segmentation step according to the present disclosure;

[0066] Figure 5 is a flowchart of an embodiment of the cell segmentation model training steps according to the present disclosure;

[0067] Figure 6 is an embodiment of an image to be identified and a corresponding schematic diagram of an abnormal cell attention visualization image;

[0068] Figure 7 is a schematic diagram of an abnormal cell attention visualization image generated by using different abnormal cell attention heat map generation methods;

[0069] Figure 8 This is a comparison of the segmentation effects before and after retraining using the CellPose cell segmentation model;

[0070] Fig. 9 is a ROC curve diagram according to a specific embodiment of the present disclosure;

[0071] Fig.10 is a schematic structural diagram of an abnormal cell detection device according to an embodiment of the present disclosure;

[0072] Fig.11 It is a structural diagram of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0073] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0074] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0075] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the abnormal cell detection method, apparatus, electronic device, and storage medium of the present disclosure can be applied.

[0076] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0077] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as abnormal cell detection applications, short video social applications, audio and video conferencing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0078] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with information input devices (e.g., keyboard, mouse, touch screen, microphone, camera, etc.) and information output devices (e.g., display screen, speaker, etc.), including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4) players, laptop computers and desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the terminal devices listed above. It can be implemented as multiple software or software modules (for example, used to provide abnormal cell detection services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0079] In some cases, the abnormal cell detection method provided by the present disclosure may be executed by the terminal devices 101, 102, 103, and accordingly, the abnormal cell detection apparatus may be disposed in the terminal devices 101, 102, 103. In this case, the system architecture 100 may not include the server 105.

[0080] In some cases, the abnormal cell detection method provided by the present disclosure can be jointly performed by the terminal devices 101, 102, 103 and the server 105. For example, the step of "obtaining an image to be identified" can be performed by the terminal devices 101, 102, 103, and the steps of "for each image block to be identified, executing the following abnormal cell attention heat map generation step" can be performed by the server 105. The present disclosure does not limit this. Accordingly, the abnormal cell detection device can also be respectively set in the terminal devices 101, 102, 103 and the server 105.

[0081] In some cases, the abnormal cell detection method provided by the present disclosure may be executed by the server 105 , and accordingly, the abnormal cell detection device may also be disposed in the server 105 . In this case, the system architecture 100 may not include the terminal devices 101 , 102 , and 103 .

[0082] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0083] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0084] Continue to refer Figure 2A , which shows a process 200 of an embodiment of an abnormal cell detection method according to the present disclosure, the abnormal cell detection method comprises the following steps:

[0085] Step 201: Obtain an image to be recognized.

[0086] Here, the image to be identified may be an immunofluorescence image obtained by photographing the same field of view including cells using a microscope.

[0087] In some embodiments, the image to be recognized may be a grayscale image.

[0088] Step 202: divide the image to be recognized into at least two image blocks to be recognized.

[0089] Here, various image segmentation methods may be used to segment the image to be identified into at least two image blocks to be identified. For example, a sliding window method may be used for segmentation, and there may be overlapping and intersecting parts between the image blocks to be identified obtained by segmentation. For another example, an average segmentation method may be used, and there may be no overlapping and intersecting parts between the image blocks to be identified obtained by segmentation.

[0090] Step 203: for each image block to be identified, a step of generating an abnormal cell attention heat map is performed.

[0091] Here, the abnormal cell attention heat map generation step may include: Figure 2B Steps 2031 to 2025 are shown:

[0092] Step 2031: input the image block to be identified into a pre-trained image block feature extraction model to obtain image block features.

[0093] Here, the image block feature extraction model can be various models used for image feature extraction, and the present disclosure does not make any specific limitation on this.

[0094] Step 2032 , determining the abnormal cell image block score of the image block to be identified based on the distance between the image block feature of the image block to be identified and each normal cell image block feature in the pre-trained normal cell image block feature set.

[0095] Here, the normal cell image block features may be features obtained by inputting a normal cell image block that is manually labeled as including only normal cell objects but not including abnormal cell objects into an image block feature extraction model.

[0096] Since the normal cell image block feature set is obtained by feature extraction based on the normal cell image block, it can only characterize the normal cell image block, but cannot characterize the features of the abnormal cell image block, that is, the detection of abnormal cells cannot be achieved by using only the features of the normal cell image block. Therefore, an unsupervised approach is adopted here. If the distance between the image block feature of the image block to be identified and the normal cell image block feature is larger, it indicates that the image block to be identified is farther away from the normal cell image block, that is, the image block to be identified is more likely to include an abnormal cell object. Conversely, the smaller the distance between the image block feature of the image block to be identified and the normal cell image block feature, it indicates that the image block to be identified is closer to the normal cell image block, that is, the image block to be identified is more likely to not include an abnormal cell object.

[0097] Therefore, the abnormal cell image block score of the image block to be identified can be determined based on the distance between the image block feature of the image block to be identified and each normal cell image block feature in the normal cell image block feature set. The abnormal cell image block score is used to characterize the degree of including abnormal cell objects. The larger the abnormal cell image block score of the image block to be identified, the higher the possibility that the image block to be identified includes abnormal cell objects. Based on the above description, it can be known that, on the whole, the abnormal cell image block score of the image block to be identified should be negatively correlated with the distance between the image block feature of the image block to be identified and each normal cell image block feature in the normal cell image block feature set.

[0098] Optionally, step 2032 may be performed as follows:

[0099] First, the distance between the image block feature of the image block to be identified and each normal cell image block feature in the normal cell image block feature set is determined.

[0100] Then, based on the smallest K distances among the calculated distances, the abnormal cell image block score of the image block to be identified is determined. Here, K is a natural number.

[0101] As an example, K may be 2.

[0102] Here, the abnormal cell image block score of the image block to be identified is positively correlated with the K minimum distances. Optionally, the sum or mean of the K minimum distances among the obtained distances can be determined as the abnormal cell image block score of the image block to be identified.

[0103] That is to say, the larger the sum or mean of the K minimum distances, the greater the distance between the image block to be identified and the closest K normal cell image block features in the normal cell image block set, and the image block to be identified is farther away from all normal cell image block features as a whole, that is, the possibility that the image block to be identified is an abnormal cell image block is higher, and thus the abnormal cell image block score of the image block to be identified is also higher. Vice versa, no further elaboration is given.

[0104] Step 2033: Determine whether the image block to be identified is an abnormal cell image block according to the abnormal cell image block score of the image block to be identified.

[0105] Specifically, step 2033 may be performed as follows: determining whether the abnormal cell image block score of the image block to be identified is greater than a preset abnormal image block score threshold.

[0106] If the abnormal cell image block score of the image block to be identified is greater than the preset abnormal image block score threshold, it can be determined that the image block to be identified is an abnormal cell image block, and the process goes to step 2034 for execution. If not, it is determined that the image block to be identified is not an abnormal cell image block, and the process goes to step 2035 for execution.

[0107] Step 2034, back-propagating the image block features in the image block feature extraction model to obtain an abnormal cell attention heat map of the image block to be identified.

[0108] If it is determined that the abnormal cell image block score of the image block to be identified is greater than the preset abnormal image block score threshold, it indicates that the image block to be identified is an abnormal cell image block, that is, the image block to be identified includes abnormal cell objects, but it is not certain which abnormal cell objects are specifically included in the image block to be identified. In order to finally give the abnormal cell detection result of each cell object in the image to be identified, the image block feature extraction model can further pay attention to each part of the image block to be identified in the process of obtaining the image block features of the image block to be identified. Specifically, various weighted heat map generation methods that are currently known or developed in the future to explain the model can be used to back-propagate the image block features of the image block to be identified in the image block feature extraction model to obtain the abnormal cell attention heat map of the image block to be identified. For example, the following methods are included but not limited to: Saliency map method, Guided Back Propagation (GBP) method, LRP (Layer-wise relevance propagation) method, Grad-CAM (Gradient-weighted Class Activation Mapping) method, Grad-CAM (Gradient-weighted Class Activation Mapping)++ method, etc.

[0109] The abnormal cell attention heat map reflects the degree of attention paid by the image block feature extraction model to each part of the image block to be identified in the process of obtaining the image block features of the image block to be identified. If the abnormal cell attention of a certain pixel in the abnormal cell attention heat map is higher, it means that the image block feature extraction model pays more attention to the corresponding pixel in the image block to be identified, and the possibility that the corresponding pixel is a pixel in the abnormal cell object is greater.

[0110] Step 2035: determine the preset low attention heat map as the abnormal cell attention heat map of the image block to be identified.

[0111] Here, if in step 2033, it is determined that the abnormal cell image block score of the image block to be identified is not greater than the preset abnormal image block score threshold, it indicates that the image block to be identified is a normal cell image block. In the process of obtaining the image block features of the image block to be identified, the image block feature extraction model should have a low overall attention to the image block to be identified. Therefore, the preset low attention heat map can be determined as the abnormal cell attention heat map corresponding to the image block features of the image block to be identified. For example, the abnormal cell attention of each pixel in the preset low attention heat map can be 0.

[0112] After step 203, a heat map of abnormal cell attention for each image block to be identified can be obtained.

[0113] It should be noted that the image size of the abnormal cell attention heat map of the image block to be identified may be the same as that of the image block to be identified.

[0114] Step 204 , merging the abnormal cell attention heat maps of each image block to be identified to obtain an overall abnormal cell attention heat map.

[0115] Here, the division method of each image block to be identified can be obtained according to the division of the image to be identified, the corresponding merging method can be determined, and the abnormal cell attention heat map of each image block to be identified can be merged according to the position of the corresponding image block to be identified in the image to be identified to obtain the overall abnormal cell attention heat map.

[0116] Step 205 , based on the overall cell segmentation result of cell segmentation of the image to be identified and the overall abnormal cell attention heat map, an abnormal cell detection result of each segmented cell in the overall cell segmentation result is generated.

[0117] Here, various cell segmentation methods can be used to perform cell segmentation on the image to be identified, and obtain the overall cell segmentation result of the image to be identified.

[0118] The overall cell segmentation result of the image to be identified may include the cell segmentation result of at least one segmented cell. The cell segmentation result of each segmented cell may be a cell segmentation mask image of the same size as the image to be identified, and the cell segmentation mask image may include cell pixels for characterizing cell objects (for example, the pixel value may be 1) and background pixels for characterizing non-cell objects (for example, the pixel value may be 0). The overall abnormal cell attention heat map is also an image of the same size as the image to be identified, and each pixel in the cell segmentation mask image of each segmented cell corresponds one-to-one to each pixel in the overall abnormal cell attention heat map.

[0119] Optionally, step 205 may include: Figure 2C Step 2051 and step 2052 are shown:

[0120] Step 2051, for each segmented cell, the abnormal cell attention degree of each cell pixel in the cell segmentation mask image of the segmented cell corresponding to the corresponding pixel in the overall abnormal cell attention degree heat map is determined as the abnormal cell score of the corresponding cell pixel.

[0121] Step 2052: Generate an abnormal cell detection result for each segmented cell according to the abnormal cell score of each cell pixel in the cell segmentation mask image of each segmented cell.

[0122] Specifically, for each segmented cell, the abnormal cell score of the segmented cell can be determined according to the abnormal cell score of each cell pixel in the cell segmentation mask image of the segmented cell, and then the abnormal cell detection result of the segmented cell can be generated according to the abnormal cell score of the segmented cell.

[0123] As an example, the maximum value among the abnormal cell scores of the cell pixels in the cell segmentation mask image of each segmented cell may be determined as the abnormal cell score of the segmented cell.

[0124] As an example, it can be first determined whether the abnormal cell score of the segmented cell is greater than a preset abnormal cell score threshold. If it is determined to be greater, an abnormal cell detection result indicating that the segmented cell is an abnormal cell is generated. If it is determined not to be greater, an abnormal cell detection result indicating that the segmented cell is a normal cell is generated.

[0125] In some optional embodiments, the pre-trained normal cell image block features and image block feature extraction model can be obtained by Figure 3A The feature extraction model training step 300 shown is pre-trained, and the feature extraction model training step 300 includes the following steps:

[0126] Step 301 : obtaining a first training sample set consisting of normal cell image blocks and a first verification sample set consisting of normal cell image blocks and abnormal cell image blocks.

[0127] Here, both the normal cell image block and the abnormal cell image block are image blocks in the immunofluorescence image, the normal cell image block is labeled as not including the abnormal cell object, and the abnormal cell image block is labeled as including the abnormal cell object. That is, the normal cell image block is manually labeled to determine that it does not include the abnormal cell object, and the abnormal cell image block is manually labeled to determine that it includes the abnormal cell object.

[0128] Step 302: Initialize the model structure and model parameters of the image block feature extraction model, and initialize the normal cell image block feature set.

[0129] Here, the image block feature extraction model may be a pre-trained model for extracting image features, and the model structure and model parameters of the pre-trained image block feature extraction model may be directly obtained.

[0130] Alternatively, the image block feature extraction model may be a model for extracting image features that has not been pre-trained, and various parameter initialization methods may be used to initialize the model parameters of the image block feature extraction model. For example, the parameters may be randomly set to smaller values.

[0131] In some optional implementations, reference may be made to Figure 3B, Figure 3B FIG. 2 shows an exemplary structural diagram of an image block feature extraction model. Figure 3B As shown in , the image block feature extraction model may include a sequentially connected Vision Transformer encoder and a nonlinear projector. The Vision Transformer encoder is composed of multiple sub-encoders consisting of sequentially connected normalization layers, multi-head attention layers, normalization layers, and multi-layer perceptrons. The nonlinear encoder includes sequentially connected fully connected layers, normalization layers, activation function layers, and fully connected layers. Figure 3B The image block feature extraction model shown can extract linear and nonlinear features of the image block to be identified from multiple dimensions.

[0132] Step 303: Set the optimal area under the curve AUC to a preset smaller AUC.

[0133] As an example, the preset smaller AUC may be 0.

[0134] Step 304: For the first training sample in the first training sample set, perform a model parameter adjustment step until a preset parameter adjustment end condition is met.

[0135] The model parameter adjustment steps include: Figure 3C The following steps 3041 to 3048 are shown:

[0136] Step 3041: perform data enhancement on the first training sample to obtain at least two enhanced training samples.

[0137] Here, the first training sample is a normal cell image block, so the normal cell image block can be processed by different data enhancement methods to obtain different enhanced training samples, that is, enhanced normal cell image blocks. For example, D data enhancement methods can be used to obtain D enhanced training samples, that is, D enhanced normal cell image blocks. Here, D is a natural number not less than 2.

[0138] Here, there is no specific limitation on the specific data enhancement method, for example, it may include but is not limited to: geometric transformation (translation, flipping, mirroring, scaling, cropping), color space transformation, kernel filtering, random erasing, feature space enhancement, etc.

[0139] Step 3042: input each enhanced training sample into the image block feature extraction model to obtain the corresponding image block features.

[0140] Step 3043: Determine the adjusted model parameters of the image block feature extraction model based on the differences between the image block features of each enhanced training sample.

[0141] It is understandable that, although different data processing methods are used for different enhanced training samples of the same first training sample, they are from the same normal cell block image. Therefore, the image block features obtained by inputting different enhanced training samples of the same first training sample into the image block feature extraction model should also be relatively close. Therefore, various loss function calculation methods can be used here to calculate the loss function between the image block features of different enhanced training samples of the same first training sample, and various optimization methods can be used to determine the adjusted model parameters of the image block feature extraction model based on the calculated loss function values.

[0142] As an example, the first training sample may be data enhanced to obtain three enhanced training samples in step 3041. Accordingly, a triple mean-shift contrastive loss calculation method may be used in step 3043 to calculate the difference between the image block features of three different enhanced training samples for the same first training sample.

[0143] It should be noted that, in step 3043, only the parameter values ​​of the adjusted model parameters of the image block feature extraction model are determined, but the model parameter values ​​of the image block feature extraction model themselves are not directly adjusted and updated.

[0144] Step 3044, based on the adjusted parameters of the image block feature extraction model, extract the adjusted image block features of each first training sample and each first verification sample respectively.

[0145] Here, according to the adjusted parameters of the image block feature extraction model determined in step 3043, each first training sample can be input into the image block feature extraction model to obtain the image block features of the first training sample after parameter adjustment, and each first verification sample can be input into the image block feature extraction model to obtain the image block features of the first verification sample after parameter adjustment.

[0146] Step 3045: For each first verification sample, determine the distance between the parameter-adjusted image block feature of the first verification sample and the parameter-adjusted image block feature of each first training sample, and determine the abnormal cell image block score of the first verification sample based on the smallest K distances among the determined distances.

[0147] Here, various feature distance calculation methods can be used to calculate the distance between the parameter-adjusted image block features of each first verification sample and the parameter-adjusted image block features of each first training sample.

[0148] For each first verification sample, the abnormal cell image block score of the first verification sample can be determined based on the smallest K distances between the parameter-adjusted image block features of the first verification sample and the parameter-adjusted image block features of each first training sample. For example, the sum or mean of the smallest K distances can be used as the abnormal cell image block score of the first verification sample.

[0149] Step 3046, calculating the current area under the curve AUC based on the difference between the abnormal cell image block score and the label of each first validation sample in the first validation sample set.

[0150] After step 3045, the abnormal cell image block score of each first verification sample has been obtained, and each first verification sample is a normal cell image block or an abnormal cell image block, and the corresponding label is not including an abnormal cell object or including an abnormal cell object.

[0151] In the process of calculating AUC, the current ROC can be obtained according to different abnormal cell image block score thresholds.

[0152] For a specific abnormal cell image block score threshold, when the abnormal cell image block score of the first verification sample is greater than the abnormal cell image block score threshold, an actual abnormal cell detection result is generated to characterize the first verification sample as an abnormal cell image block; if the abnormal cell image block score of the first verification sample is not greater than the abnormal cell image block score threshold, an actual abnormal cell detection result is generated to characterize the first verification sample as a normal cell image block. Finally, based on the difference between the actual abnormal cell detection result of the first verification sample and the label, the false positive rate (FPR) of the horizontal coordinate of the curve and the true positive rate (TPR) of the vertical coordinate of the curve are calculated.

[0153] Finally, the current AUC can be calculated based on the above current ROC.

[0154] Step 3047, determine whether the calculated AUC is greater than the optimal AUC.

[0155] If it is determined to be greater than, it indicates that the adjusted model parameters of the image block feature extraction model perform better than the unadjusted model parameters, and the process may proceed to step 3048.

[0156] If it is determined that it is not greater than, it indicates that the adjusted model parameters of the image block feature extraction model do not perform better than the model parameters before adjustment, and the process may proceed to step 3041 to continue to perform the model parameter adjustment step 300.

[0157] Step 3048, updates the optimal AUC to the current AUC, updates the model parameters of the image block feature extraction model to the adjusted model parameters, and updates the normal cell image block feature set to the adjusted image block features of each first training sample.

[0158] That is, if the adjusted model parameters of the image block feature extraction model perform better than the model parameters before adjustment, the model parameters of the image block feature extraction model are updated with the adjusted model parameters with better performance, and the normal cell image block feature set is updated with the first training sample adjusted image block features obtained based on the adjusted parameters of the image block feature extraction model in step 3044 until the preset adjustment end condition is met.

[0159] Here, the preset parameter adjustment end condition can be set according to the actual situation. For example, the preset parameter adjustment end condition may include but is not limited to at least one of the following conditions: the number of times the model parameter adjustment step is executed is greater than the preset number of times, the model parameter adjustment step has been executed for each first training sample in the first training sample set, etc.

[0160] In some optional embodiments, the overall cell segmentation result used in step 204 for performing cell segmentation on the image to be identified can be obtained by: Figure 4 The cell segmentation step 400 shown in FIG. 4 is obtained, and the cell segmentation step 400 includes the following steps 401 and 402:

[0161] Step 401 , performing cell segmentation on each image block to be identified based on a cell segmentation model to obtain a corresponding cell segmentation result.

[0162] Here, the cell segmentation model may be various known or future developed machine learning models for performing cell segmentation on immunofluorescence images including cell objects. As an example, the cell segmentation model may be a CellPose cell segmentation model.

[0163] Step 402: merge the cell segmentation results of each image block to be identified to obtain the overall cell segmentation result.

[0164] Here, the image blocks to be identified may be at least two image blocks obtained by dividing the image to be identified. By using the cell segmentation model to perform cell block segmentation on each image block to be identified in the image to be identified, a cell segmentation result of each image block to be identified is obtained, and then the cell segmentation results of each image block to be identified are merged by using a merging method corresponding to the division method of dividing the image to be identified to obtain the image blocks to be identified, so as to obtain the overall cell segmentation result of the image to be identified.

[0165] In some optional embodiments, the cell segmentation model can be obtained by Figure 5The cell segmentation model training step 500 shown in FIG. 5 is obtained, and the cell segmentation model training step includes the following steps 501 and 602:

[0166] Step 501: Obtain a second training sample set and a second verification sample set.

[0167] The segmentation result obtained by first segmenting the second training sample or the second verification sample based on the preset cell segmentation model is marked as a correct segmentation result.

[0168] Here, the preset cell segmentation model may be a pre-trained model for cell segmentation, for example, a Cellpose cell segmentation model.

[0169] In practice, a preset cell segmentation model can be used to perform the first cell segmentation on a large number of immunofluorescence image blocks including cell objects, and then a small number of immunofluorescence image blocks with good segmentation effects can be manually screened based on the first segmentation results, and marked as correct segmentation results. Finally, the immunofluorescence image blocks marked as correct segmentation results are divided into a second training sample set and a second verification sample set. In this way, the cost of manual labeling can be greatly reduced after the first segmentation of the preset cell segmentation model.

[0170] Step 502: Based on the second training sample set and the second verification sample set, a preset cell segmentation model is trained to obtain a cell segmentation model.

[0171] Here, various machine learning methods can be used to train the preset cell segmentation model based on the second training sample set and the second verification sample set to obtain a cell segmentation model. During the training process, the preset cell segmentation model is trained with the second training sample set as the training sample, and the labeled cell segmentation results of the second training sample are first segmented by the preset cell segmentation model and manually labeled as the correct cell segmentation results. The preset cell segmentation model is verified with the second verification sample set as the verification sample, and the labeled cell segmentation results of the second verification sample are first segmented by the preset cell segmentation model and manually labeled as the correct cell segmentation results.

[0172] The training method of the above-mentioned cell segmentation model training step 500 does not require direct manual labeling of cell segmentation results. Instead, it first uses a preset cell segmentation model to perform the first segmentation, and then manually selects samples with better segmentation effects as training samples and verification samples based on the first segmentation, and further trains the preset cell segmentation model. This can reduce the cost of manual labeling in the model training process, especially for the high cost of manual labeling of cell segmentation, and the cost reduction effect is significant.

[0173] In some optional embodiments, the abnormal cell detection method process 200 may further include the following steps 206:

[0174] Step 206 , superimposing the overall abnormal cell attention heat map and the image to be identified to obtain an abnormal cell attention visualization image.

[0175] Here, various image overlay methods can be used to generate abnormal cell attention visualization images.

[0176] In practice, the overall abnormal cell attention is hot Figure 1 It is generally a feature image of the same size as the image to be identified. The pixel value of each pixel in the overall abnormal cell attention heat map can be a value between 0 and 1. The closer the pixel value is to 1, the more the image block feature extraction model pays attention to the pixel at the same position as the pixel in the image to be identified, or the greater the possibility that the pixel at the same position as the pixel in the image to be identified belongs to an abnormal cell.

[0177] As an example, step 206 may be performed as follows:

[0178] First, an abnormal cell attention feature image corresponding to the abnormal cell attention heat map may be generated.

[0179] Among them, each pixel point in the generated abnormal cell attention feature image corresponds to a pixel point in the abnormal cell attention heat map one by one, and the pixel value of each pixel point in the abnormal cell attention image is determined according to the pixel value of the pixel point at the same position in the abnormal cell attention heat map. For example, the smaller the pixel value of the pixel point in the abnormal cell attention heat map, the closer the pixel value of the corresponding pixel point in the abnormal cell attention feature image is to the pixel value used to represent the first preset color (for example, red); conversely, the larger the pixel value of the pixel point in the abnormal cell attention heat map, the closer the pixel value of the corresponding pixel point in the abnormal cell attention feature image is to the pixel value used to represent the second preset color (for example, green).

[0180] Then, the abnormal cell attention feature image is overlaid on the image to be identified according to a preset transparency to obtain an abnormal cell attention visualization image.

[0181] The abnormal cell attention visualization image can intuitively show the degree to which different pixels in the image to be identified are abnormal cells, that is, the image block feature extraction model can be intuitively explained. Figure 6 , Figure 6 The middle left part is an image to be identified, and the right side is the corresponding abnormal cell attention visualization image. Figure 7The figure shows the abnormal cell attention heat map generated by different abnormal cell attention heat map generation methods, and then the different abnormal cell attention visualization images are obtained by superimposing them with the image to be identified. Figure 7 The first row from left to right includes the original image, the image to be identified after preprocessing the original image, the abnormal cell attention visualization image generated by the Grad-CAM method for the image to be identified, and the abnormal cell attention visualization image generated by the Grad-CAM++ method.

[0182] The second row, from left to right, are the abnormal cell attention visualization images generated by using the XGrad-CAM method, EigenCAM, EigenGradCAM, and LayerCAM methods for the images to be identified.

[0183] from Figure 7 It can be seen that the abnormal cell attention visualization image generated by the Grad-CAM++ method has the best effect.

[0184] The abnormal cell detection method provided by the above-mentioned embodiment of the present disclosure is firstly divided into at least two image blocks to be identified. Then, for each image block to be identified, a corresponding abnormal cell attention heat map is generated. Then, the abnormal cell attention heat maps of each image block to be identified are merged to obtain an overall abnormal cell attention heat map. Finally, based on the overall cell segmentation result of cell segmentation of the image to be identified and the overall abnormal cell attention heat map, the abnormal cell detection result of each segmented cell in the overall cell segmentation result is generated. In this way, the abnormal cell detection result of each cell can be automatically determined, and then professional and technical personnel only need to manually review and interpret the part detected as abnormal cells, and no further manual review and interpretation is required for the remaining parts, which greatly reduces the labor cost.

[0185] Embodiment 1:

[0186] In the specific experiment, the experimental process is as follows:

[0187] 1. Dataset construction:

[0188] For the cell segmentation results of the cell immunofluorescence images that have undergone cell segmentation, each cell therein is manually labeled as a normal cell or an abnormal cell, and the center point of each cell is designated as the cell position information.

[0189] 2. Data preprocessing:

[0190] Using the sliding window method, with (512*512) as the window size and 412 as the step length for each movement, each cell immune image is divided into multiple patches (image blocks). If there is no center point coordinate marked as an abnormal cell in the patch, the patch is marked as a normal cell image block. If the patch includes at least one center point coordinate marked as an abnormal cell, the patch is marked as an abnormal cell image block, and the number of abnormal cell center point coordinates in the patch is recorded as the number of abnormal cells.

[0191] According to the distribution of the number of abnormal cells in each patch marked as an abnormal cell image block, 1320 normal cell image blocks were randomly selected as the first training sample set, and 220 normal cell image blocks and 220 abnormal cell image blocks were randomly selected as the first verification sample set.

[0192] 3. Image feature extraction model training:

[0193] According to the method shown in the feature extraction model training step 300, a normal cell image block feature set and an image block feature extraction model are obtained. The image block feature extraction model is as follows: Figure 3B Model shown.

[0194] Since the first training sample set has 1320 first training samples, the normal cell image block feature set correspondingly has 1320 normal cell image block features.

[0195] 4. Retraining of cell segmentation model:

[0196] According to the training method of the cell segmentation model training step 600, a large number of immunofluorescence cell image blocks are first segmented using CellPose as the preset cell segmentation model, and then 10 image blocks (patches) with better first segmentation effects are manually selected as the second training sample set and 6 image blocks as the second verification sample set. The above second training sample input and the corresponding first segmentation cell segmentation results are input into CellPose as "pseudo-mask" annotations to retrain CellPose. After 2000 epochs, the CellPose cell segmentation model after self-supervision fine-tuning is obtained. The effects of the CellPose cell segmentation model before and after fine-tuning are shown in the figure. Figure 8 shown. Figure 8 The left side of the figure shows the segmentation result of a cell image block by the CellPose cell segmentation model before retraining, and the right side shows the segmentation result of the CellPose cell segmentation model after retraining for the same cell image block. Figure 8It can be seen that some cells were not segmented before retraining, but after retraining, these cells were also segmented. In addition, before retraining, two cells were segmented into one cell, but after retraining, these merged cells were segmented separately. That is, after retraining, the segmentation accuracy of the CellPose cell segmentation model was improved.

[0197] 5. Model performance test:

[0198] In addition to the first training sample set and the first validation sample set, a test sample set was selected to test the detection performance of abnormal cell detection, and the following results were obtained: Fig. 9 The ROC curve shown in the figure shows that the AUC is 0.934, the recall rate is 0.89, and the cell judgment rate is 0.06.

[0199] Here, the calculation formula for recall is as follows:

[0200] Recall=TP / (TP+FN)

[0201] Among them, TP (True Positive) is the number of abnormal cells that are correctly detected, and FN (False Negative) is the number of cells that are abnormal cells but are not detected as abnormal cells, that is, the number of missed abnormal cells. Recall is the calculated recall rate.

[0202] The recall rate refers to the recall rate at the field of view level, not the recall rate at the patch level.

[0203] In practice, since the number of truly abnormal cells is very small, for abnormal cells detected by algorithms or models, each cell needs to be manually interpreted to confirm whether it is a truly abnormal cell. Therefore, in order to reduce labor costs, it is necessary to reduce the number of abnormal cells that still need to be manually interpreted. In other words, the lower the rate of interpreted cells, the better. In this experiment, the rate of interpreted cells was only 0.06, which greatly reduced the cost of manual interpretation.

[0204] Further references Fig.10 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an abnormal cell detection device. Figure 2A Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0205] like Fig.10 As shown, the abnormal cell detection device 1000 of this embodiment includes: an image acquisition module 1001, a segmentation module 1002, a heat map generation module 1003, a merging module 1004 and a detection module 1005.

[0206] An image acquisition module 1001 is configured to acquire an image to be identified, where the image to be identified is an immunofluorescence image obtained by photographing the same field of view including cells;

[0207] A segmentation module 1002 is configured to segment the image to be identified into at least two image blocks to be identified;

[0208] The heat map generation module 1003 is configured to perform the following abnormal cell attention heat map generation steps for each image block to be identified: input the image block to be identified into a pre-trained image block feature extraction model to obtain image block features; determine the abnormal cell image block score of the image block to be identified based on the distance between the image block features and each normal cell image block feature in the pre-trained normal cell image block feature set, wherein the abnormal cell image block score is used to characterize the degree of including abnormal cell objects; determine whether the image block to be identified is an abnormal cell image block according to the abnormal cell image block score of the image block to be identified; in response to determining yes, back-propagating the image block features in the image block feature extraction model to obtain the abnormal cell attention heat map of the image block to be identified; in response to determining no, determining a preset low attention heat map as the abnormal cell attention heat map of the image block to be identified;

[0209] A merging module 1004 is configured to merge the abnormal cell attention heat maps of each image block to be identified to obtain an overall abnormal cell attention heat map;

[0210] The detection module 1005 is configured to generate an abnormal cell detection result for each segmented cell in the overall cell segmentation result based on the overall cell segmentation result of cell segmentation of the image to be identified and the overall abnormal cell attention heat map.

[0211] In this embodiment, the specific processing of the image acquisition module 1001, the segmentation module 1002, the heat map generation module 1003, the merging module 1004 and the detection module 1005 of the abnormal cell detection device 1000 and the technical effects thereof can be referred to respectively. Figure 2A The relevant descriptions of step 201, step 202, step 203, step 204 and step 205 in the corresponding embodiment are not repeated here.

[0212] In some optional embodiments, the overall cell segmentation result of the image to be identified may include at least one cell segmentation mask image for segmenting cells, the cell segmentation mask image including cell pixels for characterizing cell objects and background pixels for characterizing non-cell objects; and

[0213] The detection module 1005 can be further configured to: for each segmented cell, determine the abnormal cell attention degree of each cell pixel point in the cell segmentation mask image of the segmented cell corresponding to the corresponding pixel point in the overall abnormal cell attention heat map as the abnormal cell score of the corresponding cell pixel point, and generate the abnormal cell detection result of the segmented cell according to the abnormal cell score of the cell pixel point in the cell segmentation mask image of the segmented cell.

[0214] In some optional embodiments, determining the abnormal cell image block score of the image block to be identified based on the distance between the image block feature and each normal cell image block feature in a pre-trained normal cell image block feature set may include:

[0215] Determine the distance between the image block feature and each normal cell image block feature in a normal cell image block feature set;

[0216] Based on the smallest K distances among the calculated distances, the abnormal cell image block score of the image block to be identified is determined, where K is a natural number.

[0217] In some optional embodiments, the pre-trained normal cell image block features and image block feature extraction model may be pre-trained through the following feature extraction model training steps:

[0218] Acquire a first training sample set consisting of normal cell image blocks and a first verification sample set consisting of normal cell image blocks and abnormal cell image blocks, wherein the normal cell image blocks and the abnormal cell image blocks are both image blocks in immunofluorescence images, the labels of the normal cell image blocks are that they do not include abnormal cell objects, and the labels of the abnormal cell image blocks are that they include abnormal cell objects;

[0219] Initialize the model structure and model parameters of the image block feature extraction model, and initialize the normal cell image block feature set;

[0220] The optimal area under the curve AUC is set to a preset smaller AUC;

[0221] For the first training sample in the first training sample set, the following model parameter adjustment steps are performed until the preset parameter adjustment end condition is met: data enhancement is performed on the first training sample to obtain at least two enhanced training samples; each enhanced training sample is input into the image block feature extraction model to obtain corresponding image block features; based on the difference between the image block features of each enhanced training sample, the adjusted model parameters of the image block feature extraction model are determined; based on the adjusted parameters of the image block feature extraction model, the adjusted image block features of each first training sample and each first verification sample are respectively extracted; for each first verification sample, the adjusted image block features of the first verification sample are determined to be different from the adjusted image block features of each first training sample; The distance between the image block features after the abnormal cell image block is determined, and based on the smallest K distances among the determined distances, the abnormal cell image block score of the first verification sample is determined; based on the difference between the abnormal cell image block score and the label of each first verification sample in the first verification sample set, the current area under the curve AUC is calculated; it is determined whether the calculated AUC is greater than the optimal AUC; in response to determining that it is greater than, the optimal AUC is updated to the current AUC, the model parameters of the image block feature extraction model are updated to the adjusted model parameters, and the normal cell image block feature set is updated to the image block features after parameter adjustment of each of the first training samples; in response to determining that it is not greater than, continuing to execute the model parameter adjustment step.

[0222] In some optional embodiments, the overall cell segmentation result of performing cell segmentation on the image to be identified may be obtained through the following cell segmentation steps:

[0223] Performing cell segmentation on each of the image blocks to be identified based on the cell segmentation model to obtain corresponding cell segmentation results;

[0224] The cell segmentation results of each of the image blocks to be identified are combined to obtain the overall cell segmentation result.

[0225] In some optional embodiments, the cell segmentation model can be obtained through the following cell segmentation model training steps:

[0226] Acquire a second training sample set and a second verification sample set, wherein a segmentation result obtained by first segmenting the second training sample or the second verification sample based on a preset cell segmentation model is marked as a correct segmentation result;

[0227] Based on the second training sample set and the second validation sample set, the preset cell segmentation model is trained to obtain the cell segmentation model, wherein the labeled cell segmentation results of the second training sample and the second validation sample are respectively the cell segmentation results obtained by the corresponding first segmentation.

[0228] In some optional embodiments, the abnormal cell detection device 1000 may further include a visualization module 1006 configured to superimpose the overall abnormal cell attention heat map and the image to be identified to obtain an abnormal cell attention visualization image.

[0229] It should be noted that the implementation details and technical effects of each module in the abnormal cell detection device provided in the embodiments of the present disclosure can be referred to the description of other embodiments in the present disclosure, and will not be repeated here.

[0230] Reference below Fig.11 , which shows a schematic diagram of the structure of a computer system 1100 suitable for implementing the electronic device of the present disclosure. Fig.11 The computer system 1100 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0231] like Fig.11 As shown, the computer system 1100 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage device 1108 to a random access memory (RAM) 1103. Various programs and data required for the operation of the computer system 1100 are also stored in the RAM 1103. The processing device 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0232] Typically, the following devices may be connected to the I / O interface 1105: input devices 1106 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, etc.; output devices 1107 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1108 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1109. The communication devices 1109 may allow the computer system 1100 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.11 The computer system 1100 of an electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0233] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 1109, or installed from a storage device 1108, or installed from a ROM 1102. When the computer program is executed by the processing device 1101, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0234] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0235] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0236] The computer readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device implements the following Figure 2AThe illustrated embodiment and its alternative implementations illustrate a method for detecting abnormal cells.

[0237] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Python, Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0238] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0239] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, an image acquisition module may also be described as a "module for acquiring an image to be recognized."

[0240] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

Claims

1. A method for detecting abnormal cells, comprising: Acquiring an image to be identified, wherein the image to be identified is an immunofluorescence image obtained by photographing the same field of view including the cells; Dividing the image to be identified into at least two image blocks to be identified; For each image block to be identified, the following abnormal cell attention heat map generation step is performed: the image block to be identified is input into a pre-trained image block feature extraction model to obtain image block features; based on the distance between the image block features and each normal cell image block feature in the pre-trained normal cell image block feature set, the abnormal cell image block score of the image block to be identified is determined, wherein the abnormal cell image block score is used to characterize the degree of inclusion of abnormal cell objects; according to the abnormal cell image block score of the image block to be identified, it is determined whether the image block to be identified is an abnormal cell image block; in response to determining yes, the image block features are back-propagated in the image block feature extraction model to obtain the abnormal cell attention heat map of the image block to be identified; in response to determining no, a preset low attention heat map is determined as the abnormal cell attention heat map of the image block to be identified; Merge the abnormal cell attention heat maps of each image block to be identified to obtain the overall abnormal cell attention heat map; Based on the overall cell segmentation result of cell segmentation of the image to be identified and the overall abnormal cell attention heat map, an abnormal cell detection result of each segmented cell in the overall cell segmentation result is generated.

2. The method according to claim 1, wherein: The overall cell segmentation result of the image to be identified includes at least one cell segmentation mask image for segmenting cells, and the cell segmentation mask image includes cell pixels for characterizing cell objects and background pixels for characterizing non-cell objects; as well as The generating of the abnormal cell detection result of each segmented cell in the overall cell segmentation result based on the overall cell segmentation result of the image to be identified and the overall abnormal cell attention heat map comprises: For each segmented cell, the abnormal cell attention degree of each cell pixel point in the cell segmentation mask image of the segmented cell corresponds to the abnormal cell attention degree of the corresponding pixel point in the overall abnormal cell attention degree heat map, and is determined as the abnormal cell score of the corresponding cell pixel point. In addition, the abnormal cell detection result of the segmented cell is generated according to the abnormal cell score of the cell pixel point in the cell segmentation mask image of the segmented cell.

3. The method according to claim 1, wherein: The step of determining the abnormal cell image block score of the image block to be identified based on the distance between the image block feature and each normal cell image block feature in a pre-trained normal cell image block feature set comprises: Determine the distance between the image block feature and each normal cell image block feature in a normal cell image block feature set; Based on the smallest K distances among the calculated distances, the abnormal cell image block score of the image block to be identified is determined, where K is a natural number.

4. The method according to claim 3, wherein: The pre-trained normal cell image block features and image block feature extraction model are pre-trained through the following feature extraction model training steps: Acquire a first training sample set consisting of normal cell image blocks and a first verification sample set consisting of normal cell image blocks and abnormal cell image blocks, wherein the normal cell image blocks and the abnormal cell image blocks are both image blocks in immunofluorescence images, the labels of the normal cell image blocks are that they do not include abnormal cell objects, and the labels of the abnormal cell image blocks are that they include abnormal cell objects; Initialize the model structure and model parameters of the image block feature extraction model, and initialize the normal cell image block feature set; The optimal area under the curve AUC is set to a preset smaller AUC; For the first training sample in the first training sample set, the following model parameter adjustment steps are performed until the preset parameter adjustment end condition is met: data enhancement is performed on the first training sample to obtain at least two enhanced training samples; each enhanced training sample is input into the image block feature extraction model to obtain corresponding image block features; based on the difference between the image block features of each enhanced training sample, the adjusted model parameters of the image block feature extraction model are determined; based on the adjusted parameters of the image block feature extraction model, the adjusted image block features of each first training sample and each first verification sample are respectively extracted; for each first verification sample, the adjusted image block features of the first verification sample are determined to be different from the adjusted image block features of each first training sample; The distance between the image block features after the abnormal cell image block is determined, and based on the smallest K distances among the determined distances, the abnormal cell image block score of the first verification sample is determined; based on the difference between the abnormal cell image block score and the label of each first verification sample in the first verification sample set, the current area under the curve AUC is calculated; it is determined whether the calculated AUC is greater than the optimal AUC; in response to determining that it is greater than, the optimal AUC is updated to the current AUC, the model parameters of the image block feature extraction model are updated to the adjusted model parameters, and the normal cell image block feature set is updated to the image block features after parameter adjustment of each of the first training samples; in response to determining that it is not greater than, continuing to execute the model parameter adjustment step.

5. The method according to claim 1, wherein: The overall cell segmentation result of performing cell segmentation on the image to be identified is obtained through the following cell segmentation steps: Performing cell segmentation on each of the image blocks to be identified based on the cell segmentation model to obtain corresponding cell segmentation results; The cell segmentation results of each of the image blocks to be identified are combined to obtain the overall cell segmentation result.

6. The method according to claim 5, wherein: The cell segmentation model is obtained through the following cell segmentation model training steps: Acquire a second training sample set and a second verification sample set, wherein a segmentation result obtained by first segmenting the second training sample or the second verification sample based on a preset cell segmentation model is marked as a correct segmentation result; Based on the second training sample set and the second validation sample set, the preset cell segmentation model is trained to obtain the cell segmentation model, wherein the labeled cell segmentation results of the second training sample and the second validation sample are respectively the cell segmentation results obtained by the corresponding first segmentation.

7. The method according to claim 1, wherein: The method further comprises: The overall abnormal cell attention heat map and the image to be identified are superimposed to obtain an abnormal cell attention visualization image.

8. An abnormal cell detection device, comprising: An image acquisition module is configured to acquire an image to be identified, wherein the image to be identified is an immunofluorescence image obtained by photographing a same field of view including cells; A segmentation module, configured to segment the image to be identified into at least two image blocks to be identified; The heat map generation module is configured to perform the following abnormal cell attention degree heat map generation steps for each image block to be identified: input the image block to be identified into a pre-trained image block feature extraction model to obtain image block features; Based on the distance between the image block feature and each normal cell image block feature in the pre-trained normal cell image block feature set, determine the abnormal cell image block score of the image block to be identified, wherein the abnormal cell image block score is used to characterize the degree of including abnormal cell objects; according to the abnormal cell image block score of the image block to be identified, determine whether the image block to be identified is an abnormal cell image block; in response to the determination of yes, back-propagate the image block feature in the image block feature extraction model to obtain the abnormal cell attention heat map of the image block to be identified; in response to the determination of no, determine the preset low attention heat map as the abnormal cell attention heat map of the image block to be identified; A merging module is configured to merge the abnormal cell attention heat maps of each image block to be identified to obtain an overall abnormal cell attention heat map; The detection module is configured to generate an abnormal cell detection result for each segmented cell in the overall cell segmentation result based on the overall cell segmentation result of cell segmentation of the image to be identified and the overall abnormal cell attention heat map.

9. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by one or more processors, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images