Image processing methods, apparatus, devices and storage media

Abstract

This application relates to an image processing method, apparatus, device, and storage medium, and pertains to the field of artificial intelligence technology. The method includes: acquiring a recognition image of a target cell nucleus based on a medical microscopic image; the recognition image is a multi-channel image composed of a target image patch and a mask image; cell type identification is performed based on the recognition image to obtain the cell type to which the cell corresponding to the target cell nucleus belongs; this solution can more accurately extract features related to cell classification, thereby improving the accuracy of cell classification in medical microscopic images.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus, device, and storage medium. Background Technology

[0002] With the continuous development of the application of artificial intelligence (AI) in the medical field, AI can now be used to classify cells in medical microscopic images to assist medical personnel in making medical-related decisions.

[0003] In related technologies, deep neural networks can be used to process microscopic images containing cells to obtain the cell type. For example, developers can pre-train an image recognition model using a model training device. During application, the computer device inputs the microscopic image containing cells into the image recognition model, and the image recognition model outputs the cell type.

[0004] However, image recognition models in related technologies have a poor ability to extract features related to cell classification from input microscopic images, resulting in low accuracy in cell classification. Summary of the Invention

[0005] This application provides an image processing method, apparatus, device, and storage medium that can improve the accuracy of cell classification in microscopic images. The technical solution is as follows.

[0006] On the one hand, an image processing method is provided, the method comprising:

[0007] Acquire medical microscopic images, wherein the medical microscopic images contain at least one cell nucleus;

[0008] Based on the medical microscopic image, an identification image of the target cell nucleus in the at least one cell nucleus is obtained; the identification image is a multi-channel image composed of a target image block and a mask image; the target image block is an image block in the medical microscopic image that contains the target cell nucleus; the mask image is used to indicate the position of the target cell nucleus in the target image block;

[0009] Cell type identification is performed based on the identified image to obtain the classification probability distribution of the target cell nucleus; the classification probability distribution is used to indicate the probability that the cell corresponding to the target cell nucleus belongs to various cell types;

[0010] Based on the classification probability distribution of the target cell nucleus, the cell type to which the cell corresponding to the target cell nucleus belongs is obtained.

[0011] In another aspect, an image processing apparatus is provided, the apparatus comprising:

[0012] A microscopic image acquisition module is used to acquire medical microscopic images, wherein the medical microscopic images contain at least one cell nucleus;

[0013] The image acquisition module is used to acquire an identification image of a target cell nucleus in at least one cell nucleus based on the medical microscopic image; the identification image is a multi-channel image composed of a target image block and a mask image; the target image block is an image block in the medical microscopic image that contains the target cell nucleus; the mask image is used to indicate the position of the target cell nucleus in the target image block;

[0014] The probability distribution acquisition module is used to perform cell type identification based on the recognition image and obtain the classification probability distribution of the target cell nucleus; the classification probability distribution is used to indicate the probability that the cell corresponding to the target cell nucleus belongs to various cell types;

[0015] The cell type acquisition module is used to acquire the cell type of the cell corresponding to the target cell nucleus based on the classification probability distribution of the target cell nucleus.

[0016] In one possible implementation, the image acquisition module is used to:

[0017] Cell nucleus location identification is performed on the medical microscopic image to obtain the location information of the target cell nucleus in the medical microscopic image;

[0018] Based on the location information of the target cell nucleus in the medical microscopic image, the target image block is extracted from the medical microscopic image;

[0019] The mask image is generated based on the position information of the target cell nucleus in the target image block;

[0020] The target image block and the mask image are combined to generate the recognition image.

[0021] In one possible implementation, the image acquisition module is used to:

[0022] Based on the location information of the target cell nucleus in the medical microscopic image, at least two target image blocks are extracted from the medical microscopic image; the target cell nucleus is located in different target image blocks.

[0023] Based on the positional information of the target cell nucleus in at least two target image blocks, at least two mask images are generated;

[0024] At least two of the target image blocks are combined with the mask images of each of the at least two target image blocks to generate at least two recognition images.

[0025] In one possible implementation, the image acquisition module is used to randomly extract at least two target image blocks from the medical microscopic image based on the position information of the target cell nucleus in the medical microscopic image.

[0026] In one possible implementation, the image acquisition module is used to extract at least two target image blocks from the medical microscopic image based on the position information of the target cell nucleus in the medical microscopic image and at least two positional constraints.

[0027] The location constraint condition is used to limit the distance between the position of the target cell nucleus in the target image block and the edge of the target image block.

[0028] In one possible implementation, the cell type acquisition module is used for,

[0029] The classification probability distributions of at least two of the identified images are fused to obtain a fused classification probability distribution.

[0030] Based on the fusion classification probability distribution, the cell type to which the cell corresponding to the target cell nucleus belongs is obtained.

[0031] In another aspect, a computer device is provided, the computer device comprising a processor and a memory, the memory storing at least one computer instruction, the at least one computer instruction being loaded and executed by the processor to implement the above-described image processing method.

[0032] In another aspect, a computer-readable storage medium is provided, wherein at least one computer instruction is stored therein, the at least one computer instruction being loaded and executed by a processor to implement the above-described image processing method.

[0033] In another aspect, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned image processing method.

[0034] The beneficial effects of the technical solutions provided in this application include at least the following:

[0035] For a target cell nucleus to be classified in a medical microscopic image, a multi-channel recognition image is obtained, consisting of a target image patch containing the target cell nucleus in the medical microscopic image and a mask image indicating the location of the target cell nucleus in the target image patch. The multi-channel recognition image is then used to identify the cell type to which the cell corresponding to the target cell nucleus belongs. In the above process, since the input information for cell type recognition includes not only the image patch in the medical microscopic image but also the mask image indicating the location of the target cell nucleus, the image recognition model can distinguish the target cell nucleus from the background of the target cell nucleus from the input data, thereby more accurately extracting features related to cell classification and improving the accuracy of cell classification in medical microscopic images.

[0036] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0038] Figure 1 This is a system configuration diagram of a cell classification system involved in various embodiments of this application;

[0039] Figure 2 This is a flowchart illustrating an image processing method according to an exemplary embodiment;

[0040] Figure 3 This is a flowchart illustrating a cell recognition scheme according to an exemplary embodiment;

[0041] Figure 4 This is a flowchart illustrating an image processing method according to an exemplary embodiment;

[0042] Figure 5 yes Figure 4 The illustrated embodiment is a schematic diagram of HE staining.

[0043] Figure 6 yes Figure 4 The illustrated embodiment is a schematic diagram of the cell nucleus segmentation results.

[0044] Figure 7 This is the main algorithm flow of the scheme shown in this application;

[0045] Figure 8 This is a flowchart illustrating a cell recognition scheme according to an exemplary embodiment;

[0046] Figure 9This is a schematic diagram illustrating cell classification results according to an exemplary embodiment;

[0047] Figure 10 This is a structural block diagram of an image processing apparatus according to an exemplary embodiment;

[0048] Figure 11 This is a schematic diagram of the structure of a computer device according to an exemplary embodiment. Detailed Implementation

[0049] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0050] Please refer to Figure 1 This illustrates a system configuration diagram of a cell classification system according to various embodiments of this application. Figure 1 As shown, the system may include a medical microscopic image acquisition device 120, a terminal 140, and a server 160; optionally, the system may also include a database 180.

[0051] The medical microscopic image acquisition device 120 can be a medical device used to acquire medical microscopic images, such as a medical microscope. Accordingly, the aforementioned medical microscopic images can be color microscope images, etc.

[0052] The medical microscopic image acquisition device 120 may include an image output interface, such as a Universal Serial Bus (USB) interface, a High Definition Multimedia Interface (HDMI) interface, or an Ethernet interface, etc.; or, the above-mentioned image output interface may also be a wireless interface, such as a Wireless Local Area Network (WLAN) interface, a Bluetooth interface, etc.

[0053] Accordingly, depending on the type of image output interface mentioned above, operators can export medical microscopic images in various ways. For example, they can import medical microscopic images to terminal 140 via wired or short-range wireless means, or they can import medical microscopic images to terminal 140 or server 160 via local area network or Internet.

[0054] Terminal 140 can be a terminal device with certain processing capabilities and interface display functions. For example, terminal 140 can be a mobile phone, tablet computer, e-book reader, smart glasses, laptop computer, desktop computer, etc.

[0055] Terminal 140 may include terminals used by developers and terminals used by medical personnel.

[0056] When terminal 140 is implemented as a terminal for developers, developers can use terminal 140 to develop machine learning models for classifying cells in medical microscopic images and deploy the machine learning models to server 160 or terminals used by medical personnel.

[0057] When terminal 140 is used by medical personnel, an application can be installed in terminal 140 to classify cells in medical microscopic images and present the classification results. After terminal 140 acquires medical images acquired by medical microscopic image acquisition device 120, it uses the aforementioned application to classify and identify the medical microscopic images to obtain classification results and present the classification results so that doctors can perform operations such as pathological diagnosis.

[0058] exist Figure 1 In the system shown, terminal 140 and medical microscopic image acquisition device 120 are physically separate entities. Alternatively, in another possible implementation, when terminal 140 is implemented as a terminal for use by medical personnel, terminal 140 and medical microscopic image acquisition device 120 can also be integrated into a single entity; for example, terminal 140 can be a terminal device with medical microscopic image acquisition function.

[0059] Among them, server 160 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.

[0060] The aforementioned server 160 may be a server that provides background services for the application installed in the terminal 140. This background server may be responsible for application version management, background classification processing of medical microscopic images obtained by the application and returning the processing results, background training of machine learning models developed by developers, etc.

[0061] The aforementioned database 180 can be a Redis database, or it can be any other type of database. Database 180 is used to store various types of data.

[0062] Optionally, terminal 140 and server 160 are connected via a communication network. Optionally, medical microscopic image acquisition device 120 and server 160 are connected via a communication network. Optionally, this communication network can be a wired network or a wireless network.

[0063] Optionally, the system may also include a management device ( Figure 1 (Not shown), the management device is connected to the server 160 via a communication network. Optionally, the communication network can be a wired network or a wireless network.

[0064] Figure 2 This is a flowchart illustrating an image processing method according to an exemplary embodiment. The method can be executed by a computer device, such as a server, a terminal, or a combination of both. The server can be one of the aforementioned... Figure 1 The server 160 in the illustrated embodiment can be the terminal described above. Figure 1 The illustrated embodiment includes a terminal 140 used by medical personnel. This computer device can be implemented as a classification device for sorting cells in medical microscopic images. Figure 2 As shown, the image processing method may include the following steps.

[0065] Step 201: Obtain a medical microscopic image containing at least one cell nucleus.

[0066] In the embodiments of this application, the aforementioned medical microscopic images can be images obtained by acquiring images of stained cell tissue under a microscope.

[0067] For example, after medical staff collect cell tissue samples, they can stain the samples to distinguish between different tissues such as the cell nucleus and cytoplasm.

[0068] Step 202: Based on the medical microscopic image, acquire an identification image of the target cell nucleus in at least one cell nucleus; the identification image is a multi-channel image composed of a target image block and a mask image; the target image block is an image block in the medical microscopic image that contains the target cell nucleus; the mask image is used to indicate the position of the target cell nucleus in the target image block.

[0069] In the embodiments of this application, the target cell nucleus is one of at least one cell nucleus contained in a medical microscopic image.

[0070] The aforementioned recognition image has multiple image channels, which correspond to the target image block and the mask image, respectively.

[0071] For example, if the target image patch is a three-channel image and the mask image is a single-channel image, with each image channel corresponding to a color, then the recognition image can be a four-channel image, where the pixel value in each channel is the color value of the pixel in the target image patch corresponding to the color in that current channel.

[0072] The aforementioned mask image can be a binary image. For example, in this binary image, the pixel value corresponding to the target cell nucleus can be 1, and the pixel values ​​at other locations can be 0. Alternatively, the pixel values ​​in the binary image can be any values ​​other than 0 and 1, as long as the pixel value at the target cell nucleus is different from the pixel values ​​at other locations outside the target cell nucleus.

[0073] Step 203: Based on the recognition image, cell type identification is performed to obtain the classification probability distribution of the target cell nucleus; the classification probability distribution is used to indicate the probability that the cell corresponding to the target cell nucleus belongs to various cell types.

[0074] In this embodiment of the application, a computer device can use a pre-trained image recognition model to identify cell types in a recognition image and output the classification probability distribution of the target cell nucleus. The input to this image recognition model is a multi-channel image (e.g., a four-channel image), and the image recognition model treats the multi-channel image as a whole for feature extraction and classification.

[0075] Step 204: Based on the classification probability distribution of the target cell nucleus, obtain the cell type to which the cell corresponding to the target cell nucleus belongs.

[0076] For each cell nucleus in the above medical microscopic image, the computer device can perform steps 201 to 204 as the target cell nucleus to obtain the cell type to which each cell nucleus belongs in the medical microscopic image.

[0077] Optionally, the computer device can also display the cell type to which each cell nucleus belongs, corresponding to the aforementioned medical microscopic image. For example, in one exemplary embodiment, the computer device can label each cell nucleus with a box of a different color based on the medical microscopic image.

[0078] In summary, the solution presented in this application, for a target cell nucleus to be classified in a medical microscopic image, acquires a multi-channel recognition image consisting of a target image patch containing the target cell nucleus in the medical microscopic image and a mask image indicating the location of the target cell nucleus in the target image patch. The multi-channel recognition image is then used to identify the cell type corresponding to the target cell nucleus. In this process, since the input information for cell type recognition includes not only the image patch in the medical microscopic image but also the mask image indicating the location of the target cell nucleus, the image recognition model can distinguish the target cell nucleus from its background from the input data. This allows for more accurate extraction of features related to cell classification, thereby improving the accuracy of cell classification in medical microscopic images.

[0079] Furthermore, the solution shown in this application embodiment indicates the location of the target cell nucleus in the target image block by adding a mask image outside the target image block. Compared with specifying the location of the target cell nucleus through a method outside the image, the solution in this application embodiment only needs to set a single input port, thereby ensuring the simplicity of the model structure. This improves the accuracy of model classification while ensuring the training and classification efficiency of the model.

[0080] The above application Figure 2 The solutions in the illustrated embodiments can be implemented based on AI. For example, the step of identifying cell types in the image can be performed by an image recognition model trained based on AI technology.

[0081] AI (Artificial Intelligence) is the theory, methods, technology, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, artificial intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. Artificial intelligence technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0082] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0083] The above application Figure 2 The solution illustrated in the example can be applied to a medical cloud. A medical cloud refers to a cloud platform for healthcare services created using cloud computing, combining medical technology with new technologies such as cloud computing, mobile technology, multimedia, wireless communication, big data, and the Internet of Things. This enables the sharing of medical resources and expands the scope of medical services. Because of the application of cloud computing technology, the medical cloud improves the efficiency of medical institutions and facilitates access to medical care for residents. For example, hospital appointment booking and medical insurance are products of the integration of cloud computing and the medical field. The medical cloud also has advantages such as data security, information sharing, dynamic expansion, and a global deployment.

[0084] Please refer to Figure 3 This illustrates a flowchart of a cloud-based cell identification scheme based on a medical application, as shown in an exemplary embodiment of this application. Figure 3 As shown, a cell identification solution based on a medical cloud may include the following steps:

[0085] S1, after medical personnel collect medical microscopic images of cells and tissues using microscope 31, they store the medical microscopic images in the cloud.

[0086] S2, During cell classification, computer equipment extracts medical microscopic images 32 from the cloud.

[0087] S3, the computer device acquires a recognition image 33 of the target cell nucleus (including the target image block in the medical microscopic image 32 and the mask image) from the medical microscopic image 32.

[0088] S4, the computer device will input the image 33 into the image recognition model 34.

[0089] S5, Image recognition model 34 outputs classification probability distribution 35.

[0090] S6, the computer device determines the cell type of the target cell nucleus based on the classification probability distribution 35, and displays the recognition result image 36 based on the identified cell type.

[0091] In the aforementioned recognition result image 36, the target cell nucleus can be circled by lines of color corresponding to the cell type.

[0092] Figure 4 This is a flowchart illustrating an image processing method according to an exemplary embodiment. The method can be executed by a computer device, such as a server, a terminal, or a combination of both. The server can be one of the aforementioned... Figure 1 The server 160 in the illustrated embodiment can be the terminal described above. Figure 1 The illustrated embodiment includes a terminal 140 used by medical personnel. This computer device can be implemented as a classification device for sorting cells in medical microscopic images. Figure 4 As shown, the image processing method may include the following steps.

[0093] Step 401: Obtain a medical microscopic image containing at least one cell nucleus.

[0094] In one possible implementation, computer devices can acquire medical microscopic images from the cloud.

[0095] In another possible implementation, the computer device may also receive medical microscopic images sent by the medical microscopic image acquisition device.

[0096] In another possible implementation, when the computer device is a medical microscopic image acquisition device, it can directly acquire the aforementioned medical microscopic images. For example, when the computer device is a medical device integrated with a microscope, it can directly acquire medical microscopic images within the microscope's field of view through the integrated microscope.

[0097] In one possible implementation, the aforementioned medical microscopic image could be a microscopic image of a stained section of cell tissue.

[0098] For example, the aforementioned stained cell tissue sections may include hematoxylin-eosin (HE) stained sections, Thinprep Cytologic Test (TCT) sections, and Immunohistochemistry (IHC) sections, etc.

[0099] For example, HE staining is one of the most commonly used staining methods in microscopic medical analysis. It consists of two staining agents: hematoxylin stains basophilic structures such as the cell nucleus blue-purple, while eosin stains most of eosinophilic structures such as the cytoplasm pink.

[0100] Please refer to Figure 5 This illustrates a schematic diagram of HE staining according to an embodiment of this application. Figure 5 As shown, dot 51 represents the cell nucleus, which is stained blue-purple. The areas within dashed box 52, excluding the blue-purple nucleus, represent the cytoplasm, which is stained pink.

[0101] After acquiring the medical microscopic image, the computer device can then use the medical microscopic image to obtain a recognition image of at least one target cell nucleus within the cell nucleus. The process of acquiring the recognition image can be referred to subsequent steps 402 to 405.

[0102] Step 402: Perform cell nucleus location identification on the medical microscopic image to obtain the location information of the target cell nucleus in the medical microscopic image.

[0103] In the scheme shown in the embodiments of this application, the process of identifying the cell nucleus position in a medical microscopic image and obtaining the position information of the target cell nucleus in the medical microscopic image can be tried using deep learning methods. For example, a computer device can use a pre-trained cell nucleus detection model to perform feature extraction and cell nucleus segmentation on the input medical microscopic image to obtain the position of each cell nucleus in the image.

[0104] For example, taking the Hover-Net architecture-based cell nucleus detection model as an example, the computer device inputs an R (Red), G (Green), and B (Blue) image (i.e., a medical microscopic image) into the cell nucleus detection model. The output of the cell nucleus detection model is the horizontal and vertical offset of each pixel to its predicted centroid, as well as a binary classification judgment of whether each pixel is a cell nucleus. The computer device achieves cell nucleus segmentation and detection by combining the two.

[0105] For example, please refer to Figure 6 This diagram illustrates the cell nucleus segmentation results involved in an embodiment of this application. Figure 6 As shown, the computer device inputs the medical microscopic image 61 into the cell nucleus detection model 62. Based on the output of the cell nucleus detection model 62, a medical microscopic image 63 labeled with each cell nucleus can be obtained. Optionally, in Figure 6 In the medical microscopic image 63, the wireframe represents the results of cell detection and segmentation. Different cell nuclei can correspond to different colors. For example, each cell nucleus can be randomly assigned a color to distinguish the nuclei of different cells.

[0106] In this embodiment of the application, the input and output of the above-mentioned cell nucleus detection model are:

[0107] The input is an RGB image, and the output is a mask of all detected cell nuclei. .

[0108] The dimensions of the mask are the same as those of the input image, and the value of each pixel in the mask is... This means that this pixel belongs to the first... One cell nucleus, If the value is 0, it means that this pixel belongs to the background.

[0109] The architecture of the aforementioned cell nucleus detection model is not limited to Hover-Net. For example, the architecture of the aforementioned cell nucleus detection model can also be a Mask Region-Convolutional Neural Network (Mask R-CNN) architecture, etc.

[0110] The aforementioned cell nucleus detection model can be trained using pre-labeled medical microscopic image samples with pre-annotated cell nucleus locations. For example, during training, the model training device can input the medical microscopic image samples into the cell nucleus detection model to obtain the predicted cell nucleus location information. Then, the model training device calculates a loss function value based on the difference between the predicted cell nucleus locations and the labeled cell nucleus locations, and adjusts the parameters in the cell nucleus detection model based on this loss function value. The model training device can iteratively execute the above process until the cell nucleus detection model converges.

[0111] Step 403: Based on the location information of the target cell nucleus in the medical microscopic image, extract the target image block from the medical microscopic image.

[0112] In this embodiment of the application, the computer device can extract a target image block containing the target cell nucleus from the medical microscopic image based on the position information of the target cell nucleus in the medical microscopic image and according to a preset image block size.

[0113] When classifying cells, it is necessary to combine the cell's morphological characteristics with its location for judgment. Since the environment surrounding the cell nucleus (the morphology of other cells, the color difference between the foreground and background, the texture of the background, etc.) has a great influence on the classification judgment of the cell nucleus, in this embodiment of the application, the computer device can pre-set an image patch size (e.g., a pixel size of 224×224), and for each cell nucleus that needs to be classified in the medical microscopic image, a 224×224 image patch containing the corresponding cell nucleus is segmented out as part of the model input in the subsequent classification process.

[0114] In this embodiment of the application, the above image block size is only illustrated by 224×224 as an example. The computer device can also set a smaller image block size to cut out a smaller background environment and emphasize the cell itself and its surrounding background; or, the computer device can also set a larger image block size to cut out a larger background environment and further emphasize the location of the cell.

[0115] In one possible implementation, the above-mentioned extraction of a target image patch from a medical microscopic image based on the location information of the target cell nucleus in the medical microscopic image includes:

[0116] Based on the location information of the target cell nucleus in the medical microscopic image, at least two target image blocks are extracted from the medical microscopic image; the target cell nucleus is located in different target image blocks.

[0117] In the embodiments of this application, when the computer device segments the target image block, it does not limit the specific location of the cell in the background environment. That is, the computer device can acquire multiple different target image blocks containing the target cell nucleus as a data augmentation method.

[0118] In one possible implementation, based on the location information of the target cell nucleus in the medical microscopic image, at least two target image patches are extracted from the medical microscopic image, including:

[0119] Based on the location information of the target cell nucleus in the medical microscopic image, at least two target image blocks are randomly extracted from the medical microscopic image.

[0120] In this embodiment of the application, the computer device can extract at least two target image blocks from a medical microscopic image by random cropping.

[0121] In one possible implementation, based on the location information of the target cell nucleus in the medical microscopic image, at least two target image patches are extracted from the medical microscopic image, including:

[0122] Based on the location information of the target cell nucleus in the medical microscopic image, and at least two location constraints, at least two target image patches are extracted from the medical microscopic image;

[0123] The positional constraint is used to limit the distance between the position of the target cell nucleus in the target image block and the edge of the target image block.

[0124] Optionally, the aforementioned positional constraints may include at least one of the following: conditions that define the relationship between the position of the target cell nucleus in the target image block and the distance between the upper and lower edges of the target image block; and conditions that define the relationship between the position of the target cell nucleus in the target image block and the distance between the left and right edges of the target image block.

[0125] For example, the aforementioned positional constraint can restrict the distance between the target cell nucleus and the upper edge of the target image patch to be greater than or less than the distance between the target cell nucleus and the lower edge of the target image patch. Alternatively, the aforementioned positional constraint can restrict the distance between the target cell nucleus and the left edge of the target image patch to be greater than or less than the distance between the target cell nucleus and the right edge of the target image patch.

[0126] Optionally, the above-mentioned positional restriction conditions may include at least one of the following: a condition that limits the ratio of the distance between the position of the target cell nucleus in the target image block and the distance between the upper and lower edges of the target image block; and a condition that limits the ratio of the distance between the position of the target cell nucleus in the target image block and the distance between the left and right edges of the target image block.

[0127] For example, the aforementioned positional constraint can limit the ratio between the distance between the target cell nucleus's position in the target image patch and the upper edge of the target image patch, and the distance between the target cell nucleus's position in the target image patch and the lower edge of the target image patch, to be greater than or less than a first proportional threshold. Alternatively, the aforementioned positional constraint can limit the ratio between the distance between the target cell nucleus's position in the target image patch and the left edge of the target image patch, and the distance between the target cell nucleus's position in the target image patch and the right edge of the target image patch, to be greater than or less than a second proportional threshold.

[0128] Step 404: Generate a mask image based on the position information of the target cell nucleus in the target image block.

[0129] For the aforementioned target image block, the computer device can determine the position information of the target cell nucleus in the target image block based on the position information of the target cell nucleus in the medical microscopic image and the position information of the aforementioned target image block in the medical microscopic image. Then, the computer device can generate a mask image based on the position information of the target cell nucleus in the target image block. For example, in the mask image, the pixel value at the position corresponding to the position information of the target cell nucleus in the target image block is different from the pixel value at other positions in the mask image.

[0130] In one possible implementation, a mask image is generated based on the positional information of the target cell nucleus within the target image patch, including:

[0131] At least two mask images are generated based on the position information of the target cell nucleus in at least two target image blocks.

[0132] In this embodiment of the application, when the computer device acquires at least two target image blocks containing target cell nuclei, the computer device can generate a corresponding mask image for each target image block.

[0133] Step 405: Combine the target image patch and the mask image to generate a recognition image.

[0134] The computer device can combine the target image block (such as a three-channel RGB image) with a single-channel mask image to generate a four-channel image, which is used as the recognition image mentioned above.

[0135] In one possible implementation, a computer device can combine at least two target image blocks with their respective mask images to generate at least two recognition images.

[0136] Specifically, when the computer device acquires at least two target image blocks containing the target cell nucleus, the computer device combines each target image block with its corresponding mask image to generate at least two recognition images. In other words, in each recognition image, the location of the target cell nucleus indicated by the mask image is the location of the target cell nucleus within the target image block of the current recognition image.

[0137] Step 406: Based on the identified image, cell type identification is performed to obtain the classification probability distribution of the target cell nucleus.

[0138] Among them, the classification probability distribution is used to indicate the probability that the cell corresponding to the target cell nucleus belongs to various cell types.

[0139] In this embodiment of the application, the computer device can input the recognition image into the image recognition model, which will extract features from the recognition image as a whole and map the extracted features into a classification space containing various cell types to obtain the probability that the cell corresponding to the target cell nucleus belongs to various cell types.

[0140] In this embodiment, for a target image patch containing a target cell nucleus, a mask of the cell nucleus itself is overlaid to emphasize the target cell nucleus to be classified. This RGB three-channel + cell nucleus mask scheme can effectively utilize the information surrounding the cell nucleus and the features of the cell nucleus itself, thereby significantly improving the accuracy of cell nucleus classification. Specifically, the RGB three channels provide the image recognition model with information about the cell's surrounding environment and the morphology of nearby cells, helping the image recognition network better understand the cell's location and state. The cell nucleus mask channel indicates to the image recognition model which cell in the background image the cell to be classified belongs to, while also emphasizing the cell's shape characteristics. The aforementioned recognition image combines both the cell environment and the cell's own features.

[0141] The image recognition model described above can be trained using pre-labeled image samples of cell types. For example, during training, the model training device can input the image samples into the image recognition model to obtain the probability distribution of cell types predicted by the model. Then, the model training device calculates a loss function value based on the difference between the probability distribution of cell types predicted by the image recognition model and the labeled cell types, and adjusts the parameters in the image recognition model based on this loss function value. The model training device can iteratively execute the above process until the image recognition model converges.

[0142] Step 407: Based on the classification probability distribution of the target cell nucleus, obtain the cell type to which the cell corresponding to the target cell nucleus belongs.

[0143] Optionally, the computer device can determine the cell type of the cell corresponding to the target cell nucleus based on the probability that the cell corresponding to the target cell nucleus belongs to various cell types.

[0144] For example, suppose that the probability of the cell corresponding to the target cell nucleus belonging to various cell types includes the probabilities corresponding to each of the five cell types, and the probability corresponding to cell type A is the highest and greater than a pre-set probability threshold (such as 90%), then the computer device can determine that the cell corresponding to the target cell nucleus belongs to cell type A.

[0145] In one possible implementation, based on the classification probability distribution of the target cell nucleus, the cell type to which the cell corresponding to the target cell nucleus belongs is obtained, including:

[0146] The classification probability distributions of at least two recognition images are fused to obtain a fused classification probability distribution; based on the fused classification probability distribution, the cell type to which the cell corresponding to the target cell nucleus belongs is obtained.

[0147] In this embodiment of the application, when the target cell nucleus corresponds to two or more recognition images, the image recognition model will output a set of classification probability distributions for each recognition image. At this time, the computer device can fuse the classification probability distributions of each recognition image in the two or more recognition images, for example, by taking the average value, to obtain the fused classification probability distribution, and take the cell type with the highest probability that is greater than the preset probability threshold as the cell type to which the cell corresponding to the target cell nucleus belongs.

[0148] In another exemplary scheme, when the target cell nucleus corresponds to two or more recognition images, which are recognition images captured by the computer device based on different positional constraints, the computer device can also perform a weighted average of the classification probability distribution of each recognition image in the two or more recognition images to obtain a fused classification probability distribution.

[0149] For example, the computer device can be pre-set with weights corresponding to each location constraint. The computer device can multiply the classification probability distribution of each recognition image in two or more recognition images by the weights of the corresponding location constraints, and then take the average value to obtain the above-mentioned fused classification probability distribution.

[0150] In summary, the solution presented in this application, for a target cell nucleus to be classified in a medical microscopic image, acquires a multi-channel recognition image consisting of a target image patch containing the target cell nucleus in the medical microscopic image and a mask image indicating the location of the target cell nucleus in the target image patch. The multi-channel recognition image is then used to identify the cell type corresponding to the target cell nucleus. In this process, since the input information for cell type recognition includes not only the image patch in the medical microscopic image but also the mask image indicating the location of the target cell nucleus, the image recognition model can distinguish the target cell nucleus from its background from the input data. This allows for more accurate extraction of features related to cell classification, thereby improving the accuracy of cell classification in medical microscopic images.

[0151] Furthermore, the solution shown in this application embodiment indicates the location of the target cell nucleus in the target image block by adding a mask image outside the target image block. Compared with specifying the location of the target cell nucleus through a method outside the image, the solution in this application embodiment only needs to set a single input port, thereby ensuring the simplicity of the model structure. This improves the accuracy of model classification while ensuring the training and classification efficiency of the model.

[0152] The solution described in the above embodiments of this application proposes a novel input format to address the problem in cell classification using HE staining images where the input image cannot simultaneously reflect both the environment and the cell itself. By utilizing the three color channels of the cell nucleus mask and the cell nucleus background to form a four-channel image, the model's training and classification are fully utilized based on cell nucleus information. This allows for accurate classification of cells into seven categories: inflammatory cells, healthy epithelial cells, malignant epithelial cells, fibroblasts, myocytes, endothelial cells, and other cells.

[0153] Please refer to Figure 7 This illustrates the main algorithm flow of the scheme presented in this application. Figure 7As shown, the algorithm flow of the scheme presented in this application can be divided into three steps: image input 71, cell detection and segmentation 72, and cell classification 73. In the third step, cell classification, the input information consists of an image patch from the original medical microscopic image and a mask image of the target cell nucleus to be classified within the image patch.

[0154] The solutions described in the above embodiments of this application can be used in conjunction with a pathology cloud platform system. For example, the pathology cloud platform system can extract and render pathology images (i.e., medical microscopic images) from the database in real time, and then, through the solutions described in the embodiments of this application, detect, segment, and classify the images, and output the classification results to assist pathologists in viewing, learning, and diagnosis.

[0155] For example, please refer to Figure 8 This illustrates a flowchart of a cloud-based cell identification scheme based on a medical application, as shown in an exemplary embodiment of this application. Figure 8 As shown, a cell identification solution based on a medical cloud may include the following steps:

[0156] S1. After medical personnel collect medical microscopic images of cells and tissues using microscope 81, they store the medical microscopic images in the cloud.

[0157] S2, During cell classification, computer equipment extracts medical microscopic images 82 from the cloud.

[0158] S3, the computer device acquires at least two identification images 83 of the target cell nucleus (containing different target image blocks in the medical microscopic image 82 and corresponding mask images) from the medical microscopic image 82.

[0159] S4, the computer device will input the image 83 into the image recognition model 84.

[0160] S5, Image recognition model 84 outputs at least two classification probability distributions 85.

[0161] S6, the computer device performs probability fusion based on at least two classification probability distributions 85 and determines the cell type to which the target cell nucleus belongs, and displays the recognition result image 86 based on the identified cell type.

[0162] In the above-mentioned recognition result image 86, the target cell nucleus can be circled by lines of color corresponding to the cell type.

[0163] The above application Figure 8The proposed solution revolves around a pathology cloud platform system, classifying each cell nucleus in the extracted pathological images. This solution employs a unique input structure that allows the image recognition model to combine the cell's environment with the cell's own characteristics for refined cell classification. For example, based on commonly used cell nucleus classification standards, cell nuclei are categorized into the seven classes mentioned above.

[0164] In the scheme shown in the embodiments of this application, the position of the target cell nucleus in the mask image is not limited. Therefore, the mask image containing the target cell nucleus and its corresponding target image block can be arbitrarily cut, as long as the target cell nucleus is on this mask image, thereby achieving data augmentation.

[0165] The solutions shown in the various embodiments of this application, through the design of novel input patterns, can significantly improve the accuracy of cell type recognition algorithms. The base network of the aforementioned image recognition model can use any classification network, such as ViT or ResNet, and the loss function can be any type of classification loss function, such as the cross-entropy loss function.

[0166] In the embodiments of this application, the image recognition model can fully combine and utilize the characteristics of the cell nucleus's environment and the cell nucleus itself to improve the input of the classification network and enhance the classification accuracy.

[0167] The solution shown in this application embodiment can also be applied to the application scenarios of customized microscopes. By using an intelligent microscope to automatically detect, segment, and classify the cell type corresponding to each cell nucleus in HE staining images, it lays the foundation for the next step of medical analysis.

[0168] For example, please refer to Figure 9 This illustrates a schematic diagram of cell classification results shown in an exemplary embodiment of this application. Figure 9 As shown, taking the input medical microscopic image 91 as an example, which is an HE-stained section image under a microscope at 40x magnification, the detection, segmentation, and classification system shown in the above embodiments of this application can automatically output high-quality classification results 92. For example... Figure 9 As shown, different colors can be used to label different types of cells in the output result 92. For example, red represents tumor cells 92a, green represents inflammatory cells 92b, blue represents connective tissue cells 92c, tan represents dead cells 92d, dark green represents epithelial cells 92e, and so on. Accordingly, medical personnel can use the automatically output cell nuclear classification results to assess the patient's condition and guide subsequent medication.

[0169] Please refer to Table 1, which shows a comparison of the effects of the schemes shown in various embodiments of this application (i.e., the schemes that use target image patches + mask images as input) and the schemes that use only image patches as input when tested on a certain dataset.

[0170] Table 1

[0171]

[0172] As can be seen from Table 1 above, the accuracy has been greatly improved after adopting the solutions shown in the various embodiments of this application.

[0173] Optionally, the solution shown in the above embodiments of this application can be integrated into the smart microscope and scanner to directly segment and classify cells in the pathological images captured by the microscope.

[0174] In the embodiments of this application, the above-mentioned cell classification types are only illustrated by the above 7 types. Optionally, there may be more or fewer types of cell classification.

[0175] In the embodiments of this application, no requirements are placed on the magnification of the microscope for medical microscopic images. For example, it can be set to 10, 20, 40 magnification, etc.

[0176] The solutions shown in the above embodiments of this application can be implemented or executed in conjunction with blockchain. For example, some or all of the steps in the above embodiments can be executed in a blockchain system; or, the data required or generated for the execution of each step in the above embodiments can be stored in a blockchain system; for example, the training samples used for the above model training, as well as the model input data such as medical images during the model application process, can be obtained by computer devices from the blockchain system; and for another example, the parameters of the model obtained after the above model training can be stored in a blockchain system.

[0177] Figure 10 This is a structural block diagram of an image processing apparatus according to an exemplary embodiment. The apparatus can implement... Figure 2 or Figure 4 The image processing apparatus includes all or part of the steps in the method provided in the illustrated embodiment:

[0178] The microscopic image acquisition module 1001 is used to acquire medical microscopic images, wherein the medical microscopic images contain at least one cell nucleus;

[0179] The image acquisition module 1002 is used to acquire an identification image of a target cell nucleus in the at least one cell nucleus based on the medical microscopic image; the identification image is a multi-channel image composed of a target image block and a mask image; the target image block is an image block in the medical microscopic image that contains the target cell nucleus; the mask image is used to indicate the position of the target cell nucleus in the target image block;

[0180] The probability distribution acquisition module 1003 is used to perform cell type identification based on the recognition image and obtain the classification probability distribution of the target cell nucleus; the classification probability distribution is used to indicate the probability that the cell corresponding to the target cell nucleus belongs to various cell types;

[0181] The cell type acquisition module 1004 is used to acquire the cell type of the cell corresponding to the target cell nucleus based on the classification probability distribution of the target cell nucleus.

[0182] In one possible implementation, the image acquisition module is used to:

[0183] Cell nucleus location identification is performed on the medical microscopic image to obtain the location information of the target cell nucleus in the medical microscopic image;

[0184] Based on the location information of the target cell nucleus in the medical microscopic image, the target image block is extracted from the medical microscopic image;

[0185] The mask image is generated based on the position information of the target cell nucleus in the target image block;

[0186] The target image block and the mask image are combined to generate the recognition image.

[0187] In one possible implementation, the image acquisition module is used to:

[0188] Based on the location information of the target cell nucleus in the medical microscopic image, at least two target image blocks are extracted from the medical microscopic image; the target cell nucleus is located in different target image blocks.

[0189] Based on the positional information of the target cell nucleus in at least two target image blocks, at least two mask images are generated;

[0190] At least two of the target image blocks are combined with the mask images of each of the at least two target image blocks to generate at least two recognition images.

[0191] In one possible implementation, the image acquisition module is used to randomly extract at least two target image blocks from the medical microscopic image based on the position information of the target cell nucleus in the medical microscopic image.

[0192] In one possible implementation, the image acquisition module is used to extract at least two target image blocks from the medical microscopic image based on the position information of the target cell nucleus in the medical microscopic image and at least two positional constraints.

[0193] The location constraint condition is used to limit the distance between the position of the target cell nucleus in the target image block and the edge of the target image block.

[0194] In one possible implementation, the cell type acquisition module is used for,

[0195] The classification probability distributions of at least two of the identified images are fused to obtain a fused classification probability distribution.

[0196] Based on the fusion classification probability distribution, the cell type to which the cell corresponding to the target cell nucleus belongs is obtained.

[0197] In summary, the solution presented in this application, for a target cell nucleus to be classified in a medical microscopic image, acquires a multi-channel recognition image consisting of a target image patch containing the target cell nucleus in the medical microscopic image and a mask image indicating the location of the target cell nucleus in the target image patch. The multi-channel recognition image is then used to identify the cell type corresponding to the target cell nucleus. In this process, since the input information for cell type recognition includes not only the image patch in the medical microscopic image but also the mask image indicating the location of the target cell nucleus, the image recognition model can distinguish the target cell nucleus from its background from the input data. This allows for more accurate extraction of features related to cell classification, thereby improving the accuracy of cell classification in medical microscopic images.

[0198] Furthermore, the solution shown in this application embodiment indicates the location of the target cell nucleus in the target image block by adding a mask image outside the target image block. Compared with specifying the location of the target cell nucleus through a method outside the image, the solution in this application embodiment only needs to set a single input port, thereby ensuring the simplicity of the model structure. This improves the accuracy of model classification while ensuring the training and classification efficiency of the model.

[0199] Figure 11This is a schematic diagram of a computer device according to an exemplary embodiment. The computer device can be implemented as a computer device used for training a first image recognition model in the various method embodiments described above, or it can be implemented as a computer device used for midline recognition of the brain using a second image recognition model in the various method embodiments described above. The computer device 1100 includes a central processing unit (CPU) 1101, a system memory 1104 including random access memory (RAM) 1102 and read-only memory (ROM) 1103, and a system bus 1105 connecting the system memory 1104 and the central processing unit 1101. The computer device 1100 also includes a basic input / output system 1106 to facilitate information transfer between various devices within the computer, and a mass storage device 1107 for storing an operating system 1113, application programs 1114, and other program modules 1115.

[0200] The mass storage device 1107 is connected to the central processing unit 1101 via a mass storage controller (not shown) connected to the system bus 1105. The mass storage device 1107 and its associated computer-readable media provide non-volatile storage for the computer device 1100. That is, the mass storage device 1107 may include computer-readable media (not shown), such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0201] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, flash memory or other solid-state storage technologies, CD-ROM, or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 1104 and mass storage device 1107 described above can be collectively referred to as memory.

[0202] Computer device 1100 can be connected to the Internet or other network devices via network interface unit 1111 connected to the system bus 1105.

[0203] The memory also includes one or more programs, which are stored in the memory, and the central processing unit 1101 implements these programs by executing them. Figure 2 or Figure 4 All or part of the steps of any of the methods shown.

[0204] In exemplary embodiments, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory including a computer program (instructions) that can be executed by a processor of a computer device to perform the methods shown in the various embodiments of this application. For example, the non-transitory computer-readable storage medium may be a read-only memory, a random access memory, a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0205] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods shown in the various embodiments described above.

[0206] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0207] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, The method includes: Acquire medical microscopic images, wherein the medical microscopic images contain at least one cell nucleus; Based on the position information of the target cell nucleus in the at least one cell nucleus in the medical microscopic image, at least two target image blocks are extracted from the medical microscopic image; the position of the target cell nucleus is different in different target image blocks; based on the position information of the target cell nucleus in the at least two target image blocks, at least two mask images are generated; the at least two target image blocks are combined with the respective mask images of the at least two target image blocks to generate at least two recognition images; the recognition images are multi-channel images composed of target image blocks and mask images; Cell type identification is performed based on the identified image to obtain the classification probability distribution of the target cell nucleus; the classification probability distribution is used to indicate the probability that the cell corresponding to the target cell nucleus belongs to various cell types; Based on the classification probability distribution of the target cell nucleus, the cell type to which the cell corresponding to the target cell nucleus belongs is obtained.

2. The method according to claim 1, characterized in that, The step of extracting at least two target image patches from the medical microscopic image based on the position information of the target cell nucleus in the medical microscopic image includes: Based on the location information of the target cell nucleus in the medical microscopic image, at least two target image blocks are randomly extracted from the medical microscopic image.

3. The method according to claim 1, characterized in that, The step of extracting at least two target image patches from the medical microscopic image based on the position information of the target cell nucleus in the medical microscopic image includes: Based on the location information of the target cell nucleus in the medical microscopic image, and at least two location constraints, at least two target image blocks are extracted from the medical microscopic image; The location constraint condition is used to limit the distance between the position of the target cell nucleus in the target image block and the edge of the target image block.

4. The method according to claim 1, characterized in that, The step of obtaining the cell type corresponding to the target cell nucleus based on the classification probability distribution of the target cell nucleus includes: The classification probability distributions of at least two of the identified images are fused to obtain a fused classification probability distribution. Based on the fusion classification probability distribution, the cell type to which the cell corresponding to the target cell nucleus belongs is obtained.

5. An image processing apparatus, characterized in that, The device includes: A microscopic image acquisition module is used to acquire medical microscopic images, wherein the medical microscopic images contain at least one cell nucleus; The image acquisition module is configured to extract at least two target image blocks from the medical microscopic image based on the position information of the target cell nucleus in the at least one cell nucleus within the medical microscopic image; the target cell nucleus is located differently in different target image blocks; generate at least two mask images based on the position information of the target cell nucleus in the at least two target image blocks; combine the at least two target image blocks with their respective mask images to generate at least two recognition images; the recognition images are multi-channel images composed of the target image blocks and the mask images. The probability distribution acquisition module is used to perform cell type identification based on the recognition image and obtain the classification probability distribution of the target cell nucleus; the classification probability distribution is used to indicate the probability that the cell corresponding to the target cell nucleus belongs to various cell types; The cell type acquisition module is used to acquire the cell type of the cell corresponding to the target cell nucleus based on the classification probability distribution of the target cell nucleus.

6. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer instruction, which is loaded and executed by the processor to implement the image processing method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The storage medium stores at least one computer instruction, which is loaded and executed by a processor to implement the image processing method as described in any one of claims 1 to 4.

8. A computer program product, characterized in that, The computer program product includes computer instructions that are read and executed by a processor of a computer device, causing the computer device to perform the image processing method as described in any one of claims 1 to 4.

Images