A hierarchical attention mechanism-based multiple-instance classification method for childhood leukemia

Abstract

A kind of children's leukemia multi-instance classification method based on hierarchical attention mechanism, first, carry out cell segmentation task: the nucleated cell in the multiple bone marrow cell images of children's leukemia patient is extracted by cell segmentation module, and the random sampling of N cells is formed into a multi-instance package as sample input learning;Second, carry out children's leukemia classification task: the above input sample after learning is used as children's leukemia classification module, distinguish each sample belongs to acute myeloid leukemia, or acute lymphoid leukemia, and further carry out multiple hierarchical typing;The method of the present application uses multi-instance learning, encodes cells into features, and calculates the contribution of different cells to each category through the attention module, and forms hierarchical features by typing, from the attention matrix of subtyping to the coarse typing, for the loss function of cancer cell proportion, by limiting the proportion, semi-supervisedly classifies whether each cell belongs to cancer cell, so that the attention mechanism can better calculate the contribution of cells to each category, so that the classification is more accurate.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing and classification of childhood leukemia, and particularly relates to a childhood leukemia multi-instance classification method based on a hierarchical attention mechanism. BACKGROUND

[0002] In the past decade, deep learning and convolutional neural networks have shown great advantages in the field of computer vision and have been widely used in medical image analysis. Using deep learning and computer vision technology can help doctors quickly identify diseased cells, thereby greatly improving the efficiency of childhood leukemia diagnosis. However, the limited number of childhood leukemia cases makes the existing data scarce, and the prediction model inevitably has the possibility of overfitting. At the same time, childhood leukemia is a highly heterogeneous disease that requires accurate typing. Current prediction models either classify only acute lymphoid and acute myeloid leukemia or classify subtypes within a certain system, which cannot truly meet the needs of clinical diagnosis.

[0003] In machine learning, multi-instance learning is a method evolved from supervised learning algorithms, which defines a "bag" as a set of multiple instances. Instead of receiving a set of individually labeled instances, the model receives a set of labeled bags, each of which contains multiple instances. Based on the prediction results of the instances, the overall bag classification probability is obtained. Such a learning method is more in line with the process of doctors analyzing and diagnosing based on a patient's image data. Currently, multi-instance learning methods based on attention mechanisms are widely used in the field of medical images to distinguish sub-regions with high diagnostic value, thereby achieving higher classification results. For example, in childhood leukemia bone marrow cell images, certain cells play a crucial role in classification, so the model pays more attention to these cells in order to achieve better classification results. However, existing methods based on attention mechanisms are only concerned with whether a cell is important for classification, but for childhood leukemia, different cells have different importance for different types. The existing methods do not distinguish the classification contribution of cells to different types. In addition, the classification of childhood leukemia has hierarchical characteristics, which should first distinguish between acute lymphoid leukemia and acute myeloid leukemia, and then further classify more detailed types in the two major systems. This way, confusion and other negative effects will not occur in the two major systems. Existing methods based on attention mechanisms are only applied to coarse classification or fine classification of a certain major category. SUMMARY

[0004] In order to overcome the defects existing in the prior art, the present application proposes a child leukemia multi-instance classification method based on a hierarchical attention mechanism, which constructs a hierarchical classification method and introduces a class-specific attention mechanism, so that the model can not only distinguish between acute lymphoid and acute myeloid leukemia, but also can subdivide the child leukemia from the subtypes of the two major systems, thereby meeting the characteristics of multiple and fine subtypes of child leukemia; the contribution of each cell to different subtypes is distinguished, and the attention mechanism between different levels of subtypes is restricted to a certain extent, the attention matrix of the fine subtype can be aggregated into the attention matrix for the coarse subtype, so that the fine subtype and the coarse subtype can be improved, and the cells are prevented from being misclassified between the coarse subtype categories, thereby improving the accuracy of the classification.

[0005] In order to achieve the above-mentioned purpose, the technical scheme adopted by the present application is:

[0006] A child leukemia multi-instance classification method based on a hierarchical attention mechanism, specifically comprising the following steps:

[0007] Step one, cell segmentation:

[0008] 1) According to the analysis and diagnosis results of child leukemia given by doctors, the nucleated cells in the multiple bone marrow cell images of a child leukemia patient are extracted through a cell segmentation module, and then N cells are randomly sampled to obtain a multi-instance package, wherein N [50, 200];

[0009] 2) For bone marrow cell images of patients with different types of child leukemia, form multi-instance package samples according to step one 1) in turn;

[0010] Step two, child leukemia classification:

[0011] The multi-instance package obtained in step one is input into the child leukemia classification module, which distinguishes whether each sample belongs to acute myeloid leukemia or acute lymphoid leukemia, and further distinguishes different subtypes in the two major systems, and further distinguishes the subtypes or small classes in the subtypes that are not easy to distinguish, i.e. multiple hierarchical classification;

[0012] The child leukemia classification module includes an encoder, a class-based attention module and a classifier, the encoder accepts the output of the cell segmentation module to obtain a cell feature matrix F cell ∈R N×K , the cell feature matrix F cell ∈R N×K is input into the class-based attention module to obtain an attention matrix, and then the cell feature matrix F cell ∈R N×KMultiplying the attention matrix of different levels obtains a feature matrix for different levels of classifier to distinguish children's leukemia subtypes.

[0013] The step one first 1) step forms a multi-instance package, and the specific method is:

[0014] 1.1) a bone marrow cell image of a child leukemia patient is multi-center cropped with a step size of b pixels and a window size of w pixels, n1 image blocks covering the whole image are obtained, and the image block set is represented by X={x1, x2,..., x n1}, wherein a∈[15, 50], b>0, w>0, n1>0 and are all integers;

[0015] 1.2) each image block obtained in step one 1.1) is randomly subjected to image enhancement in shape, angle and color, and is center cropped to obtain a square image block with a side length of w1 pixels, and the image block set after image enhancement is represented by , wherein w1∈[128, 256];

[0016] 1.3) the image block set obtained in step one 1.2) is input into a cell segmentation module to obtain an output image block, the position of the center coordinate of the output image block in the original image is located and spliced to obtain a cell segmentation result of the original image size, all cells in the segmentation result are extracted, and N cells are randomly sampled to obtain a single cell image block set C={c1, c2,..., c N}, wherein N∈[50, 200].

[0017] The specific method of the step two is:

[0018] 1) the single cell image block set C finally obtained in step one 1.3) is subjected to boundary filling, and all cell image blocks are filled into a square with a side length of w2 pixels, so as to facilitate packaging all cell image blocks into an encoder, and the single cell image block set after filling is , wherein w2∈[128, 256];

[0019] 2) according to the clinical diagnosis report of the patient, the leukemia subtype y t of the patient is obtained, that is, the patient is y t type; suppose that the patient has l1 subtypes, and the patient label is converted into a one-hot vector , wherein y t =1 and other components are 0; the cell proportion S=(s1, s2) of the patient is obtained, wherein s2 is the proportion of cancer cells, s1=1-s2, and s1 represents the proportion of normal cells;

[0020] 3) the single cell set obtained in step two 1) is input into the encoder Input encoder to obtain cell feature matrix F cell ∈R N×K Where K represents the feature dimension of a single cell, and F cell The input is a category-based attention module, which is a neural network composed of fully connected layers that learns the classification contribution of each cell to each category. Within the attention module, the cell feature matrices are first passed through fully connected layers... and Reduce the feature dimension to Then normalize using the tanh and sigmoid functions separately, multiply the normalized results, and then pass them through a... Fully connected layer from L1 dimension Obtain the attention matrix, which is the contribution of each cell to l1 categories.

[0021] A1=W×(tanh(F cell ×U)⊙sigmoid(F cell ×V))

[0022] Where ⊙ denotes the multiplication of corresponding points within the matrix;

[0023] Finally, the result is obtained by multiplying the cell feature matrix and the attention matrix. This represents the feature matrix of a classifier for l1 categories;

[0024] Let l2 be the number of categories in the second-level coarse subdivision relative to the subdivision l1, then through the transformation matrix... The values ​​of the corresponding categories in the attention matrix for l1 categories are aggregated into l2 categories, i.e. A2 = A1 × T 1 The contribution of each cell to each category in the second-level coarse classification is obtained. The feature matrix of the classifier for l2 categories is obtained by multiplying the attention matrix and the cell feature matrix. Similarly, when reaching the third level of coarse classification, let l3 be the number of categories in the third level of coarse classification. Then, the attention matrix of the obtained second level of coarse classification can be... By transforming the matrix Attention matrix aggregated to the third level coarse fractal A3 = A2 × T 2 By multiplying the attention matrix and the cell feature matrix, the feature matrix of the classifier for l3 categories is obtained. A n =A n-1 ×T n-1For the attention matrix expression of n-level classification, the general expression of the feature matrix of the classifier of the nth level is F l ∈R K×l , which represents that the classification category of the level is l, wherein n, l1, l2, l3, l, K>0, all are integers, and l1>l2>l3, that is, l n-1 >l n ;

[0025] 4) The feature matrix F l ∈R K×l is input into l binary classifiers, and the output O patient ∈R l×2 is obtained. cell ∈R N×K is input into the binary classifier about the cell category, and the output result O cell ∈R N×2 whether it is a cancer cell is obtained.

[0026] 5) According to O cell , the proportion of cells is calculated as wherein the gradient in the proportion is reserved to adjust the network parameters, and the gumbel_softmax function needs to be used to form the probability expression of whether it is a cancer cell in the first dimension of the N*2 matrix, that is, there is , which represents the probability of being a normal cell, which represents the probability of being a cancer cell; the sum in the 0th dimension is divided by the total number N, that is, the proportion is obtained wherein and O patient ∈R l×2 , for each class j, the probability p j ∈[0, 1] that the sample belongs to the class is obtained, and the loss function of the classification learning process is represented as:

[0027]

[0028]

[0029] Compared with the prior art, the present application has the following advantages:

[0030] 1. The present application proposes a multi-instance learning framework that can be applied to the classification of childhood leukemia, including a cell segmentation module and a childhood leukemia classification module, so that the framework can be trained under weak supervision, that is, only clinical reports are needed without secondary labeling by doctors, and the accuracy can be achieved.

[0031] 2. For multiple bone marrow cell smears and clinical reports containing the proportions of various cell types from children with leukemia, an attention-based leukemia classification module was designed. This module includes classifying the patient into different subtypes and classifying each cell into either cancer cells or normal cells.

[0032] 3. Based on the classification characteristics of childhood leukemia, namely that the distinction between different categories depends on different types of cells, a category-based attention mechanism is proposed, which learns different attention matrices for different categories.

[0033] 4. For the diagnostic logic of childhood leukemia, namely the tree-like diagnostic chain at different levels, a fine-to-coarse attention matrix transformation method is proposed. This method can integrate the diagnostic logic at different levels into the leukemia classification model, thereby improving the accuracy and clinical usability of the classification, ensuring that there is no confusion between coarse classifications, and improving the accuracy of subtyping. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the cell segmentation module in step one.

[0035] Figure 2 This is a schematic diagram of the childhood leukemia classification module in step two. Detailed Implementation

[0036] The present invention will now be described in further detail with reference to the accompanying drawings.

[0037] Step 1: Perform cell segmentation task: See Figure 1

[0038] 1) Based on the doctor's analysis and diagnosis of childhood leukemia, nucleated cells are extracted from multiple bone marrow cell images of a child with leukemia using a cell segmentation module. Then, N cells are randomly sampled to form a multi-instance package, as follows:

[0039] 1.1) Take a bone marrow cell images of a child leukemia patient, perform multicenter cropping with a stride of b pixels and a window size of w pixels to obtain n1 patches covering the entire image, and use X = {x1, x2, ..., x...} n1} represents a set of tiles, where a∈[15, 50], b>0, w>0, n1>0 and all are integers;

[0040] 1.2) Randomly apply image enhancement methods with changes in shape, angle, and color to each tile obtained in step 1.1), and perform center cropping to obtain a square tile with a side length of w1 pixels. The set of image-enhanced tiles is represented as follows: Where w1∈[128, 256];

[0041] 1.3) Input the set of images obtained in step 1.2) into the cell segmentation module. The output images are located and stitched together using their center coordinates in the original image to obtain the cell segmentation result of the original image size. Then, all cells in the segmentation result are extracted. Let N cells be obtained, and the set of single-cell images is represented as C = {c1, c2, ..., c...} N}, where N∈[50, 200];

[0042] 2) For bone marrow cell images of different types of childhood leukemia patients, multiple instance packages are formed sequentially according to step 1).

[0043] See Figure 2 Step 2: Perform a childhood leukemia classification task.

[0044] The multi-instance package obtained in step one is input into the childhood leukemia classification module. The childhood leukemia classification module is a convolutional neural network consisting of an encoder, an attention module, and a classifier. The attention module is a neural network consisting of fully connected layers. It distinguishes whether a sample belongs to acute myeloid leukemia or acute lymphoblastic leukemia, and further distinguishes different subtypes within the two lineages. After classifying into each different subtype, it further distinguishes the difficult-to-distinguish subtypes or the minor classes within the subtypes, that is, it performs multi-level classification.

[0045] 1) Padded the single-cell patch set C obtained in step 1.3) by filling all cell patches with squares of side length w2 pixels. This facilitates the packaging of all cell patches for input into the encoder. The padded single-cell patch set is as follows: Where w2∈[128, 256];

[0046] 2) Obtain the leukemia subtype y from the clinical diagnosis report of the patient to which the sample generated in step 1) belongs. t This means that the patient is y t Type; Suppose there are l1 subtypes for patient classification, and convert patient labels into one-hol vectors. Where y t =1, and other components are 0; obtain the patient cell proportion S = (s1, s2), where s2 is the proportion of cancer cells, s1 = 1 - s2, and s1 represents the proportion of normal cells;

[0047] 3) Take the single-cell collection obtained in step 1) of step two. The input encoder yields the cell feature matrix F. cell ∈R N×K Where K represents the feature dimension of a single cell, and F cellThe input is a category-based attention module, which is a neural network composed of fully connected layers that learns the classification contribution of each cell to each category. Within the attention module, the cell feature matrices are first passed through fully connected layers... and Reduce the feature dimension to Then normalize using the tanh and sigmoid functions separately, multiply the normalized results, and then pass them through a... Fully connected layer from L1 dimension Obtain the attention matrix, which is the contribution of each cell to l1 categories.

[0048] A1=W×(tanh(F cell ×U)⊙sigmoid(F cell ×V))

[0049] Where ⊙ denotes the multiplication of corresponding points within the matrix;

[0050] Finally, the result is obtained by multiplying the cell feature matrix and the attention matrix. This represents the feature matrix of a classifier for l1 categories;

[0051] Let l2 be the number of categories in the second-level coarse subdivision relative to the subdivision l1, then through the transformation matrix... The values ​​of the corresponding categories in the attention matrix for l1 categories are aggregated into l2 categories, i.e. A2 = A1 × T 1 The contribution of each cell to each category in the second-level coarse classification is obtained. The feature matrix of the classifier for l2 categories is obtained by multiplying the attention matrix and the cell feature matrix. Similarly, when reaching the third level of coarse classification, let l3 be the number of categories in the third level of coarse classification. Then, the attention matrix obtained from the second level of coarse classification can be... By transforming the matrix Attention matrix aggregated to the third level coarse fractal A3 = A2 × T 2 By multiplying the attention matrix and the cell feature matrix, the feature matrix of the classifier for l3 categories is obtained. A n =A n-1 ×T n-1 For the attention matrix representation of n-level classification, the feature matrix of the classifier at the nth level is generally represented as F. l ∈R K×l, represents the hierarchical classification class of the total l, wherein n, l1, l2, l3, l, K>0, are all integers, and l1>l2>l3, that is, l n-1 >l n ;

[0052] 4) input the feature matrix F l ∈R K×l in the l binary classifiers, obtain the output O patient ∈R l×2 , and input the cell feature matrix F cell ∈R N×K in the binary classifier about the cell class, obtain the output result O cell ∈R N×2 ;

[0053] 5) according to O cell , calculate the cell proportion as wherein the gradient in the proportion is reserved, so as to adjust the network parameters, and the gumbel_softmax function needs to be used to form the probability representation of whether it is a cancer cell in the first dimension of the N*2 matrix, that is, there is , which represents the probability of being a normal cell, which represents the probability of being a cancer cell; summing up in the 0th dimension and dividing by the total number N, that is, the proportion is obtained wherein and O patient ∈R l×2 , for each class j, there is the probability p j ∈[0, 1] that the sample belongs to the class, and the loss function of the classification learning process is represented as:

[0054]

[0055]

[0056] The cell segmentation module described in the application is the prior art in the field.

[0057] The application obtains a plurality of bone marrow cells of a patient through step one, and forms a sample by sampling and packaging a certain number of samples, which meets the process of doctor analysis of bone marrow cell diagnosis typing; therefore, the label used in the method is the typing of the patient and the proportion of cancer cells, which is a multi-instance learning method.

[0058] The cells are coded into features by step two, and the contribution of different cells to each category is calculated by the attention module. The hierarchical features are formed by aggregating the attention matrices of the subtypes to the coarse subtypes. The loss function for the proportion of cancer cells in step two classifies each cell as a cancer cell or not through the limitation of the proportion, semi-supervisedly, so that the attention mechanism can better calculate the contribution of each cell to each category. The loss function for the patient subtype maintains the accuracy of the model in classifying childhood leukemia. The attention matrices of the fine classification are aggregated to the coarse classification, forming a constraint between the two, so that each cell is not easily confused between large categories, making the classification more accurate.

[0059] The hierarchical attention mechanism-based childhood leukemia classification method simulates the real clinical diagnosis process and better meets the characteristics of multiple and fine subtypes of childhood leukemia. The attention mechanism for each category integrates the contribution of each cell to different categories into the cell features, better plays the role of attention mechanism in multi-classification, and makes the model pay more attention to cells for each category. Hierarchical classification is performed, and the attention matrices of the fine subtypes are aggregated to the coarse subtypes, so that the features of the fine subtypes are not easily confused between large categories, thereby improving the accuracy of the classification.

[0060] Embodiment:

[0061] Classification is performed in n=2 levels. These two levels are referred to as coarse-grained subtypes and fine-grained subtypes, respectively. The number of categories of fine-grained subtypes is l1=7, where each subtype is M2, M3, M5, M7, L1&L2, L3, and normal (Normal). The number of categories of coarse-grained subtypes is l2=3, where each subtype is acute myeloid leukemia (AML), acute lymphoblastic leukemia (ALL), and normal (Normal). The conversion relationship between coarse-grained and fine-grained is that acute myeloid leukemia includes M2, M3, M5, and M7; acute lymphoblastic leukemia includes L1&L2 and L3.

[0062] Referring to Figure 1 , step one: perform cell segmentation task:

[0063] 1) According to the analysis and diagnosis results of childhood leukemia given by doctors, the nucleated cells in the multiple bone marrow cell images of a childhood leukemia patient are extracted by the cell segmentation module, and then randomly sampled in 50 cells per sample to form a multi-instance package, as follows:

[0064] 1.1) A childhood leukemia patient's 15 bone marrow cell images with a size of 512 pixels x 512 pixels are cropped by multi-center with a step size of 128 pixels and a window size of 256 pixels, 16 tiles covering the whole image are obtained, and the tile set is represented as X={x1, x2,..., x 16}

[0065] 1.2) Take each tile obtained in step one 1.1) randomly to take shape, angle and color change image enhancement mode, and take center crop to obtain a square tile with a side length of 256 pixels. The tile set after image enhancement is represented as

[0066] 1.3) Input the tile set obtained in step one 1.2) into the cell segmentation module to obtain the output tile. The position of the center coordinate in the original image is located and spliced to obtain the cell segmentation result of the original image. All cells in the segmentation result are extracted, and 50 cells are obtained. The single cell tile set is represented as C = {c1, c2,..., c 50};

[0067] 2) According to step one 1), form multiple-instance bag samples in turn for bone marrow cell images of patients with different types of childhood leukemia;

[0068] Referring to Figure 2 , step two, the childhood leukemia classification task is performed:

[0069] Input the multiple-instance bag processed in step one into the childhood leukemia classification module. The childhood leukemia classification module is a convolutional neural network composed of an encoder, an attention module and a classifier. The attention module is a neural network composed of a fully connected layer. It distinguishes whether a sample belongs to acute myeloid leukemia or acute lymphoid leukemia, and further distinguishes different subtypes in the two major systems. After classification into different subtypes, further distinguish the subtypes that are not easy to distinguish or small classes in the subtypes, that is, perform multiple hierarchical typing:

[0070] 1) Perform boundary padding on the single cell tile set C finally obtained in step one 1.3). Fill all cell tiles to a square with a side length of 128 pixels to facilitate packing all cell tiles into the encoder. The single cell tile set after padding is

[0071] 2) Obtain the leukemia typing y of the patient from the clinical diagnosis report of the patient to which the sample generated in step one 1) belongs t , which indicates that the patient is of y t type; suppose that the patient typing has 7 subtypes, and convert the patient label to a one-hot vector Y = (y1, y2,..., y7), where y t = 1 and the other components are 0; obtain the cell proportion S = (s1, s2) of the patient, where s2 is the proportion of cancer cells, and s1 = 1-s2, s1 represents the proportion of normal cells;

[0072] 3) Obtain the single cell set Input encoder, get cell feature matrix F cell ∈R 50×512 , where 512 represents the feature dimension of a single cell, and F cell Input the category-based attention module, i.e., a neural network composed of fully connected layers that learns the classification contribution of each cell to each category; in the attention module, the cell feature matrix is first passed through the fully connected layers U∈R 512×256 and V∈R 512×256 , the feature dimension is reduced to 256, and then the tanh and sigmoid functions are normalized respectively, the normalized results are multiplied, and then a fully connected layer W∈R 256×7 from 256 dimensions to 7 dimensions is passed through to obtain the attention matrix, i.e., the contribution A1∈R 50×7 of each cell to 7 categories:

[0073] A1=W×(tanh(F cell ×U)⊙sigmoid(F cell ×V))

[0074] where ⊙ represents the multiplication of corresponding points in the matrix;

[0075] Finally, the product of the cell feature matrix and the attention matrix is obtained F7∈R 512×7 , which represents the feature matrix of the classifier for 7 categories;

[0076] The number of categories in the second level of coarse classification is 3, and the conversion matrix T 1 ∈R 7×3 is defined as follows:

[0077]

[0078] The values of the corresponding categories in the attention matrix for 7 categories are aggregated into 3 categories, i.e., A2∈R 50×3 , A2=A1×T 1 , to obtain the contribution of each cell to each category in the second level of coarse classification, and the product of the attention matrix and the cell feature matrix is obtained to obtain the feature matrix F3∈R 512×3 of the classifier for 3 categories;

[0079] 4) Input the feature matrix F3∈R 512×3 obtained in step 2, 3) into 3 binary classifiers to obtain the output coarse classification result O patient_3 ∈R 3×2 , input the feature matrix F7∈R 512×7 into 7 binary classifiers to obtain the output fine classification result O patient_7 ∈R 7×2The cell feature matrix F cell ∈R 50×512 The output result O of whether it is a cancer cell in the binary classifier about cell categories is input cell ∈R 50×2 ;

[0080] 5) Use the gumbel_softmax function to get O cell This 50x2-dimensional matrix forms a probability representation of whether it is a cancer cell in the first dimension, that is, for each cell i, there is P (normal cell) represents the probability of being a normal cell, P (cancer cell) represents the probability of being a cancer cell; summing up in the 0th dimension and dividing by the total number 50, that is, obtaining the proportion Wherein That is, the proportion of cancer cells of the patient is obtained And O patient_7 ∈R 7×2 And O patient_3 ∈R 3×2 In O, for each category j, there is the probability p of whether the sample belongs to the category j ∈[0, 1], taking the category with the highest probability, that is, the subtypes and coarse types predicted by the present application for the patient.

[0081] Through the test comparison of the test set, the correct rate of the classification method based on the hierarchical attention mechanism can reach 79% for the fine-grained classification, and can reach 97% for the coarse-grained classification.

[0082] When the fine-grained classification and the coarse-grained classification do not use the conversion relationship T 1 , that is, the respective attention matrices A1 and A2 are obtained by using different attention modules, the fine-grained classification correct rate is only 71%, and the coarse-grained classification correct rate is only 94%.

[0083] When the coarse classification and the fine classification are predicted by different models respectively, that is, the classification method is not based on hierarchy, the fine classification correct rate is only 73%, and the coarse classification correct rate is only 91%.

Claims

1. A hierarchical attention mechanism-based multiple-instance classification method for childhood leukemia, specifically comprising the following steps: Step one: Perform cell segmentation task: 1) According to the analysis and diagnosis results of childhood leukemia given by doctors, the nucleated cells in multiple bone marrow cell images of a childhood leukemia patient are extracted through a cell segmentation module, and then randomly sampled in N cells per sample, forming a multiple-instance package, wherein N∈[50, 200]; 2) For bone marrow cell images of patients with different types of childhood leukemia, multiple-instance package samples are formed according to step one 1) in turn; Step two: perform the childhood leukemia classification task: input the multi-instance package processed in step one into the childhood leukemia classification module, including an encoder, a category-based attention module, and a classifier, the encoder accepts the output of the cell segmentation module to obtain a cell feature matrix F cell The cell feature matrix F cell is input into the category-based attention module to obtain an attention matrix, and the cell feature matrix F cell is multiplied by the attention matrix of different levels to obtain a feature matrix for different level classifiers to distinguish childhood leukemia, and distinguish whether each sample belongs to acute myeloid leukemia or acute lymphoid leukemia, and further distinguish different subtypes in the two major systems, classify into different subtypes, and further distinguish subtypes that are not easy to distinguish or small classes in the subtypes, i.e. multiple levels of classification.

2. The hierarchical attention mechanism-based multiple-instance classification method for childhood leukemia according to claim 1, characterized in that: The specific method for forming a multiple-instance package in step one 1) is: 1.1) a bone marrow cell image of a childhood leukemia patient is multi-center cropped with a step size of b pixels and a window size of w pixels to obtain n1 patches covering the whole image, and the patch set is represented as X = {x1, x2, …, xn1} where a ∈ [15, 50], b > 0, w > 0, n1 > 0 and are all integers; n1} represents the patch set, wherein a ∈ [15, 50], b > 0, w > 0, n1 > 0 and are all integers; 1.2) Randomly take shape, angle and color change image enhancement mode for each tile obtained in step one 1.1), and carry out center cutting to obtain a square tile with side length w1 pixels, and the set of tiles after image enhancement is represented as wherein w1 ∈ [128, 256]; 1.3) input the tile set obtained in step one 1.2) into the cell segmentation module, get the output tile, locate the tile set obtained by splicing through the position of the center coordinate in the original image, the size of the cell segmentation result of the original image, extract all cells in the segmentation result, and then randomly sample N cells to obtain a single cell tile set represented as C={c1, c2, …, cN}. N} 3. The hierarchical attention mechanism-based multiple-instance classification method for childhood leukemia according to claim 1, characterized in that: The specific method of step two is: 1) The single cell tile set C finally obtained in step one 1.3) is subjected to boundary padding, and all cell tiles are padded into squares with a side length of w2 pixels, facilitating the packaging of all cell tiles into the encoder. The padded single cell tile set is where w2 ∈ [128, 256]; 2) Obtain the leukemia subtype y of the patient from the clinical diagnosis report of the patient to which the sample generated in step one 1) belongs t , which means that the patient is of type y t ; suppose that the patient has l1 subtypes, and convert the patient label into a one-hot vector , where y t = 1 and other components are 0; obtain the cell proportion S = (s1, s2) of the patient, where s2 is the proportion of cancer cells and s1 = 1-s2, s1 represents the proportion of normal cells; 3) the single cell set obtained in step 2) of 1) input the encoder to obtain the cell feature matrix F cell ∈R N ×K where K represents the feature dimension of a single cell, and F cell input the category-based attention module, i.e., a neural network composed of fully connected layers, to learn the classification contribution of each cell to each category; in the attention module, the cell feature matrix is first respectively passed through the fully connected layers and reduce the feature dimension to then respectively perform normalization of the tanh and sigmoid functions, multiply the normalized results, and then pass through a fully connected layer from dimension to dimension obtain the attention matrix, i.e., the contribution of each cell to the l1 categories A1 = W x (tanh(F cell x U) O sigmoid(F cell x V)) Where ⊙ represents the multiplication of corresponding points in the matrix; The final feature matrix is obtained by multiplying the cell feature matrix and the attention matrix represents the feature matrix of the classifier for the l1 classes; Let l2 be the number of categories in the second-level coarse subdivision relative to the subdivision l1, then through the transformation matrix... The values ​​of the corresponding categories in the attention matrix for l1 categories are aggregated into l2 categories, i.e. A2 = A1 × T 1 The contribution of each cell to each category in the second-level coarse classification is obtained. The feature matrix of the classifier for l2 categories is obtained by multiplying the attention matrix and the cell feature matrix. Similarly, when reaching the third level of coarse classification, let l3 be the number of categories in the third level of coarse classification. Then, the attention matrix of the obtained second level of coarse classification can be... By transforming the matrix Attention matrix aggregated to the third level coarse fractal A3 = a2 × T 2 By multiplying the attention matrix and the cell feature matrix, the feature matrix of the classifier for l3 categories is obtained. A n =A n-1 ×T n-1 For the attention matrix representation of n-level classification, the feature matrix of the classifier at the nth level is generally represented as F. l ∈R K×l This indicates that there are a total of l classification categories at this level, where n, l1, l2, l3, l, and K > 0 are all integers, and l1 > l2 > l3, i.e., l n-1 >l n ; 4) inputting the feature matrix F obtained in step two, 3) above l ∈R K×l inputting the output O obtained in the l binary classifiers patient ∈R l×2 inputting the cell feature matrix F cell ∈R N×K inputting the output result O obtained in the binary classifier on the cell category as to whether it is a cancer cell cell ∈R N×2 ; 5) according to O cell The cell proportion is calculated as Where is the gradient in the proportion that is preserved, adjusting the network parameters, requires the use of the gumbel softmax function, which forms a probability representation of whether a cell is cancerous in the first dimension of an N x 2 dimensional matrix, i.e. for each cell i has represents the probability of being a normal cell, represents the probability of being a cancer cell; summing over the 0th dimension and dividing by the total number N, i.e. obtaining the proportion where And O patient ∈ R l×2 For each class j, there is a probability p j ∈ [0,1], the loss function of the classification learning process is represented as:

4. The hierarchical attention mechanism-based multiple-instance classification method for childhood leukemia according to claim 1, characterized in that: In the category-based attention module, the cell feature matrix is first respectively passed through a fully connected layer and The feature dimension is reduced to Then the normalized results of tanh and sigmoid functions are multiplied, and then a fully connected layer from l0 dimension to l1 dimension is passed to obtain the attention matrix, i.e. the contribution degree of each cell to l1 categories A1 = W x (tanh(F cell x U) O sigmoid(F cell x V)) Where ⊙ represents the multiplication of corresponding points in the matrix.

5. The hierarchical attention mechanism-based multiple-instance classification method for childhood leukemia according to claim 4, characterized in that: Let l2 be the number of classes of the second level of coarse classification relative to the fine classification of l1, then the contribution of each cell to each class of the second level of coarse classification is obtained by converting the matrix aggregating the values of the corresponding class in the attention matrix for l1 classes to l2 classes, i.e. A2 = A1 x T 1 .

Images