Cell classification method and device based on cytology smear image

Abstract

The invention discloses a cell classification method based on a cytology smear image, and the method comprises the steps: carrying out the reasoning of the cytology smear image through a trained neural network model, and obtaining a cell classification result used for representing the slice-level classification, the neural network model comprises a cell recognition model used for detecting each cell in the cytology smear image, and a splicing layer used for extracting cell images based on the positions of the detected cells and splicing the extracted images into an image sequence; the convolutional neural network is used for carrying out feature extraction on each image in the image sequence from the splicing layer to obtain a feature sequence, and the conversion model is used for obtaining a feature matrix used for representing relevancy among features in the feature sequence based on the feature sequence from the convolutional neural network. And a multi-instance model used for performing multi-instance learning based on the feature matrix from the conversion model to predict the cell classification result. According to the invention, the accuracy and interpretability of cell classification are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular, to a cell classification method based on cytological smear images. BACKGROUND

[0002] In the clinical diagnosis system of cytopathology, cell classification based on cytological smear images is a core technical link to realize early identification and accurate diagnosis of lesions, and the reliability and efficiency of its detection results directly affect the timeliness and accuracy of clinical diagnosis and treatment decisions. In current clinical practice, cell classification still mostly relies on traditional manual microscopic observation as the main technical path. This mode gradually exposes multi-dimensional technical limitations in long-term application, which are as follows:

[0003] Firstly, there is a significant bottleneck in detection efficiency. Manual interpretation relies on professional technicians to observe and analyze a large number of cell samples under a microscope, which not only leads to long detection time, but also requires high clinical experience accumulation, continuous concentration and visual fatigue tolerance of the operator, making it difficult to meet the high-efficiency detection demand in large-scale scenarios.

[0004] Secondly, the subjective bias of classification results is prominent. The key interpretation links such as cell morphology classification and abnormal cell recognition highly depend on the subjective experience and judgment standard of the operator. The differences in knowledge background, operation habit and pathological feature cognition among different personnel easily lead to inconsistent interpretation results of the same batch of samples in different laboratories or by different operators, seriously affecting the repeatability and reliability of the detection results. SUMMARY

[0005] Embodiments of the present application provide a cell classification method based on cytological smear images to improve the intelligence of cell classification based on cytological smear images.

[0006] The first aspect of the embodiments of the present application provides a cell classification method based on cytological smear images, which comprises: inferring the cytological smear images by a trained neural network model to obtain a cell classification result for representing slice-level classification,

[0007] wherein,

[0008] The neural network model comprises:

[0009] a cell recognition model for detecting each cell in the cytological smear image,

[0010] a stitching layer for extracting cell images based on the positions of the detected cells and stitching the extracted images into an image sequence,

[0011] a convolutional neural network configured to extract features from each image in a sequence of images from a slide to obtain a sequence of features, wherein each feature vector in the sequence of features corresponds to one of the images in the sequence of images,

[0012] a transformation model configured to obtain a feature matrix representing correlations between features in the sequence of features based on the sequence of features from the convolutional neural network, wherein each row vector in the feature matrix corresponds to a feature vector in the sequence of features, and each element in the feature matrix represents a correlation between features,

[0013] a multi-instance model configured to predict the cell classification result based on the multi-instance learning using the feature matrix from the transformation model.

[0014] In one possible implementation, the length of the sequence of images depends on the number of cells identified by the cell recognition model, and the length of the sequence of images and the length of the corresponding sequence of features are different for different cytological smear images.

[0015] In one possible implementation, the multi-instance learning is performed using the feature matrix as an instance bag and the row vectors in the feature matrix as instances.

[0016] In one possible implementation, each encoder of the transformation model further comprises a normalization layer configured to perform a normalization operation on the attention weights from the self-attention layer in the encoder.

[0017] In one possible implementation, the multi-instance model performs a global max-pooling operation on each feature vector in the feature matrix to obtain discriminative key features, and performs a global average-pooling operation on each feature vector in the feature matrix to obtain statistical distribution information of the feature vectors, fuses the results of the global max-pooling operation and the global average-pooling operation to obtain fused features, and performs cell classification based on the fused features.

[0018] In one possible implementation, the neural network model is trained in the following manner:

[0019] The cell recognition model is trained using sample cytological smear images to obtain a trained lesion cell recognition model,

[0020] The trained cell recognition model is used to obtain sample image sequences for training the convolutional neural network, the transformation model, and the multi-instance learning model.

[0021] A plurality of sample image sequences are used as a batch of training samples, each sample image sequence in the batch of training samples is traversed, and a mask image is added to each sample image sequence to pad the sequence length of each sample image sequence to an upper limit of the length of the batch of training samples.

[0022] inputting each of the padded sample image sequences into a convolutional neural network,

[0023] inputting the sample feature sequence extracted by the convolutional neural network into a conversion model,

[0024] inputting the sample feature matrix output by the conversion model into a multi-instance learning model,

[0025] adjusting model parameters of the convolutional neural network, the conversion model, and the multi-instance learning model according to a loss function value between a sample cell classification result output by the instance learning model and an expected cell classification result until a desired result is achieved.

[0026] As a possible implementation, the normalization layer in each encoder of the conversion model masks the features of the masked image from participating in the distribution of attention weights from the self-attention layer in the encoder during the training process.

[0027] As a possible implementation, the normalization layer in each encoder of the conversion model masks the features of the masked image from participating in the distribution of attention weights from the self-attention layer in the encoder during the training process, comprising:

[0028] for each attention weight of the sample feature sequence from the self-attention layer in the encoder,

[0029] in the case where the length of the sample feature sequence is less than or equal to the upper limit of the length, the weight between the i-th feature in the sample feature sequence and the j-th feature in the sample feature sequence is: the ratio of the exponential attention score of the i-th feature with respect to the j-th feature to the total exponential attention score of the i-th feature with respect to all features in the sample feature sequence, where i and j are natural numbers, and the exponential attention score is obtained according to the attention weight of the i-th feature with respect to the j-th feature from the attention layer,

[0030] in the case where the length of the sample feature sequence is greater than the upper limit of the length, the weight between the i-th feature in the sample feature sequence and the j-th feature of interest region in the sample feature sequence is 0.

[0031] As a possible implementation, the use of the trained cell recognition model to obtain the sample image sequence for training the convolutional neural network, the conversion model, and the multi-instance learning model comprises:

[0032] inputting the sample cytological smear image into the trained cell recognition model, and obtaining the image position information of each cell in the sample cytological smear image through inference of the trained cell recognition model,

[0033] According to the image position information of each cell in the sample cytological smear image, an image of each cell in the sample cytological smear image is cropped from the sample cytological smear image,

[0034] The cropped images are spliced into a sample image sequence of the sample cytological smear image.

[0035] As a possible implementation, the upper limit of the length is determined according to a maximum sequence length in sequence lengths of each sample image sequence in the batch training sample.

[0036] The second aspect of the embodiments of the present application provides a cell classification device based on a cytological smear image, which comprises:

[0037] An inference module is configured to perform inference on the cytological smear image by using the trained neural network model to obtain a cell classification result for representing slice-level classification,

[0038] wherein,

[0039] The neural network model comprises:

[0040] a cell recognition model configured to detect each cell in the cytological smear image,

[0041] a splicing layer configured to extract cell images based on positions of the detected cells and splice the extracted images into an image sequence,

[0042] a convolutional neural network configured to perform feature extraction on each image in the image sequence from the splicing layer to obtain a feature sequence, wherein a feature vector of each feature in the feature sequence corresponds to each image in the image sequence,

[0043] a conversion model configured to obtain a feature matrix for representing correlation between each feature in the feature sequence based on the feature sequence from the convolutional neural network, wherein each row vector in the feature matrix corresponds to the feature vector of each feature in the feature sequence, and an element in the feature matrix is used to represent the correlation between the features,

[0044] a multi-instance model configured to predict the cell classification result based on the feature matrix from the conversion model by using multi-instance learning.

[0045] The cell classification method based on cytological smear images provided by the embodiment of the application, by combining the convolutional neural network model and the conversion model, takes into account the local features and global features of cell morphology, which is conducive to improving the accuracy of cell classification; by combining the conversion model and the multi-instance learning model, it is conducive to integrating feature information, thereby improving the accuracy and robustness of cell classification; by the normalization layer added in each encoder layer of the conversion model, the length of the image sequence input to the convolutional neural network is not restricted, avoiding information loss and noise interference, thereby being conducive to improving the accuracy of cell classification. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 A flowchart of a process of performing cell classification on cytological smear images by the embodiment of the application using the trained neural network model.

[0047] Figure 2 A schematic diagram of a neural network model for cell classification based on cytological smear images according to the embodiment of the application.

[0048] Figure 3 A schematic diagram of the model structure of the encoder in the conversion model according to the embodiment of the application.

[0049] Figure 4 A flowchart of a process of training the neural network model according to the embodiment of the application.

[0050] Figure 5 A schematic diagram of the attention weight matrix of the sample feature sequence output by the self-attention layer after processing by the normalization layer according to the embodiment of the application.

[0051] Figure 6 A schematic diagram of a cell classification device based on cytological smear images according to the embodiment of the application.

[0052] Figure 7 Another schematic diagram of a cell classification device based on cytological smear images according to the embodiment of the application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical means and advantages of the application more clear and explicit, the application will be further described in detail below with reference to the drawings.

[0054] The cell classification method based on cytological smear images provided by the embodiment of the application, by training the neural network model including the cell recognition model, the convolutional neural network, the conversion model and the multi-instance model, performs slice-level cell classification on cytological smear images.

[0055] Referring to Figure 1 as shown, Figure 1A flowchart of a process for performing cell classification on a cytological smear image using a trained neural network model according to an embodiment of the present application. The process includes: for any cytological smear image,

[0056] Step 101, input the cytological smear image into a cell detection model in the trained neural network model, and obtain image position information of each cell in the cytological smear image through inference of the cell detection model.

[0057] As an example, the inference of the cell detection model can also obtain the category of each cell.

[0058] Step 102, according to the image position information of each cell, crop each cell image from the cytological smear image, and splice the cropped images into an image sequence of the cytological smear image.

[0059] As an example, crop single-cell images of each target category from the cytological smear image, and splice the cropped images into a single-cell image sequence.

[0060] In this embodiment, the sequence length of the obtained image sequence is different for different cytological smear images, and depends on the number of detected cells. This embodiment neither pads images to meet the sequence length nor removes images to meet the sequence length, but keeps the existing image sequence. In this way, invalid noise data introduced by padding operation is avoided, so as to avoid false negative or false positive results caused by interference with feature extraction and judgment; information loss caused by discarding images is avoided so as to ensure that the selected images fully represent the true situation; the integrity and authenticity of the input data are ensured, the information distortion caused by cropping or padding is avoided, which is conducive to fully mining and utilizing all the information provided by the images, thereby providing more comprehensive and reliable basis for subsequent cell classification, and is conducive to improving the accuracy and robustness of the cell classification result.

[0061] It should be understood that the order of splicing the cropped images can be random, and this embodiment does not limit it.

[0062] Step 103, input the image sequence into a convolutional neural network in the trained neural network model, and obtain a feature sequence of the image sequence through inference of the convolutional neural network,

[0063] As an example, each feature vector in the feature sequence corresponds to each image in the image sequence.

[0064] Step 104, input the feature sequence into a Transformer model in the trained neural network model, and obtain a feature matrix of the feature sequence through inference of the Transformer model,

[0065] As an example, each row vector in the feature matrix corresponds to a feature vector of each feature in the feature sequence, and elements in the feature matrix are used to represent the correlation between features, so that the feature matrix is used to represent the feature sequence itself and the correlation between features in the feature sequence.

[0066] In step 105, the feature matrix is input into a multi-instance learning model in the trained neural network model, and a cell classification result of the cytological smear image is obtained through inference of the multi-instance learning model. The cell classification result is used to represent the slice-level cell classification.

[0067] As an example, multi-instance learning is performed with the feature matrix as an instance bag and the row vectors in the feature matrix as instances. For example, the multi-instance model performs global max-pooling operation on each feature vector in the feature matrix to obtain key features with discriminative ability, and performs global average-pooling operation on each feature vector to obtain statistical distribution information of each feature vector in the feature matrix. The global max-pooling operation result and the global average-pooling operation result are fused to obtain fused features, and cell classification is performed based on the fused features.

[0068] In the above inference process, the cooperation of the convolutional neural network and the Transformer model realizes multi-level information capture from local features to global context; multi-instance learning meets the actual needs of clinical diagnosis in which the whole slice is taken as the interpretation unit, significantly improves the accuracy and robustness of the cell pathology diagnosis while maintaining high computational efficiency, and provides reliable technical support for clinical auxiliary diagnosis.

[0069] To facilitate understanding of the embodiments of the present application, the following examples are described.

[0070] Referring to Figure 2 As shown in the figure, Figure 2 is a schematic diagram of a neural network model for cell classification based on cytological smear images according to an embodiment of the present application. The neural network model includes:

[0071] A cell recognition model is used to obtain cell image position information such as a bounding box and a cell category based on a cytological smear image.

[0072] A splicing layer is used to crop an image within a cell bounding box from a cytological smear image based on the cell bounding box and splice the image into an image sequence.

[0073] A convolutional neural network is used to perform feature extraction on the image sequence to obtain a feature sequence.

[0074] A conversion model is used to obtain a feature matrix based on the feature sequence from the convolutional neural network, the feature matrix being used to represent the feature sequence itself and the correlation between features in the feature sequence.

[0075] a multi-instance learning model for cell classification based on the feature matrix from the conversion model in a multi-instance learning paradigm to obtain a cell classification result of the slice-level cell classification.

[0076] Referring to Figure 3 as shown, Figure 3 is a model structure diagram of an encoder in the transformer model of the embodiment. The self-attention layer in each layer of the transformer model also includes a normalization layer, and the self-attention weight from the self-attention layer is input to the normalization layer for normalization operation.

[0077] In the embodiment, the convolutional neural network is advantageous for efficiently capturing the local morphological features of the cell image, such as edges, textures, and nucleus-cytoplasm ratios, due to its hierarchical local connection and spatial inductive bias characteristics. Moreover, the convolutional neural network effectively alleviates the gradient disappearance defect in the deep network by introducing a residual connection structure, thereby facilitating the extraction of more abundant and discriminative multi-level features from the cell image, and taking into account the dual requirements of precision and efficiency.

[0078] As an example, the convolutional neural network is a CNN network, which has strong local feature extraction capability and helps to provide a solid and reliable feature basis for subsequent deep modeling based on the Transformer, avoiding the performance bottleneck caused by poor quality of shallow features. In addition, its efficient parallel computing characteristics ensure the overall response speed.

[0079] In the embodiment, the Transformer is combined with multi-instance learning to achieve accurate image classification at the slice level. After the feature sequence is input to the Transformer encoder, the inherent self-attention mechanism can dynamically model the global dependency relationship between any two features in the feature sequence, thereby capturing more complex and deeper intercell morphological context semantic information in the entire cytological smear image.

[0080] On this basis, the multi-instance learning paradigm is used for classification, which can not necessarily rely on the classification of each cell, but rather the image sequence is regarded as a whole instance package, and the features of the images in the entire image sequence are extracted by the Transformer as instances. Through the pooling operation, the information of all instances is aggregated, and finally the instance package level cell classification result is obtained.

[0081] The integration of the Transformer and the multiple-instance learning plays the ability of the Transformer to process variable-length sequences and capture global information, and overcomes the defect of the traditional convolutional neural network that the sequence length is fixed; the multiple-instance learning meets the demand of comprehensive evaluation based on the whole slide in clinical diagnosis, and effectively improves the accuracy, robustness and clinical consistency of the interpretation result by integrating all the cell image information.

[0082] Referring to Figure 4 as shown, Figure 4 A flowchart of a process for training a neural network model according to an embodiment of the present application. It includes:

[0083] Step 401, using sample cytological smear images, training a cell recognition model to obtain a trained cell recognition model.

[0084] Step 402, for each sample cytological smear image, segmenting the sample cytological smear image into multiple image blocks, and using the trained cell detection model to infer each image block to identify the category and image position information of the cells in each image block.

[0085] Step 403, according to the image position information of the identified cells, cropping each cell from the sample cytological smear image, and splicing the cropped images into a sample image sequence to obtain a sample image sequence of the sample cytological smear image.

[0086] As an example, single-cell images of a sample target category can be cropped and spliced into a sample image sequence.

[0087] Step 404, taking multiple sample cytological smear images as a batch of training samples, and padding the lengths of the sample image sequences of each sample cytological smear image in the batch of training samples to the upper limit of the length corresponding to the batch.

[0088] Since the number of cells in each sample cytological smear image is different, the sequence length of the sample image sequence of each sample cytological smear image is different. In order to realize variable-length image sequence input in the convolutional neural network in the trained neural network model, in each batch of training samples, the sample image sequences of all sample cytological smear images in the batch are traversed, the longest sample image sequence is found, the sequence length of the longest sample image sequence is taken as the length upper limit of the batch of training samples, and for the sample image sequences less than the length upper limit in the batch, a mask image with zero pixel value, that is, a pure black image, is used to fill in, until the sequence length of the sample image sequence reaches the length upper limit. In this way, the characteristics of the black image in the convolutional feature extraction are utilized, that is, the response is constant and easy for the network to identify, which effectively avoids the interference noise introduced by randomly filling invalid image content, thereby maintaining the batch training efficiency while ensuring the accuracy and stability of the feature extraction process.

[0089] Step 405, input each filled sample image sequence to the convolutional neural network, so that the convolutional neural network performs unified feature extraction on each sample image in the input sample image sequence in parallel, and converts each sample image into a feature vector. These feature vectors form a sample feature sequence of the input sample image sequence.

[0090] Step 406, input the sample feature sequence extracted by the convolutional neural network to the Transformer model, so that the Transformer model outputs sample feature matrix information of the sample feature sequence. The number of rows of the sample feature matrix information depends on the length of the sample feature sequence, and the number of columns depends on the dimension of the features in the sample feature sequence. The elements in the sample feature matrix represent the correlation between the features.

[0091] In the self-attention layer of each encoder of the Transformer model, the self-attention layer outputs the attention weight between each feature in the sample feature sequence, which represents the correlation between the features.

[0092] Referring to Figure 5 as shown, Figure 5Figure 1 is a schematic diagram of a normalized layer processing an attention weight matrix of a sample feature sequence output by a self-attention layer. In the figure, the red values are the attention weight values before normalization, and the blue values are the output of the normalized layer, i.e., the attention weight values after normalization. When a sample feature sequence of length 3 is padded to a sample feature sequence of length 4, the attention weight of the sample feature sequence is a 4x4 attention weight matrix, where the element in the ith row and the ith column represents the attention weight value between the ith feature and the jth feature in the sample feature sequence, and the attention weight value of 0 represents the attention weight value between each feature and the mask feature. When the length of the sample feature sequence is 4, the attention weight matrix does not have a 0 value.

[0093] To avoid the Softmax function in the normalization layer generating a non-zero activation when processing a zero-pixel filled image, thereby interfering with the self-attention calculation of each layer encoder in the Transformer model, the normalization layer in each encoder in the training process performs the following processing on the attention weight from the self-attention layer:

[0094] In the case where the length of the sample feature sequence is less than or equal to the length upper limit, the weight between the ith feature in the sample feature sequence and the jth feature in the sample feature sequence is the exponential attention score of the ith feature relative to the jth feature, i.e., divided by the total exponential attention score of the ith feature relative to all features in the sample feature sequence, where i and j are natural numbers.

[0095] In the case where the length of the sample feature sequence is greater than the length upper limit, the weight between the ith feature in the sample feature sequence and the jth feature in the sample feature sequence is 0.

[0096] Mathematically, it is expressed as:

[0097]

[0098] wherein, represents the weight between feature i and feature j, is the attention weight of feature i relative to feature j from the self-attention layer, represents the exponential attention score of feature i relative to feature j, represents the total exponential attention score of feature i relative to all features L, and L is the total number of features in the sample feature sequence.

[0099] For example, Figure 5 In the example of the normalized operation result = 0.57.

[0100] Through the operation of the normalization layer, the pure black image features introduced by the padding can be identified and shielded, ensuring that they do not participate in the distribution of self-attention weights, thereby effectively maintaining the purity of the feature sequence and the accuracy of the context modeling.

[0101] In step 407, the sample feature matrix information output by the Transformer model is input into the multi-instance model, so that the multi-instance model outputs the sample cell classification result of the sample cytological smear image.

[0102] The multi-instance model regards the sample feature matrix information corresponding to the sample cytological smear image as a whole instance package, and all instances in the package are jointly associated with a sample cell classification result label representing the sample cytological smear image.

[0103] Specifically, the multi-instance learning paradigm adopts a two-step feature aggregation strategy:

[0104] First, the features in the instance package are processed in parallel, one of which is subjected to global maximum pooling operation to extract key discriminative feature information in the instance package to highlight the significant differences in the feature space, and the other is subjected to global average pooling operation to obtain the overall statistical distribution information of the features in the instance package, and the global consistency of the features is preserved.

[0105] Subsequently, the features obtained after the above two pooling processes are fused to generate fusion features that can take into account both key discriminative information and statistical distribution information, and the sample cell classification result is determined based on the fusion features, thereby realizing the mapping from cell-level features to slice-level diagnosis.

[0106] In step 408, the model parameters of the to-be-trained model are adjusted according to the loss function between the sample cell classification result and the expected cell classification result, and the training is repeated until the desired result is achieved, thereby completing the training of the convolutional neural network, the Transformer model, and the multi-instance model in the neural network model.

[0107] In the training process of the present embodiment, the normalization operation of the normalization layer in each encoder of the Transformer model ensures the integrity of the feature sequence input to the convolutional neural network model, so that the convolutional neural network model in the trained neural network model is not constrained by the sequence length of the input image sequence, avoiding the mixing of features caused by fixed sequence length input and invalid padding, and is beneficial to improving the accuracy, robustness and interpretability of the cell pathology diagnosis.

[0108] Referring to Figure 6 shown, Figure 6 is a schematic diagram of a cell classification device based on cytological smear images according to an embodiment of the present application. The device comprises:

[0109] an inference module configured to perform inference on the cytological smear image by the trained neural network model to obtain a cell classification result for representing the slice-level classification,

[0110] wherein,

[0111] the neural network model comprises:

[0112] a cell recognition model configured to detect each cell in the cytological smear image,

[0113] a stitching layer configured to extract cell images based on positions of the detected cells and stitch the extracted images into an image sequence,

[0114] a convolutional neural network configured to perform feature extraction on each image in the image sequence from the stitching layer to obtain a feature sequence, wherein a feature vector of each feature in the feature sequence corresponds to each image in the image sequence,

[0115] a transformation model configured to obtain a feature matrix for representing correlation between each feature in the feature sequence based on the feature sequence from the convolutional neural network, wherein each row vector in the feature matrix corresponds to the feature vector of each feature in the feature sequence, and an element in the feature matrix is used to represent the correlation between the features,

[0116] a multi-instance model configured to predict the cell classification result based on the feature matrix from the transformation model by multi-instance learning.

[0117] Referring to Figure 7 illustrated, Figure 7 is another schematic diagram of an intelligent agent device of an embodiment of the present application. The device comprises a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to implement the steps of the cell classification method based on a cytological smear image of an embodiment of the present application.

[0118] The memory can include a random access memory (RAM) and can also include a non-volatile memory (NVM), for example at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0119] The processor described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0120] The embodiment of the present application further provides a computer readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to realize steps of the cell classification method based on a cytological smear image.

[0121] For the device / network side equipment / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts refer to the part of the method embodiment.

[0122] In this document, the terms“first” and“second” are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these entities or operations have any such actual relationship or order. Also, the terms“include”,“contain” or any other variants mean to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement“including a…” does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0123] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of classifying cells based on cytological smear images, characterized by, The method comprises: performing inference on the cytological smear image by the trained neural network model to obtain a cell classification result for representing slice-level classification, wherein, the neural network model comprises: a cell recognition model for detecting each cell in the cytological smear image, a stitching layer for extracting cell images based on the positions of the detected cells and stitching the extracted images into an image sequence, a convolutional neural network for performing feature extraction on each image in the image sequence from the stitching layer to obtain a feature sequence, wherein each feature vector in the feature sequence corresponds to each image in the image sequence, a conversion model for obtaining a feature matrix for representing the correlation between each feature in the feature sequence based on the feature sequence from the convolutional neural network, wherein each row vector in the feature matrix corresponds to each feature vector in the feature sequence, and each element in the feature matrix is used to represent the correlation between the features, a multi-instance model for predicting the cell classification result based on the multi-instance learning of the feature matrix from the conversion model.

2. The method of claim 1, wherein the step of classifying the cell comprises: The length of the image sequence depends on the number of cells recognized by the cell recognition model, and the lengths of the image sequences of different cytological smear images are different, and the lengths of the corresponding feature sequences are different. The multi-instance learning is performed by taking the feature matrix as an instance bag and the row vector in the feature matrix as an instance.

3. The method of claim 1 or 2, wherein, Each encoder of the conversion model further comprises a normalization layer for performing a normalization operation on the attention weights from the self-attention layer in the encoder.

4. The method of classifying cells according to claim 3, wherein, The multi-instance model performs a global max-pooling operation on each feature vector in the feature matrix to obtain discriminative key features, and performs a global average-pooling operation on the feature vector to obtain statistical distribution information of the feature vector, fuses the global max-pooling operation result and the global average-pooling operation result to obtain a fusion feature, and performs cell classification based on the fusion feature.

5. The method of classifying cells according to claim 4, wherein, The neural network model is trained in the following manner: training the cell recognition model using sample cytological smear images to obtain a trained lesion cell recognition model, using the trained cell recognition model to obtain sample image sequences for training the convolutional neural network, the conversion model, and the multi-instance learning model, taking a plurality of sample image sequences as a batch of training samples, traversing each sample image sequence in the batch of training samples, and adding a mask image to each sample image sequence to make the sequence length of each sample image sequence up to the upper limit of the length of the batch of training samples, inputting each padded sample image sequence into the convolutional neural network, inputting the sample feature sequence extracted by the convolutional neural network into the conversion model, inputting the sample feature matrix output by the conversion model into the multi-instance learning model, adjusting the model parameters of the convolutional neural network, the conversion model, and the multi-instance learning model according to the loss function value between the sample cell classification result output by the multi-instance learning model and the expected cell classification result until the desired result is achieved.

6. The method of classifying cells according to claim 5, wherein, The normalization layer in each encoder of the conversion model masks the features of the mask image from participating in the distribution of attention weights from the self-attention layer in the encoder during the training process.

7. The method of classifying cells according to claim 6, wherein, The normalization layer in each encoder of the conversion model masks the features of the mask image from participating in the distribution of attention weights from the self-attention layer in the encoder during the training process, including: For each attention weight of a sequence of sample features from the self-attention layer in the encoder, In the case where the length of the sequence of sample features is less than or equal to the upper limit of the length, the weight between the i-th feature in the sequence of sample features and the j-th feature in the sequence of sample features is: the exponential attention score of the i-th feature with respect to the j-th feature, divided by the total exponential attention score of the i-th feature with respect to all features in the sequence of sample features, where i and j are natural numbers, and the exponential attention score is obtained according to the attention weight of the i-th feature with respect to the j-th feature from the attention layer, In the case where the length of the sequence of sample features is greater than the upper limit of the length, the weight between the i-th feature in the sequence of sample features and the j-th feature of interest region in the sequence of sample features is 0.

8. The method of classifying cells according to claim 5, wherein, The use of the trained cell recognition model to obtain a sequence of sample images for training the convolutional neural network, the conversion model, and the multi-instance learning model includes: inputting a sample cytological smear image into the trained cell recognition model, and obtaining image position information of each cell in the sample cytological smear image through inference of the trained cell recognition model, cropping images of each cell in the sample cytological smear image according to the image position information of each cell in the sample cytological smear image, stitching the cropped images into a sequence of sample images of the sample cytological smear image.

9. The method of claim 5, wherein the step of classifying the cell comprises the step of: The upper limit of the length is determined according to the maximum sequence length in the sequence lengths of each sample image sequence in the batch training samples. ​ 10. A cytological smear image-based cell classification apparatus, characterized by, The device includes: an inference module configured to infer the cytological smear image through the trained neural network model to obtain a cell classification result for representing slice-level classification, wherein the neural network model includes: a cell recognition model configured to detect each cell in the cytological smear image, a stitching layer configured to extract cell images based on the positions of the detected cells and stitch the extracted images into an image sequence, a convolutional neural network configured to extract features from each image in the image sequence from the stitching layer to obtain a feature sequence, wherein each feature vector of each feature in the feature sequence corresponds to each image in the image sequence, a conversion model configured to obtain a feature matrix representing the correlation between each feature in the feature sequence based on the feature sequence from the convolutional neural network, wherein each row vector in the feature matrix corresponds to the feature vector of each feature in the feature sequence, and each element in the feature matrix represents the correlation between the features, a multi-instance model configured to predict the cell classification result based on the multi-instance learning of the feature matrix from the conversion model.

Images