Cervical Cell Detection and Recognition Method Based on Attention Mechanism and Detection Transformer

Abstract

The present invention discloses a cervical cell detection and recognition method based on an attention mechanism and a detection transformer, including: inputting multiple cervical cancer cell source images into a preset convolutional network to extract image features; the preset convolutional network uses a 50-layer residual network combined with a self-attention mechanism as the backbone network; performing spatial position encoding on the source images; reducing the dimension of the extracted image features, and combining the spatial position encoding for transformer processing; inputting the processing results of the transformer into two feed-forward neural networks to respectively predict detection boxes and categories, and calculating a loss function based on this; updating network parameters by minimizing the loss function to obtain an optimal model; based on the optimal model, detecting the cervical cell source image to be detected and recognized, and realizing the positioning and classification of cervical abnormal cells. This method can screen cell smears, can well solve the low accuracy and limitations of manual film reading, has lower costs, less workload, and higher accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of cervical cell information and graphic image processing, and particularly relates to a cervical cell detection and recognition method based on an attention mechanism and a detection transformer. Background Art

[0002] Currently, the diagnosis of cervical cancer cytology still mainly relies on pathologists manually reviewing slides under a microscope. With the increasing attention to cervical cancer screening, the workload of pathologists has increased dramatically, leading to overwork, seriously affecting the quality of pathological diagnosis and reducing the accuracy of screening. Due to the increasing demand for cervical cancer screening, the shortage of pathologist talents is difficult to make up in the short term, and the large differences in medical resources, etc., have promoted the rapid development of the computer-aided cervical screening industry and stimulated the emergence of new technologies and new products. Therefore, a computer-aided slide review system using object detection and recognition methods is needed to assist in screening cell smears.

[0003] As Figure 1 shown, it is a previous cell detection method in the prior art, which uses a two-stage object detection method. First, a residual network is used to extract image features, and then a region proposal network is used to generate proposal regions. After fusing the extracted features and the proposal regions, subsequent classification and regression operations are performed through a fully connected layer, and finally, the detection box of the cervical cell nucleus and the cell prediction category are obtained.

[0004] The above technical framework is a classic two-stage cervical cell detection framework, and the main problems of this solution are described as follows: The speed of the two-stage is significantly slower than that of the single-stage, and the algorithm process is also more complex. In terms of feature extraction, pathologists' judgment of cervical cancer cells often combines global information rather than a single judgment of a certain cell. The existing methods have insufficient ability to model the long-distance dependence relationship between pixels. In addition, the morphologies of abnormal cervical cells are diverse, and the existing methods also have poor performance in adapting to abnormal cells of different shapes. Summary of the Invention

[0005] The purpose of the present invention is to propose a cervical cell detection and recognition method based on an attention mechanism and a detection transformer to solve the above deficiencies in the prior art. This method can quickly and accurately locate the lesion area of cervical cell images and realize automatic auxiliary diagnosis in cervical cancer screening cytology examinations.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is:

[0007] The embodiment of the present invention provides a cervical cell detection and recognition method based on an attention mechanism and a detection transformer, including:

[0008] S1. Input multiple source images of cervical cancer cells into a preset convolutional network to extract image features. The preset convolutional network uses a 50-layer residual network combined with a self-attention mechanism as the backbone network.

[0009] S2. Perform spatial position encoding on the source images. Reduce the dimension of the extracted image features and perform transformer processing in combination with the spatial position encoding.

[0010] S3. Input the processing results of the transformer into two feed-forward neural networks to predict the detection boxes and categories respectively, and calculate the loss function accordingly. Update the network parameters by minimizing the loss function to obtain the optimal model.

[0011] S4. Based on the optimal model, detect the source images of cervical cells to be detected and recognized, and realize the positioning and classification of abnormal cervical cells.

[0012] Further, the 50-layer residual network combined with the self-attention mechanism includes: 5 convolutional blocks and 4 residual blocks. The step S1 includes:

[0013] S11. Add four self-attention mechanisms to the residual blocks in the 50-layer residual network.

[0014] S12. Modify the residual blocks in the 50-layer residual network and add deformable convolutions.

[0015] S13. Input the source images to be detected and recognized into the 50-layer residual network for convolution operations. An operation of learning the offset amount of the input feature map through a convolutional layer is added to the deformable convolution in step S12, and the output feature map is obtained through bilinear interpolation.

[0016] S14. Fuse the attention scores calculated by the four attention mechanisms in step S11 to obtain the final attention score.

[0017] S15. Fuse the output feature map and the attention score to complete the extraction of image features.

[0018] Further, in step S11, adding four self-attention mechanisms to the residual blocks in the 50-layer residual network includes:

[0019] The residual network block with the self-attention mechanism added is expressed as:

[0020]

[0021] (In formula (1), M represents the total number of attention heads; m represents the attention head index; Ω q represents the support key region of the specified query; A m (q, k, z q , xk ) represents the attention weight of the m-th attention head, where z q represents the query content, x k represents the key content, q is the index of z q and k is the index of x k ; W m and W' m represent learnable weights respectively; ⊙ represents element-wise multiplication at corresponding positions, and the attention weight normalization is restricted to:

[0022]

[0023] In equation (2),

[0024]

[0025] In equation (3), ∈ j There are four in total, ∈ 1 represents the similarity between the query and the key content; ∈ 2 represents the content of the query and the relative position; ∈ 3 represents the content of the key; ∈ 4 represents the global position bias between the key and the query.

[0026] Furthermore, the step S2 includes:

[0027] S21. Perform spatial position encoding on the source image;

[0028] S22. Reduce the dimension of the extracted image features, combine them with the spatial position encoding, and input them into the transformation encoder to obtain the encoder output;

[0029] S23. Combine the spatial position encoding and the encoder output, input them into the transformation decoder for decoding to obtain the decoder output.

[0030] Furthermore, the step S21 includes:

[0031] Performing spatial position encoding on the source image is:

[0032]

[0033]

[0034] where m represents the channel subscript, pos represents the position subscript, and d model represents the dimension of the feature.

[0035] Furthermore, the transformation encoder in the step S22 consists of a multi-head self-attention module and a feed-forward neural network;

[0036] The step S22 includes:

[0037] Dimensionality reduction is performed on the extracted image features, and combined with the spatial position encoding, a weighted feature vector is obtained through a self-attention module:

[0038]

[0039] Among them, Q, K, and V respectively represent query, key, and value content, and d k represents the dimension of the key vector.

[0040] After obtaining the feature vector, it is fed into a feed-forward neural network for spatial transformation to obtain the features encoded for N objects.

[0041] Furthermore, the feed-forward neural network in the step S3 is composed of a three-layer perceptron with a rectified linear unit activation function and a d-dimensional hidden layer and a linear layer;

[0042] The step S3 includes:

[0043] S31. Input the processing result of the transformer into two feed-forward neural networks to output the normalized center coordinates, height, and width of the predicted bounding box; the linear layer uses the SoftMax function to predict the class label;

[0044] S32. Use binary matching to match the ground truth box with the predicted box, and the matching strategy is as follows:

[0045]

[0046] N represents the number of predicted targets; y i represents the ground truth target, represents the predicted target, σ represents the correspondence from the predicted value to the ground truth, and σ(i) represents the index of the ground truth target corresponding to the i-th prediction, as follows:

[0047]

[0048] where represents the probability that the σ(i)-th object is predicted as class c i category, b i represents the ground truth target box, represents the predicted target box; the finally obtained loss function is:

[0049]

[0050] where y and respectively represent the class ground truth and the predicted value, is the bounding box loss function:

[0051]

[0052] Among them, is the GIoU loss function;

[0053] S33. Update the network parameters by minimizing the loss function to obtain an optimal model.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] The cervical cell detection and recognition method based on the attention mechanism and the detection transformer provided by the embodiment of the present invention can screen cell smears based on this method, quickly and accurately locate the lesion area of cervical cell images, and realize automatic auxiliary diagnosis in cervical cancer screening cytological examinations; it can well solve the low accuracy and limitations of manual film reading, with lower cost, less workload, and higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a structural diagram of a classic two-stage detection framework for cervical cell detection methods in the prior art.

[0057] Figure 2 is a flowchart of the cervical cell detection and recognition method based on the attention mechanism and the detection transformer provided by the embodiment of the present invention.

[0058] Figure 3 is a schematic diagram of the cervical cell detection and recognition method based on the attention mechanism and the detection transformer provided by the embodiment of the present invention.

[0059] Figure 4 is a structural diagram of a residual network block with a self-attention mechanism added provided by the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0060] In order to make the technical means, creative features, achieved purposes and effects of the present invention easy to understand, the present invention will be further described below in conjunction with specific embodiments.

[0061] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "upper", "lower", "inner", "outer", "front end", "rear end", "both ends", "one end", "the other end", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance.

[0062] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, terms such as "installation", "equipped with", "connection", etc. should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, and can be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0063] Referring to Figure 2 As shown, the embodiment of the present invention provides a cervical cell detection and recognition method based on an attention mechanism and a detection transformer, including:

[0064] S1. Input multiple cervical cancer cell source images into a preset convolutional network to extract image features; the preset convolutional network uses a 50-layer residual network combined with a self-attention mechanism as the backbone network;

[0065] S2. Perform spatial position encoding on the source image; reduce the dimension of the extracted image features, and combine the spatial position encoding to perform transformer processing;

[0066] S3. Input the processing result of the transformer into two feed-forward neural networks to predict the detection box and category respectively, and calculate the loss function accordingly; update the network parameters by minimizing the loss function to obtain an optimal model;

[0067] S4. Based on the optimal model, detect the cervical cell source image to be detected and recognized, and realize the positioning and classification of cervical abnormal cells.

[0068] In the embodiment of the present invention, this method combines a self-attention mechanism in the backbone network to perform target detection and recognition on various cells in the cervical cell image, without directly predicting the border coordinates and categories, realizing an end-to-end detection network; when using a transformer for target detection, it can enhance the capture of the feature dependence relationship of the more concerned parts and establish a more effective long-distance dependence. In addition, end-to-end automatic training and learning simplify the process of the target detection algorithm and require less prior content. For example, in actual implementation, the method provided by the embodiment of the present invention can be integrated into a computer-aided film reading system to screen cell smears, which can well solve the low accuracy and limitations of manual film reading, with lower cost, less workload, and higher accuracy.

[0069] The technical solution of the present invention will be described in more detail below.

[0070] In view of the deficiencies of the existing framework, the present invention proposes a cervical cell detection and recognition method based on the attention mechanism and the detection transformer, which can quickly and accurately locate the lesion area of cervical cell images, realize automatic auxiliary diagnosis in cervical cancer screening cytological examination, and proposes a standard detection transformer structure as the network infrastructure. At the same time, the self-attention mechanism is combined in the backbone network to detect and recognize various types of cells in cervical cell images, without directly predicting the bounding box coordinates and categories, and realizing an end-to-end detection network. The algorithm flow of the present invention is as Figure 3 shown, and the implementation steps are as follows:

[0071] 1. Feature extraction

[0072] First, multiple cervical cancer cell source images are obtained and fed into a convolutional network. The convolutional network uses a 50-layer residual network (Resnet-50) combined with the self-attention mechanism as the backbone network to extract image features. The settings of the 50-layer residual network are as follows: First, a convolution operation is performed on the input, and then 4 residual blocks are included. The specific settings are as shown in the following table:

[0073] Table 1 Parameter settings of the 50-layer residual network

[0074]

[0075] And the residual blocks in the 50-layer residual network are modified to add deformable convolution. Different from standard convolution, deformable convolution adds an operation of learning the offset amount of the input feature map through a convolutional layer, and then obtains the output feature map through bilinear interpolation. This is mainly because the abnormal cells have irregular shapes, and deformable convolution is added to improve the modeling ability of shape transformation and adapt to abnormal cells of different shapes. In addition, four self-attention mechanisms are also added, and the attention scores of the four attention mechanisms are fused to obtain the final attention score. The self-attention mechanism can model the relationship between pixels, thereby establishing more effective long-range dependencies.

[0076] As Figure 4 shown, it is the structural diagram of the residual network block with the self-attention mechanism added. In the figure, the transformer attention mechanism is the self-attention block with multiple attention heads:

[0077]

[0078] Among them, M represents the total number of attention heads; m represents the attention head index; Ω q represents the support key area for the specified query; A m (q, k, z q , x k ) represents the attention weight of the m-th attention head, where z q represents the query content, x k represents the key content, and q is zq The index of, k is x k The index of; W m And W' m Respectively represent learnable weights; ⊙ represents element-wise multiplication at corresponding positions, and the attention weight normalization is restricted to:

[0079]

[0080] Where:

[0081]

[0082] In the formula ∈ j There are four in total, ∈ 1 Depends on the similarity between the query and the key content:

[0083]

[0084] Among them, U m And Respectively are the learnable embedding matrices of the query and the key content.

[0085] ∈ 2 Depends on the content and relative position of the query:

[0086]

[0087] Among them, R k-q Calculates the sine and cosine functions of different wavelengths, projects k-q into a high-dimensional space, thereby encoding the relative position, The learnable embedding matrix for encoding the relative position R k-q ;

[0088] ∈ 3 Depends on the content of the key value:

[0089]

[0090] Among them, u m Is a learnable vector.

[0091] ∈ 4 Depends on the global position deviation between the key value and the query:

[0092]

[0093] Among them, v m Is a learnable vector.

[0094] In this step, in order to quickly and accurately locate the lesion area in cervical cell images and achieve automatic auxiliary diagnosis in cervical cancer screening cytology, it is proposed to use a detection transformer as the network infrastructure, combine the self-attention mechanism in the backbone network, and perform object detection on various types of cells in cervical cell images.

[0095] 2. Perform the transformer

[0096] Next, spatial position encoding is performed on multiple cervical cancer cell source images, which is:

[0097]

[0098]

[0099] where m is the channel subscript, pos is the position subscript, and d model is the dimension of the feature. Then, the dimensionality of the image features obtained from the feature network is reduced, combined with the spatial position encoding, and fed into the transformation encoder. The transformation encoder mainly consists of a multi-head self-attention module and a feed-forward neural network. First, a weighted feature vector is obtained through the self-attention module:

[0100]

[0101] where Q, K, and V represent the query, key, and value contents respectively, and d k represents the dimension of the key vector.

[0102] After obtaining the feature vector, it is fed into the feed-forward neural network for spatial transformation, which includes two linear transformation layers, and the intermediate activation function is the rectified linear unit function, to obtain the features encoded for N objects.

[0103] Combined with the position encoding obtained from the original image, as well as the output of the encoder, it is fed into the transformation decoder for decoding. The decoder mainly consists of three modules: a self-attention module, a traditional attention module, and a feed-forward network. The self-attention module and the feed-forward network are the same as those described above. For the traditional attention module part, the query comes from the previous output of the decoder, and the key-value comes from the output of the encoder and the position encoding. The specific calculation method is the same as that of the self-attention module.

[0104] 3. Detection network

[0105] The obtained results are input into two feedforward neural networks to predict the detection boxes and classes respectively, and the loss function is calculated accordingly for the training of the next iteration. The feedforward neural network consists of a three-layer perceptron with a rectified linear unit activation function and a d-dimensional hidden layer and a linear layer. The feedforward neural network predicts the normalized center coordinates, height, and width of the boxes, and then the linear layer uses the SoftMax function to predict the class labels. For N (a fixed value, set to 100) predictions, the number of ground truth targets corresponding to them needs to be less than N, and the ground truth boxes are matched with the predicted boxes using bipartite matching. The matching strategy is as follows:

[0106]

[0107] N represents the number of predicted targets; y i represents the ground truth target, represents the predicted target, σ represents the correspondence from the predicted value to the ground truth, and σ(i) represents the index of the ground truth target corresponding to the i-th prediction, as follows:

[0108]

[0109] where is the probability that the σ(i)-th object is predicted as class c i is the ground truth target box, i is the predicted target box. The finally obtained loss function is:

[0110] where y and

[0111]

[0112] respectively represent the class ground truth and predicted value, is the bounding box loss function:

[0113]

[0114] where is the GIoU loss function.

[0115] The network calculates the error through the loss function based on the obtained predicted detection boxes and classes, solves and evaluates the network model, and updates the network parameters by minimizing the loss function to obtain the optimal model.

[0116] The finally obtained network model is used for detection, and the final output is a series of sets, each set containing the coordinate boxes and class information of the targets, thus realizing the localization and classification of cervical abnormal cells.

[0117] An embodiment of the present invention proposes a cervical cell detection and recognition method based on an attention mechanism and a detection transformer. While using the transformer for object detection, the self-attention mechanism is combined in the stage of extracting image features, which can enhance the capture of the feature dependence of the more concerned parts and establish more effective long-range dependencies. In addition, end-to-end automatic training and learning simplify the process of the object detection algorithm and require less prior content. Using the computer-aided film reading system combined with the present invention to screen cell smears can well solve the low accuracy and limitations of manual film reading, with lower costs, less workload, and higher accuracy. In summary, the invention has broad application prospects and great commercial value.

[0118] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification only illustrates the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for cervical cell detection and recognition based on an attention mechanism and a detection transformer, characterized in that, it includes: S1. Input multiple cervical cancer cell source images into a preset convolutional network to extract image features; The preset convolutional network uses a 50-layer residual network combined with a self-attention mechanism as the backbone network; S2. Perform spatial position encoding on the source images; reduce the dimension of the extracted image features, and combine the spatial position encoding to perform transformer processing; S3. Input the processing result of the transformer into two feed-forward neural networks to predict the detection box and category respectively, and calculate the loss function accordingly; update the network parameters by minimizing the loss function to obtain the optimal model; S4. Based on the optimal model, detect the cervical cell source image to be detected and recognized, and realize the localization and classification of cervical abnormal cells.

2. The method for cervical cell detection and recognition based on an attention mechanism and a detection transformer according to claim 1, characterized in that, The 50-layer residual network combined with a self-attention mechanism includes: 5 convolutional blocks and 4 residual blocks; the step S1 includes: S11. Add four self-attention mechanisms to the residual blocks in the 50-layer residual network; S12. Modify the residual blocks in the 50-layer residual network and add deformable convolutions; S13. Input multiple cervical cancer cell source images into the 50-layer residual network for convolution operations; an operation of learning the offset of the input feature map through a convolutional layer is added to the deformable convolution in step S12, and the output feature map is obtained through bilinear interpolation; S14. Fuse the attention scores calculated by the four attention mechanisms in step S11 to obtain the final attention score; S15. Fuse the output feature map and the attention score to complete the extraction of image features.

3. The method for cervical cell detection and recognition based on an attention mechanism and a detection transformer according to claim 2, characterized in that, In the step S11, adding four self-attention mechanisms to the residual blocks in the 50-layer residual network includes: The residual network block with a self-attention mechanism added is expressed as: (1) In the formula, M represents the total number of attention heads; m represents the attention head index; Ω q represents the support key area of the specified query; A m (q, k, z q , x k ) represents the attention weight of the m-th attention head, where z q represents the query content, x k represents the key content, q is the index of z q and k is the index of x k ; W m and W′ m represent learnable weights respectively; ⊙ represents element-wise multiplication at corresponding positions, and the attention weight normalization is restricted to: In formula (2), In formula (3), ∈ j There are four in total, ∈ 1 represents the similarity between the query and the key content; ∈ 2 represents the content and relative position of the query; ∈ 3 represents the content of the key; ∈ 4 represents the global position deviation between the key and the query.

4. The method for cervical cell detection and recognition based on an attention mechanism and a detection transformer according to claim 1, characterized in that, The step S2 includes: S21. Perform spatial position encoding on the source images; S22. Reduce the dimension of the extracted image features, and combine the spatial position encoding to input into the transformation encoder to obtain the encoder output; S23. Combine the spatial position encoding and the encoder output to input into the transformation decoder for decoding to obtain the decoder output.

5. The method for cervical cell detection and recognition based on an attention mechanism and a detection transformer according to claim 4, characterized in that, The step S21 includes: Performing spatial position encoding on the source images is: Among them, m represents the channel subscript, pos represents the position subscript, and d model represents the dimension of the feature.

6. The method for cervical cell detection and recognition based on an attention mechanism and a detection transformer according to claim 4, characterized in that, The transformation encoder in the step S22 consists of a multi-head self-attention module and a feed-forward neural network; The step S22 includes: Dimensionality reduction is performed on the extracted image features, and combined with the spatial position encoding, a weighted feature vector is obtained through a self-attention module: Among them, Q, K, and V respectively represent query, key, and value content, and d k represents the dimension of the key vector; After obtaining the feature vector, it is fed into a feed-forward neural network for spatial transformation to obtain the features encoded for N objects.

7. The cervical cell detection and recognition method based on the attention mechanism and the detection transformer according to claim 1, wherein, the feed-forward neural network in the step S3 is composed of a three-layer perceptron with a rectified linear unit activation function and a d-dimensional hidden layer and a linear layer; the step S3 includes: S31. Input the processing result of the transformer into two feed-forward neural networks to output the normalized central coordinates, height, and width of the prediction box; the linear layer uses the SoftMax function to predict the class label; S32. Use binary matching to match the ground truth box with the prediction box, and the matching strategy is as follows: N represents the number of predicted targets; y i represents the true target, represents the predicted target, σ represents the correspondence between the predicted value and the true value, and σ(i) represents the true target index corresponding to the i-th prediction, as follows: Among them indicates that the σ(i)-th object is predicted to be of class c i probability, and b i represents the true target box, represents the predicted target box; the finally obtained loss function is: where y and represent the class true value and the predicted value respectively, is the bounding box loss function: Among them, is the GIoU loss function; S33. Update the network parameters by minimizing the loss function to obtain an optimal model.

Images