Cervical liquid-based cell detection method based on feature fusion and storage medium

Abstract

The present application relates to the technical field of image detection, and particularly relates to a cervical liquid-based cell detection method based on feature fusion and a storage medium, which comprises the following steps: S1, extracting coarse contour features from an input image and obtaining a coarse contour image through segmentation; S2, performing morphological feature extraction on the coarse contour image, constructing a description vector, and extracting image extraction features through coarse contour feature extraction; S3, fusing the image extraction features and the description vector to obtain a fusion vector; and S4, performing prediction based on the fusion vector to obtain the cell position and cell label of each cell to be detected. The beneficial effect is that, by adding morphological features related to the morphological features of cells in the detection process, and fusing the traditional image features to construct a fusion vector, and then performing detection based on the fusion vector, the artificial intelligence model can directly train and detect according to the morphological features related to the cells in the detection process, thereby improving the detection efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image detection, in particular to a cervical liquid-based cell detection method based on feature fusion and a storage medium. BACKGROUND

[0002] Thinprep cytologic test (TCT) is a cervical cytology examination technology, which detects cervical cells by using a liquid-based thin-layer cell detection system and performs cytological classification diagnosis. Compared with the traditional cervical scraping Papanicolaou smear examination, it significantly improves the specimen satisfaction and the detection rate of cervical abnormal cells. At the same time, it can also find some precancerous lesions, microbial infections such as mold, trichomonas, virus, chlamydia, etc.

[0003] In the prior art, in order to improve the efficiency of doctors in cell testing, there are technical solutions for image processing, detection, classification by introducing artificial intelligence models, and then adding labels to assist doctors in testing. For example, Chinese patent CN202010553018.X discloses a cervical cell automatic classification model establishment and a cervical cell automatic classification method. After processing the image, the VGG16 model is used to detect the cells in the image and classify the image based on the pre-trained model.

[0004] However, in actual implementation, the inventors found that this kind of technical solution, which is based on traditional artificial intelligence models for image feature extraction, model training and actual detection, only focuses on the rules of image features that can be extracted by each convolutional layer, and the overall training and detection process has relatively weak interpretability, which requires repeated training and verification to determine that the model can extract correct image features and perform detection. SUMMARY

[0005] In view of the above problems existing in the prior art, a cervical liquid-based cell detection method based on feature fusion is provided. On the other hand, a storage medium having computer instructions corresponding to the cervical liquid-based cell detection method is also provided.

[0006] The specific technical solutions are as follows:

[0007] A cervical liquid-based cell detection method based on feature fusion, comprising:

[0008] Step S1: obtaining an input image, extracting a rough contour feature from the input image, and segmenting to obtain a rough contour image;

[0009] Step S2: performing morphological feature extraction on the rough contour image, constructing a description vector, and further extracting the rough contour feature to obtain an image extraction feature;

[0010] Step S3: fusing the image extracted features and the description vector to obtain a fusion vector;

[0011] Step S4: predicting based on the fusion vector to obtain the cell location and cell label of each cell to be detected.

[0012] In another aspect, the step S1 comprises:

[0013] Step S11: obtaining the input image, and performing multiple feature extractions on the input image to obtain feature map information;

[0014] Step S12: constructing multiple candidate boxes in the input image according to the feature map information;

[0015] Step S13: classifying the candidate boxes to obtain the rough contour features, and segmenting the input image according to the candidate boxes corresponding to the rough contour features to obtain the rough contour image.

[0016] In another aspect, in the step S2, the method for constructing the description vector comprises:

[0017] Step A21: performing cell contour segmentation and cell nucleus contour segmentation on the rough contour image to obtain a cell contour image and a cell nucleus image;

[0018] Step A22: performing morphological feature extraction on the cell contour image and the cell nucleus image to obtain multiple morphological features, and constructing the description vector based on the morphological features;

[0019] The morphological features include cytoplasm color, nuclear membrane regularity, nuclear-cytoplasm ratio, chromatin coarse particles in the nucleus, and nuclear spacing.

[0020] The nuclear membrane regularity includes the mean of the distance from the nucleus center to the nuclear membrane edge, the standard deviation of the distance from the nucleus center to the nuclear membrane edge, and the maximum curvature of the nuclear membrane.

[0021] In another aspect, in the step S2, the method for constructing the image extracted features comprises:

[0022] Step B21: segmenting the rough contour features to obtain multiple candidate units;

[0023] Step B22: performing bilinear interpolation on each candidate unit to calculate the center pixel value corresponding to the candidate unit;

[0024] Step B23: constructing the image extracted features according to the center pixel value.

[0025] In another aspect, the step S3 comprises:

[0026] Step S31: concatenating the image features and the description vectors to obtain a combined vector;

[0027] Step S32: predicting dimension weights corresponding to each feature dimension in the combined vector according to the combined vector;

[0028] Step S33: calculating the fusion vector by weighting according to the dimension weights, the feature dimensions, and the combined vector.

[0029] In another aspect, in the step S31, the combined vector is formed by sequentially combining a group of image features and two groups of description vectors.

[0030] In another aspect, in the step S32, the dimension weights are calculated by using a weight calculation model, and the weight calculation model includes:

[0031] a first full connection layer, which receives the combined vector and predicts first weight features;

[0032] a noise linear rectification module, which is connected to the first full connection layer, and processes the first weight features to obtain second weight features;

[0033] a second full connection layer, which is connected to the noise linear rectification module, and processes the second weight features to obtain third weight features;

[0034] an activation function, which is connected to the second full connection layer, and re-maps the third weight features to obtain the dimension weights;

[0035] The number of the first weight features, the second weight features, the third weight features, and the dimension weights corresponds to the number of the feature dimensions.

[0036] In another aspect, the step S33 includes:

[0037] Step S331: calculating a plurality of weighted features according to the dimension weights and each feature dimension;

[0038] Step S332: concatenating the weighted features and the combined vector to obtain concatenated features;

[0039] Step S333: inputting the concatenated features into a full connection layer to obtain the fusion vector.

[0040] In another aspect, the step S4 includes:

[0041] Step S41: predicting a plurality of cell regions corresponding to the cell to be detected based on the fusion vector;

[0042] Step S42: classifying the cell regions to obtain the cell label, and then outputting the cell to be detected.

[0043] A storage medium, the storage medium stores computer instructions, the computer instructions are suitable for executing on a computer device, characterized in that when the computer device executes the computer instructions, the cervical liquid-based cell detection method described above is executed.

[0044] The above technical solution has the following advantages or beneficial effects:

[0045] In the prior art, the training and detection method of the artificial intelligence model only relies on image features, which needs to be repeatedly verified to realize the detection process relying on image features. In the present scheme, morphological features related to the morphological features of the cells are added in the detection process, and the traditional image features are fused to construct a fusion vector, and then the detection is carried out based on the fusion vector. The artificial intelligence model can directly train and detect according to the morphological features related to the cells in the detection process, thereby improving the detection efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0046] Reference is made to the accompanying drawings to more fully describe embodiments of the present application. However, the accompanying drawings are only used for illustration and explanation, and do not constitute a limitation on the scope of the present application.

[0047] Figure 1 is a whole schematic diagram of the embodiment of the present application;

[0048] Figure 2 is a schematic diagram of step S1 in the embodiment of the present application;

[0049] Figure 3 is a schematic diagram of the description vector construction process in the embodiment of the present application;

[0050] Figure 4 is a contour extraction schematic diagram in the embodiment of the present application.

[0051] Figure 5 is an image feature extraction schematic diagram in the embodiment of the present application;

[0052] Figure 6 is a schematic diagram of step S3 in the embodiment of the present application;

[0053] Figure 7 is a weight calculation model schematic diagram in the embodiment of the present application;

[0054] Figure 8 is a schematic diagram of step S33 in the embodiment of the present application;

[0055] Figure 9 For sub-step S4 in step S4 in the embodiment of the present application. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0057] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0058] The present application will be further described below with reference to the drawings and specific embodiments, but is not limited to the present application.

[0059] The present application includes:

[0060] A cervical liquid-based cell detection method based on feature fusion, as shown in Figure 1 includes:

[0061] Step S1: obtaining an input image, extracting a rough contour feature from the input image, and segmenting to obtain a rough contour image;

[0062] Step S2: performing morphological feature extraction on the rough contour image, constructing a description vector, and further extracting the rough contour feature to obtain image extraction features;

[0063] Step S3: fusing the image extraction features and the description vector to obtain a fusion vector;

[0064] Step S4: predicting based on the fusion vector to obtain the cell position and cell label of each cell to be detected.

[0065] Specifically, the training and detection method of the artificial intelligence model in the prior art only relies on image features, which needs to be repeatedly verified to realize the detection process relying on image features. In the embodiment, after rough contour feature extraction and image segmentation are performed on the input image, morphological feature extraction related to morphological features is performed respectively to obtain multiple feature dimensions for describing cell information, and then a description vector is assembled, and image extraction features are obtained by using a traditional neural network extraction method. When the description vector and the image extraction features are obtained, the fusion vector is obtained by fusing them. Then, the fusion vector is predicted to achieve better detection effect.

[0066] In the implementation process, the cervical liquid-based cell detection method described above is set in a computer device as a software embodiment, is used for processing an input image, detecting a cell position of each cell to be detected in the input image, and classifying a cell label obtained. The input image refers to an image obtained by scanning a cervical liquid-based slice based on a digital pathology image scanning system. The slice is processed by sampling, slicing, and staining according to an existing slice manufacturing process. The input image contains a plurality of cells to be detected, which are classified in different diagnostic methods to represent different disease processes. The coarse contour feature refers to a series of candidate boxes obtained by detecting and predicting a region that may contain a cell to be detected in the input image, which is used to roughly mark the region of the cell to be detected. The morphological feature refers to a plurality of features directly related to the morphology of the cell, which is used as each dimension of the description vector to assemble the description vector. The image extraction feature is a high-dimensional feature of the image extracted by convolution and other operations of the neural network.

[0067] In one embodiment, as shown in Figure 2 Step S1 includes:

[0068] Step S11: Obtain the input image, and perform multiple feature extractions on the input image to obtain feature map information;

[0069] Step S12: Construct a plurality of candidate boxes in the input image according to the feature map information;

[0070] Step S13: Classify the candidate boxes to obtain the coarse contour feature, and segment the input image according to the candidate box corresponding to the coarse contour feature to obtain the coarse contour image.

[0071] Specifically, to achieve better detection effect, in the embodiment, an extraction process similar to the RPN network structure is introduced. When the input image is obtained, a pre-trained convolutional network can be used to perform multiple feature extractions on the input image to obtain feature map information. Then, based on the feature map information, a plurality of candidate boxes of different sizes are constructed in the region candidate network, and the contents in the candidate boxes are classified to determine whether the cells to be detected exist in the candidate boxes, and then the coarse contour feature is obtained. After determining whether the cells to be detected exist in the candidate boxes, the input image can be segmented based on the candidate box corresponding to the classification result, and the segmented image is taken as the coarse contour image.

[0072] In one embodiment, in step S2, as shown in Figure 3 The method of constructing the description vector includes:

[0073] Step A21: performing cell contour segmentation and cell nucleus contour segmentation on the coarse contour image to obtain a cell contour image and a cell nucleus image;

[0074] Step A22: morphological feature extraction is performed on the cell contour image and the nucleus image to obtain a plurality of morphological features, and a description vector is constructed based on the morphological features;

[0075] The morphological features include cytoplasm color, nuclear membrane regularity, nucleus-plasma ratio, chromatin coarse particles in the nucleus, and nuclear spacing.

[0076] The nuclear membrane regularity includes the mean value of the distance from the nucleus center to the nuclear membrane edge, the standard deviation of the distance from the nucleus center to the nuclear membrane edge, and the maximum curvature of the nuclear membrane.

[0077] Specifically, for the artificial intelligence detection method in the prior art, which only relies on the image features extracted by the convolution layer for detection, resulting in unstable detection effect, in the embodiment, the cell contour image and the nucleus image are segmented by the method of pre-segmenting the coarse contour features based on image processing, and then morphological processing is performed on the cell contour image and the nucleus image to construct a description vector for describing the cell to be detected, so that the subsequent model prediction process can pay more attention to the changes of the cell itself, and the prediction accuracy is improved.

[0078] Further, in the process of morphological description of the cell, the prior art can obtain a large number of features, including color, texture, image features of color, image features of texture, area, aspect ratio, and perimeter of the cell. If too many feature dimensions are set in the process of constructing the description vector, the model may be over-fitted in the subsequent model training and prediction process, resulting in a decrease in detection accuracy. To solve this problem, the inventors have made a large number of regression studies to judge the influence of each feature on the prediction result, and have combined the experience and knowledge of senior pathologists to determine the above-mentioned several features which have strong explanatory power and high correlation. Figure 4 As shown in FIG. 1, after the input image is segmented and classified by the candidate box, a coarse contour image of the input image segmented by the bounding box 101 is obtained, which contains the cell to be detected. Then, the contour of the coarse contour image is extracted to obtain a cell contour image 102 and a nucleus image 103. A plurality of sampling points are selected on the nucleus image 103, and the minimum distance from each sampling point to the cell contour image 102 is calculated as the nuclear membrane edge distance. In feature calculation, the cytoplasm color is divided into red and blue, corresponding to 0 and 1; the nuclear membrane regularity is represented by the mean value of the distance from the nucleus center to the nuclear membrane edge, the standard deviation of the distance from the nucleus center to the nuclear membrane edge, and the maximum curvature of the nuclear membrane; the nucleus-plasma ratio is the ratio of the area of the nucleus to the area of the cell; the coarse particles refer to the mean value of the pixel value in the nucleus; the nuclear spacing refers to the mean value and the standard deviation of the distance from the center of the current cell to the centers of other cells in the patch, and if there is only one cell, the values are set to 0 and 0. Based on the above processing, a better morphological representation of the cell is achieved.

[0079] In one embodiment, as shown in step S2, Figure 5 the method for constructing the image extraction feature includes:

[0080] Step B21: segmenting the rough contour feature to obtain a plurality of candidate units;

[0081] Step B22: performing bilinear interpolation on each candidate unit to calculate a center pixel value corresponding to the candidate unit;

[0082] Step B23: constructing the image extraction feature according to the center pixel value.

[0083] Specifically, to achieve better feature extraction effect, in the embodiment, a method similar to ROI Align is introduced for image feature extraction. When the rough contour feature is received, the rough contour feature is segmented by a corresponding window size to obtain a plurality of candidate units, and each candidate unit does not overlap with each other. Then, the candidate units are sampled and processed based on the bilinear interpolation method to calculate the center pixel value of the candidate units. Based on the plurality of center pixel values, the original input rough contour feature can be back propagated and mapped to the original image, while correcting the coordinate offset generated in the sampling process, to realize the construction of the image extraction feature.

[0084] In one embodiment, as shown in step S3, Figure 6 includes:

[0085] Step S31: splicing the image extraction feature and the description vector to obtain a combined vector;

[0086] Step S32: predicting according to the combined vector to obtain a dimension weight corresponding to each feature dimension in the combined vector;

[0087] Step S33: weighted calculation of the fusion vector according to the dimension weight, the feature dimension and the combined vector.

[0088] Specifically, to achieve better prediction effect in the subsequent prediction process, in the embodiment, the image extraction feature and the description vector are spliced as a plurality of dimensions of the combined vector in advance to obtain the combined vector, and then the dimension weight of each feature dimension of the combined vector is calculated and weighted. At the same time, to better preserve the original feature information, the combined vector without weighting processing is also introduced in the construction of the fusion vector. Through the above settings, the model can achieve better prediction effect.

[0089] Further, in step S31, the combined vector is formed by sequentially combining the image extraction features and the two sets of description vectors. In the implementation process, it is embodied in the form of "combined vector = image extraction features + description vector + description vector". Through the above combination form, the weight of the description vector associated with the morphological features in the combined vector can be increased, so that the model can pay more attention to the morphological features of the cells.

[0090] In one embodiment, in step S32, a weight calculation model is used to calculate the dimension weight, as shown in the following formula: Figure 7 The weight calculation model includes:

[0091] A first fully connected layer C1, which receives the combined vector and predicts a first weight feature;

[0092] A noise linear rectification module C2, which is connected to the first fully connected layer C1, and processes the first weight feature to obtain a second weight feature;

[0093] A second fully connected layer C3, which is connected to the noise linear rectification module C2, and processes the second weight feature to obtain a third weight feature;

[0094] An activation function C4, which is connected to the second fully connected layer C3, and re-maps the third weight feature to obtain the dimension weight;

[0095] The number of the first weight feature, the second weight feature, the third weight feature, and the dimension weight corresponds to the feature dimension.

[0096] Specifically, to achieve better determination effect of the weight of each feature dimension, in the embodiment, the weight calculation model is designed to sequentially map the combined vector in the weight space by the first fully connected layer C1, the noise linear rectification module C2, the second fully connected layer C2, and the activation function C4, and to fit between the value domain of the fully connected layer and the target domain by the noise linear rectification module C2 and the activation function C4. To achieve better fitting effect, the noise linear rectification module C2 is implemented by using the ReLU function, and the activation function C4 is implemented by using the sigmoid function.

[0097] In one embodiment, as shown in the following formula: Figure 8 Step S33 includes:

[0098] Step S331: calculating a plurality of weighted features according to the dimension weight and each feature dimension;

[0099] Step S332: splicing the weighted features with the combined vector to obtain spliced features;

[0100] Step S333: inputting the spliced features into a full connection layer to obtain a fusion vector.

[0101] Specifically, to achieve a better fusion effect, in the embodiment, after obtaining the dimension weight, the feature dimension is calculated based on the dimension weight in advance to obtain a weighted feature. Then, the weighted feature is spliced with the combination vector to obtain spliced features, which include the feature dimension with the dimension weight and the feature dimension without the dimension weight. Then, the spliced features are input into a full connection layer to obtain a fusion vector, thereby achieving a better fusion effect.

[0102] In one embodiment, as shown in FIG. 4, step S4 includes: Figure 9

[0103] Step S41: predicting a plurality of cell regions corresponding to the cells to be detected based on the fusion vector;

[0104] Step S42: classifying the cell regions to obtain cell labels, and then outputting the cells to be detected.

[0105] Specifically, to achieve a better prediction effect, in the embodiment, after obtaining the fusion vector constructed by the above process, the input image is predicted by a full connection layer based on the fusion vector, thereby predicting the cell regions associated with the cells to be detected. Then, the intercepted cell regions are classified based on a classifier to generate cell labels corresponding to each cell to be detected, the cell labels being pre-designed labels corresponding to relevant medical requirements, used to represent the possible state of the cell. The cell labels are combined with the images corresponding to the cell regions, i.e., the cells to be detected are obtained and output.

[0106] A storage medium, the storage medium storing computer instructions, the computer instructions being suitable for being executed on a computer device, when the computer device executes the computer instructions, the cervical liquid-based cell detection method described above is executed.

[0107] The above is only the preferred embodiment of the present application, and does not limit the implementation and protection scope of the present application. It should be realized by those skilled in the art that any equivalent replacement and obvious change made according to the description and drawings of the present application should be included in the protection scope of the present application.​

Claims

1. A method for detecting cervical liquid-based cells based on feature fusion, characterized in that, The method comprises the following steps: Step S1: obtaining an input image, extracting a rough contour feature from the input image, and segmenting to obtain a rough contour image; Step S2: performing morphological feature extraction on the rough contour image, constructing a description vector, and further extracting the rough contour feature to obtain an image extraction feature; Step S3: fusing the image extraction feature and the description vector to obtain a fusion vector; Step S4: predicting based on the fusion vector to obtain the cell location and cell label of each cell to be detected; The method for constructing the description vector comprises: Step A21: performing cell contour segmentation and cell nucleus contour segmentation on the rough contour image to obtain a cell contour image and a cell nucleus image; Step A22: performing morphological feature extraction on the cell contour image and the cell nucleus image to obtain a plurality of morphological features, and constructing the description vector based on the morphological features; The morphological features comprise cytoplasm color, nuclear membrane regularity, nuclear-cytoplasm ratio, chromatin coarse particles in the nucleus, and nuclear spacing; The nuclear membrane regularity comprises the mean value of the distance from the nuclear center to the nuclear membrane edge, the standard deviation of the distance from the nuclear center to the nuclear membrane edge, and the maximum curvature of the nuclear membrane; Wherein: A plurality of sampling points are selected on the cell nucleus image, and the minimum distance from each sampling point to the cell contour image is calculated as the distance from the nuclear center to the nuclear membrane edge; The cytoplasm color is divided into red and blue, corresponding to 0 and 1.

2. The cervical liquid-based cytology test method according to claim 1, characterized by, The step S1 comprises: Step S11: obtaining the input image, and performing multiple feature extractions on the input image to obtain feature map information; Step S12: constructing a plurality of candidate boxes in the input image according to the feature map information; Step S13: classifying the candidate boxes to obtain the rough contour feature, and segmenting the input image according to the candidate box corresponding to the rough contour feature to obtain the rough contour image.

3. The cervical liquid-based cytology test method according to claim 1, characterized by, In the step S2, the method for constructing the image extraction feature comprises: Step B21: segmenting the rough contour feature to obtain a plurality of candidate units; Step B22: performing bilinear interpolation on each candidate unit to calculate the center pixel value corresponding to the candidate unit; Step B23: constructing the image extraction feature according to the center pixel value.

4. The cervical liquid-based cytology test method according to claim 1, characterized by, The step S3 comprises: Step S31: splicing the image extraction feature and the description vector to obtain a combined vector; Step S32: predicting according to the combined vector to obtain a dimension weight corresponding to each feature dimension in the combined vector; Step S33: performing weighted calculation according to the dimension weight, the feature dimension, and the combined vector to obtain the fusion vector.

5. The cervical liquid-based cytology test method according to claim 4, characterized by, In the step S31, the combined vector is formed by sequentially combining a group of image extraction features and two groups of description vectors.

6. The cervical liquid-based cytology test method according to claim 4, wherein, In the step S32, the dimension weight is calculated by using a weight calculation model, and the weight calculation model comprises: a first full connection layer, which receives the combined vector and predicts to obtain a first weight feature; a noise linear rectification module connected to the first full connection layer, the noise linear rectification module processing the first weight feature to obtain a second weight feature; a second full connection layer connected to the noise linear rectification module, the second full connection layer processing the second weight feature to obtain a third weight feature; an activation function connected to the second full connection layer, the activation function remapping the third weight feature to obtain the dimension weight; the number of the first weight feature, the second weight feature, the third weight feature, and the dimension weight corresponding to the feature dimension.

7. The cervical liquid-based cytology test method according to claim 4, wherein, The step S33 comprises: Step S331: calculating a plurality of weighted features according to the dimension weight and each feature dimension; Step S332: splicing the weighted features and the combination vector to obtain a spliced feature; Step S333: inputting the spliced feature into a full connection layer to obtain the fusion vector.

8. The method according to claim 1, wherein the method is a liquid-based cervical cytological test. The step S4 comprises: Step S41: predicting a plurality of cell regions corresponding to the cell to be detected based on the fusion vector; Step S42: classifying the cell regions to obtain the cell label, and then outputting the cell to be detected.

9. A storage medium having stored therein computer instructions adapted to be executed on a computer device, characterized in that, When the computer device executes the computer instructions, the cervical liquid-based cell detection method of any one of claims 1-8 is executed.

Images