Abnormal cell image recognition method and device in cervical cell image

Abstract

The present disclosure relates to a method and device for identifying abnormal cell images in cervical cell images. The method comprises: inputting a multi-scale cervical cell image into a pre-trained multi-scale fusion network, wherein the multi-scale cervical cell includes different sizes of the same cervical cell image; extracting key node features of the cervical cell image of each scale in the multi-scale cervical cell image through the multi-scale fusion network; determining the grading results of abnormal cell images in the cervical cell image corresponding to the key node features of the cervical cell image of each scale through the Berthesda grading system; fusing the grading results of abnormal cell images in the cervical cell image of each scale, and outputting the abnormal cell image in the cervical cell image and the abnormal cell image with the highest level in the grading results. The present method can identify and determine abnormal cell images in cervical cell images without manual work, and can grade abnormal cell images.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and in particular to a method and device for identifying abnormal cell images in cervical cell images. Background Art

[0002] Cervical cancer is a common gynecological malignancy, and its incidence ranks second among female malignant tumors, second only to breast cancer. Early detection and intervention treatment of cervical cancer can effectively prevent its occurrence and development. As the main method for detecting cervical cancer and precancerous lesions, cervical cytology can greatly reduce the incidence and mortality. Due to the lack of a large number of pathological cytology diagnostic doctors who have undergone rigorous training and rich experience, it is currently difficult for my country to meet the needs of conducting large-scale, high-level cervical cancer screening.

[0003] With the development of artificial intelligence technology and its widespread application in the field of medical imaging, a method has emerged to use computer vision technology to achieve computer-assisted automated diagnosis and screening of pathological cells, that is, abnormal cells, in cervical cells, which can greatly improve the detection rate of cervical precancerous lesions.

[0004] However, the current traditional method of screening abnormal cells in cervical cells through computer vision technology lacks an effective computer-assisted film reading system. After the abnormal cells are screened out, they still need to be manually identified through professional knowledge and fail to provide a good interpretable basis for the extraction results of abnormal cells. Summary of the invention

[0005] Based on this, it is necessary to provide a method and device for identifying abnormal cell images in cervical cell images, which does not require manual identification and determination of abnormal cell images in cervical cell images and can provide an explainable basis for the extraction results of abnormal cells, in order to address the above technical problems.

[0006] In a first aspect, the present disclosure provides a method for identifying abnormal cell images in cervical cell images, the method comprising:

[0007] Inputting a multi-scale cervical cell image into a pre-trained multi-scale fusion network, wherein the multi-scale cervical cell image includes different sizes of the same cervical cell image;

[0008] Extracting key node features of the cervical cell image at each scale in the multi-scale cervical cell image through a multi-scale fusion network;

[0009] The Berthesda grading system is used to determine the grading results of abnormal cell images in the cervical cell images corresponding to the key node features of the cervical cell images at each scale;

[0010] The grading results of abnormal cell images in the cervical cell images of each scale are fused, and the abnormal cell images in the cervical cell images and the abnormal cell images with the highest level in the grading results are output.

[0011] In one embodiment, extracting key node features of the cervical cell image at each scale in the multi-scale cervical cell image through a multi-scale fusion network includes:

[0012] Extracting features of a first-scale cervical cell image in the multi-scale cervical cell image through a residual network of a first-scale network in the multi-scale fusion network;

[0013] According to the features of the first-scale cervical cell image, key node features of the first-scale cervical cell image are obtained through the graph convolution pooling network of the first-scale network;

[0014] Determine an input image region of a second-scale cervical cell image in the multi-scale cervical cell image according to a region of interest of an image block corresponding to a key node feature of the first-scale cervical cell image;

[0015] Inputting the input image region into a second scale network of the multi-scale fusion network;

[0016] Extracting features of the input image region through a residual network of the second scale network;

[0017] According to the features of the input image region, key node features of the input image region are obtained through the graph convolution pooling network of the second scale network.

[0018] In one embodiment, the extracting the features of the first scale cervical cell image in the multi-scale cervical cell image by the residual network of the first scale network in the multi-scale fusion network also includes:

[0019] The first-scale cervical cell image is subjected to binary segmentation to remove a background area without cervical cells in the first-scale cervical cell image, and the first-scale cervical cell image after removing the background area without cervical cells is segmented into a plurality of image blocks of the same size without overlap.

[0020] In one embodiment, extracting features of the first scale cervical cell image in the multi-scale cervical cell image through the residual network of the first scale network in the multi-scale fusion network includes:

[0021] A plurality of image blocks of the same size without overlap are input into the residual network of the first scale network to obtain the features of each image block, the activation function of the residual network adopts a linear rectification function, and the residual network includes a convolutional layer and a residual module.

[0022] In one embodiment, the step of obtaining key node features of the first-scale cervical cell image according to the features of the first-scale cervical cell image through a graph convolution pooling network of the first-scale network includes:

[0023] According to the characteristics of each image block and the position information of each image block in the first-scale cervical cell image, a digital slice structure diagram of the first-scale cervical cell image is constructed to obtain node characteristics;

[0024] Calculating the L1 distance between the node features, constructing neighbor node features through the L1 distance, and obtaining a first adjacency matrix corresponding to the node features;

[0025] Updating the node feature representation through a graph convolutional network;

[0026] The key node features after updating the feature representation of the node are obtained through a single-layer neural network.

[0027] In one embodiment, updating the node feature representation through a graph convolutional network includes:

[0028] Sampling neighbor node features of each of the node features according to the first adjacency matrix;

[0029] The information of the neighbor node features obtained after sampling is aggregated according to an aggregation function to obtain the node features represented by a vector.

[0030] In one embodiment, sampling neighbor node features of each node feature according to the first adjacency matrix includes:

[0031] When the number of neighbor node features of the node feature is less than the preset node sampling number, sampling is performed using a sampling method with replacement;

[0032] When the number of neighbor node features of the node feature is greater than a preset number of node samplings, sampling is performed using a sampling method without replacement;

[0033] A first node feature of the preset node sampling quantity is obtained.

[0034] In one embodiment, aggregating the information of the neighbor node features according to an aggregation function includes:

[0035] Performing a linear transformation on the first node feature obtained after sampling by using a parameter-sharing linear transformation matrix;

[0036] Performing a self-attention operation on the first node features after the linear transformation to obtain weight values ​​between the first node features;

[0037] Sort the weight values ​​in descending order to obtain the first preset number of weight values ​​ranked first;

[0038] The first preset number of weight values ​​are normalized by a normalization function, and the normalized weight values ​​are used to fuse the corresponding first node features to obtain the node features represented by a vector.

[0039] In one embodiment, the key node features after updating the feature representation of the node through a single-layer neural network include:

[0040] Obtaining the attention weight value of the node feature represented by the vector through a self-attention operation;

[0041] Obtain the node feature represented by a vector corresponding to the attention weight value within a preset first range;

[0042] The node features represented by the vector are updated according to the attention weight value and the normalization function to obtain the key node features after the feature representation of the node is updated.

[0043] In one embodiment, the multi-scale fusion network is trained by the following methods:

[0044] Constructing a training data set and a test data set of a multi-scale fusion network, wherein the training data set and the test data set both include cervical cell images provided with annotation information, wherein the annotation information is determined by abnormal cells pre-annotated in the cervical cell images;

[0045] The multi-scale network is trained using the training data set and the test data set as well as a preset classification loss function to obtain a multi-scale fusion network.

[0046] In one embodiment, the calculation formula of the classification loss function is:

[0047]

[0048] in, is the cross entropy loss function of the first scale network; is the cross entropy loss function of the second scale network; the cross entropy loss function of the first scale network and the cross entropy loss function of the second scale network are:

[0049]

[0050] in, is the annotation information in the cervical cell image, K is the number of key node features, and y i * The predicted annotation information in the cervical cell image is calculated.

[0051] In one embodiment, the grading result of the abnormal cell image in the cervical cell image corresponding to the key node features of the cervical cell image at each scale determined by the Berthesda grading system includes:

[0052] The key node features of cervical cell images at each scale are graded according to the Berthesda grading system through the fully connected network of the multi-scale fusion network and the normalized exponential function, and the grading results of abnormal cell images in the cervical cell images corresponding to the key node features of the cervical cell images at each scale are determined.

[0053] In a second aspect, the present disclosure also provides a device for identifying abnormal cell images in cervical cell images.

[0054] The device comprises:

[0055] An image input module, used for inputting a multi-scale cervical cell image into a pre-trained multi-scale fusion network, wherein the multi-scale cervical cell image includes different sizes of the same cervical cell image;

[0056] A key node feature extraction module, used to extract key node features of the cervical cell image of each scale in the multi-scale cervical cell image through a multi-scale fusion network;

[0057] A grading result determination module, used for determining the grading result of the abnormal cell image in the cervical cell image corresponding to the key node features of the cervical cell image at each scale through the Berthesda grading system;

[0058] The grading result fusion module is used to fuse the grading results of abnormal cell images in the cervical cell images of each scale, and output the cervical cell image area corresponding to the grading results of the abnormal cell images in the cervical cell images and the highest-level abnormal cell images.

[0059] In a third aspect, the present disclosure further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0060] In a fourth aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0061] In a fifth aspect, the present disclosure further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0062] The above-mentioned abnormal cell image recognition method and device in the cervical cell image can extract key node features from cervical cell images of different sizes through a multi-scale fusion network, and can determine the cell area in the cervical cell image, and then use the Berthesda grading system to determine the corresponding abnormal cell image and grading result in the cell area for the key node feature, and then fuse the grading results corresponding to cervical cell images of different scales, so as to more accurately determine the abnormal cell image and the corresponding grading result. Finally, the abnormal cell image in the cervical cell image and the abnormal cell image with the highest level in the grading result are output. Because the Berthesda grading system is used to identify the abnormal cell image and determine the grading result, it can provide an interpretable basis for the grading result of the determined abnormal cell image. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A schematic diagram of an application environment of a method for identifying abnormal cell images in cervical cell images in an embodiment;

[0064] Figure 2 A schematic diagram of a process of a method for identifying abnormal cell images in cervical cell images in one embodiment;

[0065] Figure 3 Schematic diagram of the process of step S204 in one embodiment;

[0066] Figure 4 A schematic diagram of the structure of a multi-scale fusion network in one embodiment;

[0067] Figure 5 Schematic diagram of the architecture of a residual network in one embodiment;

[0068] Figure 6 Schematic diagram of the process of step S304 in one embodiment;

[0069] Figure 7 A schematic diagram of a sampling process in an embodiment;

[0070] Figure 8 A schematic diagram of a polymerization process in an embodiment;

[0071] Fig. 9 A schematic diagram of weight value calculation in one embodiment;

[0072] Fig.10 This is a schematic diagram of the process of step S608 in one embodiment;

[0073] Fig.11 A schematic diagram of a self-attention operation in one embodiment;

[0074] Fig.12 It is a schematic block diagram of the structure of an abnormal cell image recognition device in a cervical cell image in one embodiment;

[0075] Fig.13 Schematic diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0077] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0078] At present, some methods have emerged for assisting the identification of abnormal cells in cervical cells, such as: an intelligent assisted film reading method, which relies on diagnostic physicians to browse the cells in the slice and check the pathological nature of the cells. If the suspicious degree of cell pathological changes reaches a certain threshold, a remote expert is requested to review and inspect. Although this method improves the efficiency of film reading to a certain extent, it requires greater manual intervention and cannot well release the resources of pathology experts. Another example: an artificial intelligence assisted film reading system for cervical cell liquid-based smears. This solution first segments cervical cells and then performs graded identification to obtain diagnostic results. Although this system is more efficient, the cell nucleus segmentation method it uses is the traditional active contour model. This model uses a strict algorithm, requires greater manual intervention and professional knowledge, and has limited applicable scenarios. In addition, this film reading system only provides cell annotation maps when there is a conflict in the grading results, allowing the film reading physician to decide the final grading result, and cannot provide a good basis for interpretability of each grading result.

[0079] In summary, currently, auxiliary identification of abnormal cervical cells requires professional knowledge and manual identification, and cannot provide an explainable basis for the identified abnormal cells.

[0080] Therefore, in order to solve the above problems, the present disclosure provides a method for identifying abnormal cell images in cervical cell images, which can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store cervical cell image data and other data that the server 104 needs to process or store. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 inputs the multi-scale cervical cell image that needs to identify abnormal cells into the multi-scale fusion network pre-trained in the server 104, wherein the multi-scale cervical cell image may include the same cervical cell image of different sizes that needs to identify abnormal cells. It is also possible that the server 104 obtains the multi-scale cervical cell image in the terminal 102 and inputs the multi-scale cervical cell image into the multi-scale fusion network therein. The server 104 extracts the key node features of the cervical cell image of each scale through the multi-scale fusion network therein. After extracting the key node features, the server 104 determines the grading result of the abnormal cell image in the corresponding cervical cell image in the extracted key node features through the Berthesda grading system. After determining the grading results of abnormal cells, the server 104 merges the grading results of abnormal cell images in the cervical cell images of each scale, and after fusion, the server 104 outputs the abnormal cell images in the cervical cell images that need to identify abnormal cells, and the abnormal cell images with the highest level in the grading results. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, etc. The server 104 can be implemented as an independent server or a server cluster composed of multiple servers. It should be noted that the solution can also be implemented by a terminal or a server alone.

[0081] In one embodiment, Figure 2 As shown, a method for identifying abnormal cell images in cervical cell images is provided, and the method is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0082] S202, inputting a multi-scale cervical cell image into a pre-trained multi-scale fusion network, wherein the multi-scale cervical cell image includes different sizes of the same cervical cell image.

[0083] The cervical cell image may generally be an image containing a plurality of cervical cells. The multi-scale fusion network may generally be a network capable of extracting abnormal cell images from the cervical cell image and determining a grading result for the abnormal cell image.

[0084] Specifically, the acquired multi-scale cervical cell image is input into the multi-scale fusion network pre-trained in the server 104. The multi-scale cervical cell image may include different sizes of the same cervical cell image, and those skilled in the art may select and set the size of the cervical cell image according to the specific application scenario, which is not limited in this embodiment.

[0085] In some embodiments, the sizes may include: 10×10 cervical cell images, 20×20 cervical cell images, and the like.

[0086] S204, extracting key node features of the cervical cell image at each scale in the multi-scale cervical cell image through a multi-scale fusion network.

[0087] Among them, the key node features can usually be features corresponding to the key nodes that retain the cervical cell images of each scale. The key nodes can usually represent image blocks at more critical positions of the cervical cell images of each scale, and can usually be used to represent image blocks containing cell images in the cervical cell images.

[0088] Specifically, the cervical cell images of each scale in the multi-scale cervical cell images are input into the multi-scale fusion network, and the key node features in the cervical cell images of different scales are extracted through different sub-networks in the multi-scale fusion network to obtain the key node features of the cervical cell images of each scale.

[0089] S206, determining the grading result of the abnormal cell image in the cervical cell image corresponding to the key node features of the cervical cell image at each scale through the Berthesda grading system.

[0090] The Bethesda grading system may be a TBS (The Bethesda System) grading system. The grading results of abnormal cell images can be determined by the TBS grading system, and the grading results may include, from low to high importance: NILM (Negative for Intraepithelial Lesion or Malignancy) of cervical cells, no intraepithelial lesion cells or malignant cells, ASC-US (atypical squamous cells), ASC-H (atypical squamous cells), LSIL (low-grade squamous intraepithelial lesion), HSIL (high-grade squamous intraepithelial lesion), SCC (squamous cell carcinoma), etc.

[0091] Specifically, the key node features of each scale are obtained, and the abnormal cell image in the cervical cell image corresponding to the key node features of the cervical cell image of each scale is determined by the Berthesda grading system. Then, the grading result corresponding to the abnormal cell image is determined by the identified abnormal cell image and the Berthesda grading system.

[0092] S208, merging the grading results of the abnormal cell images in the cervical cell images of each scale, and outputting the abnormal cell images in the cervical cell images and the abnormal cell images with the highest level in the grading results.

[0093] Specifically, the grading results of abnormal cells in cervical cell images at each scale are obtained above, and the grading results of abnormal cells in cervical cell images at each scale are fused. By fusing the grading results at different scales and gradually focusing on the key area, a more accurate grading result can be obtained, and the grading results of abnormal cells in cervical cell images of different scales determined in each sub-network in the multi-scale fusion network are counted, and a distribution histogram is generated for the grading results, as well as the abnormal cell image corresponding to the highest level grading result at the largest scale.

[0094] In some embodiments, the sets of grading results of abnormal cell images in cervical cell images of each scale are respectively: Y1, Y2, and Y3, and Y1, Y2, and Y3 respectively include several grading results. Several grading results in Y1, Y2, and Y3 are fused, so as to gradually focus on the key area and obtain more accurate grading results. Each key node feature corresponds to a grading result, and the highest level grading result among all grading results is taken. For example, if the key node feature includes three classification results of ASC-US, LSIL, and HSIL, the highest cancer grade is HSIL; if the key node feature includes NILM, HSIL, and SCC, the highest cancer grade is SCC.

[0095] The above-mentioned abnormal cell image recognition method in cervical cell images can extract key node features from cervical cell images of different sizes through a multi-scale fusion network, and can determine the cell area in the cervical cell image, and then use the Berthesda grading system to determine the corresponding abnormal cell image and grading result in the cell area for the key node feature, and then fuse the grading results corresponding to cervical cell images of different scales, so as to more accurately determine the abnormal cell image and the corresponding grading result. Finally, the abnormal cell image in the cervical cell image and the abnormal cell image with the highest level in the grading result are output. Because the Berthesda grading system is used to identify the abnormal cell image and determine the grading result, it can provide an interpretable basis for the grading result of the determined abnormal cell image.

[0096] In one embodiment, the extracting the features of the first scale cervical cell image in the multi-scale cervical cell image by the residual network of the first scale network in the multi-scale fusion network also includes:

[0097] The first-scale cervical cell image is subjected to binary segmentation to remove a background area without cervical cells in the first-scale cervical cell image, and the first-scale cervical cell image after removing the background area without cervical cells is segmented into a plurality of image blocks of the same size without overlap.

[0098] Among them, binary segmentation image segmentation is an algorithm in image segmentation. It is simple to calculate and is not affected by image brightness and contrast. It divides the image into background and foreground according to the grayscale characteristics of the image. Because variance is a measure of the uniformity of grayscale distribution, the larger the inter-class variance between the background and foreground, the greater the difference between the two parts of the image. Therefore, the foreground and background areas can be separated by variance, thereby removing the cell-free white background area in the cervical cell image.

[0099] In this embodiment, by using binary segmentation, the cell-free background area can be removed, the interference caused by the cell-free background area can be reduced, and the image corresponding to the abnormal cells can be identified more accurately.

[0100] In one embodiment, Figure 3 and Figure 4 As shown, the key node features of the cervical cell image at each scale in the multi-scale cervical cell image are extracted through a multi-scale fusion network, including:

[0101] S302, extracting features of a first scale cervical cell image in the multi-scale cervical cell image through a residual network of the first scale network in the multi-scale fusion network.

[0102] Among them, the residual network can be a convolutional neural network, which is easy to optimize and can improve accuracy by increasing the depth. The residual block inside it uses skip connections to alleviate the gradient vanishing problem caused by increasing the depth in the deep neural network.

[0103] Specifically, the features of the first-scale cervical cell image are extracted through the residual network in the first-scale network in the multi-scale fusion network.

[0104] S304: Obtain key node features of the first-scale cervical cell image according to the features of the first-scale cervical cell image through a graph convolution pooling network of the first-scale network.

[0105] Among them, the graph convolution pooling network can be a network that constructs corresponding node features for features, then updates the representation of node features through the graph convolution network, and finally simplifies the graph structure of the cervical cell image, and only retains the key node features after the updated node feature representation. The graph convolution pooling network may include a graph convolution network and a graph pooling network. The graph convolution network can update the representation of node features. The graph pooling network can retain the key node features after the updated node feature representation.

[0106] Specifically, after obtaining the features of the cervical cell image of the first scale, the node features corresponding to the features of the cervical cell image of the first scale are constructed based on the features of the cervical cell image and the node position information through the graph convolutional pooling network in the first scale network, and then the graph convolutional network in the graph convolutional pooling network updates the representation of the node features, and changes the representation to a representation through a vector, and then the graph convolutional pooling network in the graph convolutional pooling network is used to project the node features after the updated node feature representation to a one-dimensional vector, and the key node features of the cervical cell image of the first scale can be obtained through the one-dimensional vector.

[0107] S306: Determine an input image region of the second-scale cervical cell image in the multi-scale cervical cell image according to a region of interest of an image block corresponding to a key node feature of the first-scale cervical cell image.

[0108] Among them, the region of interest is the ROI (region of interest), which is usually the image block corresponding to the key node of the cell image at the first scale in machine vision and image processing, outlining the area to be processed in the form of a box, circle, ellipse, irregular polygon, etc., which is called the region of interest.

[0109] Specifically, the key node features of the cervical cell image at the first scale are obtained, the image block corresponding to the key node features is found, and the region of interest in the image block is determined. The region corresponding to the region of interest is found in the cervical cell image at the second scale, and the region may be the input image region.

[0110] S308: Input the input image region into a second scale network of the multi-scale fusion network.

[0111] S310: Extract features of the input image region through the residual network of the second scale network.

[0112] S312: Obtain key node features of the input image region according to the features of the input image and through the graph convolution pooling network of the second scale network.

[0113] Specifically, the step of inputting the input image region into the second scale network is similar to the operation of the first scale cervical cell image in the first scale network. First, the input image region is divided into a number of image blocks of the same size without overlap. The image blocks of the same size are input into the second scale network.

[0114] The method for extracting features of the input image area is the same as the method for extracting features of the first-scale cervical cell image mentioned above, and the method for obtaining key node features of the input image area is the same as the method for obtaining key node features of the first cervical cell image mentioned above, and will not be repeated here.

[0115] In another implementation of this embodiment, the multi-scale fusion network may further include a third scale network.

[0116] The input image region of the third scale cervical cell image in the multi-scale cervical cell image is determined according to the region of interest of the image block corresponding to the key node features of the input image region, and the input image region of the third scale cervical cell image can be recorded as the first image region.

[0117] The first image region is input into the third scale network of the multi-scale fusion network.

[0118] The features of the first image region are extracted through the residual network in the third scale network.

[0119] According to the features of the first image, key node features in the first image region are obtained through the graph convolution pooling network of the third scale network.

[0120] For the specific implementation steps in another embodiment, please refer to the definitions in the above method and will not be repeated here.

[0121] In some embodiments, for example, the first scale cervical cell image I is firstly subjected to binary segmentation and divided into image blocks {I1, I2, I3...I n}; Then, the residual network in the first scale network is used to extract features, and a digital slice structure diagram is constructed based on the extracted image features and image block position information. The node features of the diagram can be X = {x1,…,x n}; Then the graph convolution network is trained on the constructed digital slice structure graph to obtain the optimized node features, and then the graph pooling network is trained based on the digital slice structure graph to obtain the pooled graph structure and the key node features; the input of the first scale network is the first scale cervical cell image, the input of the second scale network is the corresponding area of ​​the ROI area in the second scale cervical cell image obtained according to the key node features after pooling in the first scale network, and the input of the third scale network is the corresponding area of ​​the ROI area in the third scale cervical cell image obtained according to the key node features obtained in the second scale network after pooling in the second scale network. The key node features of the corresponding cervical cell images of different scales are obtained through the first scale network, the second scale network and the third scale network respectively.

[0122] It should be noted that here only three scale networks are used to extract key node features of cervical cell images of three different sizes as an example. Those skilled in the art can add or delete the above-mentioned networks of different scales according to actual conditions. The specific number of networks of different scales used is not limited in the embodiments of the present disclosure, and the key node features in cervical cell images of different scales can be extracted according to needs.

[0123] In this embodiment, the representation of node features can be updated by using the graph convolution network in the graph convolution pooling network, thereby capturing the interdependence between image blocks of cervical cell images, thereby obtaining global information and correlation information of cervical cell images. By using the graph pooling network in the graph convolution pooling network, the key node features of the cervical image can be retained, the cervical cell graph structure can be simplified, irrelevant node features can be removed, and the interference caused by irrelevant node features in determining abnormal cell images can be reduced. By using networks of different scales, cervical cell images of different sizes can be extracted, further improving the accuracy of determining abnormal cell images.

[0124] In one embodiment, the extracting the features of the first scale cervical cell image in the multi-scale cervical cell image through the residual network of the first scale network in the multi-scale fusion network includes:

[0125] A plurality of image blocks of the same size without overlap are input into the residual network of the first scale network to obtain the features of each image block, the activation function of the residual network adopts a linear rectification function, and the residual network includes a convolutional layer and a residual module.

[0126] Among them, the linear rectification function can usually be a RELU function, which is an activation function commonly used in artificial neural networks.

[0127] Specifically, a number of image blocks of the first-scale cervical cell images with the same size and no overlap obtained after binary segmentation are input into the residual network, and the features of each image block are extracted through the residual network to obtain the features of each image block. Figure 5 As shown, it is a schematic diagram of the architecture of the residual network. The input image I can be a number of image blocks of the same size without overlap. Each image block is sequentially passed through the convolution layer, residual module 1, residual module 2, residual module 3 and residual module 4, and the image feature X is output. X can include the features of each image block. The convolution layer is a 7×7 convolution layer, where the step size is 2 and the convolution kernel is 64 dimensions. The residual module 1 can include a 3×3 convolution layer, a 64-dimensional convolution kernel, where the step size is 2. The residual module 2 can include a 3×3 convolution layer, a 128-dimensional convolution kernel, where the step size is 2. The residual module 3 can include a 3×3 convolution layer, a 256-dimensional convolution kernel, where the step size is 2. The residual module 4 can include a 3×3 convolution layer, a 512-dimensional convolution kernel, where the step size is 2.

[0128] In this embodiment, a residual network is used to solve the problem of gradient disappearance caused by the increase in network depth, and the use of RELU function in the residual network can achieve low computational complexity, simplify the calculation process, and introduce sparsity.

[0129] In one embodiment, Figure 6 As shown, the key node features of the first-scale cervical cell image are obtained according to the features of the first-scale cervical cell image and through the graph convolution pooling network of the first-scale network, including:

[0130] S602: construct a digital slice structure diagram of the first-scale cervical cell image according to the characteristics of each image block and the position information of each image block in the first-scale cervical cell image to obtain node characteristics.

[0131] The digital slice structure diagram may be a structure diagram obtained by converting the first-scale cervical cell image into a data form.

[0132] Specifically, after extracting the features of each image block of the first-scale cervical cell image, the position information of each image block in the first-scale cervical cell image is obtained, and a digital slice structure diagram of the first-scale cervical cell image is constructed according to the features of each image block and the position information of each image block. The features of each corresponding node in the diagram are obtained according to the constructed digital slice structure diagram.

[0133] In some embodiments, when the extracted features of each image block are X={x1,…,x n}, determine the position information of each image block in the first scale cervical cell image as P = {p1,…,p n}, construct a digital slice structure diagram based on the characteristics and position information of each image block, determine the characteristics of each corresponding node in the digital slice structure diagram, and obtain the characteristics of each node as follows: i =[x i ||p i ],x i ∈R F ;

[0134] Among them, R F It refers to the F-dimensional real vector space where the feature X of the image block is located, and || represents the concatenation operation of the feature X and the position information P of each image block.

[0135] S604, calculating the L1 distance between the node features, constructing neighbor node features through the L1 distance, and obtaining a first adjacency matrix corresponding to the node features.

[0136] The L1 distance may be the Manhattan distance or the absolute value between node features.

[0137] Specifically, the L1 distance between each two node features is calculated, and the node features corresponding to the 10% L1 distances with the highest similarity calculated for each node feature are selected as the neighbor node features of the node. The L1 distance between node features is the subtraction of the elements at the corresponding positions and then the absolute value is taken, and finally the sum is calculated.

[0138] The more similar the two nodes are, the smaller the L1 distance is. A first adjacency matrix corresponding to the node features is constructed based on the neighbor node features.

[0139] In some implementations, the feature x of each node is calculated i =[x i ||p i ],x i ∈R F , the L1 distance between x1 and x2 is calculated as n The L1 distance between in, It can be x i The node x obtained by splicing the corresponding position p i Node features, the subsequent x that needs to be explained i It can refer to the node features obtained after splicing. n The L1 distance between every two features is sorted in ascending order to obtain the x1 neighbor node features corresponding to the top 10% of the L1 distances. The adjacency matrix is ​​constructed based on the x1 neighbor node features. The adjacency matrix corresponding to each node can be: N = {n ij},N∈R n×n, where i∈[1,…,n],j∈[1,…,n]. If x i With x j Adjacent n ij Can be 1, otherwise can be 0.

[0140] It should be noted that here only the neighbor node features corresponding to the top 10% of the L1 distances are taken as an example. Those skilled in the art can set different conditions to select the neighbor nodes corresponding to the L1 distances according to actual conditions, as long as it can indicate that the features of the two nodes are close.

[0141] S606: Update the node feature representation through a graph convolutional network.

[0142] Among them, the graph convolution network can be a network that can perform convolution operations on graph data in cervical cell images.

[0143] Each node feature and the adjacency matrix corresponding to the node feature are input into the graph convolutional network, and the representation of the node feature is updated through the graph convolutional network.

[0144] S608, obtaining key node features after updating the feature representation of the node through a single-layer neural network.

[0145] Among them, a single-layer neural network is a network that linearly divides the input data space.

[0146] Specifically, a self-attention mechanism is implemented through a single-layer neural network, and the node features after the feature representation of the node is updated are screened through the self-attention operation in the self-attention mechanism, and the key node features after the updated representation are retained to obtain the key node features. The key node features may be the node features after the updated representation.

[0147] In this embodiment, by updating and optimizing the representation of each node feature through the graph convolutional network, sufficient expression capability of the node features of the cervical cell image can be obtained, which can be used to capture the interdependence between image blocks, thereby obtaining global information and related information. And by implementing the attention mechanism through a single-layer neural network, the key node features in the cervical cell image can be retained, thereby removing useless or uncritical node features.

[0148] In one embodiment, updating the node feature representation through a graph convolutional network includes:

[0149] The neighbor node features of each of the node features are sampled according to the first adjacency matrix.

[0150] Among them, sampling can be a method of extracting a certain number of node features in the first adjacency matrix.

[0151] Specifically, a first adjacency matrix corresponding to each node feature is obtained, neighbor node features in the corresponding first adjacency matrix are obtained, and neighbor node features in the adjacency matrix are sampled.

[0152] The information of the neighbor node features obtained after sampling is aggregated according to an aggregation function to obtain the node features represented by a vector.

[0153] The aggregation function may generally be a function for calculating the weights between neighbor node features and fusing the weights with the neighbor node features.

[0154] Specifically, the weight of each neighbor node feature obtained after sampling is calculated through an aggregation function, and the weight of each neighbor node feature and the neighbor node feature are fused through an aggregation function to obtain the neighbor node feature represented by a vector.

[0155] In some embodiments, the input is node features First, the neighbor node features of each node in the graph are sampled according to the adjacency matrix, and then the information of the neighbor node features obtained after sampling is aggregated according to the aggregation function, so as to obtain the neighbor node features represented by the vector after the features of each node in the graph are updated.

[0156] In this embodiment, by sampling the neighbor node features of each node feature, the neighbor node features associated with each node feature can be obtained, and then these associated neighbor node features are aggregated, so that sufficient expression ability for the node features of cervical cell images can be obtained, which can be used to capture the interdependence between image blocks, thereby obtaining global information and correlation information.

[0157] In one embodiment, Figure 7 As shown, sampling the neighbor node features of each node feature according to the first adjacency matrix includes:

[0158] S702, when the number of neighbor node features of the node feature is less than a preset node sampling number, sampling is performed using a sampling method with replacement;

[0159] S704: When the number of neighbor node features of the node feature is greater than a preset number of node samplings, sampling is performed using a sampling method without replacement;

[0160] S706: Obtain the first node feature of the preset node sampling quantity.

[0161] Among them, the sampling method with replacement can be a sampling method in which when individuals are drawn one by one, each individual drawn is put back into the population before the next drawing. The sampling method without replacement can be a method in which when individuals are drawn one by one, each individual drawn is not put back into the population for the next drawing. The preset node sampling number can be the number of sampled neighbor node features. Those skilled in the art can set different node sampling numbers according to actual conditions, and the node sampling number is not limited in this embodiment.

[0162] Specifically, when the number of neighbor node features corresponding to the first adjacency matrix of the node features to be sampled is less than the preset node sampling number, a sampling method with replacement can be used to extract a neighbor node feature from the first adjacency matrix each time, record the neighbor node feature after extraction, and put the neighbor node feature back. Until the number of neighbor node features extracted is the preset node sampling number, the neighbor node features of the preset node sampling number obtained by the sampling method with replacement can be recorded as the first node feature.

[0163] When the number of adjacent node features corresponding to the node features to be sampled in the first adjacency matrix is ​​greater than the preset node sampling number, a sampling method without replacement can be used to extract one neighbor node feature from the first adjacency matrix each time. After extraction, the neighbor node feature is recorded and deleted from the extracted first adjacency matrix until the number of extracted neighbor node features is the preset node sampling number, and then the neighbor node features of the preset node sampling number obtained by the sampling method without replacement can be recorded as the first node features.

[0164] The number of the first node features may be a preset node sampling number.

[0165] In one embodiment, Figure 8 As shown, the information of the neighbor node features is aggregated according to the aggregation function, including:

[0166] S802: Perform a linear transformation on the first node features obtained after sampling through a parameter-sharing linear transformation matrix.

[0167] The linear transformation matrix may be a special matrix, which means that the matrix can be obtained through linear transformation. The linear transformation may generally be a linear mapping from a linear space to itself.

[0168] S804, performing a self-attention operation on the first node features after the linear transformation to obtain weight values ​​between the first node features.

[0169] The self-attention operation may be an operation for determining the importance of the first node feature. The weight value may represent the importance between each two first node features.

[0170] S806: Sort the weight values ​​in descending order to obtain the first preset number of weight values ​​ranked top.

[0171] S808, normalizing the first preset number of weight values ​​by using a normalization function, and using the normalized weight values ​​to fuse the corresponding first node features to obtain the node features represented by a vector.

[0172] The normalization function may be a softmax function, which is a generalization of a logical function and can "compress" a K-dimensional vector containing arbitrary real numbers into another K-dimensional real vector so that the range of each element is between (0, 1) and the sum of all elements is 1.

[0173] Specifically, in order to obtain sufficient expressive power, the first node feature is converted into a feature with a higher level of expressive power. First, the first node features of the preset node sampling number obtained above need to be linearly transformed using a parameter-sharing linear transformation matrix. Then, a self-attention operation is performed on each first node feature after the linear transformation, so as to obtain a weight value between each two first node features, and the weight value represents the corresponding importance between the two first nodes. The self-attention operation can be implemented using a single-layer network, and the activation function can be LeakyReLU. The weight values ​​are sorted in descending order according to the size of the weight values, and the weight values ​​of the first preset number ranked in descending order are obtained. It should be noted that the first preset number here can be set by a technician in this field according to the actual scenario, and is not limited in this embodiment. And the weight value ranked high is obtained, and the weight value can prove that the correlation between the two node features is large. Then the weight values ​​of the first preset number obtained are normalized by the softmax function to obtain the normalized weight value, and the normalized weight value and the corresponding first node feature are fused to obtain the node feature represented by the vector, and the node feature represented by the vector can be the first node feature.

[0174] In some embodiments, a parameter-sharing linear transformation matrix is ​​first used. Perform a linear transformation on the first node feature. Then perform a self-attention operation on each first node feature obtained after the linear transformation. like Fig. 9 As shown, the weight value e between the first node features is calculated. ij l =A(W l x i l ,Wl x j l ), the weight value can represent the first node feature x j For the first node feature x i The importance between them. Among them, the self-attention operation can be implemented using a single-layer network, and the network parameters of the single-layer network can be The activation function can use LeakyReLU, so the weight value can be expressed as: Then the obtained weight values ​​can be sorted in descending order to obtain the weight values ​​of the first preset number of weights after sorting. The first preset number is usually less than or equal to the preset number of node samples. Then the obtained weight values ​​are normalized by the softmax function. Get the normalized weight value.

[0175] Then, the normalized weight value is used to fuse the corresponding first node feature, and the representation of the first node feature is updated. The representation of the updated first node feature can be:

[0176] In this embodiment, the use of parameter sharing in linear transformation can bring strong flexibility, reduce the amount of calculation, and improve efficiency. And after the self-attention operation, obtaining a larger weight value can obtain node features with strong association information, and then determine the association relationship between the corresponding image blocks, and the obtained weight value is normalized by the softmax function, which can optimize the feature representation of each node and obtain sufficient expression ability.

[0177] In one embodiment, Fig.10 As shown, the key node features after updating the feature representation of the node through the single-layer neural network include:

[0178] S1002, obtaining the attention weight value of the node feature represented by the vector through a self-attention operation.

[0179] S1004, obtaining the node feature represented by the vector corresponding to the attention weight value within a preset first range.

[0180] S1006, updating the node features represented by the vector according to the attention weight value and the normalization function, and obtaining the key node features after the feature representation of the node is updated.

[0181] The normalization function can usually be a sigmoid function. The sigmoid function is often used as an activation function of a neural network and can map variables to between 0 and 1.

[0182] Specifically, the attention weight value corresponding to each node feature represented by the vector is obtained through the self-attention operation. In order to retain the key features in the cervical cell image as much as possible, the attention weight value within the preset first range is obtained. The preset first range can usually obtain a larger attention weight value. Finally, the obtained attention weight value of the preset first range is normalized by the normalization function, and the attention weight value of the preset first range is converted into a one-dimensional matrix. The one-dimensional matrix is ​​multiplied by the corresponding node feature represented by the vector to obtain the key node feature after updating the feature representation of the node.

[0183] In some embodiments, Fig.11 As shown, through the self-attention operation For node features represented by vectors Calculate and get the attention weight value θ att l+1 =A pool X l+1 ,θ att l+1 ∈R n In order to retain the key features, the preset first range here can be 10% of the maximum attention weight value. Further, let Then idx l+1 =rank(θ att l+1 ,k), where rank(·,k) can be expressed as returning the largest k indexes, X l+1 (idx l+1 ,:) can be expressed as the original X l+1 According to the index idx l+1 Feature extraction is performed. After extraction, key node features can be obtained. Finally, according to the attention weight Update node features The weights are normalized using the sigmoid normalization function. n ∈R n is a 1-dimensional matrix whose elements are all 1, and ⊙ is a matrix multiplication operation that multiplies elements by element. Based on the above method, the adjacency matrix of the key node features represented by the vector can also be calculated to correspond to the adjacency matrix of the key node features, which can be: N l+2 =N l+1 (idx l+1 ,idx l+1 ).

[0184] It should be noted that the node features not represented by the vector can also be calculated by the self-attention operation to obtain the key node features not represented by the vector. The key node features not represented by the vector are matched with the node features represented by the vector to obtain the key node features represented by the vector.

[0185] In this embodiment, the attention weight value of each node feature can be calculated through the self-attention operation, and a larger attention weight value can be determined through the preset first range, so as to retain the key node features. Finally, the key node features can be used to focus on the key areas in the cervical cell image, thereby improving the recognition rate of abnormal cell images in the cervical cell image.

[0186] In this embodiment, the multi-scale fusion network is trained by the following methods:

[0187] Constructing a training data set and a test data set of a multi-scale fusion network, wherein the training data set and the test data set both include cervical cell images provided with annotation information, wherein the annotation information is determined by abnormal cells pre-annotated in the cervical cell images;

[0188] The multi-scale network is trained using the training data set and the test data set as well as a preset classification loss function to obtain a multi-scale fusion network.

[0189] The annotation information may include a cancer cell annotation frame and cancer category information of the cell, and cancer information of any pathological area in the cervical cell image may be obtained.

[0190] In this embodiment, by setting up a training data set and a test data set of cervical cell images with labeled information, after the training of the multi-scale fusion network is completed, the grading results of abnormal cells in the cervical cell images can be determined through the multi-scale fusion network without the need for identification based on artificial experience, and the classification loss function can reduce the errors generated by training.

[0191] In one embodiment, the calculation formula of the classification loss function is:

[0192]

[0193] in, is the cross entropy loss function of the first scale network; is the cross entropy loss function of the second scale network; the cross entropy loss function of the first scale network and the cross entropy loss function of the second scale network can be:

[0194]

[0195] in, is the annotation information in the cervical cell image, K is the number of key node features, y i * The predicted annotation information in the cervical cell image is calculated.

[0196] In another implementation of this embodiment, the calculation formula of the classification loss function may also be:

[0197]

[0198] in, is the cross entropy loss function of the first scale network; is the cross entropy loss function of the second scale network; is the cross entropy loss function of the third scale network; the cross entropy loss function of the first scale network, the cross entropy loss function of the second scale network, and the cross entropy loss function of the third scale network can be:

[0199]

[0200] in, is the annotation information in the cervical cell image, K is the number of key node features, and y i * The predicted annotation information in the cervical cell image is calculated.

[0201] It should be noted that the annotation information here may refer to abnormal cell images or the grading results of abnormal cell images.

[0202] In this embodiment, by using different scale-independent network loss functions in each multi-scale fusion network, the training process of the multi-scale fusion network can be better guided and the accuracy of identifying abnormal cell images can be improved.

[0203] In one embodiment, the grading result of the abnormal cell image in the cervical cell image corresponding to the key node features of the cervical cell image at each scale determined by the Berthesda grading system includes:

[0204] The key node features of cervical cell images at each scale are graded according to the Berthesda grading system through the fully connected network of the multi-scale fusion network and the normalized exponential function, and the grading results of abnormal cell images in the cervical cell images corresponding to the key node features of the cervical cell images at each scale are determined.

[0205] Among them, the fully connected network can usually connect the input and output of the network of each scale in all multi-scale fusion networks.

[0206] Specifically, a fully connected network and a softmax layer were used to grade the key node features of cervical cell images using NILM, ASC-US, ASC-H, LSIL, HSIL, and SCC according to the Berthesda grading system to obtain the grading results.

[0207] In this embodiment, the grading result of cervical cells can be determined through a fully connected network and a softmax layer, without the need for manual experience-based identification, so as to determine the grading result.

[0208] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0209] Based on the same inventive concept, the disclosed embodiment also provides a device for recognizing abnormal cells in cervical cell images for implementing the above-mentioned method for recognizing abnormal cells in cervical cell images. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific implementation methods in one or more embodiments of the device for recognizing abnormal cells in cervical cell images provided below can refer to the above-mentioned embodiment of the method for recognizing abnormal cells in cervical cell images, and will not be repeated here.

[0210] In one embodiment, Fig.12 As shown, a device 1200 for identifying abnormal cells in cervical cell images is provided, comprising: an image input module 1202, a key node feature extraction module 1204, a classification result determination module 1206, and a classification result fusion module 1208, wherein:

[0211] The image input module 1202 is used to input the multi-scale cervical cell image into the pre-trained multi-scale fusion network, wherein the multi-scale cervical cell includes different sizes of the same cervical cell image.

[0212] The key node feature extraction module 1204 is used to extract the key node features of the cervical cell image of each scale in the multi-scale cervical cell image through a multi-scale fusion network.

[0213] The grading result determination module 1206 is used to determine the grading result of the abnormal cell image in the cervical cell image corresponding to the key node features of the cervical cell image at each scale through the Berthesda grading system.

[0214] The grading result fusion module 1208 is used to fuse the grading results of the abnormal cell images in the cervical cell images of each scale, and output the abnormal cell images in the cervical cell images and the abnormal cell images with the highest level in the grading results.

[0215] In one embodiment of the device, the key feature extraction module includes: a first feature extraction module, a first graph convolution pooling module, an input image area determination module, a second scale network input module, a second feature extraction module, and a second graph convolution pooling module, wherein:

[0216] The first feature extraction module is used to extract features of the first scale cervical cell image in the multi-scale cervical cell image through the residual network of the first scale network in the multi-scale fusion network.

[0217] The first graph convolution pooling module is used to obtain key node features of the first-scale cervical cell image through the graph convolution pooling network of the first-scale network according to the features of the first-scale cervical cell image.

[0218] The input image region determination module is used to determine the input image region of the second scale cervical cell image in the multi-scale cervical cell image according to the region of interest of the image block corresponding to the key node features of the first scale cervical cell image.

[0219] The second scale network input module is used to input the input image area into the second scale network of the multi-scale fusion network.

[0220] The second feature extraction module is used to extract the features of the input image area through the residual network of the second scale network.

[0221] The second graph convolution pooling module is used to obtain key node features of the input image area according to the features of the input image area through the graph convolution pooling network of the second scale network.

[0222] In one embodiment of the device, the device also includes: a binary segmentation module, used to perform binary segmentation on the first-scale cervical cell image, remove the background area without cervical cells in the first-scale cervical cell image, and segment the first-scale cervical cell image after removing the background area without cervical cells into a number of image blocks of the same size without overlap.

[0223] In one embodiment of the device, the first feature extraction module is also used to input a number of non-overlapping image blocks of the same size into the residual network of the first scale network to obtain the features of each image block, the activation function of the residual network adopts a linear rectification function, and the residual network includes a convolutional layer and a residual module.

[0224] In one embodiment of the device, the first graph convolution pooling module includes: a node feature calculation module, an adjacency matrix construction module, a graph convolution module, and a graph pooling module, wherein:

[0225] The node feature calculation module is used to construct a digital slice structure diagram of the first-scale cervical cell image according to the features of each image block and the position information of each image block in the first-scale cervical cell image to obtain the node features.

[0226] The adjacency matrix construction module is used to calculate the L1 distance between the node features, construct neighbor node features through the L1 distance, and obtain a first adjacency matrix corresponding to the node features.

[0227] A graph convolution module is used to update the node feature representation through a graph convolution network.

[0228] The graph pooling module is used to obtain key node features after updating the feature representation of the node through a single-layer neural network.

[0229] In one embodiment of the apparatus, the graph convolution module includes: a sampling module and an aggregation module;

[0230] A sampling module is used to sample neighbor node features of each of the node features according to the first adjacency matrix.

[0231] The aggregation module is used to aggregate the information of the neighbor node features obtained after sampling according to an aggregation function to obtain the node features represented by a vector.

[0232] In one embodiment of the device, the sampling module includes: a sampling module with replacement and a sampling module without replacement;

[0233] The sampling module with replacement is used to perform sampling by a sampling method with replacement when the number of neighbor node features of the node feature is less than the preset node sampling number, so as to obtain the first node feature of the preset node sampling number.

[0234] The no-replacement sampling module is used to perform sampling by a no-replacement sampling method when the number of neighbor node features of the node feature is greater than a preset node sampling number, so as to obtain the first node feature of the preset node sampling number.

[0235] In one embodiment of the device, the aggregation module includes: a linear transformation module, a weight value calculation module, a weight value acquisition module, and a node feature update module, wherein:

[0236] A linear transformation module is used to perform a linear transformation on the first node features obtained after sampling through a parameter-sharing linear transformation matrix.

[0237] A weight value calculation module is used to perform a self-attention operation on the first node features after the linear transformation to obtain the weight values ​​between the first node features.

[0238] The weight value acquisition module is used to sort the weight values ​​in descending order and obtain the weight values ​​of the first preset number of rankings.

[0239] The node feature updating module is used to normalize the first preset number of weight values ​​through a normalization function, and use the normalized weight values ​​to fuse the corresponding first node features to obtain the node features represented by a vector.

[0240] In one embodiment of the device, the graph pooling module includes: an attention weight value calculation module, an attention weight value acquisition module, and a key node feature acquisition module, wherein:

[0241] The attention weight value calculation module is used to obtain the attention weight value of the node feature represented by the vector through a self-attention operation.

[0242] The attention weight value acquisition module is used to obtain the node feature represented by a vector corresponding to the attention weight value within a preset first range.

[0243] The key node feature acquisition module is used to update the node feature represented by the vector according to the attention weight value and the normalization function to obtain the key node feature after the feature representation of the node is updated.

[0244] In one embodiment of the device, the device further comprises: a training data preparation module and a multi-scale fusion network training module, wherein:

[0245] The training data preparation module is used to construct a training data set and a test data set for a multi-scale fusion network, wherein the training data set and the test data set both include cervical cell images with annotation information, and the annotation information is determined by abnormal cells pre-annotated in the cervical cell images.

[0246] The multi-scale fusion network training module is used to train the multi-scale network through the training data set and the test data set and a preset classification loss function to obtain a multi-scale fusion network.

[0247] In one embodiment of the device, the calculation formula of the classification loss function is:

[0248]

[0249] in, is the cross entropy loss function of the first scale network; is the cross entropy loss function of the second scale network; the cross entropy loss function of the first scale network and the cross entropy loss function of the second scale network are:

[0250]

[0251] in, is the annotation information in the cervical cell image, K is the number of key node features, and y i * The predicted annotation information in the cervical cell image is calculated.

[0252] In one embodiment of the device, the grading result determination module 1206 is also used to perform grading judgment on the key node features of the cervical cell image of each scale according to the Berthesda grading system through the fully connected network of the multi-scale fusion network and the normalized exponential function, and determine the grading result of the abnormal cell image in the cervical cell image corresponding to the key node features of the cervical cell image of each scale.

[0253] Each module in the above-mentioned abnormal cell image recognition device in cervical cell images can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0254] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.13As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for identifying abnormal cell images in cervical cell images is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0255] Those skilled in the art will understand that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present disclosure, and does not constitute a limitation on the computer device to which the scheme of the present disclosure is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0256] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0257] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0258] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0259] It should be noted that the cervical cell images involved in the present disclosure are all images authorized by the user or fully authorized by all parties.

[0260] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided by the present disclosure may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided by the present disclosure may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.

[0261] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0262] The above-described embodiments only express several implementation methods of the present disclosure, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present disclosure, and these all belong to the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the attached claims.

Claims

1. A method for identifying abnormal cell images in cervical cell images, characterized in that: The method comprises: Inputting a multi-scale cervical cell image into a pre-trained multi-scale fusion network, wherein the multi-scale cervical cell image includes different sizes of the same cervical cell image; Extracting key node features of the cervical cell image at each scale in the multi-scale cervical cell image through a multi-scale fusion network; The Berthesda grading system is used to determine the grading results of abnormal cell images in the cervical cell images corresponding to the key node features of the cervical cell images at each scale; The grading results of abnormal cell images in the cervical cell images of each scale are merged, and the abnormal cell images in the cervical cell images and the abnormal cell images with the highest level in the grading results are output; The step of extracting key node features of each scale of the cervical cell image in the multi-scale cervical cell image through a multi-scale fusion network includes: Extracting features of a first-scale cervical cell image in the multi-scale cervical cell image through a residual network of a first-scale network in the multi-scale fusion network; According to the features of the first-scale cervical cell image, key node features of the first-scale cervical cell image are obtained through the graph convolution pooling network of the first-scale network; Determine an input image region of a second-scale cervical cell image in the multi-scale cervical cell image according to a region of interest of an image block corresponding to a key node feature of the first-scale cervical cell image; Inputting the input image region into a second scale network of the multi-scale fusion network; Extracting features of the input image region through a residual network of the second scale network; According to the features of the input image region, key node features of the input image region are obtained through the graph convolution pooling network of the second scale network.

2. The method according to claim 1, characterized in that The extracting the features of the first scale cervical cell image in the multi-scale cervical cell image by the residual network of the first scale network in the multi-scale fusion network also includes: The first-scale cervical cell image is subjected to binary segmentation to remove a background area without cervical cells in the first-scale cervical cell image, and the first-scale cervical cell image after removing the background area without cervical cells is segmented into a plurality of image blocks of the same size without overlap.

3. The method according to claim 1, characterized in that The extracting the features of the first scale cervical cell image in the multi-scale cervical cell image through the residual network of the first scale network in the multi-scale fusion network includes: A plurality of image blocks of the same size without overlap are input into the residual network of the first scale network to obtain the features of each image block, the activation function of the residual network adopts a linear rectification function, and the residual network includes a convolutional layer and a residual module.

4. The method according to claim 3, characterized in that The key node features of the first-scale cervical cell image are obtained according to the features of the first-scale cervical cell image and through the graph convolution pooling network of the first-scale network, including: According to the characteristics of each image block and the position information of each image block in the first-scale cervical cell image, a digital slice structure diagram of the first-scale cervical cell image is constructed to obtain node characteristics; Calculating the L1 distance between the node features, constructing neighbor node features through the L1 distance, and obtaining a first adjacency matrix corresponding to the node features; Updating the node feature representation through a graph convolutional network; The key node features after updating the feature representation of the node are obtained through a single-layer neural network.

5. The method according to claim 4, characterized in that The updating of the node feature representation method through a graph convolutional network includes: Sampling neighbor node features of each of the node features according to the first adjacency matrix; The information of the neighbor node features obtained after sampling is aggregated according to an aggregation function to obtain the node features represented by a vector.

6. The method according to claim 5, characterized in that The step of sampling neighbor node features of each node feature according to the first adjacency matrix includes: When the number of neighbor node features of the node feature is less than the preset node sampling number, sampling is performed using a sampling method with replacement; When the number of neighbor node features of the node feature is greater than the preset node sampling number, sampling is performed using a sampling method without replacement; A first node feature of the preset node sampling quantity is obtained.

7. The method according to claim 6, characterized in that The aggregating the information of the neighbor node features according to the aggregation function includes: Performing a linear transformation on the first node feature obtained after sampling by using a parameter-sharing linear transformation matrix; Performing a self-attention operation on the first node features after the linear transformation to obtain weight values ​​between the first node features; Sort the weight values ​​in descending order to obtain the first preset number of weight values ​​ranked first; The first preset number of weight values ​​are normalized by a normalization function, and the normalized weight values ​​are used to fuse the corresponding first node features to obtain the node features represented by a vector.

8. The method according to claim 4, characterized in that The key node features obtained by the single-layer neural network after updating the feature representation of the node include: Obtaining the attention weight value of the node feature represented by the vector through a self-attention operation; Obtain the node feature represented by a vector corresponding to the attention weight value within a preset first range; The node features represented by the vector are updated according to the attention weight value and the normalization function to obtain the key node features after the feature representation of the node is updated.

9. The method according to claim 2, characterized in that: The multi-scale fusion network is trained by the following methods: Constructing a training data set and a test data set of a multi-scale fusion network, wherein the training data set and the test data set both include cervical cell images with annotation information, wherein the annotation information is determined by abnormal cells pre-annotated in the cervical cell images; The multi-scale network is trained using the training data set and the test data set as well as a preset classification loss function to obtain a multi-scale fusion network.

10. The method according to claim 9, characterized in that The calculation formula of the classification loss function is: ； in, is the cross entropy loss function of the first scale network; is the cross entropy loss function of the second scale network; the cross entropy loss function of the first scale network and the cross entropy loss function of the second scale network are: ； in, is the annotation information in the cervical cell image, K is the number of key node features, The predicted annotation information in the cervical cell image is calculated.

11. The method according to claim 1, characterized in that: The grading result of the abnormal cell image in the cervical cell image corresponding to the key node features of the cervical cell image at each scale determined by the Berthesda grading system includes: The key node features of cervical cell images at each scale are graded according to the Berthesda grading system through the fully connected network of the multi-scale fusion network and the normalized exponential function, and the grading results of abnormal cell images in the cervical cell images corresponding to the key node features of the cervical cell images at each scale are determined.

12. A device for identifying abnormal cell images in cervical cell images, characterized in that: The device comprises: An image input module, used for inputting a multi-scale cervical cell image into a pre-trained multi-scale fusion network, wherein the multi-scale cervical cell image includes different sizes of the same cervical cell image; A key node feature extraction module, used to extract key node features of the cervical cell image of each scale in the multi-scale cervical cell image through a multi-scale fusion network; A grading result determination module, used for determining the grading result of the abnormal cell image in the cervical cell image corresponding to the key node features of the cervical cell image at each scale through the Berthesda grading system; A grading result fusion module is used to fuse the grading results of abnormal cell images in the cervical cell images of each scale, and output the cervical cell image area corresponding to the grading results of the abnormal cell images in the cervical cell images and the highest-level abnormal cell images; The key node feature extraction module includes: a first feature extraction module, a first graph convolution pooling module, an input image area determination module, a second scale network input module, a second feature extraction module, and a second graph convolution pooling module, wherein: The first feature extraction module is used to extract features of the first scale cervical cell image in the multi-scale cervical cell image through the residual network of the first scale network in the multi-scale fusion network; The first graph convolution pooling module is used to obtain key node features of the first-scale cervical cell image through the graph convolution pooling network of the first-scale network according to the features of the first-scale cervical cell image; The input image region determination module is used to determine the input image region of the second scale cervical cell image in the multi-scale cervical cell image according to the region of interest of the image block corresponding to the key node features of the first scale cervical cell image; The second scale network input module is used to input the input image area into the second scale network of the multi-scale fusion network; The second feature extraction module is used to extract features of the input image region through the residual network of the second scale network; The second graph convolution pooling module is used to obtain key node features of the input image area according to the features of the input image area through the graph convolution pooling network of the second scale network.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images