Cell image recognition method, device and system, computer equipment and medium

Abstract

The invention relates to a cell image recognition method, device and system, computer equipment and a medium. The method comprises the following steps: receiving a to-be-detected cell image sent by an image acquisition device; the to-be-detected cell image is a real-time image of living tissue in the visual field of the endoscope, and the image acquisition device is connected with the endoscope; based on the to-be-detected cell image, obtaining an identification result of the to-be-detected cell image through a trained cell image identification model; the identification result comprises a cell category and a cell position; wherein the cell image recognition model is based on a yolk network, and a GhostNet feature extraction network is used as a backbone network. By adopting the method, image analysis can be quickly carried out, and the method is particularly suitable for quickly diagnosing whether tumor lesions exist in a target area or not during an operation.

Description

Technical Field

[0001] The present application relates to the field of medical image recognition technology, and in particular to a cell image recognition method, device, system, computer equipment and medium. Background Art

[0002] An endoscope is a device that uses a head probe and an optical lens to inspect and treat internal human tissues. Its working principle is based on optical imaging and probe technology. It introduces a light source and a lens into the body cavity or tissue to observe the microscopic structure or diseased tissue in the body.

[0003] However, although the current optical imaging through endoscopes can provide clear images reflecting cell morphology in real time, users still need to manually analyze real-time image information when using the endoscope to determine the normal cell conditions or tumor cell conditions in the real-time image information, which inevitably distracts some attention. In addition, the image analysis efficiency is low, making it difficult to adapt to the rapid diagnosis of whether there are tumor lesions in the target area during surgery. Summary of the invention

[0004] Based on this, it is necessary to provide a cell image recognition method, device, system, computer equipment and medium that can improve the efficiency of cell image analysis in response to the above technical problems.

[0005] In a first aspect, the present application provides a cell image recognition method. The method comprises:

[0006] Receiving the image of the cell to be tested sent by the image acquisition device; the image of the cell to be tested is a real-time image of the living tissue in the field of view of the endoscope, and the image acquisition device is connected to the endoscope;

[0007] Based on the cell image to be tested, obtaining a recognition result of the cell image to be tested by a trained cell image recognition model; the recognition result includes a cell type and a cell position;

[0008] Among them, the cell image recognition model is based on the Yolo network, and the GhostNet feature extraction network is used as the backbone network.

[0009] In one embodiment, the trained cell image recognition model is obtained by training through the following steps:

[0010] Acquire an initial sample cell image set carrying cell information annotations, wherein the cell information annotations include cell category annotations and cell position annotations of the sample cell images;

[0011] Performing data enhancement on the initial sample cell image set to obtain an enhanced sample cell image set carrying cell information annotations;

[0012] Performing cell recognition on the target sample cell image using the cell image recognition model to obtain a sample cell result; the target sample cell image includes the initial sample cell image in the initial sample cell image set and the enhanced sample cell image in the enhanced sample cell image set;

[0013] Parameters of the cell image recognition model are optimized according to the sample cell results and the cell information annotations.

[0014] In one embodiment, the cell image recognition model includes a GhostNet feature extraction network, an SPP pooling network, a PANet neck network, and a head prediction network connected in sequence;

[0015] The method of using the cell image recognition model to perform cell recognition on the target sample cell image to obtain the sample cell result includes:

[0016] Based on the GhostNet feature extraction network, feature extraction is performed on the target sample cell image to obtain a first feature map, a second feature map, and a third feature map corresponding to the target sample cell image; the third feature map is the last layer feature map extracted by the GhostNet feature extraction network;

[0017] The third feature map is input into the SPP pooling network for pooling after passing through three standard convolution layers connected in sequence, and the results after the pooling process are concatenated and passed through three standard convolution layers connected in sequence to obtain a fourth feature map;

[0018] Based on the PANet neck network, feature enhancement is performed on the first feature map, the second feature map, and the fourth feature map, and features at different levels are fused to output feature vectors of multiple scales, where the feature vectors of multiple scales are used to characterize target sample regions of multiple different sizes corresponding to the target sample cell image;

[0019] Based on the head prediction network, multiple bounding boxes are predicted in target sample regions of multiple different sizes, and confidences that the multiple bounding boxes correspond to normal cells and tumor cells are determined respectively, to obtain the sample cell results.

[0020] In one embodiment, the PANet neck network further includes a plurality of combined convolutions, wherein the combined convolutions include two layers of sequentially connected depthwise separable convolutions and three layers of sequentially connected standard convolution layers; the three layers of sequentially connected standard convolution layers sequentially include a first convolution layer, a second convolution layer, and a third convolution layer; the method further includes:

[0021] Based on the convolution kernel mean of the first convolution layer, a convolution operation is performed on the feature map of the cell image to be tested input by the first convolution layer to obtain a first output; at the same time, based on the convolution kernel of the first convolution layer, a convolution operation is performed on the feature map of the cell image to be tested input by the first convolution layer to obtain a third output; and after obtaining the third output, a difference between the third output and the first output is used to obtain a fourth output;

[0022] After obtaining the first output, based on the convolution kernel mean of the second convolution layer, the first output is convolved to obtain the second output; at the same time, based on the convolution kernel of the second convolution layer, the first output is convolved to obtain the fifth output; after obtaining the fourth output, based on the convolution kernel of the second convolution layer, the fourth output is convolved to obtain the sixth output; and the fifth output and the sixth output are combined to obtain the seventh output; and the difference between the seventh output and the second output is used to obtain the ninth output;

[0023] After obtaining the second output, a convolution operation is performed on the second output based on the convolution kernel of the third layer of convolution to obtain an eighth output; after obtaining the ninth output, a convolution operation is performed on the ninth output based on the convolution kernel of the third layer of convolution to obtain a tenth output; and the eighth output and the tenth output are combined to obtain an eleventh output, which is the final output of the three standard convolution layers connected in sequence.

[0024] In one embodiment, the data enhancement of the initial sample cell image set to obtain an enhanced sample cell image set carrying cell information annotations includes:

[0025] According to the preset multi-dimensional cell characteristic parameters, the cell characteristic data of each dimension corresponding to the normal cells and the tumor cells in the initial sample cell image set are obtained;

[0026] For each cell category, correlation analysis is performed between the cell feature data of each dimension corresponding to the cell category and the cell category to obtain the correlation between the cell feature parameters of each dimension and the cell category;

[0027] Based on the correlation, the cell characteristic parameters with high correlation to the normal cell category are eliminated from the cell characteristic parameters with low correlation to the tumor cell category to obtain the target cell characteristic parameters;

[0028] Based on the target cell characteristic parameters, data perturbation is performed on the initial sample cell image set to obtain an enhanced sample cell image set carrying cell information annotations.

[0029] In one embodiment, the data perturbation of the initial sample cell image set based on the target cell characteristic parameters to obtain an enhanced sample cell image set carrying cell information annotations includes:

[0030] Based on the cell information annotation, extracting a sample tumor cell sub-image from the initial sample cell image in the initial sample cell image set;

[0031] Based on the target cell characteristic parameters, performing data perturbation on the sample tumor cell sub-image to obtain a first enhanced tumor cell sub-image;

[0032] The first enhanced tumor cell sub-image and the initial sample cell image corresponding to the first enhanced tumor cell sub-image are combined to obtain a first enhanced sample cell image carrying cell information annotations, and the first enhanced sample cell image is used to determine the enhanced sample cell image set.

[0033] In a second aspect, the present application also provides a cell image recognition device. The device comprises:

[0034] A receiving module, used for receiving the image of the cell to be tested sent by the image acquisition device; the image of the cell to be tested is a real-time image of the living tissue in the field of view of the endoscope, and the image acquisition device is connected to the endoscope;

[0035] A recognition module, used for obtaining a recognition result of the cell image to be tested based on the cell image to be tested by a trained cell image recognition model; the recognition result includes a cell type and a cell position;

[0036] Among them, the cell image recognition model is based on the Yolo network, and the GhostNet feature extraction network is used as the backbone network.

[0037] In a third aspect, the present application also provides a cell image recognition system, the system comprising:

[0038] endoscopes, which are used to obtain real-time images of living tissue in the field of view;

[0039] An image acquisition device, connected to the endoscope, for acquiring a real-time image of the living tissue in the field of view of the endoscope, and sending the real-time image of the living tissue as an image of the cell to be tested to a computer device;

[0040] A computer device is connected to the image acquisition device and is used to receive the cell image to be tested sent by the image acquisition device; based on the cell image to be tested, a recognition result of the cell image to be tested is obtained through a trained cell image recognition model; the recognition result includes cell category and cell position; wherein the cell image recognition model is based on a yolo network, and uses a GhostNet feature extraction network as a backbone network.

[0041] In a fourth aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the cell image recognition method provided in the first aspect of the present application are implemented.

[0042] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the cell image recognition method provided in the first aspect of the present application.

[0043] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the cell image recognition method provided in the first aspect of the present application are implemented.

[0044] The above-mentioned cell image recognition method, device, system, computer equipment and medium receive the cell image to be tested sent by the image acquisition device; the cell image to be tested is a real-time image of living tissue in the endoscope field of view, and the image acquisition device is connected to the endoscope; based on the cell image to be tested, the recognition result of the cell image to be tested is obtained by a trained cell image recognition model; the recognition result includes cell category and cell position; wherein the cell image recognition model is based on the yolo network, and the GhostNet feature extraction network is used as the backbone network. The present application uses an image acquisition device to collect the living tissue image in the endoscope field of view as the cell image to be tested in real time, and uses the trained cell image recognition model to identify the cell category and cell position in the cell image to be tested. The cell image recognition model is based on the yolo network, and uses the GhostNet feature extraction network as the backbone network, which can quickly perform image analysis and is particularly suitable for rapid diagnosis of whether there are tumor lesions in the target area during surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] FIG1 is a diagram showing an application environment of a cell image recognition method according to an embodiment;

[0046] FIG2 is a schematic diagram of a process of a cell image recognition method in one embodiment;

[0047] FIG3 is a schematic diagram of a process for obtaining a trained cell image recognition model in one embodiment;

[0048] FIG4 is a schematic diagram of the structure of a cell image recognition model in one embodiment;

[0049] FIG5 is a schematic diagram of a process for obtaining sample cell results in one embodiment;

[0050] FIG6 is a schematic diagram of a process for obtaining an enhanced sample cell image set in one embodiment;

[0051] FIG7 is a schematic diagram of a process for obtaining an enhanced sample cell image set in another embodiment;

[0052] FIG8 is a schematic diagram of a process for obtaining an enhanced sample cell image set in another embodiment;

[0053] FIG9 is a schematic diagram of a process flow of a three-layer convolution operation in one embodiment;

[0054] FIG10 is a block diagram of a cell image recognition device according to an embodiment;

[0055] FIG. 11 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] The cell image recognition method provided in the embodiment of the present application can be applied in the application environment shown in FIG1. ​​The image acquisition device 102 is connected to the endoscope 100, and is used to capture real-time images of living tissues in the field of view of the endoscope 100, and transmit the captured images to the computer device 104 through a wired or wireless network, and the computer device 104 is configured with a trained cell image recognition model. The image acquisition device 102 and the endoscope 100 can also be the same device, for example, an endoscope with an image acquisition function.

[0058] The computer device 104 may be a terminal or a server. The terminal may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server may be implemented as an independent server or a server cluster consisting of multiple servers.

[0059] In one embodiment, as shown in FIG2 , a cell image recognition method is provided, which is described by taking the method applied to the computer in FIG1 as an example, and includes the following steps:

[0060] Step 202: receiving the image of the cell to be tested sent by the image acquisition device.

[0061] The cell image to be tested refers to a real-time image of living tissue in the field of view of the endoscope, which can display the state distribution of multiple cells in the living tissue at the cellular level.

[0062] Exemplarily, a light source is provided to the part to be observed through an endoscope, and the light source may include a visible light source and a special light source. Specifically, the visible light source is an LED light source, which can provide multiple monochromatic lights of different wavelength ranges, such as blue light, green light, red light, etc. In some embodiments, the visible light source can also be a combination of multiple monochromatic lights, or a wide-spectrum white light source; the special light source can be a laser light source corresponding to a fluorescent agent, such as a near-infrared light or other laser. In some embodiments, a fluorescent agent can be injected into the part to be observed before endoscopic imaging is used, and the fluorescent agent can generate fluorescence after absorbing the laser generated by the laser light source.

[0063] Step 204 : Based on the cell image to be tested, obtain the recognition result of the cell image to be tested by using the trained cell image recognition model.

[0064] The recognition result includes the cell type and the cell position. It is understood that the cell image to be detected may include multiple cells, and the multiple cells may be one or more of normal cells and tumor cells.

[0065] The embodiment of the present application can configure a cell image recognition model to include at least a feature extraction network and a prediction network. The feature extraction network is used to extract the feature image of the cell image to be tested, and the prediction network is used to identify the feature image to obtain a recognition result, wherein the recognition result can be a normal cell or a tumor cell representing cell category information, and a prediction box representing cell location information, and the prediction box is used to circle the normal cell or the tumor cell on the cell image to be tested.

[0066] Taking glioma as an example, glioma cells and normal cells have certain differences in protein, nucleic acid, water, morphological structure, etc., which leads to differences in their spectral curves. Therefore, with the help of the spectral features contained in the image of the cell to be tested, the spectral features in the image of the cell to be tested are extracted to obtain the spectral feature vector corresponding to the image of the cell to be tested. The characteristic information of the cell is extracted by analyzing the spectral features, cell size and shape, etc.

[0067] In the embodiments of the present application, there is no specific limitation on the structure of the cell image recognition model. For example, the target detection model may be a model with YOLOv4 (You Only Look Once version 4) as the basic structure, or may be a model with other structures, such as a model with variant structures of YOLOv4, YOLOv5, YOLOv6, YOLOv7, or YOLOv8.

[0068] In one implementation, the cell image recognition model can adopt a model based on YOLOv4. The YOLOv4 model is a single-stage target detection model that can meet real-time requirements while maintaining a high accuracy rate. The backbone network CSPDarkNet53 of the existing YOLOv4 network model is replaced with a GhostNet feature extraction network. The GhostNet feature extraction network is a lightweight network that is used to improve the efficiency of feature extraction of cell images.

[0069] The cell images collected by the endoscope in the embodiment of the present application have high resolution and complex details. The GhostNet feature extraction network can generate rich feature maps, capture subtle differences in cell images, and help detect and identify tumor cells. In the medical field, especially in telemedicine and mobile medical devices, computing resources may be limited. The lightweight characteristics of the GhostNet feature extraction network make it extremely advantageous in these medical environments. In addition, the GhostNet feature extraction network generates more feature maps by introducing the Ghost module to simulate redundant features, thereby reducing the number of convolution operations, reducing storage requirements, and improving the efficiency of the model during reasoning.

[0070] In the above-mentioned cell image recognition method, the image of the cell to be tested sent by the image acquisition device is received; the image of the cell to be tested is a real-time image of the living tissue in the endoscope field of view, and the image acquisition device is connected to the endoscope; based on the cell image to be tested, the recognition result of the cell image to be tested is obtained by a trained cell image recognition model; the recognition result includes cell category and cell position; wherein the cell image recognition model is based on the yolo network, and the GhostNet feature extraction network is used as the backbone network. The present application uses an image acquisition device to collect the living tissue image in the endoscope field of view as the cell image to be tested in real time, and uses the trained cell image recognition model to identify the cell category and cell position in the cell image to be tested. The cell image recognition model is based on the yolo network, and uses the GhostNet feature extraction network as the backbone network, which can quickly perform image analysis and is particularly suitable for rapid diagnosis of whether there are tumor lesions in the target area during surgery.

[0071] The following will describe in detail the process of training the cell image recognition model according to the embodiment of the present application.

[0072] In one embodiment, as shown in FIG3 , the trained cell image recognition model is obtained by training in the following steps:

[0073] Step 302: Acquire an initial sample cell image set with cell information annotations.

[0074] Among them, the cell information annotation includes the cell category annotation and cell position annotation of the sample cell image.

[0075] Exemplarily, the computer device of the embodiment of the present application can obtain an initial sample cell image set with glioma cells, glioma cell locations, normal cells, and normal cell locations from a public data set or a specific medical data set. Alternatively, the computer device can also obtain an unlabeled cell image, which may include multiple cells, and the multiple cells can be one or more of glioma cells and normal cells. The computer device uses a rectangular box of appropriate size to circle the positions of glioma cells and normal cells, marks the coordinates of the diagonal vertices of the rectangular box as cell position information, and marks glioma cells or normal cells as cell category information.

[0076] Step 304 , data enhancement is performed on the initial sample cell image set to obtain an enhanced sample cell image set carrying cell information annotations.

[0077] Since the number of normal cell samples and the number of tumor cell samples are generally quite different, the embodiment of the present application can perform data enhancement on the initial sample cell image set to obtain an enhanced sample cell image set with cell information annotations.

[0078] Step 306: Perform cell recognition on the target sample cell image using the cell image recognition model to obtain a sample cell result.

[0079] The target sample cell images include initial sample cell images in the initial sample cell image set and enhanced sample cell images in the enhanced sample cell image set.

[0080] For example, after data enhancement is performed on the initial sample cell image set to obtain an enhanced sample cell image set with cell information annotations, the initial sample image set and the enhanced sample image set are merged to obtain a target sample image set. The cell image recognition model is used to perform cell recognition on the target sample images in the target sample image set to obtain the cell category and cell position corresponding to the target sample image.

[0081] Step 308: Optimize the parameters of the cell image recognition model according to the sample cell results and cell information annotations.

[0082] The embodiment of the present application calculates the loss value of the loss function through the sample cell results and the real information of the cell category and cell position marked by the cell information, and optimizes the parameters of the cell image recognition model through the loss value. The cell image recognition model is iteratively trained through the target sample cell image set, and the gap is narrowed through continuous iteration until the model converges to obtain a trained cell image recognition model.

[0083] In a possible implementation of the embodiment as shown in FIG3 , the cell image recognition model includes a GhostNet feature extraction network, an SPP pooling network, a PANet neck network, and a head prediction network connected in sequence.

[0084] Please refer to Figure 4. As an example, the cell image recognition model includes a GhostNet feature extraction network, an SPP pooling network, a PANet neck network and a head prediction network. Among them, the GhostNet feature extraction network includes a Conv2d+BN+ReLU6 structure, a first Ghost structure, a second Ghost structure, a third Ghost structure, a fourth Ghost structure and a fifth Ghost structure connected in sequence. The third Ghost structure and the fourth Ghost structure are connected to the PANet neck network through a layer of convolution respectively. The fifth Ghost structure is connected to the SPP pooling network through three layers of standard convolution layers. The SPP pooling network includes three parallel maximum pooling layers, and the pooling kernel sizes of the maximum pooling layers are 5×5, 9×9 and 13×13 respectively. The pooling results of the SPP pooling network are spliced ​​and connected to the PANet neck network through three layers of standard convolution layers. The PANet neck network includes multiple convolution upsampling and convolution downsampling. Further, the PANet neck network of the embodiment of the present application also includes multiple combined convolutions, which include two layers of depth-separable convolutions connected in sequence and three layers of standard convolution layers connected in sequence. It can be understood that the three sequentially connected standard convolutional layers in the cell image recognition model structure include the first convolution layer, the second convolution layer and the third convolution layer.

[0085] As shown in FIG5 , the cell image recognition model is used to perform cell recognition on the target sample cell image to obtain the sample cell results, including:

[0086] Step 502: Based on the GhostNet feature extraction network, feature extraction is performed on the target sample cell image to obtain a first feature map, a second feature map, and a third feature map corresponding to the target sample cell image.

[0087] Among them, the first feature map is the feature map output by the third GhostNet structure, the second feature map is the feature map output by the fourth GhostNet structure, and the third feature map is the feature map output by the fifth GhostNet structure, that is, the last layer of feature maps extracted by the GhostNet feature extraction network.

[0088] Step 504: After the third feature map passes through three sequentially connected standard convolution layers, it is input into the SPP pooling network for pooling processing, and the results after the pooling processing are concatenated with three sequentially connected standard convolution layers to obtain a fourth feature map.

[0089] Step 506: Based on the PANet neck network, the first feature map, the second feature map, and the fourth feature map are enhanced and features at different levels are fused to output feature vectors of multiple scales.

[0090] The feature vectors of multiple scales are used to characterize target sample regions of multiple different sizes corresponding to the target sample cell images.

[0091] Step 508 , based on the head prediction network, predict multiple bounding boxes in target sample regions of multiple different sizes, and determine the confidence that the multiple bounding boxes correspond to normal cells and tumor cells, respectively, to obtain sample cell results.

[0092] Exemplarily, the computer device determines a confidence score for each bounding box, the confidence score being the probability that the bounding box contains a tumor cell or a normal cell and the overlap rate between the predicted bounding box and the input target sample region. For each target sample region, the conditional probability of containing a normal cell or a tumor cell is predicted, and the confidence score is multiplied by the conditional probability to obtain the confidence of each bounding box relative to the normal cell or the tumor cell.

[0093] In one embodiment, as shown in FIG6 , the initial sample cell image set is data enhanced to obtain an enhanced sample cell image set carrying cell information annotations, including:

[0094] Step 602 , based on the preset multi-dimensional cell characteristic parameters, obtain the cell characteristic data of each dimension corresponding to the normal cells and the tumor cells in the initial sample cell image set.

[0095] The preset multi-dimensional cell characteristic parameters may be preset characteristic parameters related to normal cells and tumor cells that can be extracted from cell images, for example, various spectral characteristics, cell size, shape, texture characteristics, color characteristics, such as red-green point number ratio characteristic attribute, red-green point area ratio characteristic attribute, red-green point disorder degree characteristic attribute, etc.

[0096] The embodiments of the present application do not limit the method of obtaining cell feature data in each dimension, such as feature extraction based on geometric shapes, feature extraction based on statistical analysis, feature extraction based on frequency domain analysis, feature extraction based on wavelet transform, etc.

[0097] Step 604 , for each cell category, correlation analysis is performed between the cell feature data of each dimension corresponding to the cell category and the cell category to obtain the correlation between the cell feature parameters of each dimension and the cell category.

[0098] The computer device of the embodiment of the present application performs correlation analysis on the cell feature data of each dimension and the cell category corresponding to the normal cells and tumor cells in the initial sample cell image of the initial sample cell image set, and obtains the correlation between the cell feature parameters of each dimension and the cell category.

[0099] Step 606 , based on the correlation, the cell characteristic parameters with high correlation to the normal cell category are eliminated from the cell characteristic parameters with low correlation to the tumor cell category to obtain the target cell characteristic parameters.

[0100] The computer device of the embodiment of the present application screens out cell characteristic parameters with low correlation with tumor cell categories, and eliminates cell characteristic parameters with high correlation with normal cell categories, thereby obtaining target cell characteristic parameters.

[0101] Step 608 , based on the target cell characteristic parameters, data perturbation is performed on the initial sample cell image set to obtain an enhanced sample cell image set carrying cell information annotations.

[0102] In a possible implementation of the embodiment shown in FIG. 6 , the step of obtaining an enhanced sample cell image set carrying cell information annotations is shown in FIG. 7 , and includes:

[0103] Step 702 : extracting a sample tumor cell sub-image from an initial sample cell image in an initial sample cell image set based on the cell information annotation.

[0104] Exemplarily, based on the position information of the tumor cells in the cell information annotation, at least one sample tumor cell sub-image is extracted from each initial sample image of the initial sample cell image set. For example, the tumor cell region in each initial sample cell image is cut out from the image to obtain at least one sample tumor cell sub-image.

[0105] Step 704 : Based on the target cell characteristic parameters, data perturbation is performed on the sample tumor cell sub-image to obtain a first enhanced tumor cell sub-image.

[0106] Exemplarily, the cell feature data corresponding to the target cell feature parameter in the sample tumor cell sub-image is perturbed to obtain a first enhanced tumor cell sub-image corresponding to the sample tumor cell sub-image. For example, for the selected target cell feature parameter, the feature data corresponding to the sample tumor cell sub-image is perturbed to obtain a new sample tumor cell sub-image as the first enhanced tumor cell sub-image corresponding to the sample tumor cell sub-image.

[0107] Step 706 : Combine the first enhanced tumor cell sub-image and the initial sample cell image corresponding to the first enhanced tumor cell sub-image to obtain a first enhanced sample cell image carrying cell information annotations.

[0108] The initial sample cell image corresponding to the first enhanced tumor cell sub-image refers to the initial sample cell image where the sample tumor cell sub-image corresponding to the first enhanced tumor cell sub-image is located.

[0109] For example, the first enhanced tumor cell sub-image is pasted and fused into the initial sample cell image, and the tumor cell category and pasting and fusion position information are recorded to obtain the first enhanced sample cell image with cell information annotation. The first enhanced sample cell image can be used to determine the enhanced sample cell image set.

[0110] In a possible implementation of the embodiment shown in FIG. 7 , the process of obtaining an enhanced sample cell image set carrying cell information annotations may further include the steps shown in FIG. 8 :

[0111] Step 802 : Using the sample tumor cell subgraph as sample data and the first enhanced tumor cell subgraph corresponding to the sample tumor cell subgraph as label data, a second enhanced tumor cell subgraph model is trained.

[0112] Step 804 : Input the sample tumor cell sub-graph and the first enhanced tumor cell sub-graph into the trained second enhanced tumor cell sub-graph model respectively to obtain the second enhanced tumor cell sub-graph.

[0113] The embodiment of the present application uses the sample tumor cell sub-image in the initial sample cell image set as sample data, and uses the first enhanced tumor cell sub-image generated by data perturbation of the sample tumor cell sub-image as label data, and uses the mapping relationship between the sample tumor cell sub-image and the first enhanced tumor cell sub-image to train the second enhanced tumor cell sub-image model. The trained second enhanced tumor cell sub-image model is then used to perform data enhancement on the sample tumor cell sub-image and the first enhanced tumor cell sub-image to obtain a second enhanced tumor cell sub-image. The second enhanced tumor cell sub-image is pasted and fused to its corresponding initial sample cell image, and the tumor cell category and the location information of the pasting and fusion are recorded to obtain a second enhanced sample cell image carrying cell information annotations. Combine the first enhanced sample cell image and the second enhanced sample cell image to obtain an enhanced sample cell image set.

[0114] In this embodiment, data enhancement is performed on the tumor cell region to balance the difference in the number of normal cell samples and the number of tumor cell samples, thereby improving the accuracy of the cell image recognition model training.

[0115] When the cell image recognition model of the embodiment of the present application is actually applied after training, in the recognition result of the cell image to be tested obtained by the trained cell image recognition model based on the cell image to be tested, the computer device will execute multiple groups of three-layer standard convolution operations connected in sequence. For example, the third feature map output by the GhostNet feature extraction network needs to undergo three-layer standard convolution operations connected in sequence, the pooling results need to be spliced ​​and then subjected to three-layer standard convolution operations connected in sequence, and the PANet neck network also has multiple groups of three-layer standard convolution operations connected in sequence.

[0116] In order to further improve the application efficiency of the cell image recognition method of the embodiment of the present application and improve its response rate in actual surgery, in the process of obtaining the recognition result of the cell image to be tested through the trained cell image recognition model in the embodiment of the present application, when the computer device performs the three-layer standard convolution operation connected in sequence, as shown in FIG9, the following steps can be adopted:

[0117] Step 902: Based on the convolution kernel mean of the first convolution layer, a convolution operation is performed on the feature map of the cell image to be tested input by the first convolution layer to obtain a first output; at the same time, based on the convolution kernel of the first convolution layer, a convolution operation is performed on the feature map of the cell image to be tested input by the first convolution layer to obtain a third output; and after obtaining the third output, a difference between the third output and the first output is used to obtain a fourth output.

[0118] Step 904, after obtaining the first output, based on the mean of the convolution kernel of the second layer of convolution, perform a convolution operation on the first output to obtain a second output; at the same time, based on the convolution kernel of the second layer of convolution, perform a convolution operation on the first output to obtain a fifth output; after obtaining the fourth output, based on the convolution kernel of the second layer of convolution, perform a convolution operation on the fourth output to obtain a sixth output; and combine the fifth output and the sixth output to obtain a seventh output; use the difference between the seventh output and the second output to obtain a ninth output.

[0119] Step 906: After obtaining the second output, perform a convolution operation on the second output based on the convolution kernel of the third layer of convolution to obtain an eighth output; after obtaining the ninth output, perform a convolution operation on the ninth output based on the convolution kernel of the third layer of convolution to obtain a tenth output; and combine the eighth output and the tenth output to obtain an eleventh output, which is the final output of the three standard convolution layers connected in sequence.

[0120] Among them, using the convolution kernel mean for convolution operation means directly multiplying the convolution kernel mean with the sum of the feature map data corresponding to its sliding. Using the convolution kernel for convolution operation means the conventional convolution operation, that is, each weight in the convolution kernel is multiplied by the corresponding feature map data multiple times and then summed.

[0121] In this embodiment, since the convolution kernel in the trained cell image recognition model has been determined, the mean of each weight in the convolution kernel, that is, the convolution kernel mean, can be predetermined. The first layer of convolution kernel mean can be directly multiplied with the sum of the data in the feature map of the cell image to be tested (without multiplying each weight in the first layer of convolution kernel with the data in the corresponding feature map multiple times) to obtain the first output. Then, based on the convolution kernel mean of the second layer of convolution, the first output is convolved to obtain the second output. It can be understood that the calculation time of the first output and the second output is very short. After the second output is subjected to a normal convolution operation using the convolution kernel of the third layer of convolution, the eighth output is obtained, and then the tenth output and the eighth output are added. The tenth output is used to correct the eighth output, and the final output of the three layers of standard convolution layers connected in sequence can be quickly obtained.

[0122] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0123] Based on the same inventive concept, the embodiment of the present application also provides a cell image recognition device for implementing the cell image recognition method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more cell image recognition device embodiments provided below can refer to the limitations of the cell image recognition method above, and will not be repeated here.

[0124] In one embodiment, as shown in FIG10 , a cell image recognition device is provided, including: a receiving module 1002 and a recognition module 1004, wherein:

[0125] The receiving module 1002 is used to receive the image of the cell to be tested sent by the image acquisition device; the image of the cell to be tested is a real-time image of the living tissue in the field of view of the endoscope, and the image acquisition device is connected to the endoscope.

[0126] The recognition module 1004 is used to obtain the recognition result of the cell image to be tested through the trained cell image recognition model based on the cell image to be tested; the recognition result includes the cell category and the cell position. The cell image recognition model is based on the yolo network, and the GhostNet feature extraction network is used as the backbone network.

[0127] In one embodiment, the cell image recognition device also includes a training module, which is used to: obtain an initial sample cell image set carrying cell information annotations, wherein the cell information annotations include cell category annotations and cell position annotations of the sample cell images; perform data enhancement on the initial sample cell image set to obtain an enhanced sample cell image set carrying cell information annotations; use a cell image recognition model to perform cell recognition on target sample cell images to obtain sample cell results; the target sample cell images include the initial sample cell images in the initial sample cell image set and the enhanced sample cell images in the enhanced sample cell image set; and optimize the parameters of the cell image recognition model based on the sample cell results and the cell information annotations.

[0128] In one embodiment, the cell image recognition model includes a GhostNet feature extraction network, an SPP pooling network, a PANet neck network and a head prediction network connected in sequence; when the training module performs the cell recognition on the target sample cell image using the cell image recognition model to obtain the sample cell result, it is configured as follows: based on the GhostNet feature extraction network, feature extraction is performed on the target sample cell image to obtain the first feature map, the second feature map and the third feature map corresponding to the target sample cell image; the third feature map is the last layer feature map extracted by the GhostNet feature extraction network; the third feature map is sequentially connected through three layers of standard After the quasi-convolutional layer, the image is input into the SPP pooling network for pooling processing, and the result after the pooling processing is spliced ​​and connected to three layers of standard convolutional layers in sequence to obtain a fourth feature map; based on the PANet neck network, the first feature map, the second feature map and the fourth feature map are feature enhanced and features of different levels are fused to output feature vectors of multiple scales, and the feature vectors of multiple scales are used to characterize target sample areas of multiple different sizes corresponding to the target sample cell image; based on the head prediction network, multiple bounding boxes are predicted in the target sample areas of multiple different sizes, and the confidence that the multiple bounding boxes correspond to normal cells and tumor cells respectively is determined to obtain the sample cell result.

[0129] In one embodiment, the PANet neck network also includes a plurality of combined convolutions, the combined convolutions including two layers of depth-separable convolutions connected in sequence and three layers of standard convolutions connected in sequence; the three layers of standard convolutions connected in sequence include a first layer of convolution, a second layer of convolution and a third layer of convolution in sequence; the recognition module 1004 is configured to perform the operation of the three layers of standard convolutions connected in sequence when performing the operation of the three layers of standard convolutions connected in sequence in the process of recognizing the cell image to be tested through the trained cell image recognition model: based on the convolution kernel mean of the first layer of convolution, a convolution operation is performed on the feature map of the cell image to be tested input by the first layer of convolution to obtain a first output; at the same time, based on the convolution kernel of the first layer of convolution, a convolution operation is performed on the feature map of the cell image to be tested input by the first layer of convolution to obtain a third output; and after obtaining the third output, a fourth output is obtained by using the difference between the third output and the first output; after obtaining the third output, a fourth output is obtained. After the first output, based on the mean of the convolution kernel of the second layer of convolution, a convolution operation is performed on the first output to obtain the second output; at the same time, based on the convolution kernel of the second layer of convolution, a convolution operation is performed on the first output to obtain the fifth output; after obtaining the fourth output, based on the convolution kernel of the second layer of convolution, a convolution operation is performed on the fourth output to obtain the sixth output; and the fifth output and the sixth output are combined to obtain the seventh output; the difference between the seventh output and the second output is used to obtain the ninth output; after obtaining the second output, based on the convolution kernel of the third layer of convolution, a convolution operation is performed on the second output to obtain the eighth output; after obtaining the ninth output, based on the convolution kernel of the third layer of convolution, a convolution operation is performed on the ninth output to obtain the tenth output; and the eighth output and the tenth output are combined to obtain the eleventh output, and the eleventh output is the final output of the three standard convolution layers connected in sequence.

[0130] In one embodiment, when the training module performs data enhancement on the initial sample cell image set to obtain an enhanced sample cell image set with cell information annotations, the training module is configured to: obtain cell feature data of each dimension corresponding to normal cells and tumor cells in the initial sample cell image set for preset multi-dimensional cell feature parameters; for each cell category, perform association analysis on the cell feature data of each dimension corresponding to the cell category and the cell category to obtain the correlation between the cell feature parameters of each dimension and the cell category; based on the correlation, eliminate the cell feature parameters with high correlation to the normal cell category from the cell feature parameters with low correlation to the tumor cell category to obtain the target cell feature parameters; based on the target cell feature parameters, perform data perturbation on the initial sample cell image set to obtain an enhanced sample cell image set with cell information annotations.

[0131] In one embodiment, when the training module performs the data perturbation on the initial sample cell image set based on the target cell characteristic parameters to obtain the enhanced sample cell image set carrying cell information annotations, it is configured to: extract a sample tumor cell sub-image from the initial sample cell image in the initial sample cell image set based on the cell information annotations; perform data perturbation on the sample tumor cell sub-image based on the target cell characteristic parameters to obtain a first enhanced tumor cell sub-image; combine the first enhanced tumor cell sub-image and the initial sample cell image corresponding to the first enhanced tumor cell sub-image to obtain a first enhanced sample cell image carrying cell information annotations, and the first enhanced sample cell image is used to determine the enhanced sample cell image set.

[0132] Each module in the above-mentioned cell image recognition device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0133] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG11. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Wherein, the processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. Wherein, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a cell image recognition method is implemented.

[0134] Those skilled in the art will understand that the structure shown in FIG. 11 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0135] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0136] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0137] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0138] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0139] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0140] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A cell image recognition method, characterized in that: The method comprises: Receiving the image of the cell to be tested sent by the image acquisition device; the image of the cell to be tested is a real-time image of the living tissue in the field of view of the endoscope, and the image acquisition device is connected to the endoscope; Based on the cell image to be tested, obtaining a recognition result of the cell image to be tested by a trained cell image recognition model; the recognition result includes a cell type and a cell position; Among them, the cell image recognition model is based on the Yolo network, and the GhostNet feature extraction network is used as the backbone network.

2. The method according to claim 1, characterized in that: The trained cell image recognition model is obtained by training in the following steps: Acquire an initial sample cell image set carrying cell information annotations, wherein the cell information annotations include cell category annotations and cell position annotations of the sample cell images; Performing data enhancement on the initial sample cell image set to obtain an enhanced sample cell image set carrying cell information annotations; Using the cell image recognition model to perform cell recognition on the target sample cell image, and obtain the sample cell result; The target sample cell images include the initial sample cell images in the initial sample cell image set and the enhanced sample cell images in the enhanced sample cell image set; Parameters of the cell image recognition model are optimized according to the sample cell results and the cell information annotations.

3. The method according to claim 2, characterized in that The cell image recognition model includes a GhostNet feature extraction network, an SPP pooling network, a PANet neck network and a head prediction network connected in sequence; The method of using the cell image recognition model to perform cell recognition on the target sample cell image to obtain the sample cell result includes: Based on the GhostNet feature extraction network, feature extraction is performed on the target sample cell image to obtain a first feature map, a second feature map, and a third feature map corresponding to the target sample cell image; the third feature map is the last layer feature map extracted by the GhostNet feature extraction network; The third feature map is input into the SPP pooling network for pooling after passing through three standard convolution layers connected in sequence, and the results after the pooling process are concatenated and passed through three standard convolution layers connected in sequence to obtain a fourth feature map; Based on the PANet neck network, feature enhancement is performed on the first feature map, the second feature map, and the fourth feature map, and features at different levels are fused to output feature vectors of multiple scales, where the feature vectors of multiple scales are used to characterize target sample regions of multiple different sizes corresponding to the target sample cell image; Based on the head prediction network, multiple bounding boxes are predicted in target sample regions of multiple different sizes, and confidences that the multiple bounding boxes correspond to normal cells and tumor cells are determined respectively, to obtain the sample cell results.

4. The method according to claim 3, characterized in that The PANet neck network also includes a plurality of combined convolutions, wherein the combined convolutions include two layers of sequentially connected depthwise separable convolutions and three layers of sequentially connected standard convolution layers; the three layers of sequentially connected standard convolution layers sequentially include a first convolution layer, a second convolution layer, and a third convolution layer; the method also includes: Based on the convolution kernel mean of the first convolution layer, a convolution operation is performed on the feature map of the cell image to be tested input by the first convolution layer to obtain a first output; at the same time, based on the convolution kernel of the first convolution layer, a convolution operation is performed on the feature map of the cell image to be tested input by the first convolution layer to obtain a third output; and after obtaining the third output, a difference between the third output and the first output is used to obtain a fourth output; After obtaining the first output, based on the convolution kernel mean of the second convolution layer, the first output is convolved to obtain the second output; at the same time, based on the convolution kernel of the second convolution layer, the first output is convolved to obtain the fifth output; after obtaining the fourth output, based on the convolution kernel of the second convolution layer, the fourth output is convolved to obtain the sixth output; and the fifth output and the sixth output are combined to obtain the seventh output; and the difference between the seventh output and the second output is used to obtain the ninth output; After obtaining the second output, a convolution operation is performed on the second output based on the convolution kernel of the third layer of convolution to obtain an eighth output; after obtaining the ninth output, a convolution operation is performed on the ninth output based on the convolution kernel of the third layer of convolution to obtain a tenth output; and the eighth output and the tenth output are combined to obtain an eleventh output, which is the final output of the three standard convolution layers connected in sequence.

5. The method according to claim 2, characterized in that: The data enhancement of the initial sample cell image set to obtain an enhanced sample cell image set carrying cell information annotations includes: According to the preset multi-dimensional cell characteristic parameters, the cell characteristic data of each dimension corresponding to the normal cells and the tumor cells in the initial sample cell image set are obtained; For each cell category, correlation analysis is performed between the cell feature data of each dimension corresponding to the cell category and the cell category to obtain the correlation between the cell feature parameters of each dimension and the cell category; Based on the correlation, the cell characteristic parameters with high correlation to the normal cell category are eliminated from the cell characteristic parameters with low correlation to the tumor cell category to obtain the target cell characteristic parameters; Based on the target cell characteristic parameters, data perturbation is performed on the initial sample cell image set to obtain an enhanced sample cell image set carrying cell information annotations.

6. The method according to claim 5, characterized in that The method of performing data perturbation on the initial sample cell image set based on the target cell characteristic parameters to obtain an enhanced sample cell image set carrying cell information annotations includes: Based on the cell information annotation, extracting a sample tumor cell sub-image from the initial sample cell image in the initial sample cell image set; Based on the target cell characteristic parameters, performing data perturbation on the sample tumor cell sub-image to obtain a first enhanced tumor cell sub-image; The first enhanced tumor cell sub-image and the initial sample cell image corresponding to the first enhanced tumor cell sub-image are combined to obtain a first enhanced sample cell image carrying cell information annotations, and the first enhanced sample cell image is used to determine the enhanced sample cell image set.

7. A cell image recognition device, characterized in that: The device comprises: A receiving module, used for receiving the image of the cell to be tested sent by the image acquisition device; the image of the cell to be tested is a real-time image of the living tissue in the field of view of the endoscope, and the image acquisition device is connected to the endoscope; A recognition module, used for obtaining a recognition result of the cell image to be tested based on the cell image to be tested by a trained cell image recognition model; the recognition result includes a cell type and a cell position; Among them, the cell image recognition model is based on the Yolo network, and the GhostNet feature extraction network is used as the backbone network.

8. A cell image recognition system, characterized in that: The system comprises: endoscopes, which are used to obtain real-time images of living tissue in the field of view; An image acquisition device, connected to the endoscope, for acquiring a real-time image of the living tissue in the field of view of the endoscope, and sending the real-time image of the living tissue as an image of the cell to be tested to a computer device; A computer device is connected to the image acquisition device and is used to receive the cell image to be tested sent by the image acquisition device; based on the cell image to be tested, a recognition result of the cell image to be tested is obtained through a trained cell image recognition model; the recognition result includes cell category and cell position; wherein the cell image recognition model is based on a yolo network, and uses a GhostNet feature extraction network as a backbone network.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images