Liver balloon-like variant cell automatic detection method based on medical mechanism and deep learning

Abstract

The invention relates to an automatic hepatic balloon-like deformation cell detection method based on a medical mechanism and deep learning. Comprising the steps of establishing a liver pathology image data set; and constructing a hepatic balloon-like variant cell detection network based on a medical mechanism and a deep learning technology, and training and testing the hepatic balloon-like variant cell detection network. The detection network mainly comprises a backbone network, a segmentation convolution network, a global perception network, a bidirectional multistage aggregation network and a head network, and the segmentation convolution network is mainly used for extracting high-resolution local features such as eosinophilic sphericity, lipid droplet glycogen and loose and transparent cytoplasm; the global sensing network is mainly used for extracting global information such as liver lesion severity, and the bidirectional multi-stage aggregation network is mainly used for extracting multi-scale context information such as relative positions of cell nucleuses and relative volumes of cells. According to the automatic detection method for detecting the hepatic balloon-like deformation cells, the method is based on the medical mechanism that balloon-like deformation is prompted through the image information, the hepatic balloon-like deformation cells can be recognized more accurately, and the method has important clinical application value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent medical treatment, more particularly, to a liver ballooning cell automatic detection method based on medical mechanism and deep learning. BACKGROUND

[0002] Metabolic dysfunction-associated steatohepatitis (MASLD) has become one of the most common liver diseases worldwide. Up to 20-30% of MASLD patients eventually develop metabolic steatohepatitis (MASH), which can progress to advanced fibrosis, cirrhosis and liver cancer. Therefore, early and accurate diagnosis of MASH is crucial for controlling the progression of this common liver disease. The diagnosis of MASH usually involves the combination of multiple methods, including histological examination, liver function examination, imaging examination and serum biomarker examination. Among these diagnostic methods, liver histological examination is considered the most reliable method for diagnosing MASH.

[0003] In liver histological examination, hepatocyte ballooning is a basic feature of MASH and plays a crucial role in MASH diagnosis. Hepatocyte ballooning is one of the four key histological features for diagnosing MASH, and it is considered an indicator of cell injury and stress, which is crucial for the diagnosis and severity assessment of MASH. However, in actual clinical application, due to the large area of pathological images and the lack of automatic detection tools for liver ballooning cells, it is difficult for doctors to quantitatively analyze the severity of MASH. At the same time, due to the complex characteristics and ambiguous definition criteria of liver ballooning cells, existing traditional image processing methods are difficult to accurately and effectively identify liver ballooning cells. SUMMARY

[0004] In view of the above problems, in combination with the influence of image features such as eosinophilic spherical characteristics, cytoplasmic flocculent characteristics and nucleus position on the discrimination of liver ballooning cells, the present application provides a liver ballooning cell automatic detection method based on medical mechanism and deep learning.

[0005] A liver ballooning cell automatic detection method based on medical mechanism and deep learning, which specifically comprises the following steps:

[0006] S1: Establishing a liver pathological image dataset: Collect steatohepatitis pathological sections from multiple centers and label ballooning cells, pre-process the images to establish a liver pathological image dataset, and divide the liver pathological image dataset to generate a training set and a test set;

[0007] S2: Constructing a liver ballooning cell detection network: based on the medical mechanism of liver ballooning cells, a liver ballooning cell detection network is constructed, which mainly includes a backbone network, a segmentation convolution network, a global perception network, a bidirectional multi-level aggregation network and a head network;

[0008] S3: training a liver ballooning cell detection network: first, training the feature extraction part of the detection network separately using the training set, and then training the liver ballooning cell detection network in its entirety;

[0009] S4: testing the liver ballooning cell detection network: first, evaluating the network performance using the test set, and then connecting the network to the optical microscope system to evaluate the actual application performance of the network.

[0010] Further, the step S1 specifically comprises the following steps:

[0011] S11: Collecting pathological sections of fatty liver hepatitis in multiple centers, performing hematoxylin-eosin (HE) staining on the collected pathological sections, obtaining corresponding full-field digital section images through a digital pathology full-section scanner, and marking ballooning cells using a pathological image marking software;

[0012] S12: To solve the problem of marking omission caused by the large area of liver pathological images and the small number of liver ballooning cells, a two-step method is used during marking, first, the entire full-field digital section is divided into a region of interest, and then only the liver ballooning cells within the region of interest are marked, reducing the doctor's marking attention area and reducing the marking omission of liver ballooning cells;

[0013] S13: The preprocessing mainly includes data cleaning, staining normalization and data enhancement, wherein the data enhancement includes rotation, random noise addition, random cropping, etc.

[0014] Further, in the step S13, to solve the problem of large background of pathological images and small number of liver ballooning cells, the data enhancement method includes cropping the liver ballooning cell region by manually marking the cell position information, using the Canny edge detection algorithm to obtain a cell segmentation image, and fusing the cell segmentation image with other background images to generate an enhanced positive sample.

[0015] Further, the step S2 specifically comprises the following steps:

[0016] S21: The backbone network mainly includes a first convolution stage, a second convolution stage, a third convolution stage, a fourth convolution stage and a fifth convolution stage, and the five groups of convolution stages respectively include 2, 3, 3, 4 and 3 groups of residual convolution modules. Each stage performs down-sampling while extracting image features to generate first-stage features, second-stage features, third-stage features, fourth-stage features and fifth-stage features of different scales;

[0017] S22: The split convolution network mainly consists of a split convolution layer, a normalization layer and an activation layer, and is used to generate high-resolution local features according to the first-stage features in step S21;

[0018] S23: The global perception network mainly consists of a self-global pooling layer, an attention module, a fully connected layer, a normalization layer and an activation layer, and is used to generate global features according to the fifth-stage features in step S21;

[0019] S24: The bidirectional multi-level aggregation network mainly consists of a reshaping module, an expansion module, a convolution module and a splicing module, and is used to fuse the second, third, fourth and fifth-stage features generated in step S21, the high-resolution local features generated in step S22 and the global features generated in step S23, to generate multi-scale context features;

[0020] S25: The head network mainly consists of a reshaping layer, a fully connected layer, a normalization layer, an activation layer and a Sigmoid layer, and is used to generate a position and class vector of the hepatocellular ballooning cell according to the multi-scale context features extracted in step S24, to realize the detection of the hepatocellular ballooning cell.

[0021] Further, in step S22, the split convolution network is mainly used to extract high-resolution local features such as eosinophilic globule, lipid droplet glycogen, and cytoplasmic loose transparency.

[0022] Still further, in step S22, the split convolution is a special convolution that first splits the input matrix in the spatial dimension, thereby dividing the input matrix into multiple regions, and then uses a separate convolution kernel to process the data of each region, the process being as follows:

[0023] F 1 , F 2 ,..., F s = Split(F)

[0024]

[0025] G = Concat(G 1 , G 2 ,..., G S )

[0026] where Split(·) and Concat(·) represent the splitting and splicing processes respectively, is the input feature map, is the input feature map, is the split input feature map, To split the output feature map, I and J are the width and height of the feature map in the spatial level, M and N are the number of channels of the input and output features, p and q represent the coordinates of each point on the split feature map in the spatial level, both are integers and the ranges are and

[0027] Unlike ordinary convolution, each convolution kernel of the split convolution only extracts information of a part of the spatial level region, and the convolution kernel weight is not affected by the information of the adjacent region during back propagation, having certain independence and anti-interference ability. At the same time, since the split convolution does not introduce additional convolution calculation process and the number of parameters of the shallow layer of the basic model is small, the split convolution can greatly increase the number of shallow layer parameters without affecting the model calculation efficiency and the total number of parameters, thereby improving the extraction ability of the network to the shallow layer high resolution features.

[0028] Further, in the step S23, the global perception network is mainly used to extract global information such as the severity of liver lesions.

[0029] Further, in the step S23, the self-attention module mainly consists of a full connection layer, a Softmax layer and a normalization layer, which converts the input features into an attention vector, and then weights the input features, thereby strengthening the extraction of effective information in the global features.

[0030] Further, in the step S24, the bidirectional multi-level aggregation network is mainly used to extract local context features such as cell shape contour and relative position of cell nucleus, and global context features such as relative volume of cells.

[0031] Further, in the step S24, the reshaping module mainly consists of a reshaping layer, a 1x1 convolution layer, a normalization layer and an activation layer, the convolution module mainly consists of a convolution layer, a normalization layer and an activation layer, and the splicing module mainly consists of a splicing layer, a convolution layer, a normalization layer and an activation layer.

[0032] Further, in the step S24, the expansion module mainly consists of three groups of convolution layers, and normalization layers and activation layers, wherein the three groups of convolution layers are composed of an ordinary convolution layer and expansion convolution layers with expansion ratios of 2 and 3, respectively. The two groups of expansion convolution are mainly used to strengthen the extraction of multi-scale context information.

[0033] Further, the step S3 specifically comprises the following steps:

[0034] S31: In order to solve the problem that the number of hepatocellular ballooning cells is small and easy to cause model overfitting, the application pre-trains four feature extraction networks of the backbone network, the split convolution network, the global perception network and the bidirectional multi-level aggregation network with the training set alone;

[0035] S32: After the pre-training of the feature extraction network is completed, the liver gas ballooning target detection network is completely trained using the training set.

[0036] Further, in the step S31, when the feature extraction network is pre-trained, the feature extraction network and a classification head network are first spliced to form a temporary complete classification network, and the classification network is used to distinguish whether the full field digital section to which the input image belongs contains liver gas ballooning cells.

[0037] Further, the input image of the classification network is a cut sub-image in the region of interest of the full field section, and since the cut sub-image does not need to contain liver gas ballooning cells, the number is extremely large, so that the model can avoid falling into overfitting when learning image features.

[0038] Further, in the step S32, in the first round of network training, the parameters of the feature extraction network are frozen, and in the second to fifth rounds, the parameters of the feature extraction network are gradually unfrozen.

[0039] Further, the input image of the detection network during training is a cut sub-image in the region of interest of the full field section, 75% of which contains liver gas ballooning cells, and 25% of which does not contain liver gas ballooning cells.

[0040] Further, the step S4 specifically comprises the following steps:

[0041] S41: Using evaluation indexes such as AR50, AP50, AP75 and AP@50:5:95, the detection performance of the network on the test set is evaluated, and the effectiveness and advancement of the network are verified by comparing it with traditional image processing methods and existing deep learning methods such as YOLO v3, YOLO v8 and Swin Transformer.

[0042] S42: The network is connected to an optical microscope system to evaluate the detection accuracy and system response speed of the network in a real environment, and to verify the feasibility and effectiveness of the actual application of the system.

[0043] Compared with the prior art:

[0044] The present application designs a set of liver gas ballooning cell automatic detection system, which is trained and tested by a large number of liver pathological images in multiple centers, can effectively detect liver gas ballooning cells in pathological images, and further assist the quantitative diagnosis of MASH.

[0045] In view of the characteristics of cytoplasm forming acidophilic spherical, absence of lipid droplets or glycogen, cytoplasm loose and transparent, etc., the present application designs a split convolution network composed of split convolution, etc., to improve the local feature extraction capability of the detection network.

[0046] In view of the characteristics of balloon-like degeneration prompted by global information such as the severity of liver lesions, the global perception network composed of global pooling, self-attention modules and the like is designed to improve the global feature extraction capability of the detection network.

[0047] In view of the characteristics of balloon-like degeneration prompted by local context information such as the circular shape of cells, the uncentered position of the nucleus, and the characteristics of balloon-like degeneration prompted by global context information such as the increase in the relative volume of cells, the bidirectional multi-level aggregation network with a composite connection structure is designed to improve the multi-scale context feature extraction capability of the detection network. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 FIG. 1 is a schematic diagram of the construction process of the automatic detection method for liver balloon-like degeneration cells provided by the embodiments of the present application;

[0049] Figure 2 FIG. 2 is a schematic diagram of the automatic detection process for liver balloon-like degeneration cells provided by the embodiments of the present application;

[0050] Figure 3 FIG. 3 is a schematic diagram of the network structure for automatic detection of liver balloon-like degeneration cells provided by the embodiments of the present application;

[0051] Figure 4 FIG. 4 is a schematic diagram of a backbone network structure provided by the embodiments of the present application;

[0052] Figure 5 FIG. 5 is a schematic diagram of a split convolution network structure provided by the embodiments of the present application;

[0053] Figure 6 FIG. 6 is a schematic diagram of a global perception network structure provided by the embodiments of the present application;

[0054] Figure 7 FIG. 7 is a schematic diagram of a bidirectional multi-level aggregation network structure provided by the embodiments of the present application;

[0055] Figure 8 FIG. 8 is a schematic diagram of a head network structure provided by the embodiments of the present application. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0057] In some of the flowcharts described in the specification and claims of the present application and in the above-mentioned drawings, a plurality of operations are included which occur in a particular order, but it should be clearly understood that these operations can be performed in the order in which they appear herein or in parallel, and the serial numbers of the operations, such as S1, S2, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these flowcharts can include more or fewer operations, and the operations can be performed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. herein are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do "first" and "second" represent different types.

[0058] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0059] Figure 1 is a schematic diagram of the construction process of the liver ballooning cell automatic detection method provided by the embodiments of the present application. Specifically, the method mainly includes four steps of establishing a liver pathological image data set, constructing a liver ballooning cell detection network, training a liver ballooning cell detection network, and testing a liver ballooning cell detection network.

[0060] S1: Establish a liver pathological image data set: Collect fatty hepatitis pathological sections from multiple centers and mark ballooning cells, pre-process the images to establish a liver pathological image data set, and divide the liver pathological image data set to generate a training set and a test set.

[0061] In one embodiment, first, collect fatty hepatitis pathological sections from multiple centers, pre-screen and clean the collected pathological sections to remove unqualified sections, and perform HE staining on the qualified pathological sections; then use a digital pathology whole section scanner to obtain corresponding 40x whole field digital section images under a microscope.

[0062] In one embodiment, the balloon-like cell in the image is marked by using pathological image marking software such as Motic DSAssistant, Aperio ImageScope, etc. In view of the large area of liver pathological images and the small number of liver balloon-like cells, a two-step method is adopted for marking. First, the whole field of view digital slice is marked to define the region of interest, and then only the liver balloon-like cells in the region of interest are marked, so as to reduce the marking omission of the liver balloon-like cells. One whole field of view digital slice image generally includes 3 to 7 regions of interest, and the pixel area of the region of interest is generally between 4 million and 20 million.

[0063] In one embodiment, the preprocessing mainly includes data cleaning, staining normalization and data enhancement, wherein the data enhancement includes rotation, random noise addition, random cropping and fusion generation, etc. In view of the large background of pathological images and the small number of liver balloon-like cells, the cell position information is marked by hand to crop the liver balloon-like cell region, the Canny edge detection algorithm is used to obtain the cell segmentation image, and the cell segmentation image and other background images are fused to generate enhanced positive samples.

[0064] S2: Constructing a liver balloon-like cell detection network: based on the medical mechanism of liver balloon-like cells, a liver balloon-like cell detection network is constructed, and the detection network mainly includes a backbone network, a split convolution network, a global perception network, a bidirectional multi-level aggregation network and a head network.

[0065] In one embodiment, the liver balloon-like cell automatic detection process is as shown in Figure 2 For the whole field of view slice image input into the system, first, the whole field of view slice image is overlapped and cut according to a fixed cutting step to generate a corresponding cutting sub-image, and preferably, the cutting step is 0.5; then each cutting sub-image is input into the liver balloon-like cell detection network to generate a corresponding cell detection result.

[0066] In one embodiment, the structure of the liver balloon-like cell detection network is as shown in Figure 3 The cutting sub-image is first input into the backbone network to generate five groups of features with different scales, i.e. the first stage feature, the second stage feature, the third stage feature, the fourth stage feature and the fifth stage feature; the first stage feature is input into the split convolution network to strengthen the extraction of local information and generate high-resolution local features; the fifth stage is input into the global perception network to generate global features; then, the second, third, fourth and fifth stage features, as well as the high-resolution local features and the global features are input into the bidirectional multi-level aggregation network to generate multi-scale context features after fusion; finally, the multi-scale context features are input into the head network to generate the cell detection result.

[0067] In one specific embodiment, the backbone network structure is as shown in Figure 4 The network mainly includes a first convolution stage, a second convolution stage, a third convolution stage, a fourth convolution stage and a fifth convolution stage, and the five groups of convolution stages respectively include 2, 3, 3, 4 and 3 groups of residual convolution modules. The residual convolution module is a convolution module with a cross-layer connection structure, which mainly includes a convolution layer, a normalization layer and an activation layer, wherein the normalization is preferably instance normalization, and the activation function is preferably an LReLU function. The cross-layer connection structure of the residual convolution module is mainly used to suppress the gradient disappearance and network degradation phenomenon in the deep neural network, optimize the network learning process, and improve the effectiveness and stability of the network. After the subgraph input is cut and the backbone network, the input image is sequentially subjected to the five groups of convolution stages, and the convolution stages perform feature extraction and down-sampling to generate first, second, third, fourth and fifth stage features.

[0068] In one specific embodiment, the split convolution network structure is as shown in Figure 5 The network takes the split convolution layer as the core and is used for strengthening the extraction of high-resolution local features such as acidophilic spheres, lipid droplets, glycogen, and cytoplasmic loose transparency. Split convolution is a special convolution, which is different from the ordinary convolution that only uses a single convolution kernel to process single-layer input features. It first splits the input matrix in the spatial dimension, thereby dividing the input matrix into multiple regions, then uses a separate convolution kernel to process the data of each region respectively, and finally combines the features to generate the corresponding output features. Preferably, the split number is 64 or 256. After the first stage features extracted by the backbone network are input into the split convolution network, they are sequentially subjected to two groups of residual modules composed of a split convolution layer, an activation layer and a fully connected layer, to generate high-resolution local features.

[0069] In one specific embodiment, the global perception network structure is as shown in Figure 6 The network mainly includes a global pooling layer, an attention module, a fully connected layer, a normalization layer and an activation layer, and is used for strengthening the extraction of global information such as the severity of liver lesions. After the fifth stage features extracted by the backbone network are input into the global perception network, they are first subjected to the global pooling layer to compress the input features into a one-dimensional matrix; then, the one-dimensional matrix is input into a path composed of two groups of fully connected layers and one group of activation layers, and a path composed of two groups of self-attention modules, the outputs of the two paths and the input one-dimensional matrix are fused to generate a composite one-dimensional matrix containing attention perception information; finally, the composite one-dimensional convolution is subjected to a group of fully connected layers, normalization layers and activation layers to generate global features containing global attention perception information.

[0070] The self-attention module mainly consists of fully connected layers, softmax layers, and normalization layers. After features are input into the self-attention module, they first pass through two sets of fully connected layers to generate two sets of one-dimensional matrices. These two sets of one-dimensional matrices are multiplied element-wise and converted into attention matrices using the softmax function. Then, the input features are weighted using the attention matrix to generate an attention perception matrix. Finally, the attention perception matrix is ​​processed by a fully connected layer, summed, added to the input features, and then passed through a normalization layer to generate the output features. Preferably, the normalization layer is a multi-layer normalization layer.

[0071] In a specific embodiment, the bidirectional multi-level aggregation network structure is as follows: Figure 7 As shown, the network mainly consists of a reshaping module, a dilation module, a convolution module, and a stitching module, used to enhance the extraction of multi-scale contextual information such as the relative position of cell nuclei and the relative volume of cells. The second, third, fourth, and fifth-stage features extracted by the backbone network, as well as the high-resolution local features extracted by the segmented convolution and the global features extracted by the global perception network, are first processed by a set of reshaping modules when input into the bidirectional multi-level aggregation network. The reshaping module mainly consists of a reshaping layer, a convolutional layer with a kernel size of 1, a normalization layer, and an activation layer. The reshaping module is used to unify the planar size of the input matrix without changing the specific data, while the convolutional layer is mainly used to unify the number of channels in the matrix. After passing through the reshaping module, the six sets of features generate six corresponding matrices of the same size.

[0072] After unifying the feature sizes, the second, third, fourth, and fifth stage features, along with high-resolution local features, are input into the dilation module. The dilation module mainly consists of three sets of convolutional layers, a normalization layer, and an activation layer. The three sets of convolutional layers are ordinary convolutional layers and dilated convolutional layers with dilation ratios of 2 and 3, respectively. Compared to ordinary convolutions, the two sets of dilated convolutions offer a wider field of view and better sparsity, better capturing long-distance dependencies between features and thus strengthening the contextual information in the input features. Simultaneously, the multi-scale characteristics of the five feature sets themselves and the diversity of dilation ratios in the dilated convolutions greatly enrich the multi-scale contextual information from local to global in the output features of the bidirectional multi-level aggregation network. Specifically, because the planar information of the global features is destroyed by the global pooling layer, making it difficult to extract contextual information, the global features, after reshaping, are processed only using ordinary convolutional modules containing convolutional layers, normalization layers, and activation layers.

[0073] After being processed by the expansion module and the convolution module respectively, the six groups of features are fused by a bidirectional connection structure. The bidirectional connection structure has two paths, top-down and bottom-up, which realize multi-scale feature fusion guided by global information and local information respectively. In the top-down path, the global feature and the fifth stage feature are first added for fusion, and the fused feature is divided into two paths, one of which is first input into the convolution module and then merged into the splicing module, and the other is fused with the fourth stage feature below. The eight times of feature fusion process of the top-down and bottom-up paths generates eight groups of mixed features, which are finally input into the splicing module composed of a splicing layer, a convolution layer, a normalization layer and an activation layer to generate multi-scale context features.

[0074] In a specific embodiment, the head network structure is as shown in Figure 8 The network is mainly composed of a reshaping layer, a fully connected layer, a normalization layer, an activation layer and a Sigmoid layer, which is used to generate cell detection results according to the input multi-scale context features. The multi-scale context features are first converted into a one-dimensional matrix by a reshaping layer, and then input into two groups of fully connected layers and Sigmoid layers after being processed by a fully connected layer, a normalization layer and an activation layer, to generate the foreground-background discrimination matrix and the coordinate offset matrix of the detection frame, thereby finally obtaining the hepatocellular ballooning cell detection results.

[0075] S3: Training the hepatocellular ballooning cell detection network: first, training the feature extraction part of the detection network separately, and then training the hepatocellular ballooning cell detection network completely.

[0076] In an embodiment, the step of training the feature extraction part of the detection network separately is based on constructing a corresponding classification network according to the feature extraction network, and then training the classification network. First, the four feature extraction networks of the backbone network, the split convolution network, the global perception network and the bidirectional multi-level aggregation network are spliced with a classification head network to form a temporary complete classification network. Then, the full field digital slice is divided into two groups containing hepatocellular ballooning cells and not containing hepatocellular ballooning cells, and then the full field digital slice is divided by a fixed step to construct a classification data set for predicting whether the cut subgraph belongs to the full field pathological section containing hepatocellular ballooning cells. Finally, the cut subgraph is input into the classification network to realize the pre-training of the hepatocellular ballooning cell feature extraction network.

[0077] In an embodiment, the step of training the hepatocellular ballooning cell detection network is to first import the feature extraction pre-training parameters, and then train the entire hepatocellular ballooning cell detection network. After importing the feature extraction pre-training parameters, in the first round of network training, the parameters of the feature extraction network are frozen, and in the second to fifth rounds, the parameters of the feature extraction network are gradually unfrozen in the order of the bidirectional multi-level aggregation network, the global perception network, the split convolution network and the backbone network.

[0078] In network training, 75% of the input cut subgraphs contain liver ballooning cells, and 25% do not contain liver ballooning cells. This strategy enables the network to understand negative samples that do not contain liver ballooning cells while strengthening learning of positive samples that contain liver ballooning cells, inhibits degradation of the network, and improves the generalization ability of the network.

[0079] S4: Test liver ballooning cell detection network: First, use the test set to evaluate network performance, and then connect the network to an optical microscope system to evaluate the actual application performance of the network.

[0080] In one specific embodiment, we compared the detection performance of the traditional image processing method, the existing deep learning method, and the method of the present application. Among them, the traditional image processing method based on edge detection and other technologies has an AR50, AP50, AP75, and AP@50:5:95 of 8.1%, 10.4%, 7.3%, and 6.8% on the test set, respectively. In existing deep learning, the results of YOLO v3 are 49.8%, 54.9%, 49.4%, and 42.7%, respectively, the results of YOLO v8 are 51.5%, 53.7%, 46.9%, and 47.2%, respectively, and the results of Swin Transformer are 52.6%, 54.8%, 51.5%, and 48.7%, respectively. The results of the method of the present application are 64.7%, 65.1%, 60.4%, and 58.9%, respectively. The AR50, AP50, AP75, and AP@50:5:95 of the method of the present application are improved by 12.1%, 10.3%, 8.9%, and 10.2%, respectively, compared with the best existing deep learning method.

[0081] The experimental results show that in the task of liver ballooning cell detection, deep learning methods have a great advantage over traditional image processing methods, and the method of the present application is significantly better than the best existing deep learning method on this data set. This proves the effectiveness and advancement of the method proposed in the present application, which can effectively identify liver ballooning cells in liver pathological images.

[0082] In one embodiment, the network is connected to an optical microscope system to evaluate the actual application performance of the network. The network is implanted in a computer or mobile device, and the network directly obtains pathological images through imaging software or device display output to evaluate the detection accuracy and system response speed of the network in a real environment and verify the feasibility and effectiveness of the actual application of the network.

[0083] It will be apparent to those skilled in the art that the application is not limited to the details of the above-exemplified embodiments and that the present application can be implemented in other particular forms without departing from the spirit or essential characteristics of the present application. The presently disclosed embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. No reference to an item in the claims is to be construed as an admission that such item is admitted prior art.

[0084] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any slight modification, equivalent replacement and improvement made to the above embodiments according to the technical essence of the present application shall be included in the protection scope of the technical scheme of the present application.

Claims

1. An automated detection method for hepatic ballooning degeneration cells based on medical mechanisms and deep learning, characterized in that, The specific steps include: Establish a liver pathology image dataset: Collect pathological sections of fatty liver disease from multiple centers, label and preprocess ballooning degeneration cells, and establish a liver pathology image dataset; Construction of a detection network for hepatic ballooning degeneration cells: Based on the medical mechanism of hepatic ballooning degeneration cells, a detection network for hepatic ballooning degeneration cells was constructed. Training the hepatic ballooning degeneration cell detection network: First, the feature extraction part of the detection network was trained separately using the training set, and then the entire hepatic ballooning degeneration cell detection network was trained. Testing the hepatic ballooning degeneration cell detection network: First, the network performance was evaluated using a test set. Then, the network was connected to an optical microscope system to evaluate its performance in practical applications.

2. The automatic detection method for hepatic ballooning degeneration cells based on medical mechanisms and deep learning according to claim 1, characterized in that, The process of establishing the liver pathology image dataset includes the collection, staining, and scanning of pathological sections of steatohepatitis, as well as labeling and preprocessing. The labeling process adopts a two-step method: first, the region of interest is delineated on the complete full-field digital slice, and then only the ballooning cells of the liver within the region of interest are labeled to reduce the omission of ballooning cells. The preprocessing includes data cleaning, staining normalization, and data enhancement, with data enhancement focusing on enhancing the fusion of the cell foreground and background.

3. The automatic detection method for hepatic ballooning degeneration cells based on medical mechanisms and deep learning according to claim 1, characterized in that, The liver ballooning degeneration cell detection network includes a backbone network, a segmented convolutional network, a global sensing network, a bidirectional multi-level aggregation network, and a head network.

4. The automatic detection method for hepatic ballooning degeneration cells based on medical mechanisms and deep learning according to claim 3, characterized in that, The backbone network includes a first convolutional stage, a second convolutional stage, a third convolutional stage, a fourth convolutional stage, and a fifth convolutional stage. The five convolutional stages each include 2, 3, 3, 4, and 3 residual convolutional modules, respectively. The backbone network is used to extract and downsample the input pathological images to generate first-stage features, second-stage features, third-stage features, fourth-stage features, and fifth-stage features at different scales.

5. The automatic detection method for hepatic ballooning degeneration cells based on medical mechanisms and deep learning according to claim 3, characterized in that, The segmented convolutional network uses segmented convolution as its core. Based on the first-stage features extracted by the backbone network, it generates high-resolution local features that enhance information on eosinophilic spherical, lipid droplet glycogen, and loose, transparent cytoplasm. Segmented convolution is a special type of convolution. It first segments the input matrix in space, thereby dividing the input matrix into multiple regions. Then, it uses a separate convolution kernel to process the data in each region separately.

6. The automatic detection method for hepatic ballooning degeneration cells based on medical mechanisms and deep learning according to claim 3, characterized in that, The global perception network consists of a global pooling layer, an attention module, a fully connected layer, a normalization layer, and an activation layer. Based on the fifth-stage features extracted from the backbone network, it generates global features that enhance information such as the severity of liver lesions.

7. The automatic detection method for hepatic ballooning degeneration cells based on medical mechanisms and deep learning according to claim 3, characterized in that, The bidirectional multi-level aggregation network consists of a reshaping module, an expansion module, a convolution module, and a splicing module. It integrates the second, third, fourth, and fifth stage features extracted by the backbone network, as well as the high-resolution local features extracted by the segmentation convolution network and the global features extracted by the global perception network, to generate multi-scale contextual features that enhance information such as the relative position of the cell nucleus and the relative volume of the cell. The dilation module consists of three sets of convolutional layers, a normalization layer, and an activation layer. The three sets of convolutional layers are ordinary convolutional layers and dilated convolutional layers with dilation ratios of 2 and 3, respectively.

8. The automatic detection method for hepatic ballooning degeneration cells based on medical mechanisms and deep learning according to claim 3, characterized in that, The head network consists of a remodeling layer, a fully connected layer, a normalization layer, an activation layer, and a sigmoid layer. It is used to generate the location and category vectors of hepatic ballooning cells based on the multi-scale contextual features extracted by the bidirectional multi-level aggregation network, thereby enabling the detection of hepatic ballooning cells.

9. The automatic detection method for hepatic ballooning degeneration cells based on medical mechanisms and deep learning according to claim 3, characterized in that, First, four feature extraction networks—the backbone network, the segmented convolutional network, the global perception network, and the bidirectional multi-level aggregation network—and a classification head network are concatenated to form a temporary complete classification network. This classification network is trained separately to obtain the pre-training parameters of the feature extraction network. Then, the pre-training parameters are transferred to the detection network and the feature extraction network is frozen. During training, the four feature extraction networks are unfrozen sequentially to train the liver ballooning degeneration cell detection network.

10. The automatic detection method for hepatic ballooning degeneration cells based on medical mechanisms and deep learning according to claim 1, characterized in that, The detection performance of the network on the test set was evaluated using AR50, AP50, AP75, and AP@50:5:95 metrics. The network was compared with traditional image processing methods and existing deep learning methods to verify its effectiveness and advancement. The network was also integrated into an optical microscope system to evaluate its detection accuracy and efficiency in practical applications.

Images