Method, device, equipment and medium for performing three-level lymphatic structure data annotation on tissue image

Abstract

The invention provides a method, a device, equipment and a medium for performing three-level lymphatic structure data annotation on a tissue image, which can be applied to the technical field of deep learning and the technical field of medical image data annotation. The method comprises the following steps: cutting an initial tissue dyeing image of a target object, and determining a plurality of cut image blocks; processing the plurality of cut image blocks according to a residual network model to obtain a plurality of tumor region marking image blocks; processing the tumor area marking image block according to a semantic segmentation model to obtain a lymphocyte mask image block corresponding to the tumor area marking image block; processing the plurality of lymphocyte mask image blocks based on a morphological algorithm to obtain a lymphocyte density map corresponding to the initial tissue staining image, and representing a lymphocyte density distribution result by lymphocyte density pixels of the lymphocyte density map; and labeling the initial tissue staining image based on a labeling rule and lymphocyte density pixels to obtain a labeled image representing the three-level lymph structure data.

Description

Technical Field

[0001] The present disclosure relates to the fields of deep learning technology and medical image data annotation technology, and more specifically, to a method, device, equipment and medium for annotating tertiary lymphatic structure data of tissue images. Background Art

[0002] As an important ectopic lymphoid organ (ELO), tertiary lymphoid structures (TLSs) are composed of lymphocytes. The detection and quantification of tertiary lymphoid structures have important clinical significance for promoting tumor treatment.

[0003] Commonly used methods for detecting tertiary lymphatic structures include multiple immunohistochemistry, multiple immunofluorescence, and real-time fluorescence quantification of chemokines, all of which are time-consuming and labor-intensive, and have strong subjectivity in manual identification. It is also difficult to make an effective and unified measurement of the aggregation of lymphocytes in tertiary lymphatic structures. In addition, there are many tertiary lymphatic structures and their composition is complex. Manual labeling often has strong subjectivity, mislabeling, and other problems, which reduces the accuracy of pathological image labeling and treatment efficiency. Summary of the invention

[0004] In view of this, the present disclosure provides a method, apparatus, device and medium for annotating three-level lymphatic structure data of a tissue image.

[0005] One aspect of the present disclosure provides a method for annotating three-level lymphatic structure data on a tissue image, comprising:

[0006] The initial tissue staining image of the target object is cropped to determine a plurality of cropped image blocks; the plurality of cropped image blocks are processed according to a residual network model to obtain a plurality of tumor region labeled image blocks; the tumor region labeled image blocks are processed according to a semantic segmentation model to obtain a lymphocyte mask image block corresponding to the tumor region labeled image block, wherein the lymphocyte pixel points marked in the lymphocyte mask image block are located in the tumor region labeled image block; the plurality of lymphocyte mask image blocks are processed based on a morphological algorithm to obtain a lymphocyte density map corresponding to the initial tissue staining image, wherein the lymphocyte density pixels of the lymphocyte density map represent the lymphocyte density distribution result; the initial tissue staining image is labeled based on the labeling rules and the lymphocyte density pixels to obtain a labeled image representing the tertiary lymphatic structure data.

[0007] According to an embodiment of the present disclosure, processing multiple cropped image blocks according to a residual network model to obtain multiple tumor area marked image blocks includes: processing multiple cropped image blocks according to a residual network model to obtain prediction results corresponding to each of the multiple cropped image blocks, wherein the prediction results characterize the probability that a pixel point in the cropped image block is a tumor tissue; when the prediction results meet a preset probability condition, determining the cropped image block as a tumor area marked image block.

[0008] According to an embodiment of the present disclosure, the residual network model includes a convolutional network layer, a maximum pooling layer, a residual layer, an average pooling layer, and a fully connected layer; wherein, processing multiple cropped image blocks according to the residual network model to obtain prediction results corresponding to each of the multiple cropped image blocks includes: inputting the cropped image blocks into the convolutional network layer and outputting convolution vector features; inputting the convolution vector features into the maximum pooling layer and outputting the first pooling vector features; inputting the first pooling vector features into the residual layer and outputting the residual vector features; inputting the residual vector features into the average pooling layer and outputting the second pooling vector features; inputting the second pooling vector features into the fully connected layer and outputting the prediction results corresponding to the cropped image blocks.

[0009] According to an embodiment of the present disclosure, processing a tumor region marking image block according to a semantic segmentation model to obtain a lymphocyte mask image block corresponding to the tumor region marking image includes: using the semantic segmentation model to identify lymphocyte nuclear pixels in the tumor region marking image block to obtain a recognition result; and pixel marking the lymphocyte nuclear pixels based on the recognition result to obtain a lymphocyte mask image block corresponding to the tumor region marking image block.

[0010] According to an embodiment of the present disclosure, a plurality of lymphocyte mask image blocks are processed based on a morphological algorithm to obtain a lymphocyte density map corresponding to an initial tissue staining image, including: filtering candidate pixel points in the lymphocyte mask image block to obtain a filtered image block, wherein the pixel value of the candidate pixel point is less than a preset pixel value threshold; based on pixel shape parameters, noise threshold parameters, and pixel boundary parameters, the filtered image block is processed using a corrosion algorithm to obtain a lymphocyte target image block; and the lymphocyte target image blocks corresponding to each of the plurality of lymphocyte mask image blocks are processed according to a morphological connected domain algorithm to obtain a lymphocyte density map corresponding to the initial tissue staining image.

[0011] According to an embodiment of the present disclosure, lymphocyte target image blocks corresponding to each of a plurality of lymphocyte mask image blocks are processed according to a morphological connected domain algorithm to obtain a lymphocyte density map corresponding to an initial tissue staining image, including: counting lymphocytes in the lymphocyte target image block using the morphological connected domain algorithm to obtain the number of lymphocytes; obtaining a local lymphocyte density corresponding to the lymphocyte target image block according to the number of lymphocytes corresponding to the lymphocyte target image; and obtaining a lymphocyte density map corresponding to the initial tissue staining image according to the local lymphocyte density corresponding to each of the plurality of lymphocyte target images.

[0012] According to an embodiment of the present disclosure, the initial tissue staining image is annotated based on the annotation rules and lymphocyte density pixels to obtain an annotated image representing the tertiary lymphatic structure data, including:

[0013] Based on lymphocyte density pixels, the aggregation degree of lymphocyte pixels in the initial tissue staining image is divided to determine the aggregation area; based on lymphocyte density pixels, the maturity degree of lymphocyte pixels in the aggregation area is identified to obtain the recognition result; based on the labeling rules and the recognition result, the initial tissue staining image is labeled to obtain a labeled image representing the tertiary lymphatic structure data.

[0014] Another aspect of the present disclosure provides a device for annotating three-level lymphatic structure data on a tissue image, comprising:

[0015] A cropping module, used for cropping the initial tissue staining image of the target object and determining a plurality of cropped image blocks;

[0016] A first processing module is used to process multiple cropped image blocks according to a residual network model to obtain multiple tumor area marked image blocks;

[0017] A second processing module is used to process the tumor region marked image block according to the semantic segmentation model to obtain a lymphocyte mask image block corresponding to the tumor region marked image block, wherein the lymphocyte pixel points marked in the lymphocyte mask image block are located in the tumor region marked image block;

[0018] A third processing module is used to process the multiple lymphocyte mask image blocks based on a morphological algorithm to obtain a lymphocyte density map corresponding to the initial tissue staining image, wherein the lymphocyte density pixels of the lymphocyte density map represent the lymphocyte density distribution result;

[0019] The annotation module is used to annotate the initial tissue staining image based on the annotation rules and lymphocyte density pixels to obtain an annotated image representing the tertiary lymphatic structure data.

[0020] Another aspect of the present disclosure provides an electronic device, comprising:

[0021] one or more processors;

[0022] a memory for storing one or more programs,

[0023] When one or more programs are executed by one or more processors, the one or more processors implement the above method for annotating the three-level lymphatic structure data of the tissue image.

[0024] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the above method for annotating tertiary lymphatic structure data for a tissue image.

[0025] According to the method, device, equipment and medium for annotating the three-level lymphatic structure data of tissue images provided by the present disclosure, multiple cropped image blocks are processed by a residual neural network model to obtain multiple tumor area marked image blocks, and then the tumor area marked image blocks are processed according to the semantic segmentation model to obtain lymphocyte mask image blocks, lymphocyte density maps are generated based on the lymphocyte mask image blocks, and the initial tissue staining image is annotated based on the annotation rules and lymphocyte density pixels to obtain an annotated image representing the three-level lymphatic structure data. Due to the combination of the residual neural network model and the semantic segmentation model, accurate characterization of the tumor area in the initial tissue staining image and the degree of lymphocyte aggregation in the tumor area is achieved. The lymphocyte density map provides accurate assistance for lymphocyte annotation, thereby obtaining an annotated image representing the three-level lymphatic structure data with high accuracy, reducing the occurrence of mislabeling and missing labels while ensuring the quality of data annotation, improving data annotation efficiency, and improving pathological image annotation and treatment efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0027] Figure 1 A flow chart of a method for annotating three-level lymphatic structure data on a tissue image according to an embodiment of the present disclosure is shown;

[0028] Figure 2 An example schematic diagram of determining a tumor region marking image block according to an embodiment of the present disclosure is shown;

[0029] Figure 3 An example schematic diagram of generating a lymphocyte density map according to an embodiment of the present disclosure is shown;

[0030] Figure 4 An example schematic diagram showing the effect of a lymphocyte mask image block according to an embodiment of the present disclosure is shown;

[0031] Figure 5 An example schematic diagram of generating local lymphocyte density according to an embodiment of the present disclosure is shown;

[0032] Figure 6 An example schematic diagram of generating a labeled image according to an embodiment of the present disclosure is shown;

[0033] Figure 7 An example schematic diagram of generating a medical image data set according to an embodiment of the present disclosure is shown;

[0034] Figure 8 A block diagram of an apparatus for annotating three-level lymphatic structure data on a tissue image according to an embodiment of the present disclosure is shown; and

[0035] Fig. 9 A block diagram of an electronic device suitable for implementing a method for annotating three-level lymphatic structure data on a tissue image according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0036] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0037] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0038] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0039] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0040] In the process of realizing the present disclosure, it was found that tertiary lymphoid structures (TLSs) are an important ectopic lymphoid organ (ELO). Tertiary lymphoid structures are composed of lymphocytes. The detection and quantification of tertiary lymphoid structures are of great clinical significance for promoting tumor treatment. Commonly used tertiary lymphoid structure detection methods include multiple immunohistochemistry and multiple immunofluorescence, real-time fluorescence quantification of chemokines, etc., all of which have the disadvantages of being time-consuming and labor-intensive, and having strong subjectivity in manual identification, and it is difficult to make an effective and unified measurement of the aggregation degree of lymphocytes in tertiary lymphoid structures. In addition, the number of tertiary lymphoid structures is large and the composition is complex. Manual labeling often has problems such as strong subjectivity, mislabeling and missing labels, which reduces the accuracy of pathological image labeling and treatment efficiency.

[0041] In view of this, the embodiments of the present disclosure provide a method, apparatus, device and medium for annotating the three-level lymphatic structure data of tissue images. The method includes: cropping the initial tissue staining image of the target object to determine multiple cropped image blocks; processing the multiple cropped image blocks according to the residual network model to obtain multiple tumor area marked image blocks; processing the tumor area marked image blocks according to the semantic segmentation model to obtain lymphocyte mask image blocks corresponding to the tumor area marked image blocks; processing the multiple lymphocyte mask image blocks based on the morphological algorithm to obtain a lymphocyte density map corresponding to the initial tissue staining image, and the lymphocyte density pixels of the lymphocyte density map represent the lymphocyte density distribution result; annotating the initial tissue staining image based on the annotation rules and the lymphocyte density pixels to obtain an annotated image representing the three-level lymphatic structure data.

[0042] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0043] Figure 1 A flow chart of a method for annotating three-level lymphatic structure data on a tissue image according to an embodiment of the present disclosure is shown.

[0044] like Figure 1 As shown, the method 100 includes operations S110 to S150.

[0045] In operation S110 , the initial tissue staining image of the target object is cropped to determine a plurality of cropped image blocks.

[0046] According to an embodiment of the present disclosure, the target object may be a tumor patient object, and the histological image of the target object is stained using hematoxylin-eosin staining (HE) to obtain an initial tissue staining image. The histological image may be a tumor tissue pathology section, for example, a colon cancer tissue pathology section.

[0047] According to an embodiment of the present disclosure, the cropping model can set model cropping slice size parameters, input the initial tissue staining image stored in a high-resolution storage format (Supported Vector Graphics format, SVS) into the cropping model, obtain multiple cropped image blocks, and output the multiple cropped image blocks to a designated folder.

[0048] In operation S120, the plurality of cropped image blocks are processed according to the residual network model to obtain a plurality of tumor region labeled image blocks.

[0049] According to an embodiment of the present disclosure, the residual network model may be a deep learning model (Residual Networks, ResNetsResNet18), and multiple cropped image blocks are input into the residual network model to determine the prediction result of each cropped image block. The prediction result may be a result that the pixel point in the cropped image block is a tumor tissue result or a result that the pixel point in the cropped image block is a non-tumor tissue result.

[0050] According to an embodiment of the present disclosure, when the prediction result is that the pixel point in the cropped image block is a tumor tissue, the cropped image block is determined as a tumor region marking image block.

[0051] According to an embodiment of the present disclosure, multiple tumor region marking image blocks are screened and determined from multiple cropped image blocks. The multiple tumor region marking image blocks constitute the tumor region in the initial tissue staining image.

[0052] In operation S130 , the tumor region labeled image block is processed according to the semantic segmentation model to obtain a lymphocyte mask image block corresponding to the tumor region labeled image block.

[0053] According to an embodiment of the present disclosure, the semantic segmentation model may be an image segmentation model (Convolutional Networks for Biomedical Image Segmentation, U-Net).

[0054] According to an embodiment of the present disclosure, each tumor region marked image block is input into a semantic segmentation model, and after identifying and segmenting the lymphocyte nuclear pixels in the tumor region marked image block, a lymphocyte mask image block corresponding to the tumor region marked image block is obtained.

[0055] According to an embodiment of the present disclosure, the lymphocyte mask image block represents a mask image block used to distinguish lymphocyte pixels from other pixels in the tumor region marking image block. The lymphocyte pixels marked in the lymphocyte mask image block are located in the tumor region marking image block.

[0056] In operation S140 , the plurality of lymphocyte mask image blocks are processed based on a morphological algorithm to obtain a lymphocyte density map corresponding to the initial tissue staining image.

[0057] According to an embodiment of the present disclosure, each lymphocyte mask image block is processed using a morphological algorithm, such as a noise point filtering operation, a texture refinement operation, etc. Then, by counting the number of lymphocytes in each lymphocyte mask image block, the lymphocyte density corresponding to each lymphocyte mask image block is obtained.

[0058] According to an embodiment of the present disclosure, a lymphocyte density map corresponding to the initial tissue staining image is generated according to the lymphocyte densities corresponding to each of the multiple lymphocyte mask image blocks. The lymphocyte density pixels of the lymphocyte density map represent the lymphocyte density distribution result.

[0059] According to an embodiment of the present disclosure, the initial tissue staining image is composed of a plurality of cropped image blocks, and the more lymphocytes there are in the lymphocyte mask image blocks corresponding to the cropped image blocks, the greater the lymphocyte density.

[0060] According to an embodiment of the present disclosure, when the prediction result is that the pixel point in the cropped image block is non-tumor tissue, the cropped image block is determined as a non-tumor area marked image block, and the lymphocyte density corresponding to the non-tumor area marked image block is 0.

[0061] In operation S150, the initial tissue staining image is annotated based on the annotation rules and lymphocyte density pixels to obtain an annotated image representing the tertiary lymphatic structure data.

[0062] According to an embodiment of the present disclosure, the degree of lymphocyte aggregation is determined based on the lymphocyte density distribution result, thereby determining the maturity of the lymphocytes, and the marking rule can be determined based on the maturity of the lymphocytes. For example, the first maturity level can be marked with a red outline.

[0063] According to an embodiment of the present disclosure, tertiary lymphoid structures (TLSs) are an important ectopic lymphoid organ (ELO), and the lymphocyte density map can reflect the aggregation degree and maturity degree of lymphocytes, thereby providing a basis for the annotation of tertiary lymphoid structures. The initial tissue staining image is annotated based on the annotation rules and lymphocyte density pixels to obtain an annotated image representing the tertiary lymphoid structure data.

[0064] According to the embodiments of the present disclosure, the combination of the residual neural network model and the semantic segmentation model realizes the accurate characterization of the tumor area and the degree of lymphocyte aggregation in the tumor area in the initial tissue staining image. The lymphocyte density map provides accurate assistance for lymphocyte labeling, thereby obtaining a highly accurate labeled image representing the tertiary lymphatic structure data, reducing the occurrence of mislabeling and missing labels while ensuring the quality of data labeling, improving data labeling efficiency, and improving pathological image labeling and treatment efficiency.

[0065] According to an embodiment of the present disclosure, processing multiple cropped image blocks according to a residual network model to obtain multiple tumor area marked image blocks includes: processing multiple cropped image blocks according to a residual network model to obtain prediction results corresponding to each of the multiple cropped image blocks, wherein the prediction results characterize the probability that a pixel point in the cropped image block is a tumor tissue; when the prediction results meet a preset probability condition, determining the cropped image block as a tumor area marked image block.

[0066] According to the embodiment of the present disclosure, the residual network model is trained using a sample tumor data set, and the sample prediction results of the model are evaluated and the parameters are optimized. The tumor data set comes from a subset of the "100,000 histological images of human colorectal cancer and healthy tissues" data set. The tumor data set contains 11,977 histological image samples and 94 colon cancer tissue staining images from 81 patients. It consists of three types of tissue types: ADIMUC (adipose tissue and mucus, i.e., loose non-tumor tissue), STRMUS (stroma and muscle, i.e., dense non-tumor tissue) and TUMSTU (colon cancer epithelial tissue and gastric cancer epithelial tissue, i.e., tumor tissue). Each colon cancer tissue staining image is manually annotated with different classification labels, which are tumor image labels and non-tumor image labels. The size of all images is 512x512 pixels, and the resolution is 0.5µm / px.

[0067] According to an embodiment of the present disclosure, the model parameters are selected as follows: the learning rate is 0.0001, the learning rate adjustment strategy is the Learning Rate Scheduler (StepLR) method, the learning rate is multiplied by 0.1 every 10 epochs, the batch size of the model is 32, the maximum number of iterations is set to 25, the optimizer selects Adaptive Moment Estimation (Adam), and the loss function selects the cross entropy loss function (CrossEntropyLoss).

[0068] According to an embodiment of the present disclosure, multiple cropped image blocks are input into a trained residual network model for identification, and prediction results corresponding to each of the multiple cropped image blocks are obtained. The prediction result is a binary classification probability value, and a prediction result of 1 indicates that the pixel point in the cropped image block is a tumor tissue, and a prediction result of 0 indicates that the pixel point in the cropped image block is a non-tumor tissue.

[0069] According to an embodiment of the present disclosure, the preset probability condition may be a probability value of 1. When the prediction result of the cropped image block is 1, the cropped image block is determined as a tumor region marked image block. When the prediction result of the cropped image block is 0, the cropped image block is determined as a non-tumor region marked image block.

[0070] According to an embodiment of the present disclosure, the tumor region of the initial tissue staining image is constructed based on the image regions of each of the plurality of tumor region labeling image blocks.

[0071] According to an embodiment of the present disclosure, the residual network model includes a convolutional network layer, a maximum pooling layer, a residual layer, an average pooling layer, and a fully connected layer; wherein, processing multiple cropped image blocks according to the residual network model to obtain prediction results corresponding to each of the multiple cropped image blocks includes: inputting the cropped image blocks into the convolutional network layer and outputting convolution vector features; inputting the convolution vector features into the maximum pooling layer and outputting the first pooling vector features; inputting the first pooling vector features into the residual layer and outputting the residual vector features; inputting the residual vector features into the average pooling layer and outputting the second pooling vector features; inputting the second pooling vector features into the fully connected layer and outputting the prediction results corresponding to the cropped image blocks.

[0072] According to an embodiment of the present disclosure, the convolutional network layer includes a convolution sublayer, a batch normalization sublayer, and an activation sublayer. The input layer of the residual network model receives a tumor region labeled image block of size H*W, with a dimension of (H, W, 3), and the convolution sublayer contains a 7x7 convolution kernel with a step size of 2 and a padding of 3, consisting of 64 convolution kernels, and an output dimension of (H / 2, W / 2, 64).

[0073] According to an embodiment of the present disclosure, the tumor area marked image block is input into the convolution sublayer, batch normalization layer, and activation sublayer (Linear rectification function, ReLu) to obtain the convolution vector feature, and the convolution network layer accelerates the training and model generalization capabilities.

[0074] According to an embodiment of the present disclosure, the maximum pooling layer has a step size of 2 and a size of 3x3. The convolution vector feature is input into the maximum pooling layer to obtain a first pooling vector feature with reduced dimension (H / 4, W / 4, 64).

[0075] According to an embodiment of the present disclosure, the residual layer is composed of 4 residual blocks, each residual block contains two convolution sublayers and an identity mapping sublayer (jump connection), the number of output channels of the 4 residual blocks are 64, 128, 256, and 512 respectively, the first pooled vector feature is input into the residual layer, and the residual vector feature is output.

[0076] According to an embodiment of the present disclosure, the average pooling layer reduces the spatial dimension of each residual vector feature to 1x1, thereby obtaining a second pooling vector feature of a fixed length, and the length of the second pooling vector feature is equal to 512.

[0077] According to an embodiment of the present disclosure, the second pooled vector feature is flattened and passed through a fully connected layer, which maps the second pooled vector feature to an output category output node to obtain a prediction result corresponding to the cropped image block, corresponding to the score or probability of whether the pixel point in the cropped image block is a tumor tissue category.

[0078] Figure 2 An example schematic diagram of determining a tumor region marking image block according to an embodiment of the present disclosure is shown.

[0079] like Figure 2 As shown, a histological image of the colon cancer tissue of the target object is obtained, and an initial tissue staining image is obtained after staining. The initial tissue staining image is cropped to determine a plurality of cropped image blocks; the plurality of cropped image blocks are processed according to a residual network model to obtain a plurality of tumor region labeled image blocks and a plurality of non-tumor region labeled image blocks, and the tumor region of the initial tissue staining image is determined based on the plurality of tumor region labeled image blocks to obtain an initial tissue staining image with the tumor region identified, wherein the tumor region is a blue region.

[0080] According to an embodiment of the present disclosure, processing a tumor region marking image block according to a semantic segmentation model to obtain a lymphocyte mask image block corresponding to the tumor region marking image includes: using the semantic segmentation model to identify lymphocyte nuclear pixel points in the tumor region marking image block to obtain a recognition result; and pixel marking the lymphocyte nuclear pixel points based on the recognition result to obtain a lymphocyte mask image block corresponding to the tumor region marking image block.

[0081] According to the embodiments of the present disclosure, a labeled sample lymphocyte mask dataset is used to train the semantic segmentation model, and the model results are evaluated and the parameters are optimized. The sample lymphocyte mask dataset comes from a public dataset, which contains 1358 sample tumor region labeled image blocks from 66 patients and their corresponding sample lymphocyte mask image blocks. The dataset contains 17582 tumor cells, 22550 lymphocytes and 10675 other non-malignant cells.

[0082] According to an embodiment of the present disclosure, the model parameters are selected as follows: the learning rate is 0.001, the learning rate adjustment strategy is ExponentialLR, that is, after each round, the learning rate will decrease at a rate of 0.9, the batch size of the model is 16, the maximum number of iterations is set to 50, the optimizer selects Adaptive Moment Estimation (Adam), and the loss function selects the maximum likelihood function (Negative Log Likelihood Loss, LogNLLLoss).

[0083] According to an embodiment of the present disclosure, the tumor region marked image block is input into a trained semantic segmentation model to perform lymphocyte nuclear pixel point recognition to obtain a recognition result, which includes a lymphocyte recognition result and a non-lymphocyte recognition result.

[0084] According to an embodiment of the present disclosure, segmenting the lymphocyte nuclei of the tumor area marking image block based on the recognition result may include: marking the lymphocyte nucleus pixels with white pixels and marking the non-lymphocyte nucleus pixels with black pixels, thereby obtaining a lymphocyte mask image block corresponding to the tumor area marking image block.

[0085] According to an embodiment of the present disclosure, a plurality of lymphocyte mask image blocks are processed based on a morphological algorithm to obtain a lymphocyte density map corresponding to an initial tissue staining image, including: filtering candidate pixel points in the lymphocyte mask image block to obtain a filtered image block, wherein the pixel value of the candidate pixel point is less than a preset pixel value threshold; based on pixel shape parameters, noise threshold parameters, and pixel boundary parameters, the filtered image block is processed using a corrosion algorithm to obtain a lymphocyte target image block; and the lymphocyte target image blocks corresponding to each of the plurality of lymphocyte mask image blocks are processed according to a morphological connected domain algorithm to obtain a lymphocyte density map corresponding to the initial tissue staining image.

[0086] According to an embodiment of the present disclosure, the candidate pixel points represent noise points in the lymphocyte mask image block, and the pixel value of the candidate pixel points is less than a preset pixel value threshold. The filtered image block represents the image block after noise removal.

[0087] According to an embodiment of the present disclosure, based on pixel shape parameters, the filter image block is processed using a corrosion algorithm to remove small protruding parts of lymphocyte pixels in the filter image block, thereby simplifying the shape of the lymphocyte pixels.

[0088] According to an embodiment of the present disclosure, based on the noise point threshold parameter, the erosion algorithm is used to process the filter image block, thereby refining the boundary of the filter image block to make it smoother and clearer.

[0089] According to an embodiment of the present disclosure, based on pixel boundary parameters, the erosion algorithm is used to process the filter image block so that the boundary of the lymphocyte pixel points in the filter image block can shrink inward, thereby reducing the area of ​​the lymphocyte pixel points.

[0090] According to an embodiment of the present disclosure, the lymphocyte target image block represents an image block with higher quality such as smoothness and clarity.

[0091] According to an embodiment of the present disclosure, a plurality of lymphocyte target image blocks are processed using a morphological connected domain algorithm to obtain the lymphocyte density corresponding to each lymphocyte target image block, thereby synthesizing a global lymphocyte density map for displaying the lymphocyte distribution in the entire initial tissue staining image.

[0092] According to an embodiment of the present disclosure, lymphocyte target image blocks corresponding to each of a plurality of lymphocyte mask image blocks are processed according to a morphological connected domain algorithm to obtain a lymphocyte density map corresponding to an initial tissue staining image, including: counting lymphocytes in the lymphocyte target image block using the morphological connected domain algorithm to obtain the number of lymphocytes; obtaining a local lymphocyte density corresponding to the lymphocyte target image block according to the number of lymphocytes corresponding to the lymphocyte target image; and obtaining a lymphocyte density map corresponding to the initial tissue staining image according to the local lymphocyte density corresponding to each of the plurality of lymphocyte target images.

[0093] According to an embodiment of the present disclosure, the morphological connected domain algorithm may be a computer morphological connected domain counting algorithm, which is used to count the number of lymphocytes on each lymphocyte target image block.

[0094] According to an embodiment of the present disclosure, each lymphocyte target image block is taken as a unit, so as to obtain the local lymphocyte density corresponding to each lymphocyte target image block, and the local lymphocyte density represents the number of lymphocytes on each lymphocyte target image.

[0095] According to an embodiment of the present disclosure, based on the local lymphocyte densities corresponding to each of the multiple lymphocyte target images, a python heat map tool is used to generate a global lymphocyte density map corresponding to the initial tissue staining image.

[0096] Figure 3 An example schematic diagram of generating a lymphocyte density map according to an embodiment of the present disclosure is shown.

[0097] like Figure 3 As shown, the initial tissue staining image is cropped to determine a plurality of cropped image blocks; a plurality of tumor region marking image blocks are obtained based on the residual network model; the plurality of tumor region marking image blocks are input into the semantic segmentation model to obtain lymphocyte mask image blocks corresponding to the tumor region marking image blocks, and the white pixels in the lymphocyte mask image blocks are lymphocyte pixels; after filtering and corrosion operations are performed on the plurality of lymphocyte mask image blocks, the connected domains of the lymphocytes in the plurality of lymphocyte target image blocks are counted according to the morphological connected domain algorithm to obtain a lymphocyte density map corresponding to the initial tissue staining image.

[0098] Figure 4 An example schematic diagram showing the effect of a lymphocyte mask image block according to an embodiment of the present disclosure is shown.

[0099] like Figure 4 As shown, tumor region marking image block 401 generates lymphocyte mask image block 402, tumor region marking image block 403 generates lymphocyte mask image block 404, tumor region marking image block 405 generates lymphocyte mask image block 406, tumor region marking image block 407 generates lymphocyte mask image block 408, and blank image block 409 generates lymphocyte mask image block 410. There are no lymphocyte pixels in lymphocyte mask image block 410, and the white pixels in the lymphocyte mask image block are lymphocyte pixels. Among them, lymphocyte mask image block 406 has the most lymphocyte pixels, and lymphocyte mask image block 402 has the least lymphocyte pixels.

[0100] Figure 5 An example schematic diagram of generating local lymphocyte density according to an embodiment of the present disclosure is shown.

[0101] like Figure 5 As shown, the number of lymphocytes on each lymphocyte target image block is counted using a computer morphological connected domain counting algorithm, thereby obtaining the local lymphocyte density corresponding to each lymphocyte target image block. A lymphocyte target image block is generated based on cropped image block 501, and a local lymphocyte density 502 is obtained based on the connected domain count; a lymphocyte target image block is generated based on cropped image block 503, and a local lymphocyte density 504 is obtained based on the connected domain count; a lymphocyte target image block is generated based on cropped image block 505, and a local lymphocyte density 506 is obtained based on the connected domain count. It can be concluded that the lymphocyte density in cropped image block 503 is generally high, and the lymphocyte density in cropped image block 501 and cropped image block 505 is generally low.

[0102] According to an embodiment of the present disclosure, the initial tissue staining image is annotated based on the annotation rules and lymphocyte density pixels to obtain an annotated image representing the tertiary lymphatic structure data, including: based on the lymphocyte density pixels, the aggregation degree of the lymphocyte pixels in the initial tissue staining image is divided to determine the aggregation area; based on the lymphocyte density pixels, the maturity degree of the lymphocyte pixels in the aggregation area is identified to obtain an identification result; based on the annotation rules and the identification result, the initial tissue staining image is annotated to obtain a annotated image representing the tertiary lymphatic structure data.

[0103] For example, the labeling rule can be: the contour area of ​​the first maturity level (TLS1) is outlined in red, the contour area of ​​the second maturity level (TLS2) is outlined in blue, and the contour area of ​​the third maturity level (TLS3) is outlined in green. The outlined area curve is closed, and the same area corresponds to only one TLS maturity level. The TLS maturity level represents the maturity level of lymphocytes. The greater the density of lymphocytes, the higher the maturity level.

[0104] According to an embodiment of the present disclosure, based on the lymphocyte density pixels in the lymphocyte density map, the positions and aggregation degrees of the lymphocyte pixels in the initial tissue staining image are preliminarily divided to determine the aggregation area of ​​the lymphocytes.

[0105] According to an embodiment of the present disclosure, the greater the lymphocyte density, the higher the maturity level. The maturity level of the lymphocyte pixels in the aggregation area is identified based on the density value of the lymphocyte density pixels to obtain an identification result, which includes a first-level maturity level result, a second-level maturity level result, and a third-level maturity level result.

[0106] According to an embodiment of the present disclosure, contour areas of different maturity levels in the initial tissue staining image are labeled with different colors according to the labeling rules and the recognition results, and finally a labeled image representing the tertiary lymphatic structure data is obtained.

[0107] Figure 6 An example schematic diagram of generating a labeled image according to an embodiment of the present disclosure is shown.

[0108] like Figure 6As shown, 31 initial tissue staining images were obtained, the aggregation area of ​​lymphocytes on each initial tissue staining image was located, and different maturity levels in the aggregation area were divided based on the lymphocyte density pixels in the lymphocyte density map. The area was outlined using annotation software according to the annotation rules. The outline area of ​​the first maturity level (TLS1) was outlined in red, the outline area of ​​the second maturity level (TLS2) was outlined in blue, and the outline area of ​​the third maturity level (TLS3) was outlined in green to generate an annotated image. The annotation results showed that the number of TLS1 types in all initial tissue staining images was 816, the number of TLS2 types was 194, and the number of TLS3 types was 124.

[0109] Figure 7 An example schematic diagram of generating a medical image dataset according to an embodiment of the present disclosure is shown.

[0110] like Figure 7 As shown in the figure, the processing of the annotation results includes: writing a program to read the XML document generated by the annotation software, identifying the maturity of TLSs and the contour boundaries of TLSs, independently constructing a dataset of annotated images in SVS format, and forming a medical imaging dataset of 512x512 size through uniform scaling, and dividing it into training set and validation set to make the final dataset.

[0111] Based on the above method for annotating three-level lymphatic structure data on tissue images, the present disclosure also provides a device for annotating three-level lymphatic structure data on tissue images. Figure 8 The device is described in detail.

[0112] Figure 8 A structural block diagram of an apparatus for annotating three-level lymphatic structure data on a tissue image according to an embodiment of the present disclosure is shown.

[0113] like Figure 8 As shown, the apparatus 800 for annotating the three-level lymphatic structure data of a tissue image in this embodiment includes a cropping module 810 , a first processing module 820 , a second processing module 830 , a third processing module 840 and an annotation module 850 .

[0114] The cropping module 810 is used to crop the initial tissue staining image of the target object to determine a plurality of cropped image blocks. In one embodiment, the cropping module 810 can be used to perform the operation S110 described above, which will not be described in detail herein.

[0115] The first processing module 820 is used to process the multiple cropped image blocks according to the residual network model to obtain multiple tumor region marked image blocks. In one embodiment, the first processing module 820 can be used to perform the operation S120 described above, which will not be repeated here.

[0116] The second processing module 830 is used to process the tumor region marked image block according to the semantic segmentation model to obtain a lymphocyte mask image block corresponding to the tumor region marked image block, wherein the lymphocyte pixel points marked in the lymphocyte mask image block are located in the tumor region marked image block. In one embodiment, the second processing module 830 can be used to perform the operation S130 described above, which will not be repeated here.

[0117] The third processing module 840 is used to process boundary points and non-boundary points using a triangulated network growth algorithm and a surface rendering algorithm to obtain a rendering section, wherein the rendering section includes a boundary between a fluid and a solid in the flow field scene. In one embodiment, the third processing module 840 can be used to perform the operation S140 described above, which will not be described in detail here.

[0118] The labeling module 850 is used to label the initial tissue staining image based on the labeling rules and the lymphocyte density pixels to obtain a labeled image representing the tertiary lymphatic structure data. In one embodiment, the labeling module 850 can be used to perform the operation S150 described above, which will not be repeated here.

[0119] According to an embodiment of the present disclosure, the first processing module 820 includes a prediction submodule and a determination submodule.

[0120] The prediction submodule is used to process multiple cropped image blocks according to the residual network model to obtain prediction results corresponding to each of the multiple cropped image blocks, wherein the prediction results represent the probability that a pixel point in the cropped image block is a tumor tissue.

[0121] The determination submodule is used to determine the cropped image block as a tumor region marked image block when the prediction result meets a preset probability condition.

[0122] According to an embodiment of the present disclosure, the prediction submodule includes a first prediction unit, a second prediction unit, a third prediction unit, a fourth prediction unit, and a fifth prediction unit.

[0123] The first prediction unit is used to input the cropped image block into the convolutional network layer and output the convolution vector feature.

[0124] The second prediction unit is used to input the convolution vector feature into the maximum pooling layer and output the first pooling vector feature.

[0125] The third prediction unit is used to input the first pooling vector feature into the residual layer and output the residual vector feature.

[0126] The fourth prediction unit is used to input the residual vector feature into the average pooling layer and output the second pooling vector feature.

[0127] The fifth prediction unit is used to input the second pooled vector feature into the fully connected layer, and output a prediction result corresponding to the cropped image block.

[0128] According to an embodiment of the present disclosure, the second processing module 830 includes an identification submodule and a marking submodule.

[0129] The recognition submodule is used to recognize lymphocyte nucleus pixels in the tumor area marked image block using a semantic segmentation model to obtain a recognition result.

[0130] The marking submodule is used to mark the lymphocyte nuclear pixels based on the recognition result to obtain a lymphocyte mask image block corresponding to the tumor area marking image block.

[0131] According to an embodiment of the present disclosure, the third processing submodule 840 includes a filtering submodule, a corrosion submodule, and a generating submodule.

[0132] The filtering submodule is used to filter candidate pixel points in the lymphocyte mask image block to obtain a filtered image block, wherein the pixel value of the candidate pixel point is less than a preset pixel value threshold.

[0133] The corrosion submodule is used to process the filter image block using the corrosion algorithm based on the pixel shape parameters, the noise threshold parameters and the pixel boundary parameters to obtain the lymphocyte target image block.

[0134] The generation submodule is used to process the lymphocyte target image blocks corresponding to each of the multiple lymphocyte mask image blocks according to the morphological connected domain algorithm to obtain a lymphocyte density map corresponding to the initial tissue staining image.

[0135] According to an embodiment of the present disclosure, the generation submodule includes a first generation unit, a second generation unit and a third generation unit.

[0136] The first generating unit is used to count the lymphocytes in the lymphocyte target image block by using a morphological connected domain algorithm to obtain the number of lymphocytes.

[0137] The second generating unit is used to obtain the local lymphocyte density corresponding to the lymphocyte target image block according to the number of lymphocytes corresponding to the lymphocyte target image.

[0138] The third generating unit is used to obtain a lymphocyte density map corresponding to the initial tissue staining image according to the local lymphocyte densities corresponding to each of the multiple lymphocyte target images.

[0139] According to an embodiment of the present disclosure, the labeling module 850 includes a first labeling submodule, a second labeling submodule and a third labeling submodule.

[0140] The first annotation submodule is used to divide the aggregation degree of lymphocyte pixels in the initial tissue staining image based on lymphocyte density pixels to determine the aggregation area.

[0141] The second annotation submodule is used to identify the maturity of lymphocyte pixels in the aggregation area based on lymphocyte density pixels to obtain an identification result.

[0142] The third annotation submodule is used to annotate the initial tissue staining image based on the annotation rules and the recognition results to obtain an annotated image representing the tertiary lymphatic structure data.

[0143] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging circuits, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0144] For example, any of the trimming module 810, the first processing module 820, the second processing module 830, the third processing module 840, and the marking module 850 can be combined in one module / unit / sub-unit for implementation, or any of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the trimming module 810, the first processing module 820, the second processing module 830, the third processing module 840, and the marking module 850 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a suitable combination of any of them. Alternatively, at least one of the cropping module 810 , the first processing module 820 , the second processing module 830 , the third processing module 840 and the labeling module 850 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be performed.

[0145] Fig. 9 A block diagram of an electronic device suitable for implementing a method for annotating three-level lymphatic structure data for a tissue image according to an embodiment of the present invention is shown.

[0146] Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0147] like Fig. 9 As shown, a computer electronic device 900 according to an embodiment of the present invention includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0148] In RAM 903, various programs and data required for the operation of electronic device 900 are stored. Processor 901, ROM 902 and RAM 903 are connected to each other via bus 904. Processor 901 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 902 and / or RAM 903. It should be noted that the program can also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in one or more memories.

[0149] Optionally, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the input / output (I / O) interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 908 including a hard disk, etc.; and a communication portion 909 including a network interface card such as a LAN card, a modem, etc. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage portion 908 as needed.

[0150] Optionally, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. Optionally, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0151] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method for annotating the three-level lymphatic structure data of the tissue image according to the embodiment of the present invention is implemented.

[0152] Optionally, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0153] For example, optionally, the computer-readable storage medium may include the ROM 902 and / or the RAM 903 described above and / or one or more memories other than the ROM 902 and the RAM 903 .

[0154] An embodiment of the present invention also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the method for annotating three-level lymphatic structure data for tissue images provided by the embodiment of the present invention.

[0155] When the computer program is executed by the processor 901, the above functions defined in the system / device of the embodiment of the present invention are performed. Optionally, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0156] In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and downloaded and installed through the communication part 909, and / or installed from a removable medium 911. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0157] Optionally, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages, specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, such as Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0158] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0159] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for annotating three-level lymphatic structure data of a tissue image, characterized in that: The method comprises: cropping an initial tissue staining image of the target object to determine a plurality of cropped image blocks; Processing the plurality of cropped image blocks according to the residual network model to obtain a plurality of tumor region marked image blocks; Processing the tumor region marked image block according to the semantic segmentation model to obtain a lymphocyte mask image block corresponding to the tumor region marked image block, wherein the lymphocyte pixel points marked in the lymphocyte mask image block are located in the tumor region marked image block; Processing the plurality of lymphocyte mask image blocks based on a morphological algorithm to obtain a lymphocyte density map corresponding to the initial tissue staining image, wherein the lymphocyte density pixels of the lymphocyte density map represent a lymphocyte density distribution result; The initial tissue staining image is annotated based on the annotation rules and the lymphocyte density pixels to obtain an annotated image representing the tertiary lymphatic structure data.

2. The method according to claim 1, characterized in that Processing the plurality of cropped image blocks according to the residual network model to obtain a plurality of tumor region marked image blocks includes: Processing the plurality of cropped image blocks according to the residual network model to obtain prediction results corresponding to each of the plurality of cropped image blocks, wherein the prediction results represent the probability that a pixel point in the cropped image block is a tumor tissue; When the prediction result meets a preset probability condition, the cropped image block is determined as a tumor region marked image block.

3. The method according to claim 2, characterized in that The residual network model includes a convolutional network layer, a maximum pooling layer, a residual layer, an average pooling layer, and a fully connected layer; Wherein, processing the plurality of cropped image blocks according to the residual network model to obtain prediction results corresponding to each of the plurality of cropped image blocks includes: Inputting the cropped image block into the convolutional network layer and outputting convolution vector features; Inputting the convolution vector feature into the maximum pooling layer, and outputting a first pooling vector feature; Inputting the first pooled vector feature into the residual layer, and outputting the residual vector feature; Inputting the residual vector feature into the average pooling layer, and outputting a second pooling vector feature; The second pooled vector feature is input into the fully connected layer, and the prediction result corresponding to the cropped image block is output.

4. The method according to claim 1, characterized in that: Processing the tumor region marked image block according to the semantic segmentation model to obtain a lymphocyte mask image block corresponding to the tumor region marked image includes: Using a semantic segmentation model to identify lymphocyte nucleus pixels in the tumor region marked image block to obtain an identification result; The lymphocyte nucleus pixels are pixel-marked based on the recognition result to obtain a lymphocyte mask image block corresponding to the tumor region marked image block.

5. The method according to claim 1, characterized in that Processing the plurality of lymphocyte mask image blocks based on a morphological algorithm to obtain a lymphocyte density map corresponding to the initial tissue staining image includes: Filtering candidate pixel points in the lymphocyte mask image block to obtain a filtered image block, wherein the pixel value of the candidate pixel point is less than a preset pixel value threshold; Based on pixel shape parameters, noise threshold parameters, and pixel boundary parameters, the filtered image block is processed using a corrosion algorithm to obtain a lymphocyte target image block; The lymphocyte target image blocks corresponding to each of the plurality of lymphocyte mask image blocks are processed according to a morphological connected domain algorithm to obtain a lymphocyte density map corresponding to the initial tissue staining image.

6. The method according to claim 5, characterized in that Processing the lymphocyte target image blocks corresponding to each of the plurality of lymphocyte mask image blocks according to the morphological connected domain algorithm to obtain a lymphocyte density map corresponding to the initial tissue staining image includes: Counting the lymphocytes in the lymphocyte target image block using a morphological connected domain algorithm to obtain the number of lymphocytes; Obtaining a local lymphocyte density corresponding to the lymphocyte target image block according to the number of lymphocytes corresponding to the lymphocyte target image; A lymphocyte density map corresponding to the initial tissue staining image is obtained according to the local lymphocyte densities corresponding to each of the plurality of lymphocyte target images.

7. The method according to claim 1, characterized in that The initial tissue staining image is annotated based on the annotation rule and the lymphocyte density pixels to obtain an annotated image representing the tertiary lymphatic structure data, including: Based on the lymphocyte density pixels, dividing the aggregation degree of lymphocyte pixels in the initial tissue staining image to determine the aggregation area; Based on the lymphocyte density pixels, the maturity of the lymphocyte pixels in the aggregation area is identified to obtain an identification result; Based on the annotation rule and the recognition result, the initial tissue staining image is annotated to obtain an annotated image representing the tertiary lymphatic structure data.

8. A device for annotating three-level lymphatic structure data of a tissue image, comprising: A cropping module, used for cropping the initial tissue staining image of the target object and determining a plurality of cropped image blocks; A first processing module, used for processing the plurality of cropped image blocks according to a residual network model to obtain a plurality of tumor region marked image blocks; A second processing module is used to process the tumor region marked image block according to the semantic segmentation model to obtain a lymphocyte mask image block corresponding to the tumor region marked image block, wherein the lymphocyte pixel points marked in the lymphocyte mask image block are located in the tumor region marked image block; A third processing module is used to process the plurality of lymphocyte mask image blocks based on a morphological algorithm to obtain a lymphocyte density map corresponding to the initial tissue staining image, wherein the lymphocyte density pixels of the lymphocyte density map represent a lymphocyte density distribution result; The labeling module is used to label the initial tissue staining image based on the labeling rules and the lymphocyte density pixels to obtain a labeled image representing the tertiary lymphatic structure data.

9. An electronic device, comprising: one or more processors; a memory for storing one or more programs, When one or more programs are executed by one or more processors, the one or more processors implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to implement the method of any one of claims 1 to 7.

Images