A three-level lymphatic structure pathology image classification model construction and classification method

Abstract

The embodiment of the present invention discloses a classification model and classification method for three-level lymphatic structure pathology images. The construction method includes: inputting each HE whole-slice image and HE image of the three-level lymphatic structure into a first branch, using a first feature extraction network to extract features from each HE whole-slice image and HE image, respectively, to obtain image features at the full slice level and image features at the three-level lymphatic structure level; using a second feature extraction network to extract features from each HE image block, to obtain image features at the cell level; fusing the three levels of image features of the same three-level lymphatic structure, and performing a primary classification of the maturity of the three-level lymphatic structure based on the fused features; inputting the fluorescence image of each three-level lymphatic structure into a second branch for secondary classification; and training each feature extraction network based on the differences in multiple recognition results to obtain a classification model for the three-level lymphatic structure pathology images.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of medical image processing, and in particular to a three-level lymphatic structure pathology image classification model construction and classification method. Background Art

[0002] Pathological images are an important basis for doctors to make medical diagnoses. In the treatment of lung cancer, there are three main treatment methods: radiotherapy, chemotherapy, and neoadjuvant drug therapy. The treatment effects and prognostic effects of each treatment method vary from patient to patient, and the differences can even be huge. Studies have found that the number of tertiary lymphatic structures in patient tumor sections is closely related to their maturity (category), but the maturity of tertiary lymphatic structures can only be obtained through immunofluorescence staining, which is costly and difficult to obtain. However, HE images are low-cost and easy to obtain. Therefore, a classification method is needed to accurately classify the maturity of tertiary lymphatic structures based on HE images, reduce the cost of obtaining tertiary lymphatic structure pathological parameters, help doctors and patients develop personalized treatment plans, and improve survival. Summary of the Invention

[0003] The embodiment of the present invention provides a three-level lymphatic structure pathology image classification model construction and classification method to solve the above technical problems.

[0004] In a first aspect, an embodiment of the present invention provides a method for constructing a three-level lymphatic structure pathology image classification model, comprising:

[0005] Acquire multiple HE whole-slice images, as well as HE images and fluorescence images of the tertiary lymphatic structures in each HE whole-slice image;

[0006] Each HE whole-slice image and each HE image of the tertiary lymphatic structure are input into a first branch, where the following processing is performed: a first feature extraction network is used to extract features from each HE whole-slice image and each HE image, respectively, to obtain image features at the whole-slice level and image features at the tertiary lymphatic structure level; each HE image is divided into multiple HE image blocks, and a second feature extraction network is used to extract features from each HE image block to obtain image features at the cellular level; the three levels of image features for the same tertiary lymphatic structure are fused, and the maturity of the tertiary lymphatic structure is classified based on the fused features;

[0007] Inputting the fluorescence images of each tertiary lymphatic structure into the second branch, performing feature extraction on each fluorescence image using a third feature extraction network in the second branch, and performing secondary classification of the maturity of the tertiary lymphatic structure based on each extracted feature;

[0008] Based on the differences between the two classification results of the same lymphatic structure and the classification results of manual identification in fluorescence images, each feature extraction network was trained, and the trained first branch was used as the classification model for the three-level lymphatic structure pathology images.

[0009] In a second aspect, an embodiment of the present invention provides a three-level lymphatic structure pathology image classification method, comprising:

[0010] Acquire HE whole-slice images and HE images of the third-level lymphatic structures to be classified;

[0011] The HE whole slice and HE image are input into the classification model of the three-level lymphatic structure pathological image constructed by the above method to obtain the maturity of the three-level lymphatic structure.

[0012] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising:

[0013] one or more processors;

[0014] a memory for storing one or more programs,

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the three-level lymphatic structure pathology image classification model construction method described in any embodiment.

[0016] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for constructing a three-level lymphatic structure pathology image classification model described in any embodiment.

[0017] Embodiments of the Invention: This invention aims to provide a deep learning method for accurately predicting the maturity of tertiary lymphatic structures from HE pathology images. This method constructs a modular, dual-branch, multi-task pathology image classification network based on feature fusion. The immunofluorescence image processing branch guides and supervises the training of the HE image processing branch, effectively extracting feature information from pathology images. The system classifies tertiary lymphatic structures by fusing features at the full-slice level, the tertiary lymphatic structure level, and the cell image block level. This system can accurately predict tertiary lymphatic structures in pathological HE images, reduce the cost of obtaining physiological parameters of tertiary lymphatic structures for patients, facilitate widespread application, and contribute to personalized diagnosis and treatment for patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 This is a flow chart of a method for constructing a three-level lymphatic structure pathology image classification model provided by an embodiment of the present invention;

[0020] Figure 2 is a schematic diagram of a three-level lymphatic structure classification standard provided by an embodiment of the present invention;

[0021] Figure 3 This is a flowchart of another method for constructing a three-level lymphatic structure pathology image classification model provided by an embodiment of the present invention;

[0022] Figure 4 An operational flow chart of a feature fusion attention module provided by an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of the structure of a feature extraction network provided by an embodiment of the present invention;

[0024] Figure 6 This is a flow chart of a three-level lymphatic structure pathology image classification method provided by an embodiment of the present invention;

[0025] Figure 7 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention are described clearly and completely below. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are also within the scope of protection of the present invention.

[0027] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0028] In the description of the present invention, it should also be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood broadly. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0029] As described in the background art, there is an urgent need for a classification method that can accurately classify the maturity of tertiary lymphatic structures based on HE images. At present, there are the following difficulties in predicting the maturity of tertiary lymphatic structures from HE images: 1) The HE image staining of each patient has staining differences due to the non-repeatability of the doctor's operation. 2) The classification of tertiary lymphatic structures in HE images is related to the physiological state of the entire section, the morphology and cell density of each tertiary lymphatic structure, and the constituent cells in the tertiary lymphatic structure. How to extract effective features is a difficulty. 3) In addition to serving as the gold standard for the maturity of tertiary lymphatic structures, what other effective information in immunofluorescence images can be used to predict tertiary lymphatic structures.

[0030] Based on the above difficulties, Figure 1 This is a flow chart of a method for constructing a three-level lymphatic structure pathology image classification model provided by an embodiment of the present invention. By constructing a feature extraction framework and an immunofluorescence image pairing processing framework, HE images and immunofluorescence images are processed simultaneously to extract effective features and achieve accurate classification of three-level lymphatic structures. The method is executed by an electronic device, such as Figure 1 As shown, the specific steps include:

[0031] S110 , acquiring multiple HE whole-slice images, as well as HE images and fluorescence images of the tertiary lymphatic structures in each HE whole-slice image.

[0032] In this example, HE whole-slice images, HE images (with most areas outside the tertiary lymphatic structures removed), and immunofluorescence images of the tertiary lymphatic structures were obtained as training and test sets for the tertiary lymphatic structure classification model. Optionally, the maturity categories of the tertiary lymphatic structures (primary, secondary, and mature) were manually annotated using immunofluorescence images and used as the gold standard for tertiary lymphatic structure classification in model training.

[0033] In a specific embodiment, first, HE whole-slice images and immunofluorescence images of the patient can be collected to ensure that these images cover the different morphologies and cell densities of the tertiary lymph node structures. Then, the doctor uses an image annotation tool to manually outline the contours of the tertiary lymph node structures in the HE whole-slice images and immunofluorescence images to obtain the HE images and immunofluorescence images of the tertiary lymph node structures for each patient. At the same time, the doctor uses the image annotation tool to manually outline the contours of the tertiary lymph node structures in the HE whole-slice images and immunofluorescence images. Figure 2 The three-level lymphatic structure classification standard shown in the figure uses immunofluorescence images to determine the classification of each tertiary lymphatic structure. Specifically, this standard distinguishes three categories based on the CD20, CD21, and CD23 channels. Immunofluorescence images allow doctors to accurately identify the maturity of tertiary lymphatic structures. Finally, the labeled HE full-section images, HE images, immunofluorescence images, and corresponding classification labels are organized into a database to ensure database balance and diversity, thereby improving the model's generalization ability.

[0034] S120. Each HE whole-slice image and each HE image of the tertiary lymphatic structure are input into a first branch, where the following processing is performed: a first feature extraction network is used to extract features from each HE whole-slice image and each HE image, respectively, to obtain image features at the whole-slice level and image features at the tertiary lymphatic structure level; each HE image is divided into multiple HE image blocks, and a second feature extraction network is used to extract features from each HE image block to obtain image features at the cellular level; the three levels of image features of the same tertiary lymphatic structure are fused, and the maturity of the tertiary lymphatic structure is classified based on the fused features.

[0035] Figure 3 This is a flowchart of another method for constructing a three-level lymphatic structure pathology image classification model provided by an embodiment of the present invention, combined with Figure 3 The entire method can be divided into two branches: one branch is used to process HE whole-slice images and HE images, called the first branch; the other branch is used to process immunofluorescence images, called the second branch. This step describes the specific operations in the first branch.

[0036] like Figure 3As shown, the first branch includes a multi-scale feature extraction module, a feature fusion module, and a classification network. The multi-scale feature extraction module includes three feature extraction networks, one for extracting image features at the whole slice, the third-level lymphatic structure, and the other at the cellular level from HE image data. Optionally, each feature extraction network can employ a Swim-Transformer architecture, with two feature extraction networks sharing the same parameters. These networks are collectively referred to as the first feature extraction network. The whole HE slice image is fed into one first feature extraction network to generate slice-level image features. The entire HE image of the third-level lymphatic structure is fed into another first feature extraction network to generate lymphatic structure-level image features, which contain morphological information about the third-level lymphatic structure. Simultaneously, the HE image of the third-level lymphatic structure is segmented into multiple small image blocks and fed into yet another feature extraction network to generate cellular-level image features, including information about cell density and cell composition. In particular, since this network processes cellular-level images, the visual effects are very similar to those of conventional visual images. Therefore, pre-trained weights from public visual image datasets can be used as fixed weights to reduce model parameters. For ease of distinction and description, this network is referred to as the second feature extraction network.

[0037] The feature fusion module is used to fuse the three scale image features and feed them into the classification network, which then classifies the maturity of the three-level lymphatic structures based on the fused features. Optionally, this module introduces an attention mechanism to focus the network on important features during learning. Figure 4 This is an operational flow chart of a feature fusion attention module provided by an embodiment of the present invention, such as Figure 4 As shown in the figure, after the three levels of image features of the same three-level lymphatic structure are input into the module, the following operations are performed in sequence within the module: two levels of image features are arbitrarily extracted as query vectors and key vectors, and the corresponding value vectors are substituted into the self-attention mechanism to calculate the corresponding value vectors; the extraction is repeated until all combinations are extracted, and the three resulting value vectors are used as three primary fusion features. Then, two of the three primary fusion features are arbitrarily extracted as query vectors and key vectors, and the corresponding value vectors are substituted into the self-attention mechanism to calculate the corresponding value vectors. The extraction is repeated until all combinations are extracted, and the two resulting value vectors are used as two secondary fusion features; finally, the three primary fusion features are added together to obtain another secondary fusion feature; the three secondary fusion features are spliced ​​with the image features of the three sectors to obtain the final fusion feature.

[0038] In a specific embodiment, each feature extraction network can use Figure 5The structure shown in the figure is that the second feature extraction network is pre-trained with ImageNet data, and the weights are fixed and not updated. The weights of the attention mechanism in the first feature extraction network and the feature fusion module are not fixed and are directly related to the task. The parameters are updated during the training process. Let the full slice image of a sample be X, the three-level lymphatic structure image be T, and the n small image block level images be P. n , the three-scale features y1, y2 and n×y can be obtained through the feature extraction network 3i , respectively 1×768 dimensions, 1×768 dimensions, n×768 dimensions, the small image block features are averaged to obtain y3, and the primary fusion features obtained by the attention block are y 12 、y 23 and y 13 , the two secondary fusion features obtained by the attention block are y a and y b , another secondary fusion feature obtained by directly adding the three primary fusion features is y c , and the final fusion feature is y. The operation process of the above feature fusion module can be expressed by the following formula:

[0039] y1=X@S

[0040] y2=T@S

[0041]

[0042] y 12 =y1#y2

[0043] y 13 =y1#y3

[0044] y 23 =y2#y3

[0045] y a =y 12 #y 13

[0046] y b =y 13 #y 23

[0047] y c =y 12 +y 12 +y 23

[0048] y=y a ~y b ~y c

[0049] Among them, @ represents model feature extraction calculation, # represents attention block calculation, + represents the addition of values ​​at corresponding positions of vectors or tensors, ~ represents vector or tensor splicing, and the variable before × only represents the number of variables after × and does not represent numerical multiplication.

[0050] S130. Input the fluorescence images of each tertiary lymphatic structure into the second branch, perform feature extraction on each fluorescence image using a third feature extraction network in the second branch, and perform secondary classification on the maturity of the tertiary lymphatic structure based on each extracted feature.

[0051] Combine Figure 3 , the second branch includes a feature extraction network, called the third feature extraction network, which is used to extract features from the fluorescence image of the lymphatic structure, and to predict the maturity of the tertiary lymphatic structure again based on the features of the fluorescence image. It is worth mentioning that this branch assists the first branch in extracting effective features during the training phase, and can directly shut down the immunofluorescence pathway during testing and application without affecting the test results and use. Optionally, the first feature extraction network and the third feature extraction network can share part of the weights, that is, the third feature network is equivalent to the first feature extraction network cascaded with a classification network. Since the identification of the tertiary lymphatic structure in the fluorescence image is relatively easy, by sharing weights, the structural information in the fluorescence image can be fully utilized to guide the parameter update of the first feature network.

[0052] S140. Based on the difference between the two classification results of the same lymphatic structure and the classification result of manual identification in the fluorescence image, each feature extraction network is trained, and the trained first branch is used as a classification model for the three-level lymphatic structure pathology image.

[0053] In this step, a multi-task training gradient backpropagation loss is designed. The classification results of manual identification in S110 are used as the gold standard for predicting the maturity of the three-level lymphatic structure. The prediction results obtained by fluorescence images in the second branch are used as the secondary standard. Together with the classification results output by the first branch, a backpropagation loss function is constructed.

[0054] Specific, combined Figure 3, the entire method process includes two tasks: ① Use HE multi-scale attention fusion features to perform three-level lymphatic structure classification prediction (corresponding to the first branch, to obtain a primary classification result), ② Use immunofluorescence three-level lymphatic structure images to perform three-level lymphatic structure classification prediction (corresponding to the second branch, to obtain a secondary classification result, which is used to constrain the feature extraction and classification of the first branch). Then, from the two classification results of the same lymphatic structure and the classification results manually identified in the fluorescence image, any two classification results can be extracted as the difference, and the extraction can be repeated until all combinations are extracted; the three differences obtained are weighted and fused as the loss function for training each feature extraction network. Specifically, the HE multi-scale attention fusion feature prediction result is recorded as Pred1, the immunofluorescence prediction result is recorded as Pred2, and the gold standard is recorded as Label, then

[0055] LOSS1=(Pred1,Pred2)

[0056] LOSS2=(Pred1,Label)

[0057] LOSS3=(Pred2,Label)

[0058] LOSS=αLOOS1+βLOSS2+γLOSS3

[0059] Here, α, β, and γ are hyperparameters representing the weights of the three LOSSs. Since LOSS2 is the final target task, β is set to the maximum. The non-fixed parameters in the first and second branches are updated using the above loss function. After the parameters stabilize, the second branch is paused, and the first branch is used as the classification model for the three-level lymphatic structure pathology images.

[0060] Based on the above classification model, Figure 6 This is a flow chart of a method for classifying pathological images of three-level lymphatic structures provided by an embodiment of the present invention, which is applicable to classifying the maturity level of three-level lymphatic structures based on HE full-slice images and HE images of three-level lymphatic structures. Figure 6 As shown, the method specifically includes the following steps:

[0061] S210 , obtaining a HE whole-slice image and a HE image of the third-level lymphatic structure to be classified.

[0062] In this step, a full-slice image of the tertiary lymphatic structure to be classified is obtained, and most information other than the tertiary lymphatic structure is removed from the HE full-slice image to obtain a HE image of the tertiary lymphatic structure.

[0063] S220 , inputting the HE full slice and HE image into the classification model of the three-level lymphatic structure pathology image to obtain the maturity of the three-level lymphatic structure.

[0064] The maturity here refers to the classification result obtained by the first branch, including three categories: primary, secondary, and mature.

[0065] In summary, the embodiment of the present invention discloses a method for constructing a classification model for pathological images of three-level lymphatic structures and a classification method, which is mainly used for the automatic classification of three-level lymphatic structures in pathological images, and provides a basis for subsequent prognostic processing, etc. Specifically, the embodiment of the present invention can extract effective features from multiple scales such as the entire slice of the HE image, the morphology and cell density of the three-level lymphatic structure, and the constituent cells in the three-level lymphatic structure, and use the attention module to perform feature cross-fusion to maximize feature utilization. A modular two-branch network is used during model training to fully exploit information other than immunofluorescence as the gold standard, and to process HE image data and immunofluorescence images simultaneously to assist the HE image processing branch in extracting effective features; at the same time, the prediction results of the immunofluorescence branch are used as a secondary standard for multi-task learning, so that the network is guided by the immunofluorescence pathway in both feature extraction and classifier. During the model testing and use stage, the processing branch of the immunofluorescence image can be directly closed without affecting the test results and use at all. This method optimizes the processing flow and network from three aspects: multi-scale, multi-modal, and multi-task assistance, to achieve accurate classification of tertiary lymphatic structures, reduce the cost of patients obtaining physiological parameters of tertiary lymphatic structures, facilitate promotion and application, and contribute to personalized diagnosis and treatment of patients.

[0066] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 7 As shown, the device includes a processor 60, a memory 61, an input device 62 and an output device 63; the number of processors 60 in the device can be one or more. Figure 7 In the embodiment, a processor 60 is used as an example; the processor 60, the memory 61, the input device 62 and the output device 63 in the device can be connected by a bus or other means. Figure 7 The bus connection is taken as an example.

[0067] Memory 61, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the method for constructing a three-level lymphatic structure pathology image classification model in the embodiments of the present invention, or the program instructions / modules corresponding to the three-level lymphatic structure pathology image classification method. Processor 60 executes the software programs, instructions, and modules stored in memory 61 to execute various functional applications and data processing of the device, thereby implementing the aforementioned method for constructing a three-level lymphatic structure pathology image classification model or the three-level lymphatic structure pathology image classification method.

[0068] The memory 61 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data generated based on the use of the terminal. Furthermore, the memory 61 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some instances, the memory 61 may further include memory remotely located relative to the processor 60, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0069] The input device 62 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 63 may include a display device such as a display screen.

[0070] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method for constructing a three-level lymphatic structure pathology image classification model or the method for classifying three-level lymphatic structure pathology images of any embodiment is implemented.

[0071] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by an instruction execution system, device or device or used in combination with it.

[0072] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0073] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0074] Computer program code for performing the operations of the present invention can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a three-level lymphatic structure pathology image classification model, characterized in that: include: Acquire multiple HE whole-slice images, as well as HE images and fluorescence images of the tertiary lymphatic structures in each HE whole-slice image; Each HE whole-slice image and the HE image of the three-level lymphatic structure are input into the first branch, and the following processing is performed in the first branch: the first feature extraction network is used to extract features from each HE whole-slice image and each HE image, respectively, to obtain image features at the whole-slice level and image features at the three-level lymphatic structure level; each HE image is divided into multiple HE image blocks, and the second feature extraction network is used to extract features from each HE image block to obtain features of each HE image block; the features of multiple HE image blocks in the same HE image are averaged to obtain image features at the cellular level of the HE image; from the three levels of image features of the same three-level lymphatic structure, two levels of image features are arbitrarily extracted as query vectors and key vectors, and the corresponding value vectors are substituted into the self-attention mechanism to calculate the corresponding value vectors; the extraction is repeated until all combinations are extracted, and the three obtained value vectors are used as three primary fusion features; Extract any two of the three primary fusion features as the query vector and key vector, substitute them into the self-attention mechanism to calculate the corresponding value vector, repeat the extraction until all combinations are extracted, and use the obtained two value vectors as two secondary fusion features; Adding the three primary fusion features to obtain another secondary fusion feature; The three secondary fusion features are concatenated with the three levels of image features to obtain a final fusion feature, and the maturity of the three-level lymphatic structure is classified once according to the final fusion feature; Inputting the fluorescence images of each tertiary lymphatic structure into the second branch, performing feature extraction on each fluorescence image using a third feature extraction network in the second branch, and performing secondary classification of the maturity of the tertiary lymphatic structure based on each extracted feature; Based on the differences between the two classification results of the same lymphatic structure and the classification results of manual identification in fluorescence images, each feature extraction network was trained, and the trained first branch was used as the classification model for the three-level lymphatic structure pathology images.

2. The method according to claim 1, characterized in that The first feature extraction network and the third feature extraction network share weights.

3. The method according to claim 1, characterized in that The second feature extraction network uses weights pre-trained with a public visual image dataset.

4. The method according to claim 1, wherein The training of each feature extraction network based on the difference between the two classification results of the same lymphatic structure and the classification result of manual identification in the fluorescence image includes: From the two classification results of the same lymphatic structure and the classification results of manual identification in the fluorescence image, two classification results were randomly selected and subtracted. The subtraction was repeated until all combinations were extracted. The three obtained differences are weighted and fused as the loss function for training each feature extraction network.

5. The method according to claim 1, wherein Each feature extraction network adopts the Swim-Transformer structure.

6. A three-level lymphatic structure pathology image classification method, characterized in that: include: Acquire HE whole-slice images and HE images of the third-level lymphatic structures to be classified; The HE whole slice and HE image are input into the classification model of the three-level lymphatic structure pathological image constructed by the method according to any one of claims 1 to 5 to obtain the maturity of the three-level lymphatic structure.

7. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the three-level lymphatic structure pathology image classification model construction method described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the program is executed by a processor, the method for constructing a three-level lymphatic structure pathology image classification model as described in any one of claims 1-5 is implemented.

Images