Endometrial joint classification method, device and equipment and storage medium

Abstract

The invention discloses an endometrium joint classification method, device and equipment and a storage medium, and is applied to the technical field of computer vision, and the method comprises the steps: dividing a pathological section image of an endometrium into a plurality of sub-images, extracting a first feature of each sub-image through a histopathologic feature extractor, extracting a second feature of each sub-image through a molecular typing feature extractor; determining a histopathologic feature of the pathological section image according to the first feature of each sub-image, and determining a molecular typing feature of the pathological section image according to the second feature of each sub-image; the molecular typing category of the endometrium is predicted through the combination of the plurality of first features and the molecular typing features, and the histopathologic category of the endometrium is predicted through the combination of the plurality of second features and the histopathologic features. The problem that in the prior art, it is difficult to form a comprehensive and unified diagnosis conclusion through histopathology observation results and molecular detection results is solved, and the diagnosis accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device, equipment and storage medium for combined endometrial classification. Background Art

[0002] In modern medicine, the study of endometrial pathology is of great significance to women's health, especially in the early diagnosis and accurate classification of endometrial cancer (EC), which is crucial for the development of effective treatment plans. Traditionally, pathologists rely on microscopes to observe slices and use their rich experience and expertise to make diagnoses. However, this method has obvious limitations when dealing with large sample sizes and complex pathological features, which may lead to misdiagnosis or missed diagnosis, thus affecting the treatment effect and prognosis of patients.

[0003] In recent years, molecular classification has been shown to be of great value in cancer diagnosis, prognosis, and personalized treatment planning. Through genome sequencing and biomarker detection, researchers can gain a deeper understanding of the biological characteristics of tumors. Endometrial cancer originating from atypical endometrial hyperplasia (AEH) or endometrioid intraepithelial neoplasia (EIN) may belong to four different molecular subtypes. These molecular subtypes are: POLE hypermutation (POLEmut), MMRd, NSMP, and p53 abnormal. AEH / EIN associated with POLEmut and MMRd ECs often have POLE mutations and loss of MMR expression, respectively, which suggest that these genetic changes are early events in tumorigenesis. In addition, serous endometrial carcinoma (SEC) and carcinosarcoma are mostly derived from serous endometrial intraepithelial carcinoma (SEIC), but some cases are derived from AEH / EIN with PTEN mutations. The association between these molecular classifications and early pathological changes provides a new perspective for clinicians, making it particularly important to consider personalized strategies based on molecular characteristics in the formulation of treatment plans.

[0004] Current molecular typing methods mainly rely on genome sequencing or molecular marker detection outside the tissue, such as gene mutation, gene expression or immunohistochemical analysis. Although these methods can deeply reveal the molecular characteristics of tumors, they are separated from traditional histological observation methods and fail to achieve an organic combination of the two. This means that when making diagnostic and treatment decisions, clinicians must refer to the morphological characteristics of pathological sections and molecular test results separately, which may lead to information fragmentation and make it difficult to form a comprehensive and unified diagnostic conclusion. In addition, in actual operations, the high cost and complexity of molecular testing limit its widespread application, especially in hospitals or laboratories with limited resources. Summary of the invention

[0005] The present application provides a combined classification method, device, equipment and storage medium for endometrium, which can predict the histopathological category and molecular typing category of endometrium by combining the histopathological characteristics and molecular typing characteristics of pathological slice images, solve the problem that it is difficult to form a comprehensive and unified diagnostic conclusion from the histopathological observation results and molecular detection results in the prior art, and improve the accuracy of diagnosis. Moreover, the molecular typing detection using pathological slice images can effectively reduce the implementation cost of molecular detection and improve the applicability.

[0006] In a first aspect, the present application provides a combined endometrial classification method, comprising: Divide the pathological section image of the endometrium into a plurality of sub-images, extract a first feature of each of the sub-images by a histopathological feature extractor, and extract a second feature of each of the sub-images by a molecular typing feature extractor; Determine the histopathological feature of the pathological section image according to the first feature of each of the sub-images, and determine the molecular typing feature of the pathological section image according to the second feature of each of the sub-images; The molecular typing category of the endometrium is predicted by combining the first features with the molecular typing features, and the histopathological category of the endometrium is predicted by combining the second features with the histopathological features.

[0007] In a second aspect, the present application provides an endometrial joint classification device, comprising: A sub-image feature extraction module is configured to divide the endometrial pathological section image into a plurality of sub-images, extract a first feature of each of the sub-images by a histopathological feature extractor, and extract a second feature of each of the sub-images by a molecular typing feature extractor; A slice image feature extraction module is configured to determine the histopathological feature of the pathological slice image according to the first feature of each of the sub-images, and to determine the molecular typing feature of the pathological slice image according to the second feature of each of the sub-images; The joint classification module is configured to jointly predict the molecular typing category of the endometrium by using multiple first features and the molecular typing features, and to jointly predict the histopathological category of the endometrium by using multiple second features and the histopathological features.

[0008] In a third aspect, the present application provides an endometrial joint classification device, comprising: One or more processors; a storage device storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the endometrial joint classification method as described in the first aspect.

[0009] In a fourth aspect, the present application provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to perform the endometrial joint classification method as described in the first aspect.

[0010] In the present application, the pathological section image of the endometrium is divided into multiple sub-images, the first feature of each sub-image is extracted by a histopathological feature extractor, and the second feature of each sub-image is extracted by a molecular typing feature extractor; the histopathological feature of the pathological section image is determined according to the first feature of each sub-image, and the molecular typing feature of the pathological section image is determined according to the second feature of each sub-image; the molecular typing category of the endometrium is predicted by combining multiple first features and molecular typing features, and the histopathological category of the endometrium is predicted by combining multiple second features and histopathological features. Through the above technical means, the histopathological features of the pathological section image can be generated by fusing the first features of each sub-image with respect to the histopathological category, and the molecular typing category of the pathological section image can be generated by fusing the second features of each sub-image with respect to the molecular typing category, thereby realizing accurate extraction of the histopathological features and molecular typing features of the pathological section image. The prediction of the endometrial histopathological category is predicted by fusing the histopathological features and the second feature related to the molecular typing category, and the prediction of the endometrial molecular typing category is predicted by fusing the molecular typing features and the first feature related to the histopathological category, so that the prediction model can not only capture the significant features of the image but also learn features in other directions, enriching the feature information obtained by the prediction model and effectively improving the prediction accuracy of the prediction model. In addition, the prediction model realizes the combination of molecular typing detection and histopathological classification, provides more comprehensive diagnostic information, solves the problem in the prior art that it is difficult to form a comprehensive and unified diagnostic conclusion between histopathological observation results and molecular detection results, and improves the diagnostic accuracy. The use of pathological section images for molecular typing detection can effectively reduce the implementation cost of molecular detection and improve applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1is a flow chart of a combined endometrial classification method provided in an embodiment of the present application; Figure 2 is a flow chart of generating histopathological features of pathological section images provided by an embodiment of the present application; Figure 3 It is a flow chart of generating molecular typing features of pathological section images provided in an embodiment of the present application; Figure 4 This is a flow chart of the combined prediction of molecular typing categories provided in the embodiments of the present application; Figure 5 is a schematic diagram of the network structure of a first joint classification network provided in an embodiment of the present application; Figure 6 is a flowchart of the joint prediction of histopathology categories provided in the embodiments of the present application; Figure 7 is a schematic diagram of the network structure of a second joint classification network provided in an embodiment of the present application; Figure 8 is a schematic diagram of the classification prediction process of pathological slice images provided in an embodiment of the present application; Fig. 9 It is a structural schematic diagram of an endometrial joint classification device provided in an embodiment of the present application; Fig.10 It is a structural schematic diagram of an endometrial combined classification device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0012] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only part of the present application is shown in the accompanying drawings, but not all of the content. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0013] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0014] In a related implementation, the histopathological observation method can automatically analyze the pathological section images of the endometrium through a computer-aided diagnosis system to determine the histopathological category of the endometrium. In addition, molecular typing detection technology can be used to detect genome sequencing or molecular markers outside the tissue to determine the molecular typing category of the endometrium. Although both methods can diagnose the category of the endometrium, the two methods are separate, which means that clinicians must refer to the morphological characteristics of the pathological sections and the molecular detection results separately when making diagnostic and treatment decisions, which may lead to information fragmentation and difficulty in forming a comprehensive and unified diagnostic conclusion. In addition, in actual operations, the high cost and complexity of molecular testing limit its widespread application, especially in hospitals or laboratories with limited resources.

[0015] To solve the above problems, this embodiment provides an endometrial joint classification method, which predicts the endometrial histopathological category and molecular typing category by combining the histopathological features and molecular typing features of the pathological section image, so as to integrate the histopathological features and molecular typing features into a framework for predicting the diagnostic results of the endometrium, realize the effective combination of histopathology and molecular typing, and improve the accuracy of the diagnostic results. In addition, the use of pathological section images for molecular typing detection can effectively reduce the implementation cost of molecular detection, shorten the detection cycle of molecular detection, and improve the diagnostic efficiency, thereby improving the applicability of the diagnostic method.

[0016] The endometrium joint classification method provided in this embodiment can be performed by an endometrium joint classification device, which can be implemented by software and / or hardware. The endometrium joint classification device can be composed of two or more physical entities, or can be composed of one physical entity. For example, the endometrium joint classification device can be a computer device with strong processing capabilities such as a computer and a server. Among them, the server can be implemented by an independent server or a server cluster composed of multiple servers.

[0017] The endometrium joint classification device is installed with at least one type of operating system, and the endometrium joint classification device can install at least one application based on the operating system, and the application can be an application provided by the operating system, or an application downloaded from a third-party device or server. In this embodiment, the endometrium joint classification device is installed with at least an application that can execute the endometrium joint classification method.

[0018] For ease of understanding, this embodiment is described by taking a computer device as an example of a subject that executes the endometrial joint classification method.

[0019] Figure 1 A flowchart of a combined endometrial classification method provided in an embodiment of the present application is given. Figure 1 The combined endometrial classification method specifically includes: S110, dividing the pathological section image of the endometrium into a plurality of sub-images, extracting a first feature of each sub-image by a histopathological feature extractor, and extracting a second feature of each sub-image by a molecular typing feature extractor.

[0020] The pathological slice image is a digital pathological slice of the endometrium (WSI, Whole Slide Imaging). Due to the large size of the pathological slice image, it is difficult for the histopathological feature extractor and the molecular typing feature extractor to directly extract the histopathological features and molecular typing features of the pathological slice image. Therefore, the pathological slice image can be divided into multiple sub-images of uniform size, so that the features of the sub-images can be extracted respectively by the histopathological feature extractor and the molecular typing feature extractor, and then the features of the sub-images are fused into the histopathological features and molecular typing features of the pathological slice image.

[0021] Exemplarily, the pathological slice image can be slid and cropped by a sliding window of a preset size, and the image obtained by each sliding window cropping is used as a sub-image. When the sliding window traverses the entire pathological slice image, the pathological slice image can be obtained The multiple sub-images corresponding to the pathological slice image are obtained. Among them, the size and step size of the sliding window can be set according to actual conditions. This embodiment can quickly divide the pathological slice image into multiple sub-images of equal size in the form of a sliding window, effectively improving the efficiency of obtaining sub-images.

[0022] Optionally, a valid area can be determined in the pathological slice image according to the pixel value of each pixel point in the pathological slice image, and the valid area is a non-blank area with information. In the valid area, a sliding cropping is performed through a window of a preset size, and the image cropped by the window each time is used as a sub-image. When the entire valid area is traversed, the multiple sub-images corresponding to the pathological slice image can be obtained.

[0023] In this embodiment, the histopathology feature extractor is a pre-trained neural network model for extracting relevant features of the histopathology category in the image, and the molecular typing feature extractor is a pre-trained neural network model for extracting relevant features of the molecular typing category in the image. Optionally, the histopathology feature extractor and the molecular typing feature extractor can be convolutional neural networks or Transformers networks. The first feature is the feature related to the histopathology category extracted by the histopathology feature extractor in the sub-image, and the second feature is the feature related to the molecular typing category extracted by the molecular typing feature extractor in the sub-image.

[0024] Exemplarily, in order to train a histopathological feature extractor, sample pathological section images of the endometrium may be collected in advance, the sample pathological section images may be divided into a corresponding plurality of sample sub-images, the plurality of sample sub-images may be input into a pre-constructed histopathological feature extractor to obtain a first sample feature extracted by the histopathological feature extractor in the sample sub-image, the plurality of first sample features may be fused to generate the histopathological feature of the sample pathological section image, the histopathological feature of the sample pathological section image may be input into a first classification network to obtain a classification result output by the first classification network, the loss value between the classification result and the histopathological category annotated by the sample pathological section image may be calculated using a loss function, and the network parameters of the first classification network and the histopathological feature extractor may be updated by back propagation according to the loss value.

[0025] Similarly, to train a molecular typing feature extractor, multiple sample sub-images can be input into a pre-built molecular typing feature extractor to obtain a second sample feature extracted by the molecular typing feature extractor in the sample sub-image, and multiple second sample features can be fused to generate a molecular typing feature of the sample pathological section image, and the molecular typing feature of the sample pathological section image can be input into a second classification network to obtain a classification result output by the second classification network. The loss value between the classification result and the molecular typing category annotated by the sample pathological section image is calculated through a loss function, and the network parameters of the second classification network and the molecular typing feature extractor are updated by backpropagation according to the loss value.

[0026] It should be noted that the first classification network and the second classification network are different models of the same network structure, and the first classification network and the second classification network can use a multi-layer perceptron. The first classification network and the second classification network are only used to train the histopathology feature extractor and the molecular typing feature extractor, and in the actual application process, other trained classification networks will be used to predict the molecular typing category and histopathology category of the pathology section image.

[0027] S120, determining a histopathological feature of the pathological section image according to the first feature of each sub-image, and determining a molecular typing feature of the pathological section image according to the second feature of each sub-image.

[0028] Exemplarily, based on the pixel coordinates of each sub-image in the pathological slice image, the first features of each sub-image are fused to obtain the histopathological features of the pathological slice image. Based on the pixel coordinates of each sub-image in the pathological slice image, the second features of each sub-image are fused to obtain the molecular typing features of the pathological slice image.

[0029] Optionally, the feature weight of the first feature or the second feature can be calculated by a pre-trained weight calculation model, so that the first feature or the second feature can be weighted by the feature weight to obtain a histopathological feature or a molecular typing feature, so that the subsequent classification network can focus on the features containing key information, thereby improving the performance and efficiency of the classification network.

[0030] Figure 2 : is a flow chart of generating histopathological features of pathological slice images provided by an embodiment of the present application. Figure 2 As shown, the step of generating the histopathological features of the pathological slice image specifically includes S1201-S1203: S1201. Determine a first weight of a first feature in each histopathology category using a first weight calculation model.

[0031] The histopathological categories include at least two of endometrial polyps, normal endometrium, hyperplastic endometrium without atypical hyperplasia, irregular hyperplastic reactive endometrium, atypical endometrial hyperplasia, endometrioid adenocarcinoma, and other cancers. The first weight calculation model is a neural network model for calculating the first weight of the first feature in each histopathological category.

[0032] Assuming Xj is the first feature of the jth sub-image, the calculation formula of the first weight of Xj is as follows:

[0033] in, is the first weight of the first feature in category c. are the network parameters of the first weight calculation model, M is an adjustable hyperparameter, L is the intermediate feature length, C is the number of histopathology categories, is the normalized calculation formula, so after After calculation , that is, the accumulation of the first weights of the first feature in each histopathology category is equal to one.

[0034] The first weight calculation model can determine which histopathology category's feature information is focused on in the first feature based on the feature information about each histopathology category in the first feature, thereby assigning a high weight corresponding to the histopathology category that is focused on to the first feature, so that when the first feature is subsequently fused to generate a histopathology feature, the histopathology feature can include the key features of each histopathology category, so that the classification network can pay close attention to the key features of each histopathology category, thereby accurately predicting the histopathology category of the endometrium and improving the accuracy of the prediction results.

[0035] Since the first weight calculation model also belongs to a neural network model, it has the function of accurately determining the first weight of the first feature only after training. Therefore, the histopathological feature extractor, the first weight calculation model and the first classification network can be used as a network model for unified training. After the training is completed, the histopathological feature extraction and the first weight calculation model are used to extract the histopathological features of the pathological section image. For example, sample pathological section images of the endometrium are collected in advance, the sample pathological section images are divided into corresponding multiple sample sub-images, and the multiple sample sub-images are input into the pre-built histopathological feature extractor to obtain the first sample feature extracted by the histopathological feature extractor in the sample sub-image. The first sample weight of each first sample feature is determined by the first weight calculation model, and the first sample feature is weighted and summed based on the first sample weight to obtain the histopathological feature of the sample pathological section image. The histopathological feature of the sample pathological section image is input into the first classification network to obtain the classification result output by the first classification network. The loss value between the classification result and the histopathological category annotated by the sample pathological section image is calculated by the loss function, and the network parameters of the first classification network, the first weight calculation model and the histopathological feature extractor are updated by back propagation according to the loss value. Among them, the loss function can be FocalLoss or CrossEntropyLoss, etc.

[0036] S1202 , performing weighted summation on each first feature based on the first weight of the same histopathological category to obtain a third feature of the corresponding histopathological category.

[0037] Exemplarily, each first feature is multiplied by the first weight in the category of endometrial polyps, and the products corresponding to each first feature are accumulated to obtain the third feature of the category of endometrial polyps. The calculation process of the third feature of other histopathological categories is similar. In order to more intuitively understand the calculation process of the third feature, the following calculation formula of the third feature can be referred to:

[0038] in, is the third feature of the pathological slice image in the histopathology category of category c, is the first feature, i.e., the total number of sub-images.

[0039] S1203: Determine the third feature of each histopathology category as the histopathology feature of the pathology slice image.

[0040] Exemplarily, the third feature of each histopathological category is used as the histopathological feature of the pathological section image, that is, when the number of histopathological categories is equal to C, C histopathological features of the pathological section image can be obtained, and each histopathological feature can correspond to the key feature that characterizes the pathological section image in the corresponding histopathological category.

[0041] This embodiment determines the first weights of each first feature in each tissue pathology category through a first weight calculation model, and then performs weighted sum processing on each first feature according to the first weights of each tissue pathology category to obtain the key features of the pathological section image in each tissue pathology category, so that the subsequent classification network can focus on the key features of the pathological section image in each tissue pathology category, thereby improving the prediction accuracy of the classification network.

[0042] Figure 3 : is a flow chart of generating molecular typing features of pathological slice images provided in an embodiment of the present application. Figure 3 As shown, the step of generating molecular typing features of the pathological section image specifically includes S1204-S1206: S1204. Determine the second weight of the second feature in each molecular typing category through a second weight calculation model.

[0043] The molecular typing categories include at least two of POLE hypermutation type, MMRd type, NSMP type, and p53 abnormal type. The second weight calculation model is a neural network model for calculating the second weight of the second feature in each molecular typing category.

[0044] Assuming Yi is the second feature of the i-th sub-image, the calculation formula of the second weight of Yj is as follows:

[0045] in, is the first weight of the first feature in category d. are the network parameters of the second weight calculation model, M is an adjustable hyperparameter, L is the intermediate feature length, is the number of molecular typing categories, is the normalized calculation formula, so after After calculation , that is, the accumulation of the second weights of the second feature in each molecular typing category is equal to one.

[0046] The second weight calculation model can determine which molecular typing category's feature information the second feature focuses on based on the feature information about each molecular typing category in the second feature, thereby assigning a high weight corresponding to the molecular typing category that is focused on to the second feature, so that when the second feature is subsequently fused to generate a molecular typing feature, the molecular typing feature can include the key features of each molecular typing category, so that the classification network can pay close attention to the key features of each molecular typing category, thereby accurately predicting the molecular typing category of the endometrium and improving the accuracy of the prediction results.

[0047] Since the second weight calculation model also belongs to a neural network model, it has the function of accurately determining the second weight of the second feature only after training. For this purpose, the molecular typing feature extractor, the second weight calculation model and the second classification network can be used as a network model for unified training. After the training is completed, the molecular typing feature extraction and the second weight calculation model are used to extract the molecular typing features of the pathological section image. For example, the sample pathological section image of the endometrium is collected in advance, the sample pathological section image is divided into a corresponding plurality of sample sub-images, and the plurality of sample sub-images are input into the pre-constructed molecular typing feature extractor to obtain the second sample feature extracted by the molecular typing feature extractor in the sample sub-image. The second sample weight of each second sample feature is determined by the second weight calculation model, the molecular typing feature of the sample pathological section image is obtained by weighted summing the second sample features based on the second sample weight, the molecular typing feature of the sample pathological section image is input into the second classification network to obtain the classification result output by the second classification network, the loss value between the classification result and the molecular typing category annotated by the sample pathological section image is calculated by the loss function, and the network parameters of the second classification network, the second weight calculation model and the molecular typing feature extractor are updated according to the loss value by back propagation.

[0048] S1205 . Perform weighted summation on each second feature based on the second weight of the same molecular typing category to obtain a fourth feature corresponding to the molecular typing category.

[0049] Exemplarily, each second feature is multiplied by the second weight of the MMRd type, and the products corresponding to each second feature are accumulated to obtain the fourth feature of the MMRd type. The calculation process of the fourth feature of other molecular typing categories is similar. In order to more intuitively understand the calculation process of the fourth feature, the following calculation formula of the fourth feature can be referred to:

[0050] in, is the fourth feature of the molecular classification category of the pathological section image in category d, is the second feature, namely the total number of sub-images.

[0051] S1206. Determine the fourth feature of each molecular typing category as the molecular typing feature of the pathological section image.

[0052] Exemplarily, the fourth feature of each molecular typing category is used as the molecular typing feature of the pathological section image, that is, when the number of molecular typing categories is equal to D, D molecular typing features of the pathological section image can be obtained, and each molecular typing feature can correspond to the key feature that characterizes the pathological section image in the corresponding molecular typing category.

[0053] This embodiment determines the second weight of each second feature in each molecular typing category through a second weight calculation model, and then performs weighted summation processing on each second feature according to the second weight of each molecular typing category to obtain the key features of the pathological section image in each molecular typing category, so that the subsequent classification network can focus on the key features of the pathological section image in each molecular typing category, thereby improving the prediction accuracy of the classification network.

[0054] S130. Predict the molecular typing category of the endometrium by combining multiple first features and molecular typing features, and predict the histopathological category of the endometrium by combining multiple second features and histopathological features.

[0055] Exemplarily, multiple first features and molecular typing features may be input into a pre-trained first joint classification network, and the molecular typing category of the endometrium may be determined by the first joint classification network. The first joint classification network may be a multi-layer perceptron or a multi-layer perceptron with a multi-head attention mechanism.

[0056] Optionally, not every first feature among the multiple first features is useful for predicting the molecular typing category, that is, some first features not only cannot improve the prediction accuracy of the molecular typing category, but even interfere with the effective analysis of the molecular typing features, affecting the accuracy of the prediction results. In this regard, first features related to the molecular typing category can be obtained from the multiple first features, and the molecular typing category of the endometrium can be predicted based on these first features and the molecular typing features, thereby improving the accuracy of the prediction results.

[0057] For example, some studies have shown that mutated molecules such as POLE hypermutation, MMRd, NSMP, and p53 abnormality can be detected in the slice images of endometrial atypical hyperplasia, endometrial adenocarcinoma, and other cancers. Therefore, the first feature of endometrial atypical hyperplasia, endometrial adenocarcinoma, and other cancers can be used to assist in the classification of molecular typing. Prior knowledge can be introduced on the basis of deep learning, so that the first joint classification network can learn the characteristic information of molecular typing from more aspects, thereby improving the classification accuracy of molecular typing.

[0058] For example, Figure 4 : is a flow chart of the combined prediction of molecular typing categories provided in the embodiments of the present application. Figure 4 As shown, the steps of jointly predicting molecular typing categories specifically include S1301-S1303: S1301. Obtain a first target weight of a first category from first weights corresponding to various histopathological categories, where the first category includes at least one of atypical endometrial hyperplasia, endometrioid adenocarcinoma, and other cancers.

[0059] Exemplarily, the first weight of each first feature in the category of endometrial atypical hyperplasia is obtained as the first target weight of the category of endometrial atypical hyperplasia. The first weight of each first feature in the category of endometrial adenocarcinoma is obtained as the first target weight of the category of endometrial adenocarcinoma. The first weight of the category of other cancers is obtained as the first target weight of the category of other cancers.

[0060] S1302: Sort the first features in descending order based on the first target weights of the first category, and obtain a plurality of first features with the highest sorting order as first target features.

[0061] Exemplarily, the first features of the category of atypical endometrial hyperplasia are sorted in descending order according to the first target weight, and the top-ranked first features are obtained as the first target features of the category of atypical endometrial hyperplasia. The first features of the category of endometrioid adenocarcinoma are sorted in descending order according to the first target weight, and the top-ranked first features are obtained as the first target features of the category of endometrioid adenocarcinoma. The first features of the category of other cancers are sorted in descending order according to the first target weight, and the top-ranked first features are obtained as the first target features of the category of other cancers.

[0062] It can be understood that the higher the first weight of the first feature for a certain histopathology category, the more feature information corresponding to the histopathology category the first feature contains, that is, the higher the probability that the sub-image corresponding to the first feature belongs to the histopathology category. The lower the first weight of the first feature for a certain histopathology category, the less feature information corresponding to the histopathology category the first feature contains, that is, the lower the probability that the sub-image corresponding to the first feature belongs to the histopathology category. Therefore, after sorting the first features from large to small based on the first target weights of the first category, the first features with a lower sorting have less feature information about the first category, and the first features with a higher sorting have more feature information about the first category. By obtaining multiple first features with a higher sorting, the first features carrying molecular typing characteristics can be obtained.

[0063] S1303. Predict the molecular typing category of the endometrium by combining multiple first target features and molecular typing features.

[0064] Exemplarily, multiple first target features and molecular typing features of each first category may be combined and input into a pre-trained first joint classification network, and the molecular typing category of the endometrium may be output through the first joint classification network.

[0065] Optionally, only some of the multiple first target features and molecular typing features may contain key information of the molecular typing category. In order to improve the prediction accuracy of the subsequent first joint classification network, a third target feature containing key information of the molecular typing category may be screened out from the multiple first target features and molecular typing features, so that the molecular typing category can be predicted based on the third target feature through the first joint classification network. Exemplarily, the steps of obtaining the third target feature and predicting the molecular typing category specifically include S13031-S13033: S13031. Determine third weights of the plurality of first target features and molecular typing features in each molecular typing category through a third weight calculation model.

[0066] Among them, the third weight calculation model is a neural network model for calculating the third weight of the first target feature and the molecular typing feature in each molecular typing category. The network structure of the third weight calculation model is roughly the same as the network structure of the second weight calculation model, but the network parameters are different. The process of calculating the third weight by the third weight calculation model is similar to the process of calculating the second weight by the second weight calculation model. The calculation process of the third weight can refer to the process of calculating the second weight by the second weight calculation model in step S1204, which will not be repeated here.

[0067] It should be noted that the third weight calculation model can be trained together with the first joint classification network, that is, the third weight calculation model and the first joint classification network are trained uniformly as a network model. However, since the third weight calculation model and the first joint classification network process the features output by the molecular typing feature extractor, the histopathology feature extractor, the first weight calculation model and the second weight calculation model, after the molecular typing feature extractor, the histopathology feature extractor, the first weight calculation model and the second weight calculation model are trained, the third weight calculation model and the first joint classification network are trained based on the features of the sample slice images extracted by the molecular typing feature extractor, the histopathology feature extractor, the first weight calculation model and the second weight calculation model.

[0068] The specific training process of the third weight calculation model and the first joint classification network is as follows: the molecular typing features of the sample slice image are extracted by the trained molecular typing feature extractor and the second weight calculation model, the first sample features of the sample sub-image in each histopathology category are extracted by the histopathology feature extractor, the first sample weights of each first sample feature in each histopathology category are calculated by the first weight calculation model, and a plurality of first sample features with higher first sample weights are selected from the first sample features corresponding to the first category as the first target sample features. The third sample weights of the first target sample features and the molecular typing features in each molecular typing category are calculated by the third weight calculation model, a plurality of features with higher third sample weights are selected from the first target sample features and the molecular typing features and input into the first joint classification network to obtain the classification result output by the first joint classification network, the loss value between the classification result and the molecular typing category annotated by the sample pathology slice image is calculated by the loss function, and the network parameters of the first joint classification network and the third weight calculation model are updated according to the back propagation of the loss value.

[0069] S13032. Sort the plurality of first target features and molecular typing features in descending order based on the third weight of the same molecular typing category, and obtain the plurality of features with the highest sorting as the third target features.

[0070] Exemplarily, the plurality of first target features and molecular typing features are sorted in descending order based on their third weights in the MMRd type, and the first target features and / or molecular typing features with the highest sorting are used as the third target features of the MMRd type. The acquisition process of the third target features of other molecular typing categories is similar.

[0071] It can be understood that the higher the third weight of the first target feature or molecular typing feature in a certain molecular typing category, the more feature information corresponding to the molecular typing category is contained in the first target feature or molecular typing feature. Therefore, after sorting the first target features and molecular typing features in descending order based on the third weights of the molecular typing categories, the features with lower rankings have less feature information about the molecular typing category, and the features with higher rankings have more feature information about the molecular typing category. The multiple features with higher rankings are obtained as the third target features input into the first joint classification network, so that the first joint classification network can focus on the feature information of the molecular typing category in the third target feature, thereby accurately predicting the molecular typing category of the endometrium.

[0072] S13033. Combine the multiple third target features and input them into the first joint classification network to obtain the molecular typing category output by the first joint classification network.

[0073] Exemplarily, multiple third target features are spliced ​​and input into a first joint classification network, and the molecular classification category of the endometrium is predicted based on the multiple third target features by the first joint classification network.

[0074] Optional, Figure 5 Schematic diagram of the network structure of the first joint classification network provided in the embodiment of the present application. Figure 5 As shown, the first joint classification network includes a first normalization model, a first multi-head attention model, a second normalization model, a first multi-layer perceptron layer and a first multi-layer perceptron. Accordingly, the step of predicting the molecular typing category through the first joint classification network specifically includes S130331-S130334: S130331. Concatenate multiple third target features with the first category label to obtain a first feature sequence, and normalize the first feature sequence.

[0075] The first category marker is a CLS token, which is used to represent the global information of the entire feature sequence. Multiple third target features are concatenated with the first category marker to form a first feature sequence containing all input information, and the first feature sequence is normalized by a first normalization model, which helps to improve training stability and accelerate convergence. The first normalization model can be Layer Normalization.

[0076] S130332. Input the normalized first feature sequence into the first multi-head attention model to obtain a second feature sequence output by the first multi-head attention model.

[0077] Exemplarily, the normalized first feature sequence is input into a first multi-head attention model, and the first multi-head attention model calculates the correlation between each feature in the first feature sequence and captures global context information to output a second feature sequence.

[0078] S130333. Normalize the second feature sequence, input the normalized second feature sequence into the first multi-layer perceptron layer, and obtain a third feature sequence output by the first multi-layer perceptron layer.

[0079] Exemplarily, the second feature sequence is normalized by a second normalization model to ensure the stability of feature distribution. The second normalization model may be Layer Normalization.

[0080] Afterwards, the second feature sequence is input into the first multi-layer perceptron layer, and higher-level features are extracted through the first multi-layer perceptron layer to output a third feature sequence.

[0081] S130334. Input the third feature sequence into the first multi-layer perceptron to obtain the molecular typing category output by the first multi-layer perceptron.

[0082] Exemplarily, the third feature sequence is input into a first multi-layer perceptron, and the first multi-layer perceptron predicts the molecular typing category of the endometrium based on the third feature sequence.

[0083] This embodiment adopts a multi-layer perceptron with a multi-head attention mechanism as the first joint classification network, and uses the multi-head attention mechanism to effectively capture the correlation between the molecular typing features and the first target features, so that the first multi-layer perceptron can accurately predict the molecular typing category of the endometrium based on the molecular typing features, the first target features and the correlation between the two.

[0084] Furthermore, when the histopathological category of the endometrium is predicted by combining multiple second features and histopathological features, the multiple second features and histopathological features can be input into a pre-trained second joint classification network, and the histopathological category of the endometrium is determined by the second joint classification network. The second joint classification network can be a multi-layer perceptron or a multi-layer perceptron with a multi-head attention mechanism.

[0085] Similarly, not every second feature among the multiple second features is useful for predicting the histopathological category, that is, some second features not only cannot improve the prediction accuracy of the histopathological category, but even interfere with the effective analysis of the histopathological features, affecting the accuracy of the prediction results. In this regard, second features related to the histopathological category can be obtained from the multiple second features, and the histopathological category of the endometrium can be predicted based on these second features and the histopathological features, thereby improving the accuracy of the prediction results.

[0086] For example, some studies have shown that MMRd-type molecules are helpful in diagnosing endometrial carcinoma. Therefore, the second feature of the MMRd type can be used to assist in the classification of histopathology. By introducing prior knowledge based on deep learning, the second joint classification network can learn the characteristic information of histopathology from more aspects, thereby improving the accuracy of histopathology classification.

[0087] For example, Figure 6 : is a flowchart of the joint prediction of histopathology categories provided in the embodiment of the present application. Figure 6 As shown, the steps of jointly predicting the histopathological category specifically include S1304-S1306: S1304. Obtain a second target weight of a second category from the second weights corresponding to each molecular typing category, where the second category includes the MMRd type.

[0088] Exemplarily, the second weight of each second feature in the MMRd type is obtained as the second target weight of the MMRd type.

[0089] S1305. Sort the second features in descending order based on the second target weights of the second category, and obtain a plurality of second features with the highest sorting order as second target features.

[0090] Exemplarily, the second features of the MMRd type are sorted in descending order of the second target weights, and a plurality of second features with the highest sorting are obtained as the second target features of the MMRd type.

[0091] It can be understood that the higher the first weight of the second feature for a certain molecular typing category, the more feature information corresponding to the molecular typing category the second feature contains, that is, the higher the probability that the sub-image corresponding to the second feature belongs to the molecular typing category. The lower the first weight of the second feature for a certain molecular typing category, the less feature information corresponding to the molecular typing category the second feature contains, that is, the lower the probability that the sub-image corresponding to the second feature belongs to the molecular typing category. Therefore, after sorting the second features in descending order based on the second target weights of the second category, the second features with a lower ranking have less feature information about the second category, and the second features with a higher ranking have more feature information about the second category. By obtaining multiple second features with a higher ranking, the second features carrying histopathological characteristics can be obtained.

[0092] S1306. Predict the histopathological category of the endometrium by combining multiple second target features and histopathological features.

[0093] Exemplarily, a plurality of second target features and histopathological features of the second category may be combined and input into a pre-trained second joint classification network, and the histopathological category of the endometrium may be outputted through the second joint classification network.

[0094] Optionally, only some of the multiple second target features and histopathological features may contain key information of the histopathological category. In order to improve the prediction accuracy of the subsequent second joint classification network, a fourth target feature containing key information of the histopathological category may be screened out from the multiple second target features and histopathological features, so that the histopathological category can be predicted based on the fourth target feature through the second joint classification network. Exemplarily, the steps of obtaining the fourth target feature and predicting the histopathological category specifically include S13061-S13063: S13061. Determine fourth weights of the plurality of second target features and histopathological features in each histopathological category through a fourth weight calculation model.

[0095] Among them, the fourth weight calculation model is a neural network model for calculating the fourth weight of the second target feature and the histopathological feature in each histopathological category. The network structure of the fourth weight calculation model is roughly the same as the network structure of the first weight calculation model, but the network parameters are different. The process of calculating the fourth weight by the fourth weight calculation model is similar to the process of calculating the first weight by the first weight calculation model. The calculation process of the fourth weight can refer to the process of calculating the first weight by the first weight calculation model in step S1204, which will not be repeated here.

[0096] It should be noted that the fourth weight calculation model can be trained together with the second joint classification network, that is, the fourth weight calculation model and the second joint classification network are trained uniformly as a network model. However, since the fourth weight calculation model and the second joint classification network process the features output by the molecular typing feature extractor, the histopathology feature extractor, the first weight calculation model and the second weight calculation model, after the molecular typing feature extractor, the histopathology feature extractor, the first weight calculation model and the second weight calculation model are trained, the fourth weight calculation model and the second joint classification network are trained based on the features of the sample slice images extracted by the molecular typing feature extractor, the histopathology feature extractor, the first weight calculation model and the second weight calculation model.

[0097] The specific training process of the fourth weight calculation model and the second joint classification network is as follows: extract the histopathological features of the sample slice image through the trained histopathological feature extractor and the first weight calculation model, extract the first sample features of the sample sub-image in each molecular typing category through the molecular typing feature extractor, calculate the second sample weights of each second sample feature in each molecular typing category through the second weight calculation model, and select multiple second sample features with higher second sample weights from the second sample features corresponding to the second category as the second target sample features. Calculate the fourth sample weights of the second target sample features and the histopathological features in each histopathological category through the fourth weight calculation model, select multiple features with higher fourth sample weights from the second target sample features and the histopathological features and input them into the second joint classification network to obtain the classification result output by the second joint classification network, calculate the loss value between the classification result and the histopathological category annotated by the sample pathological slice image through the loss function, and update the network parameters of the second joint classification network and the fourth weight calculation model based on the back propagation of the loss value.

[0098] S13062. Sort the plurality of second target features and the histopathological features in descending order based on the fourth weight of the same histopathological category, and obtain the plurality of features with the highest sorting as the fourth target features.

[0099] Exemplarily, the plurality of second target features and histopathological features are ranked in descending order based on their fourth weights in the category of endometrial polyps, and the second target features and / or histopathological features ranked higher are used as the fourth target features in the category of endometrial polyps. The process of obtaining the fourth target features of other histopathological categories is similar.

[0100] It can be understood that the higher the fourth weight of the second target feature or histopathological feature in a certain histopathological category, the more feature information corresponding to the histopathological category the second target feature or histopathological feature contains. Therefore, after sorting the second target features and histopathological features in descending order based on the fourth weights of the histopathological categories, the features with lower rankings have less feature information about the histopathological category, and the features with higher rankings have more feature information about the histopathological category. Multiple features with higher rankings are obtained as the fourth target features of the second joint classification network, so that the second joint classification network can focus on the feature information of the histopathological category in the fourth target feature, thereby accurately predicting the histopathological category of the endometrium.

[0101] S13063. Combine the multiple fourth target features and input them into the second joint classification network to obtain the tissue pathology category output by the second joint classification network.

[0102] Exemplarily, multiple fourth target features are concatenated and input into a second joint classification network, and the second joint classification network predicts the histopathological category of the endometrium based on the multiple fourth target features.

[0103] Optional, Figure 7 Schematic diagram of the network structure of the second joint classification network provided in the embodiment of the present application. Figure 7 As shown, the second joint classification network includes a third normalization model, a second multi-head attention model, a fourth normalization model, a second multi-layer perceptron layer, and a second multi-layer perceptron. Accordingly, the step of predicting the histopathology category through the second joint classification network specifically includes S130631-S130634: S130631. Concatenate multiple fourth target features with the second category label to obtain a fourth feature sequence, and normalize the fourth feature sequence.

[0104] The second category marker is CLS token, which is used to represent the global information of the entire feature sequence. Multiple fourth target features are concatenated with the second category marker to form a fourth feature sequence containing all input information, and the fourth feature sequence is normalized by the third normalization model, which helps to improve training stability and accelerate convergence. The third normalization model can be Layer Normalization.

[0105] S130632. Input the normalized fourth feature sequence into the second multi-head attention model to obtain the fifth feature sequence output by the second multi-head attention model.

[0106] Exemplarily, the normalized fourth feature sequence is input into the second multi-head attention model, and the second multi-head attention model calculates the correlation between each feature in the fourth feature sequence and captures the global context information to output the fifth feature sequence.

[0107] S130633. Normalize the fifth feature sequence, input the normalized fifth feature sequence into the second multi-layer perceptron layer, and obtain a sixth feature sequence output by the second multi-layer perceptron layer.

[0108] Exemplarily, the fifth feature sequence is normalized by a fourth normalization model to ensure the stability of feature distribution. The fourth normalization model may be Layer Normalization.

[0109] Afterwards, the fifth feature sequence is input into the second multi-layer perceptron layer, and higher-level features are extracted through the second multi-layer perceptron layer to output a sixth feature sequence.

[0110] S130634. Input the sixth feature sequence into a second multi-layer perceptron to obtain a tissue pathology category output by the second multi-layer perceptron.

[0111] Exemplarily, the sixth feature sequence is input into a second multi-layer perceptron, and the second multi-layer perceptron predicts the histopathological category of the endometrium based on the sixth feature sequence.

[0112] This embodiment adopts a multi-layer perceptron with a multi-head attention mechanism as the second joint classification network, and uses the multi-head attention mechanism to effectively capture the correlation between the histopathological features and the second target features, so that the second multi-layer perceptron can accurately predict the histopathological category of the endometrium based on the histopathological features, the second target features and the correlation between the two.

[0113] In one embodiment, Figure 8 Schematic diagram of the classification prediction process of pathological slice images provided in the embodiment of the present application. Figure 8As shown, the pathological section image is divided into multiple sub-images by sliding window clipping, the first feature of each sub-image is extracted by the histopathological feature extractor, and the second feature of each sub-image is extracted by the molecular typing feature extractor. The first weight of the first feature is determined by the first weight calculation model, and the first weight and the first feature are weighted to obtain the histopathological feature. The second weight of the second feature is determined by the second weight calculation model, and the second weight and the second feature are weighted to obtain the molecular typing feature. The first target feature is screened out from the first feature according to the first weight, the third weight of the first target feature and the molecular typing feature is calculated by the third weight calculation model, the third target feature is screened out from the first target feature and the molecular typing feature according to the third weight, and the third target feature is input into the first joint classification network to obtain the molecular typing category. The second target feature is screened out from the second feature according to the second weight, the fourth weight of the second target feature and the histopathological feature is calculated by the fourth weight calculation model, the fourth target feature is screened out from the second target feature and the histopathological feature according to the fourth weight, and the fourth target feature is input into the second joint classification network to obtain the histopathological category.

[0114] In summary, the endometrium joint classification method provided in the embodiment of the present application is to divide the pathological section image of the endometrium into multiple sub-images, extract the first feature of each sub-image by a histopathological feature extractor, and extract the second feature of each sub-image by a molecular typing feature extractor; determine the histopathological feature of the pathological section image according to the first feature of each sub-image, and determine the molecular typing feature of the pathological section image according to the second feature of each sub-image; jointly predict the molecular typing category of the endometrium by multiple first features and molecular typing features, and jointly predict the histopathological category of the endometrium by multiple second features and histopathological features. Through the above-mentioned technical means, the histopathological features of the pathological section image can be generated by fusing the first features of each sub-image with respect to the histopathological category, and the molecular typing category of the pathological section image can be generated by fusing the second features of each sub-image with respect to the molecular typing category, thereby realizing accurate extraction of the histopathological features and molecular typing features of the pathological section image. The prediction of the endometrial histopathological category is predicted by fusing the histopathological features and the second feature related to the molecular typing category, and the prediction of the endometrial molecular typing category is predicted by fusing the molecular typing features and the first feature related to the histopathological category, so that the prediction model can not only capture the significant features of the image but also learn features in other directions, enriching the feature information obtained by the prediction model and effectively improving the prediction accuracy of the prediction model. In addition, the prediction model realizes the combination of molecular typing detection and histopathological classification, provides more comprehensive diagnostic information, solves the problem in the prior art that it is difficult to form a comprehensive and unified diagnostic conclusion between histopathological observation results and molecular detection results, and improves the diagnostic accuracy. The use of pathological section images for molecular typing detection can effectively reduce the implementation cost of molecular detection and improve applicability.

[0115] Based on the above embodiments, Fig. 9 This is a schematic diagram of the structure of an endometrial joint classification device provided in an embodiment of the present application. Fig. 9 The endometrium joint classification device provided in this embodiment specifically includes: a sub-image feature extraction module 21, a slice image feature extraction module 22 and a joint classification module 23.

[0116] The sub-image feature extraction module 21 is configured to divide the pathological section image of the endometrium into a plurality of sub-images, extract a first feature of each sub-image by a histopathological feature extractor, and extract a second feature of each sub-image by a molecular typing feature extractor; The slice image feature extraction module 22 is configured to determine the histopathological feature of the pathological slice image according to the first feature of each sub-image, and determine the molecular typing feature of the pathological slice image according to the second feature of each sub-image; The joint classification module 23 is configured to jointly predict the molecular typing category of the endometrium by using a plurality of first features and molecular typing features, and to jointly predict the histopathological category of the endometrium by using a plurality of second features and histopathological features.

[0117] On the basis of the above embodiment, the slice image feature extraction module 22 includes: a first weight calculation submodule, configured to determine the first weight of the first feature in each histopathological category through a first weight calculation model, and the histopathological categories include endometrial polyps, normal endometrium, hyperplastic endometrium, at least two of endometrial polyps, normal endometrium, hyperplastic endometrium without atypical hyperplasia, irregular hyperplasia reaction endometrium, atypical endometrial hyperplasia, endometrioid adenocarcinoma and other cancers; a first weighted summation submodule, configured to perform weighted summation on each first feature based on the first weight of the same histopathological category to obtain a third feature of the corresponding histopathological category; a pathological feature determination submodule, configured to determine the third feature of each histopathological category as the histopathological feature of the pathological slice image.

[0118] On the basis of the above-mentioned embodiment, the joint classification module 23 includes: a first target weight acquisition submodule, configured to acquire the first target weight of the first category from the first weights corresponding to each histopathological category, the first category including at least one of endometrial atypical hyperplasia, endometrioid adenocarcinoma and other cancers; a first target feature acquisition submodule, configured to sort the first features in descending order based on the first target weight of the first category, and acquire multiple first features with the highest sorting as the first target features; a first category prediction submodule, configured to jointly predict the molecular typing category of the endometrium through multiple first target features and molecular typing features.

[0119] On the basis of the above embodiment, the first category prediction submodule includes: a third weight calculation unit, configured to determine the third weights of multiple first target features and molecular typing features in each molecular typing category through a third weight calculation model, and the molecular typing categories include at least two of POLE hypermutation type, MMRd type, NSMP type, and p53 abnormal type; a third target feature acquisition unit, configured to sort the multiple first target features and molecular typing features in descending order based on the third weights of the same molecular typing category, and obtain multiple features with the highest sorting as the third target features; the first category prediction unit is configured to merge the multiple third target features into the first joint classification network to obtain the molecular typing category output by the first joint classification network; In the method, the first category prediction unit also includes: a first normalization subunit, configured to concatenate multiple third target features with the first category label to obtain a first feature sequence, and normalize the first feature sequence; a first attention subunit, configured to input the normalized first feature sequence into the first multi-head attention model to obtain a second feature sequence output by the first multi-head attention model; a second normalization subunit, configured to normalize the second feature sequence, input the normalized second feature sequence into the first multi-layer perceptron layer, and obtain a third feature sequence output by the first multi-layer perceptron layer; the first category prediction subunit, configured to input the third feature sequence into the first multi-layer perceptron, and obtain the molecular typing category output by the first multi-layer perceptron.

[0120] On the basis of the above embodiment, the slice image feature extraction module 22 includes: a second weight calculation submodule, configured to determine the second weight of the second feature in each molecular typing category through a second weight calculation model; a second weighted summation submodule, configured to perform weighted summation on each second feature based on the second weight of the same molecular typing category to obtain a fourth feature of the corresponding molecular typing category; a molecular typing feature determination submodule, configured to determine the fourth feature of each molecular typing category as the molecular typing feature of the pathological slice image.

[0121] On the basis of the above embodiment, the joint classification module 23 includes: a second target weight acquisition submodule, configured as a first preprocessing subunit, configured to obtain the second target weight of the second category from the second weights corresponding to each molecular typing category, the second category including the MMRd type; a second target feature acquisition submodule, configured to sort each second feature in descending order based on the second target weight of the second category, and obtain multiple second features with the top sorting as second target features; a second category prediction submodule, configured to jointly predict the histopathological category of the endometrium through multiple second target features and histopathological features.

[0122] On the basis of the above embodiment, the second category prediction submodule includes: a fourth weight calculation unit, configured to determine the fourth weights of multiple second target features and histopathological features in each histopathological category through a fourth weight calculation model; a fourth target feature acquisition unit, configured to sort the multiple second target features and histopathological features in descending order based on the fourth weights of the same histopathological category, and obtain multiple features with the highest sorting as the fourth target features; the second category prediction unit, configured to merge the multiple fourth target features into the second joint classification network to obtain the histopathological category output by the second joint classification network; wherein the second category prediction unit also includes: a third normalization The subunit is configured to concatenate multiple fourth target features with the second category label to obtain a fourth feature sequence, and normalize the fourth feature sequence; the second attention subunit is configured to input the normalized fourth feature sequence into the second multi-head attention model to obtain a fifth feature sequence output by the second multi-head attention model; the fourth normalization subunit is configured to normalize the fifth feature sequence, input the normalized fifth feature sequence into the second multi-layer perceptron layer, and obtain a sixth feature sequence output by the second multi-layer perceptron layer; the second category prediction subunit is configured to input the sixth feature sequence into the second multi-layer perceptron to obtain the tissue pathology category output by the second multi-layer perceptron.

[0123] As mentioned above, the endometrium joint classification device provided in the embodiment of the present application divides the pathological section image of the endometrium into multiple sub-images, extracts the first feature of each sub-image by a histopathological feature extractor, and extracts the second feature of each sub-image by a molecular typing feature extractor; determines the histopathological feature of the pathological section image according to the first feature of each sub-image, and determines the molecular typing feature of the pathological section image according to the second feature of each sub-image; predicts the molecular typing category of the endometrium by combining multiple first features and molecular typing features, and predicts the histopathological category of the endometrium by combining multiple second features and histopathological features. Through the above technical means, the histopathological features of the pathological section image can be generated by fusing the first features of each sub-image with respect to the histopathological category, and the molecular typing category of the pathological section image can be generated by fusing the second features of each sub-image with respect to the molecular typing category, thereby realizing accurate extraction of the histopathological features and molecular typing features of the pathological section image. The prediction of the endometrial histopathological category is predicted by fusing the histopathological features and the second feature related to the molecular typing category, and the prediction of the endometrial molecular typing category is predicted by fusing the molecular typing features and the first feature related to the histopathological category, so that the prediction model can not only capture the significant features of the image but also learn features in other directions, enriching the feature information obtained by the prediction model and effectively improving the prediction accuracy of the prediction model. In addition, the prediction model realizes the combination of molecular typing detection and histopathological classification, provides more comprehensive diagnostic information, solves the problem in the prior art that it is difficult to form a comprehensive and unified diagnostic conclusion between histopathological observation results and molecular detection results, and improves the diagnostic accuracy. The use of pathological section images for molecular typing detection can effectively reduce the implementation cost of molecular detection and improve applicability.

[0124] The endometrial combined classification device provided in the embodiment of the present application can be used to execute the endometrial combined classification method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0125] Fig.10 is a schematic diagram of the structure of an endometrial joint classification device provided in an embodiment of the present application, with reference to Fig.10 The endometrial joint classification device includes: a processor 31, a memory 32, a communication device 33, an input device 34, and an output device 35. The number of processors 31 in the endometrial joint classification device can be one or more, and the number of memories 32 in the endometrial joint classification device can be one or more. The processor 31, the memory 32, the communication device 33, the input device 34, and the output device 35 of the endometrial joint classification device can be connected via a bus or other methods.

[0126] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the endometrial joint classification method of any embodiment of the present application (for example, the sub-image feature extraction module 21, the slice image feature extraction module 22 and the joint classification module 23 in the endometrial joint classification device). The memory 32 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the device, etc. In addition, the memory 32 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0127] The communication device 33 is used for data transmission.

[0128] The processor 31 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 32, that is, realizes the above-mentioned endometrial joint classification method.

[0129] The input device 34 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 35 may include a display device such as a display screen.

[0130] The endometrial joint classification device provided above can be used to execute the endometrial joint classification method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0131] An embodiment of the present application also provides a storage medium containing computer executable instructions, which, when executed by a computer processor, are used to execute an endometrial joint classification method, the endometrial joint classification method comprising: obtaining a breast pathological section image of the detection object, and extracting pathological features of the breast pathological section image; obtaining a breast magnetic resonance imaging of the detection object, and extracting image features of the breast magnetic resonance imaging; and predicting the therapeutic effect of neoadjuvant chemotherapy for breast cancer based on the pathological features and image features.

[0132] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations (for example, in different computer systems connected by a network). The storage medium may store program instructions (for example, embodied as a computer program) that can be executed by one or more processors.

[0133] Of course, the storage medium containing computer executable instructions provided in the embodiment of the present application is not limited to the above endometrial combined classification method, and the computer executable instructions can also execute related operations in the endometrial combined classification method provided in any embodiment of the present application.

[0134] The endometrial combined classification device, endometrial combined classification system, storage medium and endometrial combined classification equipment provided in the above embodiments can execute the endometrial combined classification method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the endometrial combined classification method provided in any embodiment of the present application.

[0135] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the protection scope of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A combined endometrial classification method, characterized in that: include: Divide the pathological section image of the endometrium into a plurality of sub-images, extract a first feature of each of the sub-images by a histopathological feature extractor, and extract a second feature of each of the sub-images by a molecular typing feature extractor; Determine the histopathological feature of the pathological section image according to the first feature of each of the sub-images, and determine the molecular typing feature of the pathological section image according to the second feature of each of the sub-images; The molecular typing category of the endometrium is predicted by combining the first features with the molecular typing features, and the histopathological category of the endometrium is predicted by combining the second features with the histopathological features.

2. The endometrial combined classification method according to claim 1, characterized in that: Determining the histopathological feature of the pathological slice image according to the first feature of each of the sub-images includes: Determining, by a first weight calculation model, a first weight of the first feature in each histopathological category, the histopathological category comprising at least two of endometrial polyps, normal endometrium, hyperplastic endometrium without atypical hyperplasia, irregular hyperplastic reaction endometrium, atypical endometrial hyperplasia, endometrioid adenocarcinoma, and other cancers; Performing weighted summation on each of the first features based on a first weight of the same histopathology category to obtain a third feature corresponding to the histopathology category; The third feature of each histopathology category is determined as the histopathology feature of the pathology section image.

3. The combined endometrial classification method according to claim 2, characterized in that: The method of predicting the molecular typing category of the endometrium by combining the plurality of the first features and the molecular typing features comprises: Obtaining a first target weight of a first category from the first weights corresponding to the respective histopathological categories, the first category comprising at least one of atypical endometrial hyperplasia, endometrioid adenocarcinoma, and other cancers; Sort the first features in descending order based on the first target weights of the first categories, and obtain a plurality of first features with the highest sorting as first target features; The molecular typing category of the endometrium is predicted by combining a plurality of the first target features and the molecular typing features.

4. The combined endometrial classification method according to claim 3, characterized in that: The method of predicting the molecular typing category of the endometrium by combining the plurality of the first target features and the molecular typing features comprises: Determining third weights of the plurality of first target features and the molecular typing features in each molecular typing category by a third weight calculation model, wherein the molecular typing category includes at least two of POLE hypermutation type, MMRd type, NSMP type, and p53 abnormal type; sorting the first target features and the molecular typing features in descending order based on the third weight of the same molecular typing category, and obtaining the top ranked features as the third target features; The plurality of third target features are combined and input into the first joint classification network to obtain the molecular typing category output by the first joint classification network; the method includes: concatenating the plurality of third target features with the first category label to obtain a first feature sequence, and normalizing the first feature sequence; inputting the normalized first feature sequence into the first multi-head attention model to obtain a second feature sequence output by the first multi-head attention model; normalizing the second feature sequence, and inputting the normalized second feature sequence into the first multi-layer perceptron layer to obtain a third feature sequence output by the first multi-layer perceptron layer; and inputting the third feature sequence into the first multi-layer perceptron to obtain the molecular typing category output by the first multi-layer perceptron.

5. The combined endometrial classification method according to claim 1, characterized in that: Determining the molecular typing feature of the pathological section image according to the molecular typing feature of each sub-image includes: Determining the second weight of the second feature in each molecular typing category by a second weight calculation model; Based on the second weight of the same molecular typing category, weighted summation is performed on each of the second features to obtain a fourth feature of the corresponding molecular typing category; The fourth feature of each molecular typing category is determined as the molecular typing feature of the pathological section image.

6. The combined endometrial classification method according to claim 5, characterized in that: The method of predicting the histopathological category of the endometrium by combining a plurality of the second features and the histopathological features comprises: Obtaining a second target weight of a second category from the second weights corresponding to each molecular typing category, wherein the second category includes MMRd type; Sort the second features in descending order based on the second target weights of the second category, and obtain a plurality of second features with top rankings as second target features; The histopathological category of the endometrium is predicted by combining a plurality of the second target features and the histopathological features.

7. The combined endometrial classification method according to claim 6, characterized in that: The method of predicting the histopathological category of the endometrium by combining the plurality of the second target features and the histopathological features comprises: Determining fourth weights of the plurality of second target features and the histopathological features in each histopathological category by a fourth weight calculation model; sorting the plurality of second target features and the plurality of histopathological features in descending order based on the fourth weight of the same histopathological category, and acquiring a plurality of features with the highest sorting as the fourth target features; The plurality of fourth target features are combined and input into the second joint classification network to obtain the tissue pathology category output by the second joint classification network; the method includes: concatenating the plurality of fourth target features with the second category label to obtain a fourth feature sequence, and normalizing the fourth feature sequence; inputting the normalized fourth feature sequence into the second multi-head attention model to obtain a fifth feature sequence output by the second multi-head attention model; normalizing the fifth feature sequence, and inputting the normalized fifth feature sequence into the second multi-layer perceptron layer to obtain a sixth feature sequence output by the second multi-layer perceptron layer; and inputting the sixth feature sequence into the second multi-layer perceptron to obtain the tissue pathology category output by the second multi-layer perceptron.

8. An endometrial joint classification device, characterized in that: include: A sub-image feature extraction module is configured to divide the endometrial pathological section image into a plurality of sub-images, extract a first feature of each of the sub-images by a histopathological feature extractor, and extract a second feature of each of the sub-images by a molecular typing feature extractor; A slice image feature extraction module is configured to determine the histopathological feature of the pathological slice image according to the first feature of each of the sub-images, and to determine the molecular typing feature of the pathological slice image according to the second feature of each of the sub-images; The joint classification module is configured to jointly predict the molecular typing category of the endometrium by using multiple first features and the molecular typing features, and to jointly predict the histopathological category of the endometrium by using multiple second features and the histopathological features.

9. An endometrial joint classification device, characterized in that: include: one or more processors; A storage device stores one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the endometrial joint classification method as described in any one of claims 1-7.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to perform the endometrial joint classification method as described in any one of claims 1-7 when executed by a computer processor.

Images