A head and neck squamous cell carcinoma early identification method based on imagingomics and deep learning

Abstract

The application relates to the technical field of medical image processing and intelligent tumor grading recognition, and discloses a head and neck squamous carcinoma early recognition method based on imaging and deep learning, which comprises the following steps: acquiring head and neck CT, MRI and PET three-mode image data; adopting a two-stage dynamic structure alignment mechanism for registration; extracting fused imaging features; constructing a tumor sub-feature map; outputting a tumor grade and a prediction confidence through an uncertainty prediction model; and generating a saliency explanation heat map. Compared with the prior art which depends on single-mode images or shallow texture feature extraction models, especially under the condition that the boundary of a heterogeneous tumor region is fuzzy, different modes exist significant structure offset, the technical problem that high confidence precision discrimination of a real tumor staging state cannot be realized. Since the saliency guided registration mechanism and the improved graph convolutional neural network are introduced and combined with Dropout reasoning, the accuracy of the head and neck squamous carcinoma early stage recognition is improved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of medical image processing and intelligent tumor grading recognition, and particularly relates to a head and neck squamous cell carcinoma early recognition method based on imaging and deep learning. BACKGROUND

[0002] At present, head and neck squamous cell carcinoma (HNSCC) is a common type of malignant tumor in clinical, and early accurate staging is of key significance for formulating individualized radiotherapy and chemotherapy schemes and evaluating prognosis. With the development of medical image acquisition technology, CT, MRI and PET modalities have been widely used in head and neck imaging diagnosis: CT can provide bone structure and density information, MRI can depict soft tissue boundaries, nerve and blood vessel infiltration, and PET can provide metabolic activity and tumor functional state. However, the existing medical auxiliary diagnosis system is still mainly based on single modality or weakly fused manual radiomics features, and it is difficult to fully exert the complementary advantages of multi-modal information.

[0003] On the one hand, there are problems such as different acquisition times, patient position differences, and non-rigid deformation of anatomical structures among multi-modal images, which makes it difficult to align CT, MRI and PET images in space. The traditional registration method based on rigid or affine transformation often cannot align the key structures such as tumor boundary, metabolic hot spot or low-density necrotic area, resulting in inaccurate fusion features and error amplification. Especially in the scene of complex soft tissue structure and blurred boundary between organs in the head and neck region, the existing registration strategy cannot perform local enhancement or saliency guidance on the tumor area.

[0004] On the other hand, tumor tissue itself has high spatial heterogeneity, and there may be multiple pathological sub-regions such as active area, necrotic area and calcified area inside the tumor. Different stages of tumors have significant differences in regional combination patterns. However, the traditional feature extraction method based on global average or manual region extraction cannot effectively express the "semantic difference of internal structure combination". Although some works introduce deep convolutional neural networks to extract features, most models still treat the tumor as a single entity for processing, lack of regional level expression structure, and do not consider the relationship and interaction between regions.

[0005] In addition, in the design of staging discrimination model, the current mainstream method regards the staging task as a general multi-classification problem, ignoring the natural order structure and hierarchical semantic information between I stage<II stage<III stage<IV stage, which makes the model prone to "II stage→IV stage" jumping errors, seriously affecting the clinical credibility. For some image performance fuzzy, boundary period samples (such as II / III stage), the existing model outputs a single point class label, lacks of quantitative and expression mechanism for prediction confidence, and it is difficult to provide doctors with "whether to review" or "whether it is a high-risk fuzzy example" judgment basis.

[0006] Therefore, there is an urgent need for an early-stage classification method for head and neck squamous cell carcinoma, which can still achieve structure-aware registration fusion, high-expression regional feature modeling, sequential reasonable uncertainty output and explainable prediction mechanism in the comprehensive scenario of significant structural mismatch of multi-modal images, strong tumor area expression heterogeneity, fuzzy staging boundary and high model credibility requirement. SUMMARY

[0007] In view of the above technical deficiencies, the purpose of the present application is to provide an early-stage identification method for head and neck squamous cell carcinoma based on imageomics and deep learning, aiming to solve the technical problem that the existing technology relies too much on single-mode images or shallow texture feature extraction models, especially under the condition of fuzzy boundary of heterogeneous tumor area and significant structural offset of different modalities, which cannot achieve high-precision discrimination of tumor real staging state with high confidence.

[0008] To solve the above technical problems, the present application adopts the following technical solutions: the present application provides an early-stage identification method for head and neck squamous cell carcinoma based on imageomics and deep learning,

[0009] The early-stage identification method for head and neck squamous cell carcinoma based on imageomics and deep learning comprises:

[0010] Step S10: Obtain three-mode imageomics data including CT, MRI and PET, and use a two-stage dynamic structure alignment mechanism to perform image registration on the three-mode imageomics data, and output fused imageomics data; wherein the dynamic structure alignment mechanism comprises a global rigid registration stage and a non-rigid deformation registration stage, and a saliency guidance mechanism is introduced in the non-rigid deformation registration stage;

[0011] Step S20: Extract features from the fused imageomics data, and input the extracted features into a pre-set multi-branch neural network structure to output a deep feature tensor of the tumor area; process the deep feature tensor of the tumor area based on a cross-modal guided attention mechanism and a modal adaptive fusion mechanism to obtain a fused imageomics feature F fusion ;

[0012] Step S30: According to the fused imageomics feature F fusion , use k-means method for segmentation clustering to obtain k sub-feature regions ROIl, ROI2,..., ROI k , construct a tumor sub-feature graph G with the k sub-feature regions ROIl, ROI2,..., ROI k , design an improved graph convolutional neural network GCN based on the tumor sub-feature graph G, and the improved graph convolutional neural network GCN outputs a tumor heterogeneity representation vector h;

[0013] Step S40: jointly fusing the imageomics feature F based on the tumor heterogeneity representation vector h fusion training the preset order perception uncertainty grading identification model, and the model output includes a predicted grade label y pred combined with the corresponding uncertainty indicator H

[0014] Step S50: generating a saliency explanation heat map based on the predicted grade label y pred combined with the corresponding uncertainty indicator H

[0015] Preferably, feature extraction is performed on the fused imageomics data, and the extracted features are input into a preset multi-branch neural network structure to output a deep feature tensor of the tumor region; the deep feature tensor of the tumor region is processed by a cross-modal guided attention mechanism and a modal adaptive fusion mechanism to obtain the fused imageomics feature F fusion .

[0016] Preferably, in step S10, a saliency guiding mechanism is introduced in the non-rigid deformation registration stage, and the non-rigid deformation registration process outputs a step of registering the fused image, which specifically includes:

[0017] The PET high-metabolic uptake region, the MRI contrast distribution region, and the CT density abnormal region are extracted, and the high-metabolic uptake region, the contrast distribution region, and the density abnormal region are combined to generate a tumor saliency weight map S(x) using a saliency map fusion method based on local region consistency enhancement, where x is defined in a three-dimensional space and represents the current image position index to be registered, which is used for integration and local calculation;

[0018] A registration energy function E(T) is defined in the process of non-rigid deformation registration for the local region of the tumor, E(T) = ∫ Ω w(x)·D(I CT (T(x)),I MRI (x))dx, where T is the deformation field to be optimized; E(T) represents the defined registration energy function, D(·) represents the distance metric between two images; Ω is the integral region defined in the preset reference modality image coordinate system; w(x) is a weight function for emphasizing the importance of the tumor saliency region; I CT (T(x)) is the CT image value; I MRI (x) is the MRI image value; T(x) is a spatial deformation function for representing the optimization quantity;

[0019] Let the weight function w(x) = 1 + a S(x), wherein a is an adjustment parameter, a > 0; the tumor significance weight map S(x) is used to give higher values to the internal and edge regions of the tumor, and the non-rigid deformation registration process outputs the registration fusion image by applying the weight function w(x).

[0020] Preferably, in step S20, in step S20, the fusion image omics data is subjected to feature extraction, and the extracted features are input into a preset multi-branch neural network structure to output a deep feature tensor of the tumor region; the deep feature tensor of the tumor region is processed based on a cross-modal guided attention mechanism and a modal adaptive fusion mechanism to obtain a fusion image omics feature F fusion The steps specifically include:

[0021] Step S201: feature extraction is performed on the fusion image omics data to obtain a feature extraction function of each modality; the feature extraction function of each modality is input into a multi-branch neural network structure, the multi-branch neural network structure includes a CT subnetwork layer, an MRI subnetwork layer and a PET subnetwork layer, and a deep feature tensor of a tumor region is output, the deep feature tensor includes a deep feature tensor F CT in the CT modality, a deep feature tensor F MRI in the MRI modality, and a deep feature tensor F PET in the PET modality;

[0022] Step S202: first, the deep feature tensor F PET in the PET modality is set as a guide modality, a spatial attention matrix A PET of the guide modality is calculated, the spatial attention matrix A PET is used to represent the importance of each position in the PET modality, and the spatial attention matrix A PET is obtained by channel weighted average or additional small network prediction of PET features;

[0023] Step S203: then the spatial attention matrix A PET is applied to the CT modality and the MRI modality to realize modality guided enhancement, and optimized deep feature tensors F * MRI and F * PET are output;

[0024] Step S204: finally, the optimized deep feature tensors F * MRI and F * PET and the deep feature tensor F MRI F MRI, the fusion image features F are obtained by fusing the image features F and the graph features G using a self-attention Transformer mechanism fusion .

[0025] Preferably, in step S30, the improved graph convolutional neural network GCN specifically comprises a graph attention enhancement layer for introducing an attention mechanism in the sub-region graph structure, dynamically calculating the semantic dependence strength between different regions inside the tumor, and giving different edges different propagation weights; a residual gate layer for relieving the problem of information oversmoothing caused by too many stacked graph convolution layers; a cross-scale context fusion layer for capturing multi-scale dependence patterns between local adjacency and global topological structure in the tumor subgraph; a graph structure attribution layer for outputting a significance score of each sub-region on the overall staging prediction; and an output layer for outputting a tumor heterogeneity representation vector h.

[0026] Preferably, in step S40, a hierarchical regression unit containing 3 ordered Sigmoid output nodes is constructed in the uncertainty hierarchical identification model, which respectively outputs the tumor staging prediction probabilities wherein, P(y>I) is a probability prediction value of upgrading from grade I to a higher grade; P(y>II) is a probability prediction value of upgrading from grade II to a higher grade; P(y>III) is a probability prediction value of upgrading from grade III to a higher grade; P(y>I) represents a probability estimate value of sample y not belonging to grade I, but belonging to grade II / III / IV; P(y>II) represents a probability estimate value of sample y not belonging to grade I or II, but belonging to grade III / IV; and P(y>III) represents a probability estimate value of sample y belonging to the latest grade IV.

[0027] Preferably, in step S40, the training objective function of the uncertainty hierarchical identification model is a weighted ordered loss function L.

[0028]

[0029] wherein, y ≥j ∈{0,1} represents whether the current sample belongs to a label indicator variable greater than or equal to grade j; P(y>j) is a prediction probability of the jth ordered grade judgment Sigmoid output node output by the model; y is a grade label predicted by the model, defined as the minimum grade j satisfying k true ∈{1,2,3,4}, k true is the true staging grade of the sample; λ>0 is an adjustment coefficient of the staging distance penalty term, used to strengthen the punishment for a large cross-grade misjudgment;

[0030] The uncertainty classification model also introduces a Dropout mechanism during the training phase. Under the control of the Dropout mechanism, the prediction mean and prediction entropy are sampled and inferred multiple times during the inference phase. The prediction entropy value is used as the corresponding uncertainty index H. in The average prediction distribution after c forward inference sampling is used to quantify the degree of confidence fluctuation in the current periodization judgment.

[0031] This invention also provides an early identification system for head and neck squamous cell carcinoma based on radiomics and deep learning, comprising:

[0032] The multimodal image registration and fusion module is used to acquire trimodal radiomics data including CT, MRI and PET. It uses a two-stage dynamic structural alignment mechanism to perform image registration for the trimodal radiomics data and outputs fused radiomics data. The dynamic structural alignment mechanism includes a global rigid registration stage and a non-rigid deformation registration stage, and a saliency guidance mechanism is introduced in the non-rigid deformation registration stage.

[0033] The deep feature extraction and modality fusion module is used to extract features from the fused radiomics data and input the extracted features into a pre-defined multi-branch neural network structure, outputting a deep feature tensor of the tumor region. The deep feature tensor of the tumor region is then processed using a cross-modal guided attention mechanism and a modality adaptive fusion mechanism to obtain the fused radiomics features F. fusion ;

[0034] The graph construction and graph convolution representation module is used to represent the fused image omics features F fusion The k-means method is used for segmentation and clustering to obtain k sub-feature regions R011, R02, ..., R010. k Given k sub-feature regions ROL1, ROI2, ..., ROI k A tumor sub-feature map G is constructed for the nodes. An improved graph convolutional neural network GCN is designed based on the tumor sub-feature map G. The improved graph convolutional neural network GCN outputs a tumor heterogeneity representation vector h.

[0035] A sequence-aware uncertainty prediction module is used to jointly fuse radiomics features F based on tumor heterogeneity representation vector h. fusion Train a pre-defined sequence-aware uncertainty-based hierarchical recognition model. The model output includes the predicted level label y. pred And the corresponding uncertainty index H;

[0036] A saliency-interpretable heatmap generation module is used to generate heatmaps based on predicted level labels y. predThe tumor sub-feature map G is combined with the corresponding uncertainty index H to generate a saliency explanation heat map, which includes three-mode salient region highlighting, sub-region importance visualization and output explanatory sentences.

[0037] The application further provides a head and neck squamous carcinoma early identification device based on radiomics and deep learning, comprising a memory, a processor and a head and neck squamous carcinoma early identification program based on radiomics and deep learning stored on the memory and executable on the processor, which realizes the head and neck squamous carcinoma early identification method based on radiomics and deep learning when executed by the processor.

[0038] The application further provides a computer program product comprising a head and neck squamous carcinoma early identification program based on radiomics and deep learning, which realizes the head and neck squamous carcinoma early identification method based on radiomics and deep learning when executed by a processor.

[0039] The application has the beneficial effect that, compared with the prior art which relies on single-mode images or shallow texture feature extraction models, especially under the condition that the boundaries of heterogeneous tumor regions are fuzzy and different modes have significant structural offsets, the technical problem of high-confidence accurate identification of the true staging state of the tumor cannot be solved. Due to the introduction of the saliency guided registration mechanism and the improvement of the graph convolutional neural network combined with Dropout reasoning, the accuracy of the head and neck squamous carcinoma early stage identification is improved. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and those skilled in the art can obtain other drawings according to these drawings without creative effort.

[0041] Figure 1 The flowchart of the first embodiment of the head and neck squamous carcinoma early identification method based on radiomics and deep learning of the application.

[0042] Figure 2 The device schematic diagram of the head and neck squamous carcinoma early identification method based on radiomics and deep learning of the application. DETAILED DESCRIPTION

[0043] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be clearly and completely described in order to make the technical solutions in the embodiments of the present application apparent to those skilled in the art. Obviously, the described embodiments are only a part but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts should fall into the scope of the present application.

[0044] Embodiment one: as shown in the figure, it is a flowchart of a first embodiment of the head and neck squamous carcinoma early identification method based on imageomics and deep learning of the present application, and the first embodiment of the head and neck squamous carcinoma early identification method based on imageomics and deep learning of the present application is proposed. Figure 1

[0045] In the first embodiment, the head and neck squamous carcinoma early identification method based on imageomics and deep learning comprises:

[0046] Step S10: acquiring three-modality imageomics data including CT, MRI and PET, performing image registration on the three-modality imageomics data by using a two-stage dynamic structure alignment mechanism, and outputting fused imageomics data; wherein the dynamic structure alignment mechanism comprises a global rigid registration stage and a non-rigid deformation registration stage, and a saliency guidance mechanism is introduced in the non-rigid deformation registration stage;

[0047] It should be noted that the registration process not only focuses on the geometric alignment of the three-modality images, but also introduces a guidance mechanism based on tumor structure saliency, so that the tumor region obtains a higher fitting weight in the alignment process, and the spatial registration accuracy of the key lesions is ensured. The global registration stage is mainly used to correct the image center offset and scale difference between the modalities, and the non-rigid registration stage is more focused on the deformation modeling and fitting of the local tumor region boundary and internal structure.

[0048] It can be understood that in the conventional technology, a unified template or an alignment algorithm based on anatomical global features is often used, which is easy to cause the tumor boundary position to be blurred, or to incorrectly register the high-metabolism lesions to the background tissue, and seriously affects the accuracy of subsequent segmentation and identification tasks. Through the dynamic structure alignment mechanism adopted by the present application, especially after introducing the tumor saliency guidance in the non-rigid registration stage, the structures directly related to the lesions in different modalities, such as the abnormal metabolism region in PET, the high-contrast enhanced region in MRI and the density abnormal region in CT, will be given higher structure weight, so as to guide the whole registration process to focus more on the information region directly related to tumor diagnosis.

[0049] ​It should be understood that the saliency guidance mechanism improves the consistency of the lesion structure in the registered image, effectively avoids the structure mismatch problem caused by signal intensity differences in different modalities, especially in cases with fuzzy tumor edges or strong heterogeneity. In addition, the mechanism can adapt to changes in tumor morphology, metabolic activity and anatomical position in different cases, and has stronger generalization ability.

[0050] For example, in practical applications, the tumor in the MRI image of a certain patient is located in the left neck soft tissue area and shows obvious T2 high signal; in the PET image, there is a strong metabolic hotspot in the corresponding area; and in the CT, the area only shows mild low density change. Through the saliency guidance mechanism proposed in the present application, the information of the structures in the three modalities that are not completely consistent but point to the same lesion area is effectively extracted and used for registration guidance, so that the final fusion image forms a clear and consistent spatial expression in the three modalities in the area, significantly improving the availability and accuracy of subsequent segmentation and classification tasks.

[0051] Step S20: performing feature extraction on the fusion imageomics data, inputting the extracted features into a pre-set multi-branch neural network structure, and outputting a deep feature tensor of the tumor region; processing the deep feature tensor of the tumor region based on a cross-modal guided attention mechanism and a modal adaptive fusion mechanism to obtain fusion imageomics features F fusion ;

[0052] It should be noted that the multi-branch feature extraction network used in this step is different from the traditional single encoding path. It sets up independent convolutional encoding branches for each modality (such as CT, MRI, and PET), fully retains the expression ability of the original features of each modality, and enhances the expression ability of different scale lesion structures through multi-scale convolution kernels and residual connections. The features of each modality will not be directly spliced or weighted averaged before entering the fusion stage, but will be guided through the "cross-modal guided attention mechanism", in which the high metabolic hotspot area in the PET branch will give higher attention coefficients to the corresponding areas in the CT and MRI branches, guiding the network to focus on the active area of the potential lesion.

[0053] It can be understood that traditional imageomics feature fusion methods mostly use simple modal stacking or feature averaging operations, which lack structured modeling of complementary information between modalities, especially when the signal intensity of the same lesion area in multi-modal images is significantly different (such as weak contrast in CT and strong metabolic signal in PET), which can easily cause key information to be weakened or misjudged. The present application introduces a "modal adaptive fusion mechanism" in the fusion stage, i.e. setting a learnable weight gating structure for each modality feature branch, so that the network dynamically adjusts the feature fusion weight under different sample and lesion expression conditions, and has stronger generalization ability and adaptive ability.

[0054] It should be understood that through the introduction of the above two mechanisms, the output fused image omics features not only retain the diagnostic specificity of each modality image (such as the density resolution of CT, the soft tissue contrast of MRI, and the metabolic activity of PET), but also realize the organic combination of "structure consistency enhancement" and "cross-modality saliency guidance" at the fusion level, effectively solving the problems of information inconsistency and attention dispersion caused by modality distribution differences in multi-modality images, and providing a unified and high-expression tumor representation vector for subsequent graph structure modeling and classification tasks.

[0055] For example, in a certain actual case, the tumor contour in the CT image is not clear, the MRI image shows soft tissue abnormalities but no obvious enhancement, and the PET image shows strong metabolic signals in the region. The traditional fusion method may ignore the potential lesion features in the non-salient region of MRI and CT, while the method of the present application automatically enhances the attention distribution of this region in the CT and MRI feature branch through the PET guiding mechanism, and finally the region shows a high activation region in the comprehensive feature map after fusion, ensuring that the downstream model will not ignore its potential malignant features, thereby realizing more sensitive and accurate early lesion identification and staging evaluation.

[0056] Step S30: according to the fused image omics features F fusion The k-means method is used for segmentation clustering to obtain k sub-feature regions ROI1, ROI2,..., ROIk. k The k sub-feature regions ROI1, ROI2,..., ROIkare used as nodes to construct a tumor sub-feature graph G. k An improved graph convolutional neural network GCN is designed based on the tumor sub-feature graph G, and the improved graph convolutional neural network GCN outputs a tumor heterogeneity representation vector h.

[0057] It should be noted that in this step, the k-means clustering is no longer simply based on the image pixel space, but acts on the high-dimensional embedding space of the fused image omics features, which integrates the expression information of CT, MRI and PET three modalities and has stronger discriminant ability. The sub-feature regions formed reflect the potential differences of the tumor region internal structure in terms of metabolic activity, tissue density and boundary morphology, and can distinguish the spatial heterogeneity of tumor tissue at a finer granularity.

[0058] It can be understood that the traditional global image features or ROI (Region of Interest) level representation often ignores the microstructure heterogeneity inside the tumor region, resulting in insufficient expression ability of the recognition model for the fuzzy boundary of stage II / III. The method introduces the structural constraint and interaction relationship modeling mechanism between sub-regions by constructing a feature graph G based on sub-regions, so that the model can explicitly capture the spatial organization pattern and multi-scale structure distribution of tumor tissue in different biological regions, thereby improving the discriminability and stability of early recognition and classification.

[0059] For example, in a real PET-MRI fusion data sample, the tumor region is divided into 5 sub-regions by k-means, where: region 1 is the central metabolic active area, region 2 is the MRI enhanced edge area, region 3 is the low density necrosis area, region 4 is the CT high density calcification area, and region 5 is the tissue transition area. After constructing the graph G, the graph attention mechanism automatically identifies that region 2 and region 5 are similar in feature but far apart in space, and the model successfully captures the important structural information of the fuzzy boundary transition area in early staging by narrowing the graph embedding representation of the two. The traditional method usually shows a large feature boundary jump and unstable prediction in this area. The improved GCN can stably output low uncertainty of stage II in this scenario, avoiding the "II→IV" jumping error.

[0060] Step S40: Jointly fuse the image-based features F based on the tumor heterogeneity representation vector h fusion Train the pre-set order-aware uncertainty classification recognition model, and the model output includes the predicted grade label y pred and the corresponding uncertainty index H.

[0061] It should be noted that the "order-aware model" refers to a model that explicitly models the natural sequence relationship between different stages of tumors, unlike traditional independent class recognition methods. The model structure used in this step combines the order regression structure and the Bayesian neural network framework, and considers the grade order of the output label and the prediction distribution of the model output, thereby improving the uncertainty expression ability of the model when identifying the critical point of staging (such as II / III).

[0062] It should be understood that the ordinal perception structure model used in this step treats each level as a threshold decision problem, i.e., modeling the staging level as the position of a continuous variable between multiple decision boundaries. This step solves two points in the prior art: (1) It is difficult to model the ambiguity between categories: the traditional cross-entropy loss does not consider the natural order relationship between staging levels, resulting in error propagation in the identification of II / III stages; (2) The lack of uncertainty leads to the risk of misdiagnosis: in practical applications, some cases have low model confidence due to image artifacts, large tumor heterogeneity, or poor data quality, and without an uncertainty indicator output, the diagnosis decision may be misled.

[0063] By integrating the ordinal learning mechanism and the uncertainty estimation mechanism, the robustness of the model in the presence of fuzzy boundaries and data noise can be significantly improved, and a reliability reference can be provided for clinicians to guide whether to perform manual review.

[0064] For example, in a PET-MRI fusion image case, the model outputs a prediction result of II stage for a patient, but the corresponding prediction standard deviation is 0.32 (much higher than the average value 0.12 of the training samples), indicating that the model is less certain about this prediction. Combined with the significance heat map analysis, it is found that the metabolic level of the tumor edge region of the patient is similar to that of the typical III stage, while the central structure is similar to the II stage characteristics, indicating that it is a “II / III boundary fuzzy sample”. The introduction of the model uncertainty indicator prompts the suggestion “further biopsy or expert review is needed”, avoiding false positives.

[0065] Step S50: Based on the predicted level label y pred Combined with the tumor sub-feature map G, the significance explanation heat map is generated, which includes three modal significant region highlights, sub-region importance visualization, and output explanatory sentences.

[0066] It should be noted that the “significance explanation heat map” in this step is not a pseudo-color visualization map in the traditional sense, but a joint explanation atlas that integrates multi-modal response information and sub-graph structure contribution. This explanation atlas not only shows the spatial regions that are focused on by the model in the three modal images, but also points out which tumor sub-regions play a dominant role in staging identification in the form of a graph structure. Among them, the predicted level label is used to anchor the current reasoning result, and the uncertainty indicator is used to estimate the confidence interval in the reasoning process.

[0067] It can be understood that, due to the high spatial heterogeneity of head and neck squamous cell carcinoma tumors, doctors need to know which areas and modalities the model identification result mainly refers to, and why these areas are of key value for the diagnosis of the current stage. The present application traces the prediction output back to the original fusion image step by step through the inversion processing of the intermediate results in the model, so as to obtain an explanatory area. Further, in combination with the weight distribution of the subgraph nodes, it can be shown which sub-area represents the heterogeneity feature performance that the model focuses on. The saliency explanation not only presents visual results, but also outputs structured explanatory sentences, such as: "This case is identified as stage II, with a prediction uncertainty of 0.18, mainly based on the metabolic activity image features of the left parapharyngeal region in PET and the high soft tissue boundary ambiguity region in MRI."

[0068] For example, in a certain actual patient case, the model output is stage III, and the uncertainty is high (about 0.36). The saliency explanation heat map shows that the MRI image has a high weight in the left mandibular region with a soft tissue boundary ambiguity, and the PET image has a metabolic activity region partially overlapping with the MRI abnormal region. In the corresponding graph structure, the contribution degrees of the 3rd and 5th nodes are the highest, corresponding to the mandibular and parapharyngeal regions respectively. The output explanatory sentence is: "This case is determined to be stage III, mainly referring to the MRI abnormal boundary and its metabolic abnormal response in PET, the current uncertainty is high, and it is recommended to combine clinical auxiliary examination for further confirmation." Doctors can make clinical corrections to AI judgment accordingly, or use it for case review and teaching demonstration.

[0069] Embodiment two: In addition, the present application also provides a head and neck squamous cell carcinoma early identification system based on imageomics and deep learning, which adopts the head and neck squamous cell carcinoma early identification method based on imageomics and deep learning in the above embodiment, and can solve the technical problem of the head and neck squamous cell carcinoma early identification method based on imageomics and deep learning. Compared with the prior art, the head and neck squamous cell carcinoma early identification system based on imageomics and deep learning provided by the present application has the same beneficial effects as the head and neck squamous cell carcinoma early identification method based on imageomics and deep learning provided by the above embodiment, and other technical features in the head and neck squamous cell carcinoma early identification system based on imageomics and deep learning are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0070] Embodiment three: The present application also provides a head and neck squamous cell carcinoma early identification device based on imageomics and deep learning, please refer to Figure 2A head and neck squamous cell carcinoma early identification device based on radiomics and deep learning includes at least one processor, and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the head and neck squamous cell carcinoma early identification method based on radiomics and deep learning in the above-mentioned embodiment one. The head and neck squamous cell carcinoma early identification device based on radiomics and deep learning in the embodiment of the present application can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. The head and neck squamous cell carcinoma early identification device based on radiomics and deep learning is only an example, and should not bring any limitation to the function and use range of the embodiment of the present application. The head and neck squamous cell carcinoma early identification device based on radiomics and deep learning can include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to programs stored in a read-only memory 1002 or loaded into a random access memory 1004 from a storage device 1003. In the random access memory 1004, various programs and data required for the operation of the head and neck squamous cell carcinoma early identification device based on radiomics and deep learning are also stored. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other through a bus 1005. An I / O interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touch screens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 can allow the head and neck squamous cell carcinoma early identification device based on radiomics and deep learning to communicate with other devices wirelessly or by wire to exchange data. Although the head and neck squamous cell carcinoma early identification device based on radiomics and deep learning with various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or fewer systems can be alternatively implemented or possessed.

[0071] Embodiment Four: The present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of a head and neck squamous cell carcinoma early identification method based on radiomics and deep learning as described above. The computer program product provided by the present application can solve the technical problem of a head and neck squamous cell carcinoma early identification method based on radiomics and deep learning. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the head and neck squamous cell carcinoma early identification method based on radiomics and deep learning provided by the above-mentioned embodiments, and are not described here.

[0072] In particular, according to the embodiments disclosed by the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed by the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by a processing device 1001, the above-mentioned functions defined in the method of the embodiments of the present application are executed.

[0073] It should be understood that various parts of the present application can be realized by hardware, software, firmware, or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0074] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for early identification of head and neck squamous cell carcinoma based on radiomics and deep learning, characterized in that, The methods include: Step S10: Acquire trimodal radiomics data including CT, MRI and PET, perform image registration using a two-stage dynamic structural alignment mechanism for the trimodal radiomics data, and output fused radiomics data; wherein, the dynamic structural alignment mechanism includes a global rigid registration stage and a non-rigid deformation registration stage, and introduces a saliency guidance mechanism in the non-rigid deformation registration stage; Step S20: Extract features from the fused radiomics data and input the extracted features into a pre-defined multi-branch neural network structure to output a depth feature tensor of the tumor region; process the depth feature tensor of the tumor region using a cross-modal guided attention mechanism and a modality adaptive fusion mechanism to obtain fused radiomics features. ; Step S30: Based on the fused image omics features The k-means method is used for segmentation and clustering to obtain k sub-feature regions. With k sub-feature regions A tumor sub-feature map G is constructed for each node. An improved graph convolutional neural network (GCN) is designed based on the tumor sub-feature map G. The improved graph convolutional neural network GCN outputs a tumor heterogeneity representation vector. ; Step S40: Based on tumor heterogeneity representation vector Joint fusion of radiomics features Train a pre-defined sequence-aware uncertainty-based hierarchical recognition model. The model output includes predicted level labels. Corresponding uncertainty indicators ; The uncertainty classification identification model incorporates a Dropout mechanism during the training phase. Under the control of the Dropout mechanism, the prediction mean and prediction entropy are sampled and inferred multiple times during the inference phase, and the prediction entropy value is used as the corresponding uncertainty index. , ,in The average prediction distribution after c forward inference sampling is used to quantify the degree of confidence fluctuation in the current stage judgment; Step S50: Based on predicted level labels Corresponding uncertainty indicators By combining the tumor sub-feature map G, a saliency interpretation heatmap is generated, which includes highlighting of saliency regions in three modes, visualization of the importance of sub-regions, and output of explanatory statements.

2. The method for early identification of head and neck squamous cell carcinoma based on radiomics and deep learning as described in claim 1, characterized in that, In step S10, three-modal radiomics data including CT, MRI, and PET are acquired. A two-stage dynamic structural alignment mechanism is used for image registration of the three-modal radiomics data, and the fused radiomics data is output. Specifically, this includes: Step S101: Acquire trimodal radiomics data including CT, MRI and PET, and use a multimodal rigid registration method based on the baseline modality to correct the initial position and scale differences of the global region under different modal image coordinate systems; Step S102: The non-rigid deformation registration stage specifically includes: performing non-rigid deformation registration on the local area of ​​the tumor, including free deformation of B-splines; and introducing a saliency guidance mechanism in the non-rigid deformation registration stage, outputting a registration and fusion image from the non-rigid deformation registration process; Step S103: Standardize the registered and fused images, use an automatic segmentation algorithm to extract the coarse tumor region, and output the coarse tumor region as fused radiomics data.

3. The method for early identification of head and neck squamous cell carcinoma based on radiomics and deep learning as described in claim 2, characterized in that, Step S10, which introduces a saliency guidance mechanism in the non-rigid deformation registration stage, and the step of outputting the registered and fused image in the non-rigid deformation registration process, specifically includes: The high metabolic uptake areas from PET, the contrast distribution areas from MRI, and the density abnormality areas from CT were extracted. A tumor saliency weight map was generated by integrating these three areas using a saliency map fusion method based on enhanced local regional consistency. Where x is defined in three-dimensional space, representing the current image position index to be registered, used for integration and local calculations; In the process of non-rigid deformation registration targeting local areas of the tumor, a registration energy function is defined. Where T is the deformation field to be optimized; This represents the defined registration energy function. Indicates the distance measurement between two images; This is the integration region defined in the preset reference modal image coordinate system; This is a weighting function used to emphasize the importance of significant tumor regions; CT image values; Values ​​from MRI images; This is a spatial deformation function used to represent the quantity to be optimized; Let the weight function ,in, To adjust the parameters, α>0; Tumor significance weighting plot A weighting function is used to assign higher values ​​to the tumor interior and periphery regions. The non-rigid deformation registration process outputs a registered and fused image.

4. The method for early identification of head and neck squamous cell carcinoma based on radiomics and deep learning as described in claim 1, characterized in that, In step S30, the improved graph convolutional neural network GCN specifically includes a graph attention enhancement layer, which is used to introduce an attention mechanism into the sub-region graph structure, dynamically calculate the semantic dependency strength between different regions inside the tumor, and assign different propagation weights to different connections. The residual gate layer is used to alleviate the problem of information oversmoothing caused by excessive stacking of graph convolutional layers; the cross-scale context fusion layer is used to capture multi-scale dependency patterns between local adjacency and global topology in tumor subgraphs; and the graph structure attribution layer is used to output the significance score of each sub-region for overall staging prediction. The output layer is used to output the tumor heterogeneity representation vector. .

5. The method for early identification of head and neck squamous cell carcinoma based on radiomics and deep learning as described in claim 1, characterized in that, In step S40, the uncertainty grading identification model constructs a grading regression unit containing three ordered Sigmoid output nodes, which output the tumor stage prediction probability respectively. ,in, This is the predicted probability value for promotion from Level I to a higher level; To be from the level The predicted probability of being promoted to a higher level; For the level The predicted probability of being promoted to a higher level; Indicates sample The probability estimate that it does not belong to stage I, but is in stage II / III / IV; Indicates sample The probability estimate of being in stage III / IV but not in stage I or II; Indicates sample The probability estimate of being in the latest stage IV.

6. The method for early identification of head and neck squamous cell carcinoma based on radiomics and deep learning as described in claim 5, characterized in that, In step S40, the training objective function of the uncertainty classification identification model is the weighted ordered loss function L; ； in, This indicates whether the current sample belongs to the greater than or equal to level. The label indicates the variable; The first output of the model The ordered levels determine the predicted probability of the Sigmoid output node; The stage grade label predicted by the model is defined as satisfying minimum level ; , This represents the true stage level of the sample; , This is an adjustment coefficient for the period distance penalty term, used to strengthen the penalty for large misjudgments across different levels.

7. A system for early identification of head and neck squamous cell carcinoma based on radiomics and deep learning, applied to the method for early identification of head and neck squamous cell carcinoma based on radiomics and deep learning as described in any one of claims 1 to 6, characterized in that, The head and neck squamous cell carcinoma early identification system based on radiomics and deep learning includes: The multimodal image registration and fusion module is used to acquire trimodal radiomics data including CT, MRI and PET. It uses a two-stage dynamic structural alignment mechanism to perform image registration for the trimodal radiomics data and outputs fused radiomics data. The dynamic structural alignment mechanism includes a global rigid registration stage and a non-rigid deformation registration stage, and a saliency guidance mechanism is introduced in the non-rigid deformation registration stage. The deep feature extraction and modality fusion module is used to extract features from the fused radiomics data and input the extracted features into a pre-defined multi-branch neural network structure, outputting a deep feature tensor of the tumor region. The deep feature tensor of the tumor region is then processed using a cross-modal guided attention mechanism and a modality adaptive fusion mechanism to obtain the fused radiomics features. ; The graph construction and graph convolution representation module is used to construct graphs based on fused image omics features. The k-means method is used for segmentation and clustering to obtain k sub-feature regions. With k sub-feature regions A tumor sub-feature map G is constructed for each node. An improved graph convolutional neural network (GCN) is designed based on the tumor sub-feature map G. The improved graph convolutional neural network GCN outputs a tumor heterogeneity representation vector. ; A sequence-aware uncertainty prediction module is used for prediction based on tumor heterogeneity representation vectors. Joint fusion of radiomics features Train a pre-defined sequence-aware uncertainty-based hierarchical recognition model. The model output includes predicted level labels. Corresponding uncertainty indicators ; The uncertainty classification identification model incorporates a Dropout mechanism during the training phase. Under the control of the Dropout mechanism, the prediction mean and prediction entropy are sampled and inferred multiple times during the inference phase, and the prediction entropy value is used as the corresponding uncertainty index. , ,in The average prediction distribution after c forward inference sampling is used to quantify the degree of confidence fluctuation in the current stage judgment; A saliency-interpretable heatmap generation module is used for generating heatmaps based on predicted grade labels. Corresponding uncertainty indicators By combining the tumor sub-feature map G, a saliency interpretation heatmap is generated, which includes highlighting of saliency regions in three modes, visualization of the importance of sub-regions, and output of explanatory statements.

8. A device for early identification of head and neck squamous cell carcinoma based on radiomics and deep learning, characterized in that, The device for early identification of head and neck squamous cell carcinoma based on radiomics and deep learning includes: a memory, a processor, and a program for early identification of head and neck squamous cell carcinoma based on radiomics and deep learning stored in the memory and executable on the processor. When the program for early identification of head and neck squamous cell carcinoma based on radiomics and deep learning is executed by the processor, it implements the method for early identification of head and neck squamous cell carcinoma based on radiomics and deep learning as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, The computer program product includes a radiomics and deep learning-based early identification program for head and neck squamous cell carcinoma. When the radiomics and deep learning-based early identification program for head and neck squamous cell carcinoma is executed by a processor, it implements the method for early identification of head and neck squamous cell carcinoma based on radiomics and deep learning as described in any one of claims 1 to 6.

Images