A gene rearrangement state prediction method and device based on pathological images

Abstract

The application discloses a gene rearrangement state prediction method and device based on pathological images and an apparatus. The method comprises the following steps: acquiring HE-stained pathological section images, performing multi-scale image blocking processing, extracting image features through a pre-trained visual coding model, and unifying dimensions through linear transformation and an activation function; constructing a multi-expert structure to independently model different scale features, fusing and outputting multi-scale fusion features through a gating mechanism; constructing attention branches for different gene markers, generating marker-specific embedding vectors by using a gated attention mechanism, and realizing marker-specific feature interaction modeling through a Transformer structure; and finally outputting the rearrangement state probability of each gene marker through a classifier. The application improves the accuracy and interpretability of gene rearrangement state prediction through multi-scale feature modeling and marker-specific attention mechanisms, and has the advantages of high efficiency, non-invasiveness and suitability for clinical auxiliary diagnosis.

Description

Technical Field

[0001] This invention relates to the technical field of medical image analysis and artificial intelligence-assisted diagnosis, specifically to a method, device, and apparatus for predicting gene rearrangement status based on pathological images. Background Technology

[0002] In the current field of molecular diagnostics for lymphoma, the MYC, BCL2, and BCL6 gene rearrangements in patients with diffuse large B-cell lymphoma (DLBCL) are typically detected using fluorescence in situ hybridization (FISH). While this method is highly accurate, it suffers from drawbacks such as high cost, complex operation, and long processing time, limiting its widespread application in primary healthcare institutions. Furthermore, FISH testing requires specialized probes and equipment and provides only limited histological contextual information, making comprehensive pathological assessment difficult. In recent years, with the development of digital pathology and artificial intelligence technologies, some studies have attempted to extract potential molecular features from conventional HE-stained slide images using deep learning models. However, most methods still have limitations in feature expression capabilities, model interpretability, and multi-gene joint prediction. Existing models often focus only on single gene markers, lacking effective modeling mechanisms for multi-marker synergistic prediction, and also lack systematic design in integrating clinical information to improve model robustness and clinical applicability. Therefore, there is an urgent need for an AI system that can efficiently, accurately, and interpretably predict multiple gene rearrangement states to reduce reliance on FISH testing and improve the accessibility and clinical applicability of DLBCL molecular subtyping. Summary of the Invention

[0003] Based on this, in order to solve the technical problem of low clinical diagnostic efficiency and high misdiagnosis rate caused by the lack of an efficient, accurate and non-invasive method for predicting gene rearrangement status in pathological images in the existing technology, this invention proposes a method, device and apparatus for predicting gene rearrangement status based on pathological images.

[0004] This invention protects a method for predicting gene rearrangement states based on pathological images. The prediction method includes: acquiring HE-stained pathological slide images and performing multi-scale image block processing to obtain image block sets at different magnifications; using a pre-trained visual coding model to extract features from image blocks at each scale to obtain initial image feature vectors, and performing uniform dimensionality processing on features at different scales through linear transformation and activation functions; constructing a multi-expert structure to independently model image features at different scales, and selectively fusing features output by expert networks through a gating mechanism to form a multi-scale fused feature representation; constructing attention branches for different gene markers, performing weighted aggregation of image blocks through a gating attention mechanism to generate marker-specific embedding vectors, and implementing feature interaction modeling between markers through a Transformer structure; and based on the modeling results, performing classification prediction for each gene marker and outputting the corresponding gene rearrangement state probability.

[0005] Further, HE-stained pathological slide images are acquired, and the images are processed into multi-scale image blocks to obtain image block sets at different magnifications. A pre-trained visual coding model is used to extract features from the image blocks at each scale to obtain initial image feature vectors. The features at different scales are then processed to unify the dimension through linear transformation and activation functions, including: dividing the entire HE-stained slide image into first image blocks at magnifications of 4 to 6 (e.g., 5), and dividing the entire HE-stained slide image into second image blocks at magnifications of 15 to 25 (e.g., 20), and aligning the first and second image blocks spatially; inputting the first and second image blocks into a pre-trained general visual encoder to extract scale-specific initial feature vectors, obtaining a first initial feature vector based on the first image block and a second initial feature vector based on the second image block; and performing linear transformation and activation function processing on the first and second initial feature vectors to obtain unified-dimensional embedded features for multi-scale modeling.

[0006] Furthermore, a multi-expert structure is constructed to independently model image features at different scales, and the features output by the expert networks are selectively fused through a gating mechanism to form a multi-scale fused feature representation. This includes: constructing multiple scale-specific expert networks, where each network focuses on processing image features at different magnifications; for each scale of image features, the corresponding expert network independently models and outputs a scale-specific feature representation; dynamically calculating the weighting coefficients of each expert network based on the features of the input image, and introducing a gating mechanism with an entropy regularization loss term to perform weighted fusion of the features output by the expert networks; and concatenating the feature vectors output by the expert networks at different scales to form a multi-scale fused feature representation that includes both macroscopic structure and microscopic details.

[0007] Furthermore, attention branches are constructed for different gene markers, and image patches are weighted and aggregated using a gated attention mechanism to generate marker-specific embedding vectors. This includes: constructing independent gated attention branches for each molecular marker MYC, BCL2, and BCL6; calculating the attention weight of the image patch using a gated attention scoring function for each marker, where the attention weight is determined by calculating the feature vector of the image patch and the gated scoring function corresponding to the marker; and weighting and summing the image patch features based on the attention weights to generate marker-specific embedding vectors.

[0008] Furthermore, after generating the marker-specific embedding vectors, the process further includes: concatenating the marker-specific embedding vectors generated for the BCL2, MYC, and BCL6 gene markers respectively; modeling the concatenated marker features through stacked Transformer layers; and outputting the updated marker feature representation for subsequent gene rearrangement states after processing by the Transformer layers.

[0009] Furthermore, based on the modeling results, classification prediction is performed for each gene marker, and the corresponding gene rearrangement state probability is output. This includes: in the classification output layer, an independent fully connected classifier is configured for each gene marker. The fully connected classifier receives the corresponding marker-specific embedding vector as input and outputs the rearrangement state probability value of the corresponding marker through an activation function. The probability value ranges from 0 to 1. During the model prediction process, the attention weight corresponding to each marker is extracted, and the key region used for prediction is determined based on the attention weight. The key region is then displayed in a visual manner.

[0010] Furthermore, it also includes integrating clinical data to construct a multimodal prediction model, specifically including: collecting patients' clinical data, standardizing the clinical data and inputting it into a multilayer perceptron to generate low-dimensional clinical feature vectors; concatenating the clinical feature vectors with label-specific embedding vectors to form multimodal fusion features; and processing the fusion features through an independent classifier to output the rearrangement probability of each gene label.

[0011] Furthermore, the multimodal prediction model is trained in stages: in the first stage, the image branch model is trained and its parameters are frozen; in the second stage, the clinical data branch and the classifier head are trained separately and optimized using a binary cross-entropy loss function and an optimizer.

[0012] This invention protects a gene rearrangement state prediction device based on pathological images. The device includes: an image processing module for acquiring HE-stained pathological slide images and performing multi-scale image block processing on the images to obtain image block sets at different magnifications; using a pre-trained visual coding model to extract features from image blocks at each scale to obtain initial image feature vectors, and performing uniform dimensionality processing on features at different scales through linear transformation and activation functions; a multi-scale fusion module for constructing a multi-expert structure to independently model image features at different scales, and selectively fusing features output by expert networks through a gating mechanism to form a multi-scale fused feature representation; a label feature modeling module for constructing attention branches for different gene labels, performing weighted aggregation of image blocks through a gating attention mechanism to generate label-specific embedding vectors, and realizing feature interaction modeling between labels through a Transformer structure; and a classification prediction module for performing classification prediction on each gene label based on the modeling results, and outputting the corresponding gene rearrangement state probability.

[0013] This invention protects an electronic device, comprising: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus, and the machine-readable instructions are executed by the processor to perform the steps of the gene rearrangement state prediction method based on pathological images.

[0014] This invention protects a gene rearrangement state prediction method based on pathological images. It extracts and fuses multi-scale features from HE-stained pathological slide images through multi-scale image segmentation combined with a deep learning model, thereby achieving classification and prediction of the rearrangement states of multiple gene markers. The method first performs multi-scale image segmentation and extracts image features at each scale using a pre-trained visual coding model. Linear transformation and activation functions are used to unify the feature dimensions, thereby enhancing the model's adaptability and expressive ability to different tissue structures. Furthermore, a multi-expert structure is constructed to independently model features at different scales, and a gating mechanism is used to selectively fuse expert outputs, improving the model's efficiency and robustness in utilizing multi-scale image information. In addition, the method introduces attention branches for different gene markers, uses a gated attention mechanism to weighted aggregate image patches, generates marker-specific embedding vectors, and utilizes a Transformer structure to model the interaction relationships between markers, thereby improving the accuracy and interpretability of classification. In its implementation, a multi-expert network processes image features at different magnifications and performs feature fusion through a gating mechanism that dynamically calculates weighting coefficients and combines them with an entropy regularization loss term. This results in a multi-scale fused feature representation encompassing both macroscopic structure and microscopic details, effectively enhancing the model's ability to model complex tissue structures in pathological images. Furthermore, independent gated attention branches are constructed for molecular markers such as MYC, BCL2, and BCL6. Attention weights for image patches are calculated using a scoring function, and marker-specific embedding vectors are generated accordingly. This allows the model to focus on key regions related to specific gene markers, thereby improving prediction accuracy and biological interpretability. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0016] Figure 1 A flowchart illustrating a gene marker classification prediction method based on multi-scale fusion feature representation provided in this application embodiment;

[0017] Figure 2 A flowchart of a multi-scale image feature extraction method provided in this application embodiment;

[0018] Figure 3 A flowchart illustrating a multi-scale feature fusion neural network provided in an embodiment of this application;

[0019] Figure 4 A flowchart illustrating gene biomarker feature modeling provided in this application embodiment;

[0020] Figure 5A flowchart illustrating a marker-specific embedding vector generation method provided in this application embodiment;

[0021] Figure 6 A flowchart illustrating gene marker classification prediction provided in this application embodiment;

[0022] Figure 7 A flowchart illustrating gene marker rearrangement state prediction provided in this application embodiment;

[0023] Figure 8 A schematic diagram of a device for predicting gene marker rearrangement status provided in an embodiment of this application;

[0024] Figure 9 A schematic diagram of an electronic device for predicting gene marker rearrangement status provided in an embodiment of this application;

[0025] Figure 10 This is a performance evaluation graph of the internal test set (Beijing queue) of the system in this application;

[0026] Figure 11 The verification results of the external test set 1 (Langfang queue) of the system in this application are shown in the figure;

[0027] Figure 12 The verification results for external test set 2 (Shanxi queue) are shown in the figure;

[0028] Figure 13 The figure shows the verification results of the external test set 3 (Shenzhen queue) of the system in this application.

[0029] Figure 14 The figure shows the ROC curve (Beijing queue) of the multimodal model of the system in this application.

[0030] Figure 15 The figure shows the ROC curve (Shanxi queue) of the multimodal model of the system in this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0032] Research has revealed several limitations in existing methods for predicting gene rearrangement status based on HE-stained pathological images. Traditional FISH detection methods are costly and time-consuming, failing to meet the needs of rapid clinical diagnosis. While some AI models have attempted to extract morphological features related to gene mutations or rearrangements from HE images, their models are relatively simple, focusing primarily on predicting single gene markers and lacking the ability to model potential associations between multiple gene markers. Furthermore, existing methods often employ simple stitching or averaging strategies when processing multi-scale image features, failing to effectively integrate key information at different scales, thus limiting the model's ability to perceive both microscopic and macroscopic features of tissue structures. Simultaneously, the lack of interpretability mechanisms hinders the provision of reliable decision-making support for physicians, limiting their practical clinical application value.

[0033] Based on this, please refer to Figure 1 This application provides a method for predicting gene rearrangement states based on pathological images, comprising: acquiring HE-stained pathological slide images and performing multi-scale image block processing on the images to obtain image block sets at different magnifications; using a pre-trained visual coding model to extract features from image blocks at each scale to obtain initial image feature vectors, and performing unified dimensional processing on features at different scales through linear transformation and activation functions; constructing a multi-expert structure to independently model image features at different scales, and selectively fusing features output by expert networks through a gating mechanism to form a multi-scale fused feature representation; constructing attention branches for different gene markers, performing weighted aggregation of image blocks through a gating attention mechanism to generate marker-specific embedding vectors, and implementing feature interaction modeling between markers through a Transformer structure; and based on the modeling results, classifying and predicting each gene marker separately, and outputting the corresponding gene rearrangement state probability.

[0034] This application provides a gene rearrangement state prediction method based on pathological images. Through multi-scale image segmentation and unified feature embedding, it effectively enhances the model's ability to perceive microscopic cell morphology and macroscopic tissue arrangement within tissue structures. The introduction of a multi-expert structure and gating mechanism enhances the model's specialized modeling and dynamic fusion capabilities for features at different scales, improving the diversity and robustness of feature expression. By constructing a marker-specific attention branch and Transformer interactive modeling, it achieves joint prediction of multiple gene markers, enhancing the model's ability to capture potential relationships between genes. Finally, by outputting the rearrangement probability of each gene marker through an independent classification head, it supports multi-label joint prediction, significantly improving prediction accuracy and clinical applicability. Furthermore, the gating attention mechanism supports the visual interpretation of key regions, enhancing the model's interpretability and clinical credibility.

[0035] In step S101, the system acquires the pathological slide image stained with HE and performs multi-scale image block processing on the image to obtain a set of image blocks at different magnifications; then, a pre-trained visual coding model is used to extract features from the image blocks at each scale to obtain the initial image feature vector, and the features at different scales are processed to unify the dimensions through linear transformation and activation function.

[0036] Specifically, S101 includes the following steps:

[0037] S1011. Divide the entire HE-stained slice image into a first image block at 5x magnification, divide the entire HE-stained slice image into a second image block at 20x magnification, and spatially align the first image block and the second image block.

[0038] In this embodiment, the input HE-stained pathological slide image is a whole-slide image (WSI) of a patient with diffuse large B-cell lymphoma (DLBCL). The system uses image processing tools (such as CLAM) to systematically divide the entire image into non-overlapping small image patches at two magnification levels: 5× and 20×. Each 5× patch is 224×224 pixels, and each 5× patch corresponds to a 4×4 20× patch grid, ensuring that image patches of different scales remain spatially aligned, thereby preserving the structural information of the image.

[0039] S1012. Input the first image block and the second image block into the pre-trained general visual encoder respectively, extract the scale-specific initial feature vector, and obtain the first initial feature vector based on the first image block and the second initial feature vector based on the second image block.

[0040] After image segmentation, the system inputs 5× and 20× image blocks into a pre-trained general visual encoder (such as the UNI model) to extract scale-specific initial feature vectors. Specifically, the 5× image block outputs the first initial feature vector after encoder processing. The 20× image patch outputs the second initial feature vector. This encoder is pre-trained on a large-scale natural image dataset and has good general feature extraction capabilities, effectively capturing the morphological features of images.

[0041] S1013. Perform linear transformation and activation function processing on the first initial feature vector and the second initial feature vector respectively to obtain a unified dimension of embedded features for multi-scale modeling.

[0042] To enable effective fusion of image patch features at different scales in subsequent modeling, the system unifies the dimensions of the extracted initial feature vectors. Specifically, a linear transformation and ReLU activation function are applied to the feature vector of each image patch to map them to a unified feature space. For example, through linear transformation W∈R^(1536×256) and ReLU activation function, the original 1536-dimensional features are compressed to 256 dimensions while retaining non-linear expressive power. The final output embedded features h_i^(5×) and h_j^(20×) will serve as the basic input for subsequent multi-scale modeling, ensuring effective fusion of features at different scales while maintaining dimensionality consistency.

[0043] In step S102, the system constructs a multi-expert structure, independently models image features at different scales, and selectively fuses the features output by the expert networks through a gating mechanism to form a multi-scale fused feature representation. Figure 3 The process of constructing multi-scale fusion feature representations is given. Specifically, S102 includes the following steps:

[0044] S1021. Construct multiple scale-specific expert networks, where multiple expert networks focus on processing image features at different magnifications.

[0045] In this embodiment, the system constructs a multi-expert (MoE) architecture comprising eight expert networks to process image features at both 5× and 20× scales. Four expert networks specifically handle features of 5× scale image patches, while the other four specifically handle features of 20× scale image patches. Each expert network is an independent deep neural network module with its own parameter space, enabling specialized modeling of image features at its corresponding scale.

[0046] S1022. For image features at each scale, the corresponding expert network independently models and outputs scale-specific feature representations.

[0047] After constructing the expert networks, the system inputs the unified dimension embedding features h_i^(5×) and h_j^(20×) output from S1013 into the corresponding expert networks. Each expert network performs nonlinear transformation and feature abstraction on the input image features, outputting scale-specific feature representations. For example, eight scale-specific expert networks are constructed, four focusing on the 5× scale and four focusing on the 20× scale. Each expert independently models the image features at its corresponding scale.

[0048] ;

[0049] ;

[0050] Through this step, each expert network focuses on feature modeling at its corresponding scale, improving the model's ability to express multi-scale information.

[0051] S1023. Calculate the weighting coefficients of each expert network dynamically based on the features of the input image, and introduce a gating mechanism of entropy regularization loss term to perform weighted fusion of the features output by the expert networks.

[0052] To avoid redundant learning among expert networks and improve model efficiency, the system introduces a gating mechanism to dynamically and selectively activate expert networks. Specifically, the gating mechanism calculates the activation weights of each expert network based on the input image features and performs a weighted fusion of the outputs of each expert. The weight calculation process is implemented through a lightweight gating network, whose output is the normalized activation probability of each expert. Furthermore, during training, the system introduces a gated entropy regularization loss term, L_gate, to encourage a balanced distribution among expert networks and prevent some experts from being overused or ignored.

[0053] S1024. The feature vectors output by expert networks at different scales are concatenated to form a multi-scale fusion feature representation that includes both macroscopic structure and microscopic details.

[0054] After weighted fusion of the expert network outputs, the system concatenates the feature vectors from expert networks at different scales to generate a multi-scale fused feature representation. For example, for a certain image region, the system concatenates its corresponding 5× expert output with the 20× expert output to form the final fused feature.

[0055] ;

[0056] This fusion feature contains both macroscopic tissue structure information (5× scale) and cellular level detail information (20× scale), providing rich semantic expression for subsequent marker-specific modeling.

[0057] In step S103, the system constructs attention branches for different gene markers, performs weighted aggregation of image patches through a gated attention mechanism to generate marker-specific embedding vectors, and models feature interactions between markers using a Transformer structure. Figure 4 The specific process of feature interaction modeling is given.

[0058] Specifically, S103 includes the following steps:

[0059] S1031. For each molecular marker MYC, BCL2, and BCL6, construct an independent gated attention branch.

[0060] In this embodiment, the system constructs independent gated attention branches for each target gene marker (i.e., MYC, BCL2, and BCL6). Each attention branch consists of a gated scoring function and a weighted aggregation mechanism, used to identify the image region most relevant to the gene rearrangement state. Through this design, the system can achieve specific modeling of different gene markers, improving the independence and accuracy of the model's predictions for each marker.

[0061] In one specific implementation, Figure 5 A method for generating tag-specific embedding vectors is presented, including:

[0062] S10311. For each marker, the attention weight of the image patch is calculated using a gated attention scoring function, wherein the attention weight is determined by the feature vector of the image patch and the gated scoring function corresponding to the marker.

[0063] In each gated attention branch, the system first processes the multi-scale fusion feature E_n∈R^(1×2048) output by S1024, and calculates the attention weight of each image patch to the current gene marker through the gated scoring function g_m(·) (m∈{BCL2, MYC, BCL6}). Specifically, the attention weight a_(m,n) is calculated by the following formula:

[0064] ;

[0065] Among them, g m The · function is a gate function consisting of fully connected layers used to dynamically adjust the weights of image patches based on the semantic information of the current label m. This mechanism enables the model to automatically identify the image regions most relevant to specific gene rearrangement states, thereby improving the interpretability of predictions.

[0066] S10312. The image patch features are weighted and summed according to the attention weights to generate a label-specific embedding vector.

[0067] After calculating the attention weight for each image patch, the system performs a weighted summation of the multi-scale fusion feature E_n of each image patch with its corresponding attention weight a_(m,n) to generate a label-specific embedding vector z_m:

[0068] z_m=∑_n a_(m,n) E_n ∈R^(1×2048)

[0069] The embedding vector z_m reflects the overall feature representation most relevant to gene marker m in the current image, serving as the basic input for subsequent modeling of inter-marker interactions.

[0070] S1032. The marker-specific embedding vectors generated for the BCL2, MYC and BCL6 gene markers are spliced ​​together.

[0071] After obtaining the marker-specific embedding vectors z_BCL2, z_MYC, and z_BCL6 for the three gene markers, the system concatenates them to form a joint feature representation:

[0072] z_concat = [z_BCL2, z_MYC, z_BCL6] ∈R^(1×6144)

[0073] This concatenation operation integrates the independent feature representations of the three tags into a unified high-dimensional feature space, providing a foundation for subsequent modeling of interactions between tags.

[0074] S1033. The spliced ​​label features are modeled by stacked Transformer layers. After processing by the Transformer layers, the updated label feature representation is output for subsequent gene rearrangement states.

[0075] The system inputs the concatenated joint feature z_concat into stacked Transformer layers for cross-label feature interaction modeling. The Transformer layers consist of a multi-head self-attention mechanism and a feedforward network, effectively capturing the complex dependencies between BCL2, MYC, and BCL6. After processing through multiple Transformer layers, the system outputs the updated labeled feature representation:

[0076] ;

[0077] Here, T(·) represents the mapping function of the Transformer backbone network. The updated feature representation not only preserves the specific information of each label, but also incorporates the interaction relationships between labels, providing richer semantic information for subsequent classification prediction.

[0078] In step S104, based on the modeling results, the system performs classification prediction for each gene marker and outputs the corresponding gene rearrangement state probability. The gene rearrangement state probability is then... Figure 6 The flowchart implementation,

[0079] Specifically, S104 includes the following steps:

[0080] S1041. In the classification output layer, an independent fully connected classifier is configured for each gene marker. The fully connected classifier receives the corresponding marker-specific embedding vector as input and outputs the rearrangement probability value of the corresponding marker through an activation function. The probability value ranges from 0 to 1.

[0081] In this embodiment, the system configures an independent classification head module for each target gene marker (BCL2, MYC, BCL6). Each classification head consists of a fully connected layer and a sigmoid activation function, and receives the updated marker feature representation z'_m output from the Transformer layer in S1033 as input. Specifically, the classification head calculates the predicted probability of the rearranged state of marker m in the following manner:

[0082] ;

[0083] Where ϕm(·) represents the fully connected classifier head corresponding to label m, and σ(·) is the Sigmoid activation function, used to map the output to the 0~1 interval, representing the probability of the label being rearranged. This design makes the prediction process of each label independent, while obtaining cross-label contextual information support through feature interaction modeling of the Transformer layer in S103, thereby improving the accuracy and robustness of prediction.

[0084] S1042. During the model prediction process, extract the attention weight corresponding to each label, determine the key region for prediction based on the attention weight, and display the key region in a visual way.

[0085] To further enhance the model's interpretability, the system retains the attention weights α_(m,n) calculated in S10311 during the prediction process. These weights reflect the contribution of each image patch n to the prediction result of gene marker m. The system maps the attention weights back to the spatial location of the original pathological image, generating a heatmap that highlights the image regions with the greatest impact on the prediction results. For example, for the BCL2 marker, the system visualizes its corresponding attention heatmap, enabling pathologists to intuitively understand the model's decision-making basis and assisting them in manual review and diagnostic decision-making.

[0086] This visualization mechanism not only enhances the transparency of the model but also increases doctors' trust in the AI ​​prediction results, providing important support for the practical application of the model in clinical scenarios.

[0087] Building upon the above-described implementation methods, to further improve the accuracy and clinical applicability of gene rearrangement state prediction, this invention also introduces a multimodal fusion strategy. This strategy combines pathological image features with patient clinical data to construct a multimodal prediction model. Without altering the core image modeling process, this strategy enhances the model's ability to discriminate complex cases by fusing structured clinical information.

[0088] Specifically, after step S104, the process of predicting the gene marker rearrangement state is executed; see Appendix for details. Figure 7The system then performs the following operations:

[0089] S1051. Collect patients' clinical data, standardize the clinical data, input it into a multilayer perceptron, and generate low-dimensional clinical feature vectors.

[0090] The system collects structured clinical data from DLBCL patients, including but not limited to: age, sex, Ann Arbor stage, IPI score, tumor location, cell origin (COO) subtype, and immunohistochemical (IHC) results (such as Ki-67, CD10, MUM1, BCL2, MYC, etc.). All clinical variables are first standardized to eliminate biases caused by different dimensions. Subsequently, the standardized clinical data are input into a multilayer perceptron (MLP), which consists of several fully connected layers and a ReLU activation function, used to map the raw clinical information into a low-dimensional, compact clinical feature vector z_clinic ∈ R^(1×1024), facilitating subsequent fusion modeling.

[0091] S1052. The clinical feature vector is concatenated with the label-specific embedding vector to form a multimodal fusion feature.

[0092] To achieve complementary fusion of image features and clinical features, the system concatenates the marker-specific embedding vector z_m generated in S10312 with the clinical feature vector z_clinic extracted in S1051 to form a multimodal fusion feature representation for each gene marker m:

[0093] z_m^* = Concat(z_m, z_clinic) ∈ R^(1×3072)

[0094] Here, Concat(·) represents the vector concatenation operation. This fusion feature retains both the morphological information related to gene rearrangement in the image and incorporates the patient's clinical background information, thereby improving the model's generalization ability to complex cases.

[0095] S1053. Process the fusion features using an independent classifier and output the rearrangement probability of each gene marker.

[0096] The system configures a new independent classification head for each gene marker m. This classification head receives multimodal fusion features as input and outputs the final rearranged state prediction probability through a fully connected layer and a Sigmoid activation function.

[0097] This classification head exists in parallel with the image-based classification head in S1041, and multimodal input can be enabled or disabled according to actual deployment requirements.

[0098] Furthermore, to ensure the stability of the image feature extraction part, the parameters of the image backbone network (i.e., the image modeling modules constructed from S101 to S103) are frozen during the multimodal model training process, and training is only performed on the clinical branch and the new classification head. The training uses the same binary cross-entropy loss function as the image model, combined with an early stopping mechanism to prevent overfitting.

[0099] In summary, by introducing a multimodal fusion mechanism of clinical data, this invention achieves high-precision prediction of the BCL2, MYC, and BCL6 gene rearrangement status in DLBCL patients without relying on FISH detection, further improving the applicability and reliability of the model in clinical practice.

[0100] Please see Figure 8 , Figure 8 This is a schematic diagram of a gene rearrangement state prediction device based on pathological images, provided as an embodiment of this application. Figure 8 As shown, the prediction device 200 includes:

[0101] Image processing module 210 is used to acquire HE-stained pathological slide images and perform multi-scale image block processing on the images to obtain image block sets at different magnifications; a pre-trained visual coding model is used to extract features from image blocks at each scale to obtain initial image feature vectors, and the features at different scales are uniformly processed through linear transformation and activation function;

[0102] The multi-scale fusion module 220 is used to construct a multi-expert structure, independently model the image features at different scales, and selectively fuse the features output by the expert network through a gating mechanism to form a multi-scale fusion feature representation.

[0103] The marker feature modeling module 230 is used to construct attention branches for different gene markers, perform weighted aggregation of image patches through a gated attention mechanism to generate marker-specific embedding vectors, and realize feature interaction modeling between markers through a Transformer structure.

[0104] The classification prediction module 240 is used to perform classification prediction for each gene marker based on the modeling results and output the corresponding gene rearrangement state probability.

[0105] Furthermore, when the image processing module 210 is used to acquire HE-stained pathological slide images, perform multi-scale image block processing on the images to obtain image block sets at different magnifications, extract features from image blocks at each scale using a pre-trained visual coding model to obtain initial image feature vectors, and perform unified dimensionality processing on features at different scales through linear transformation and activation functions, the image processing module 210 is used to:

[0106] The entire HE-stained slice image is divided into a first image block at 5x magnification, and the entire HE-stained slice image is divided into a second image block at 20x magnification. The first and second image blocks are spatially aligned.

[0107] The first image patch and the second image patch are respectively input into a pre-trained general visual encoder to extract scale-specific initial feature vectors, thereby obtaining a first initial feature vector based on the first image patch and a second initial feature vector based on the second image patch;

[0108] Linear transformation and activation function processing are performed on the first and second initial feature vectors respectively to obtain a unified dimension of embedded features for multi-scale modeling.

[0109] Furthermore, when the multi-scale fusion module 220 is used to construct a multi-expert structure, independently model image features at different scales, and selectively fuse the features output by the expert network through a gating mechanism to form a multi-scale fused feature representation, the multi-scale fusion module 220 is used to:

[0110] Construct multiple scale-specific expert networks, where each network focuses on processing image features at different magnifications;

[0111] For each scale of image features, a corresponding expert network independently models and outputs scale-specific feature representations;

[0112] The weighting coefficients of each expert network are dynamically calculated based on the features of the input image, and a gating mechanism of entropy regularization loss term is introduced to perform weighted fusion of the features output by the expert networks.

[0113] The feature vectors output by expert networks at different scales are concatenated to form a multi-scale fusion feature representation that includes both macroscopic structure and microscopic details.

[0114] Furthermore, when the marker feature modeling module 230 is used to construct attention branches for different gene markers, perform weighted aggregation of image patches through a gated attention mechanism to generate marker-specific embedding vectors, and realize feature interaction modeling between markers through a Transformer structure, the marker feature modeling module 230 is used to:

[0115] For each molecular marker MYC, BCL2, and BCL6, an independent gated attention branch was constructed.

[0116] For each marker, the attention weight of the image patch is calculated using a gated attention scoring function, where the attention weight is determined by the feature vector of the image patch and the gated scoring function corresponding to the marker.

[0117] The image patch features are weighted and summed according to the attention weights to generate a label-specific embedding vector;

[0118] The marker-specific embedding vectors generated for the BCL2, MYC, and BCL6 gene markers were spliced ​​together.

[0119] The concatenated label features are modeled by stacked Transformer layers. After processing by the Transformer layers, the updated label feature representation is output for subsequent gene rearrangement states.

[0120] Furthermore, when the classification prediction module 240 is used to perform classification prediction for each gene marker based on the modeling results and output the corresponding gene rearrangement state probability, the classification prediction module 240 is used to:

[0121] In the classification output layer, an independent fully connected classifier is configured for each gene marker. The fully connected classifier receives the corresponding marker-specific embedding vector as input and outputs the rearrangement probability value of the corresponding marker through an activation function. The probability value ranges from 0 to 1.

[0122] During the model prediction process, the attention weights corresponding to each label are extracted, the key regions for prediction are determined based on the attention weights, and the key regions are displayed in a visual way.

[0123] Furthermore, the prediction device 200 also includes a multimodal fusion module (not shown in the figure), which is used to integrate clinical data to construct a multimodal prediction model, specifically including:

[0124] Collect patients' clinical data, standardize the clinical data and input it into a multilayer perceptron to generate low-dimensional clinical feature vectors;

[0125] The clinical feature vector is concatenated with the label-specific embedding vector to form a multimodal fusion feature;

[0126] The fused features are processed by an independent classifier, and the rearrangement probability of each gene marker is output.

[0127] The multimodal prediction model is trained in stages: in the first stage, the image branch model is trained and its parameters are frozen; in the second stage, the clinical data branch and the classifier head are trained separately and optimized using a binary cross-entropy loss function and an optimizer.

[0128] The gene rearrangement state prediction device based on pathological images provided in this application acquires HE-stained pathological slide images and performs multi-scale image block processing. A pre-trained visual coding model extracts image features and unifies dimensions. A multi-expert structure is constructed to independently model image features at different scales and fuses them through a gating mechanism to form a multi-scale fused feature representation. Attention branches are constructed for different gene markers, and image blocks are weighted and aggregated through a gating attention mechanism to generate marker-specific embedding vectors. A Transformer structure is used to model the interaction of features between markers. Finally, based on the modeling results, each gene marker is classified and predicted, outputting the corresponding gene rearrangement state probability. This method improves the accuracy and interpretability of gene rearrangement state prediction through multi-scale modeling, gating attention mechanisms, and a Transformer structure. It can also integrate clinical data to achieve multi-modal prediction, showing promising clinical application prospects.

[0129] The prediction device 200 used the HE2FISH model determined by the above method, and used 6 plots from the test set ( Figures 10 to 15 This is used to verify the performance of the HE2FISH model in different independent queues. Figure 10 The model's predictions on the internal test set (Beijing cohort) are shown, and its robustness is evaluated using five-fold cross-validation. Figures 11 to 13 The model's generalization ability across cross-center data was verified using three external test sets in Langfang, Shanxi, and Shenzhen, respectively, to ensure its applicability in different regions and medical institutions. Figure 14 and Figure 15 The presentation will focus on showcasing the ROC curve of the multimodal HE2FISH model (which integrates clinical information), demonstrating its performance improvement compared to the pure image model and further supporting the clinical value of multimodal design.

[0130] These charts visually present the model's predictive performance, including key metrics such as AUROC and AUPRC, with an average AUC of 0.867 compared to a multi-center external dataset. Overall, these charts not only validate the model's accuracy and robustness using its predictive capabilities but also provide empirical evidence for its application in clinical practice.

[0131] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 2 As shown, the electronic device 300 includes a processor 310, a memory 320, and a bus 330.

[0132] The memory 320 stores machine-readable instructions executable by the processor 310. When the electronic device 300 is running, the processor 310 and the memory 320 communicate via the bus 330. When the machine-readable instructions are executed by the processor 310, they can perform the operations described above. Figure 1 The steps of the gene rearrangement state prediction method based on pathological images in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.

[0133] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the gene rearrangement state prediction method based on pathological images in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.

[0134] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0136] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0137] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0138] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0139] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for predicting gene rearrangement status based on pathological images, characterized in that, The prediction method includes: HE-stained pathological slide images were acquired, and the images were processed into multi-scale image blocks to obtain image block sets at different magnifications. A pre-trained visual coding model was used to extract features from the image blocks at each scale to obtain initial image feature vectors. The features at different scales were then processed to unify the dimensions through linear transformation and activation functions. A multi-expert structure is constructed to independently model image features at different scales, and the features output by the expert network are selectively fused through a gating mechanism to form a multi-scale fused feature representation. Attention branches are constructed for different gene markers, and image patches are weighted and aggregated through a gated attention mechanism to generate marker-specific embedding vectors. The feature interaction model between markers is realized through the Transformer structure. Based on the interactive modeling results, each gene marker is classified and predicted, and the corresponding gene rearrangement state probability is output. The process involves acquiring HE-stained pathological slide images, performing multi-scale image block processing to obtain image block sets at different magnifications, and using a pre-trained visual coding model to extract features from image blocks at each scale to obtain initial image feature vectors. Furthermore, it includes unifying the dimensionality of features at different scales through linear transformation and activation functions. The entire HE-stained slice image is divided into a first image block at 5x magnification, and the entire HE-stained slice image is divided into a second image block at 15 to 25x magnification. The first and second image blocks are spatially aligned. The first image and the second image blocks are respectively input into a pre-trained general visual encoder to extract scale-specific initial feature vectors, thereby obtaining a first initial feature vector based on the first image block and a second initial feature vector based on the second image block; Linear transformation and activation function processing are performed on the first and second initial feature vectors respectively to obtain a unified dimension of embedded features for multi-scale modeling; The construction of a multi-expert structure, which independently models image features at different scales and selectively fuses the features output by the expert networks through a gating mechanism to form a multi-scale fused feature representation, includes: Construct multiple scale-specific expert networks, where each network focuses on processing image features at different magnifications; For each scale of image features, a corresponding expert network independently models and outputs scale-specific feature representations; The weighting coefficients of each expert network are dynamically calculated based on the features of the input image, and a gating mechanism of entropy regularization loss term is introduced to perform weighted fusion of the features output by the expert networks. The feature vectors output by expert networks at different scales are concatenated to form a multi-scale fused feature representation. The process of constructing attention branches for different gene markers and weighting and aggregating image patches using a gated attention mechanism to generate marker-specific embedding vectors includes: For each molecular marker MYC, BCL2, and BCL6, an independent gated attention branch was constructed. For each marker, the attention weight of the image patch is calculated using a gated attention scoring function, where the attention weight is determined by the feature vector of the image patch and the gated scoring function corresponding to the marker. The image patch features are weighted and summed according to the attention weights to generate a label-specific embedding vector; After generating the marker-specific embedding vector, the process further includes: The marker-specific embedding vectors generated for the BCL2, MYC, and BCL6 gene markers were spliced ​​together. The concatenated label features are modeled by stacked Transformer layers. After processing by the Transformer layers, the updated label feature representation is output for subsequent gene rearrangement states. The classification and prediction of each gene marker based on the interactive modeling results, and the output of the corresponding gene rearrangement state probability include: In the classification output layer, an independent fully connected classifier is configured for each gene marker. The fully connected classifier receives the corresponding marker-specific embedding vector as input and outputs the rearrangement probability value of the corresponding marker through an activation function. The probability value ranges from 0 to 1. During the model prediction process, the attention weights corresponding to each label are extracted, the key regions for prediction are determined based on the attention weights, and the key regions are displayed in a visual way.

2. The gene rearrangement state prediction method based on pathological images according to claim 1, characterized in that, This also includes integrating clinical data to build multimodal predictive models, specifically including: Collect patients' clinical data, standardize the clinical data and input it into a multilayer perceptron to generate low-dimensional clinical feature vectors; The clinical feature vector is concatenated with the label-specific embedding vector to form a multimodal fusion feature; The fusion features are processed by an independent classifier, and the rearrangement probability of each gene marker is output.

3. The gene rearrangement state prediction method based on pathological images according to claim 2, characterized in that, The multimodal prediction model is trained in stages: The first stage involves training the image branching model and freezing its parameters. In the second stage, the clinical data branch and classifier head are trained separately and optimized using a binary cross-entropy loss function and an optimizer.

4. A gene rearrangement state prediction device based on pathological images, characterized in that, The apparatus includes a computer program that performs the gene rearrangement state prediction method based on pathological images as described in any one of claims 1 to 3, comprising the following modules: The image processing module is used to acquire HE-stained pathological slide images and perform multi-scale image block processing to obtain image block sets at different magnifications. A pre-trained visual coding model is used to extract features from image blocks at each scale to obtain initial image feature vectors, and features at different scales are processed to unify the dimensions through linear transformation and activation functions. The multi-scale fusion module is used to construct a multi-expert structure, which independently models the image features at different scales and selectively fuses the features output by the expert network through a gating mechanism to form a multi-scale fused feature representation. The marker feature modeling module is used to construct attention branches for different gene markers, weighted aggregate image patches through a gated attention mechanism to generate marker-specific embedding vectors, and realize feature interaction modeling between markers through a Transformer structure; The classification prediction module is used to classify and predict each gene marker based on the modeling results, and output the corresponding gene rearrangement state probability.

5. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the gene rearrangement state prediction method based on pathological images as described in any one of claims 1 to 3.

Images