Prediction model and device for large B-cell lymphoma gene rearrangement

Abstract

The invention discloses a gene rearrangement state prediction method, equipment and device based on a pathological image. Comprising the steps of obtaining an HE staining pathological section image, performing multi-scale image partitioning processing, extracting image features through a pre-training visual coding model, and unifying dimensions through linear transformation and an activation function; constructing a multi-expert structure to perform independent modeling on different scale features, and fusing and outputting multi-scale fusion features through a gating mechanism; the method comprises the following steps: constructing attention branches for different gene markers, generating a marker specificity embedding vector by utilizing a gating attention mechanism, and realizing feature interaction modeling between markers through a Transform structure; and finally, outputting the rearrangement state probability of each gene marker through a classifier. According to the method, the accuracy and interpretability of gene rearrangement state prediction are improved through multi-scale feature modeling and a marker specific attention mechanism, and the method has the advantages of being efficient, non-invasive and suitable for clinical auxiliary diagnosis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image analysis and artificial intelligence assisted diagnosis, and in particular to a gene rearrangement state prediction method, device and apparatus based on pathological images. BACKGROUND

[0002] In the existing field of lymphoma molecular diagnosis, the MYC, BCL2 and BCL6 gene rearrangement states of diffuse large B-cell lymphoma (DLBCL) patients are usually detected by fluorescence in situ hybridization (FISH) technology. Although this method has high accuracy, it has the disadvantages of high cost, complex operation and long time-consuming, which limits its wide application in primary medical institutions. In addition, FISH detection requires special probes and equipment, and can only provide limited histological context information, making it difficult to achieve comprehensive pathological evaluation. In recent years, with the development of digital pathology and artificial intelligence technology, some studies have attempted to extract potential molecular features from conventional HE-stained section images through deep learning models, but most of these methods still have limitations in feature expression ability, model interpretability and multi-gene joint prediction. Existing models often only focus on single gene markers, lack effective modeling mechanisms for multi-marker collaborative prediction, and also lack systematic design in integrating clinical information, improving model robustness and clinical applicability. Therefore, there is an urgent need for an AI system that can efficiently, accurately and interpretably predict multiple gene rearrangement states, in order to reduce the dependence on FISH detection and improve the accessibility and clinical applicability of DLBCL molecular typing. SUMMARY

[0003] Therefore, in order to solve the technical problem of the lack of an efficient, accurate and non-invasive method for predicting gene rearrangement states in pathological images in the prior art, resulting in low clinical diagnosis efficiency and high misdiagnosis rate, the present application proposes a gene rearrangement state prediction method, device and apparatus based on pathological images.

[0004] The application protects a pathological image-based gene rearrangement state prediction method, which comprises the following steps: obtaining a HE-stained pathological section image, and performing multi-scale image block processing on the image to obtain an image block set under different magnifications; using a pre-trained visual coding model to extract features of each scale image block to obtain an initial image feature vector, and performing unified dimension processing on different scale features through linear transformation and an activation function; constructing a multi-expert structure, independently modeling different scale image features, and selectively fusing features output by the expert network through a gating mechanism to form a multi-scale fused feature representation; constructing an attention branch for different gene markers, weighting and aggregating image blocks through a gated attention mechanism to generate marker-specific embedding vectors, and realizing feature interaction modeling between markers through a Transformer structure; based on the modeling results, each gene marker is classified and predicted respectively to output the corresponding gene rearrangement state probability.

[0005] Further, the HE-stained pathological section image is obtained, and multi-scale image block processing is performed on the image to obtain an image block set under different magnifications; a pre-trained visual coding model is used to extract features of each scale image block to obtain an initial image feature vector, and unified dimension processing is performed on different scale features through linear transformation and an activation function, which includes: dividing the entire HE-stained section image into first image blocks at a magnification of 4 to 6 (e.g., 5), dividing the entire HE-stained section image into second image blocks at a magnification of 15 to 25 (e.g., 20), and aligning the first image blocks and the second image blocks in space; the first image and the second image blocks are respectively input into a pre-trained general visual encoder to extract scale-specific initial feature vectors, and first initial feature vectors based on the first image blocks and second initial feature vectors based on the second image blocks are obtained; the first initial feature vectors and the second initial feature vectors are respectively subjected to linear transformation and activation function processing to obtain embedding features of a unified dimension for multi-scale modeling.

[0006] Further, the multi-expert structure is constructed, different scale image features are independently modeled, and features output by the expert network are selectively fused through a gating mechanism to form a multi-scale fused feature representation, which includes: constructing multiple scale-specific expert networks, wherein the multiple expert networks focus on processing image features of different magnifications; for image features of each scale, the corresponding expert network is independently modeled to output scale-specific feature representation; the weighting coefficients of each expert network are dynamically calculated according to the features of the input image, and the gating mechanism of the entropy regularization loss term is introduced to weight and fuse the features output by the expert network; the feature vectors output by the different scale expert networks are spliced to form a multi-scale fused feature representation containing macroscopic structures and microscopic details.

[0007] Further, the attention branch is constructed for different genetic markers respectively, and the image blocks are weighted and aggregated through the gating attention mechanism to generate the marker-specific embedding vector, including: for each molecular marker MYC, BCL2 and BCL6, an independent gating attention branch is constructed respectively; for each marker, the attention weight of the image block is calculated through the gating attention score function, wherein the attention weight is determined through the feature vector of the image block and the gating score function corresponding to the marker; the image block features are weighted and summed according to the attention weight to generate the marker-specific embedding vector.

[0008] Further, after generating the marker-specific embedding vector, further including: splicing the marker-specific embedding vectors respectively generated for the BCL2, MYC and BCL6 genetic markers; modeling the spliced marker features through stacked Transformer layers, and outputting the updated marker feature representation for subsequent gene rearrangement state after the Transformer layer processing.

[0009] Further, based on the modeling result, each genetic marker is classified and predicted respectively, and the corresponding gene rearrangement state probability is output, including: in the classification output layer, an independent fully connected classifier is configured for each genetic marker, the fully connected classifier receives the corresponding marker-specific embedding vector as input, and outputs the rearrangement state probability value of the corresponding marker through the activation function, the probability value ranges from 0 to 1; in the model prediction process, the attention weight corresponding to each marker is extracted, the key area used for prediction is determined based on the attention weight, and the key area is displayed through a visual method.

[0010] Further, it further includes integrating clinical data to construct a multi-modal prediction model, specifically including: collecting the clinical data of the patient, inputting the standardized clinical data into a multi-layer perception to generate a low-dimensional clinical feature vector; splicing the clinical feature vector and the marker-specific embedding vector to form a multi-modal fusion feature; processing the fusion feature through an independent classifier to output the rearrangement state probability of each genetic marker.

[0011] Further, the multi-modal prediction model is trained in stages: in the first stage, the image branch model is trained and its parameters are frozen; in the second stage, the clinical data branch and the classifier head are trained separately, and are optimized through a binary cross-entropy loss function and an optimizer.

[0012] The application protects a kind of gene rearrangement state prediction device based on pathological image, device includes: image processing module, for obtaining HE dyed pathological section image, and carry out multi-scale image block processing to image, obtain the image block set under different magnification;Using pre-trained visual coding model carries out feature extraction to each scale image block, obtains initial image feature vector, and carries out uniform dimension processing to different scale features by linear transformation and activation function;Multi-scale fusion module is used to construct multi-expert structure, different scale image features are modeled independently respectively, and the features output by expert network are selectively fused by gate mechanism, and multi-scale fusion feature representation is formed;Label feature modeling module is used to construct attention branch for different gene labels respectively, image block is weighted aggregation by gate attention mechanism, generates label specificity embedding vector, and realizes the feature interaction modeling between labels by the structure of transformer;Classification prediction module is used to carry out classification prediction to each gene label respectively based on modeling result, and the corresponding gene rearrangement state probability is output.

[0013] The application protects an electronic device, comprising: a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor communicates with the memory through the bus, and the machine readable instructions are executed by the processor to perform the steps of the gene rearrangement state prediction method based on pathological image according to any one of claims 1 to 8.

[0014] The application protects a pathological image-based gene rearrangement state prediction method, which extracts and fuses multi-scale features from HE-stained pathological section images through multi-scale image block processing combined with a deep learning model, and then realizes classification and prediction of the rearrangement state of multiple gene markers. The method first performs multi-scale block processing on the image, and uses a pre-trained visual coding model to extract image features at each scale, and unifies the feature dimension through linear transformation and activation function, thereby enhancing the adaptability and expression ability of the model to different tissue structures. Further, a multi-expert structure is constructed to model different scale features independently, and the expert output is selectively fused through a gating mechanism, which improves the utilization efficiency and robustness of the model for multi-scale image information. In addition, the method introduces attention branches for different gene markers, which aggregate image blocks through a gated attention mechanism to generate marker-specific embedding vectors, and model the interaction between markers through a Transformer structure, thereby improving the accuracy and interpretability of classification. In specific implementation, the multi-expert network processes image features of different magnifications, and fuses the features through a gating mechanism that dynamically calculates a weighting coefficient and combines an entropy regularization loss term, to finally form a multi-scale fused feature representation containing macroscopic structures and microscopic details, effectively improving the modeling capability of the model for complex tissue structures in pathological images. Further, independent gated attention branches are constructed for MYC, BCL2, BCL6 and other molecular markers, the attention weights of image blocks are calculated through a scoring function, and marker-specific embedding vectors are generated accordingly, so that the model can focus on key areas related to specific gene markers, thereby improving the accuracy and biological interpretability of prediction. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings are included to provide a further understanding of the application, and constitute a part of the specification, illustrate embodiments of the application, and are used to explain the application, and do not constitute a limitation on the application. In the drawings:

[0016] Figure 1 A flowchart of a multi-scale fused feature representation gene marker classification and prediction method provided by an embodiment of the application;

[0017] Figure 2 A flowchart of a multi-scale image feature extraction method provided by an embodiment of the application;

[0018] Figure 3 A flowchart of a multi-scale feature fusion neural network provided by an embodiment of the application;

[0019] Figure 4 A flowchart of a gene marker feature modeling method provided by an embodiment of the application;

[0020] Figure 5A flowchart of a marker-specific embedded vector generation method provided by an embodiment of the present application;

[0021] Figure 6 A flowchart of gene marker classification prediction provided by an embodiment of the present application;

[0022] Figure 7 A flowchart of gene marker rearrangement state prediction provided by an embodiment of the present application;

[0023] Figure 8 A schematic diagram of a device for gene marker rearrangement state prediction provided by an embodiment of the present application;

[0024] Figure 9 An electronic device schematic diagram for gene marker rearrangement state prediction provided by an embodiment of the present application;

[0025] Figure 10 A performance evaluation diagram of an internal test set (Beijing team) of the system of the present application;

[0026] Figure 11 A verification result diagram of an external test set 1 (Langfang team) of the system of the present application;

[0027] Figure 12 A verification result diagram of an external test set 2 (Shanxi team);

[0028] Figure 13 A verification result diagram of an external test set 3 (Shenzhen team) of the system of the present application;

[0029] Figure 14 A multi-modal model ROC curve (Beijing team) diagram of the system of the present application;

[0030] Figure 15 A multi-modal model ROC curve (Shanxi team) diagram of the system of the present application. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, every other embodiment obtained by a person skilled in the art without creative work falls within the scope of the present application.

[0032] It is found through research that the method for predicting gene rearrangement state based on HE stained pathological images in the prior art has many limitations. Traditional FISH detection means is high in cost and long in cycle, and is difficult to meet the needs of rapid clinical diagnosis. Some AI models have attempted to mine morphological features related to gene mutation or rearrangement from HE images, but their model structures are relatively simple, and they are mostly focused on single gene marker prediction, lacking the ability to model the potential relationship between multiple gene markers. In addition, existing methods often use simple splicing or averaging strategies when processing multi-scale image features, failing to effectively fuse key information at different scales, resulting in limited model perception of microscopic and macroscopic features of tissue structure. At the same time, the model lacks an explainable mechanism, making it difficult to provide doctors with reliable decision-making basis, limiting its practical application value in the clinic.

[0033] Based on this, please refer to Figure 1 The embodiment of the present application provides a gene rearrangement state prediction method based on pathological images, comprising: acquiring an HE stained pathological section image, and performing multi-scale image block processing on the image to obtain a set of image blocks at different magnifications; using a pre-trained visual coding model to extract features of each scale image block to obtain an initial image feature vector, and performing uniform dimension processing on features of different scales through linear transformation and activation function; constructing a multi-expert structure, independently modeling different scale image features, and selectively fusing features output by the expert network through a gating mechanism to form a multi-scale fusion feature representation; constructing an attention branch for different gene markers, weighting and aggregating image blocks through a gated attention mechanism to generate marker-specific embedding vectors, and realizing feature interaction modeling between markers through a Transformer structure; based on the modeling result, each gene marker is classified and predicted respectively to output the corresponding gene rearrangement state probability.

[0034] The embodiment of the present application provides a gene rearrangement state prediction method based on pathological images, which effectively improves the perception ability of the model to the microscopic cell morphology and macroscopic tissue arrangement in the tissue structure through multi-scale image block processing and uniform feature embedding; introduces a multi-expert structure and a gating mechanism to enhance the professional modeling and dynamic fusion ability of the model to features of different scales, and improve the diversity and robustness of feature expression; by constructing a marker-specific attention branch and a Transformer interaction modeling, joint prediction of multiple gene markers is realized, and the ability to capture the potential relationship between genes is enhanced; finally, the rearrangement probability of each gene marker is output through an independent classification head, supporting multi-label joint prediction, which significantly improves the prediction accuracy and clinical applicability. In addition, the gated attention mechanism also supports the visualization of key areas, enhancing the explainability and clinical credibility of the model.

[0035] In step S101, the system obtains a HE-stained pathological section image, and performs multi-scale image patching processing on the image to obtain a set of image blocks at different magnifications; then a pre-trained visual encoding model is used to extract features from each scale image block to obtain an initial image feature vector, and linear transformation and activation function are used to process the features of different scales to the same dimension.

[0036] Specifically, S101 includes the following steps:

[0037] S1011, divide the whole HE-stained section image into first image blocks at 5 times, and divide the whole HE-stained section image into second image blocks at 20 times, and align the first image blocks and the second image blocks in space.

[0038] In this embodiment, the input HE-stained pathological section image is a whole field digital section image (Whole Slide Image, WSI) of a diffuse large B-cell lymphoma (DLBCL) patient. The system uses an image processing tool (such as CLAM) to systematically divide the whole image into non-overlapping small image blocks (patches) at two magnifications of 5x and 20x. Among them, the size of the 5x image block is 224x224 pixels, each 5x image block corresponds to a 4x4 20x image block grid, ensuring that the image blocks of different scales are aligned in space, thereby preserving the structural information of the image.

[0039] S1012, respectively input the first image blocks and the second image blocks into a pre-trained general visual encoder to extract scale-specific initial feature vectors, and obtain a first initial feature vector based on the first image blocks and a second initial feature vector based on the second image blocks.

[0040] After image patching, the system inputs the 5x and 20x image blocks into a pre-trained general visual encoder (such as the UNI model) to extract scale-specific initial feature vectors. Specifically, the 5x image block is processed by the encoder to output a first initial feature vector The 20x image block outputs a second initial feature vector The encoder is pre-trained on a large-scale natural image dataset and has good general feature extraction capability, which can effectively capture the morphological features of the image.

[0041] S1013, linearly transform and activate the first initial feature vector and the second initial feature vector, respectively, to obtain embedded features of the same dimension for multi-scale modeling.

[0042] In order to enable the image block features of different scales to be effectively fused in subsequent modeling, the system performs uniform dimension processing on the extracted initial feature vectors. Specifically, a linear transformation and a ReLU activation function are respectively applied to the feature vectors of each image block to map them to a uniform feature space. For example, through linear transformation W e R^(1536x256) and ReLU activation function processing, the original 1536-dimensional features are compressed to 256-dimensional features while retaining the nonlinear expression capability. The finally output embedded features h_i^(5x) and h_j^(20x) will serve as the basis input for subsequent multi-scale modeling, ensuring that different scale features are effectively fused under the premise of consistent dimensions.

[0043] In step S102, the system constructs a multi-expert structure, independently models different scale image features, and selectively fuses the features output by the expert network through a gating mechanism to form a multi-scale fused feature representation, which is Figure 3 The construction process of the multi-scale fused feature representation is given. Specifically, S102 includes the following steps:

[0044] S1021, a plurality of scale-specific expert networks are constructed, wherein the plurality of expert networks focus on processing image features of different multiples.

[0045] In this embodiment, the system constructs a multi-expert structure (Mixture-of-Experts, MoE) containing 8 expert networks for processing image features of 5x and 20x scales respectively. Among them, 4 expert networks are specifically for processing features of 5x scale image blocks, and the other 4 expert networks are specifically for processing features of 20x scale image blocks. Each expert network is an independent deep neural network module with an independent parameter space, which can model the image features of its corresponding scale.

[0046] S1022, for image features of each scale, independently model by the corresponding expert network, and output scale-specific feature representation.

[0047] After the construction of the expert network is completed, the system inputs the uniform dimension embedded features h_i^(5x) and h_j^(20x) output in S1013 into the corresponding expert network respectively. Each expert network performs nonlinear transformation and feature abstraction on the input image features, and outputs scale-specific feature representation. For example, 8 scale-specific expert networks are constructed, 4 of which focus on 5x scale and 4 of which focus on 20x scale. Each expert independently models the image features of its corresponding scale: E k (k=1,…,8)

[0048]

[0049] Through this step, each expert network focuses on feature modeling at its corresponding scale, improving the model's ability to express multi-scale information.

[0050] S1023. Calculate the weighting coefficients of each expert network dynamically based on the features of the input image, and introduce a gating mechanism of entropy regularization loss term to perform weighted fusion of the features output by the expert networks.

[0051] To avoid redundant learning among expert networks and improve model efficiency, the system introduces a gating mechanism to dynamically and selectively activate expert networks. Specifically, the gating mechanism calculates the activation weights of each expert network based on the input image features and performs a weighted fusion of the outputs of each expert. The weight calculation process is implemented through a lightweight gating network, whose output is the normalized activation probability of each expert. Furthermore, during training, the system introduces a gated entropy regularization loss term, L_gate, to encourage a balanced distribution among expert networks and prevent some experts from being overused or ignored.

[0052] S1024. The feature vectors output by expert networks at different scales are concatenated to form a multi-scale fusion feature representation that includes both macroscopic structure and microscopic details.

[0053] After weighted fusion of the expert network outputs, the system concatenates the feature vectors from expert networks at different scales to generate a multi-scale fused feature representation. For example, for a certain image region, the system concatenates its corresponding 5× expert output with the 20× expert output to form the final fused feature.

[0054]

[0055] This fusion feature contains both macroscopic tissue structure information (5× scale) and cellular level detail information (20× scale), providing rich semantic expression for subsequent marker-specific modeling.

[0056] In step S103, the system constructs attention branches for different gene markers, performs weighted aggregation of image patches through a gated attention mechanism to generate marker-specific embedding vectors, and models feature interactions between markers using a Transformer structure. Figure 4 The specific process of feature interaction modeling is given.

[0057] Specifically, S103 includes the following steps:

[0058] S1031. For each molecular marker MYC, BCL2, and BCL6, construct an independent gated attention branch.

[0059] In this embodiment, the system constructs an independent gated attention branch for each target gene marker (i.e., MYC, BCL2, and BCL6) respectively. Each attention branch is composed of a gating score function and a weighted aggregation mechanism, which is used to identify the image region most relevant to the rearrangement status of this gene. Through this design, the system can achieve specific modeling for different gene markers, and improve the independence and accuracy of the model's prediction for each marker.

[0060] In particular, in one embodiment, Figure 5 A marker-specific embedding vector generation method is given, including:

[0061] S10311、For each marker, the attention weight of the image block is calculated by the gated attention score function, wherein the attention weight is determined by the feature vector of the image block and the gating score function corresponding to the marker.

[0062] In each gated attention branch, the system first processes the multi-scale fusion features E_n∈R^(1×2048) output by S1024, and calculates the attention weight of each image block for the current gene marker by the gating score function g_m(·)(m∈{BCL2,MYC,BCL6}). Specifically, the attention weight a_(m,n) is calculated by the following formula:

[0063]

[0064] where g m (·) is a gating function composed of a fully connected layer, which is used to dynamically adjust the weight of the image block according to the semantic information of the current marker m. This mechanism enables the model to automatically identify the image region most relevant to the specific gene rearrangement status, thereby improving the interpretability of the prediction.

[0065] S10312、According to the attention weight, the image block features are weighted and summed to generate a marker-specific embedding vector.

[0066] After calculating the attention weight of each image block, the system performs weighted sum on the multi-scale fusion features E_n and the corresponding attention weight a_(m,n) of each image block to generate a marker-specific embedding vector z_m:

[0067] z_m=∑_n a_(m,n)E_n∈R^(1×2048)

[0068] This embedding vector z_m reflects the overall feature representation in the current image that is most relevant to the gene marker m, and serves as the basis input for subsequent modeling of inter-marker interaction.

[0069] S1032、The marker-specific embedding vectors generated for BCL2, MYC, and BCL6 gene markers are spliced.

[0070] After obtaining the marker-specific embedding vectors z_BCL2, z_MYC, and z_BCL6 for the three gene markers, the system concatenates them to form a joint feature representation:

[0071] z_concat=[z_BCL2,z_MYC,z_BCL6]∈R^(1×6144)

[0072] This concatenation operation integrates the independent feature representations of the three tags into a unified high-dimensional feature space, providing a foundation for subsequent modeling of interactions between tags.

[0073] S1033. The spliced ​​label features are modeled by stacked Transformer layers. After processing by the Transformer layers, the updated label feature representation is output for subsequent gene rearrangement states.

[0074] The system inputs the concatenated joint feature z_concat into stacked Transformer layers for cross-label feature interaction modeling. The Transformer layers consist of a multi-head self-attention mechanism and a feedforward network, effectively capturing the complex dependencies between BCL2, MYC, and BCL6. After processing through multiple Transformer layers, the system outputs the updated labeled feature representation:

[0075]

[0076] Here, T(·) represents the mapping function of the Transformer backbone network. The updated feature representation not only preserves the specific information of each label, but also incorporates the interaction relationships between labels, providing richer semantic information for subsequent classification prediction.

[0077] In step S104, based on the modeling results, the system performs classification prediction for each gene marker and outputs the corresponding gene rearrangement state probability. The gene rearrangement state probability is then... Figure 6 The flowchart implementation,

[0078] Specifically, S104 includes the following steps:

[0079] S1041. In the classification output layer, an independent fully connected classifier is configured for each gene marker. The fully connected classifier receives the corresponding marker-specific embedding vector as input and outputs the rearrangement probability value of the corresponding marker through an activation function. The probability value ranges from 0 to 1.

[0080] In this embodiment, the system configures an independent classification head module for each target gene marker (BCL2, MYC, BCL6). Each classification head is composed of a fully connected layer and a Sigmoid activation function, and receives the updated marker feature representation z'_m output by the Transformer layer in S1033 as input. Specifically, the classification head calculates the rearrangement state prediction probability of marker m by the following way:

[0081]

[0082] where φm(·) represents the fully connected classification head corresponding to marker m, σ(·) is the Sigmoid activation function for mapping the output to the interval [0, 1], and represents the probability of rearrangement of the marker. This design makes the prediction process of each marker independent of each other, while obtaining cross-marker context information support through the feature interaction modeling of the Transformer layer in S103, thereby improving the accuracy and robustness of the prediction.

[0083] In the model prediction process, S1042 extracts the attention weight corresponding to each marker, determines the key region used for prediction based on the attention weight, and displays the key region through a visual way.

[0084] To further improve the explainability of the model, the system retains the attention weight α_(m,n) calculated in S10311 during the prediction process. These weights reflect the contribution of each image block n to the prediction result of gene marker m. The system maps the attention weight back to the spatial position of the original pathology image to generate a heat map (Heatmap) that highlights the image area that has the greatest impact on the prediction result. For example, for the BCL2 marker, the system visualizes its corresponding attention heat map, allowing pathologists to intuitively understand the model's decision basis and assisting them in manual review and diagnosis decision-making.

[0085] This visualization mechanism not only enhances the transparency of the model, but also improves the trust of doctors in the AI prediction results, providing important support for the landing application of the model in clinical scenarios.

[0086] On the basis of the above-mentioned embodiments, to further improve the accuracy and clinical applicability of gene rearrangement state prediction, the present application also introduces a multi-modal fusion strategy, which combines pathology image features with patient clinical data to construct a multi-modal prediction model. This strategy does not change the image backbone modeling process, but enhances the model's ability to distinguish complex cases by fusing structured clinical information.

[0087] Specifically, after step S104, the process of predicting the rearrangement state of the gene marker is performed, which is described in detail with reference to FIG. 10B. Figure 7, the system further performs the following operations:

[0088] S1051, collect clinical data of the patient, and input the standardized clinical data into a multi-layer perception to generate a low-dimensional clinical feature vector.

[0089] The system collects structured clinical data of DLBCL patients, including but not limited to: age, gender, Ann Arbor stage, IPI score, tumor site, cell origin (COO) subtype, and immunohistochemical (IHC) results (such as Ki-67, CD10, MUM1, BCL2, MYC, etc.). All clinical variables are first standardized to eliminate the influence of different dimensions. Subsequently, the standardized clinical data is input into a multi-layer perception (MLP) composed of several fully connected layers and ReLU activation functions, which is used to map the original clinical information to a low-dimensional, compact clinical feature vector z clinic ∈ R^(1×1024) for subsequent fusion modeling.

[0090] S1052, splicing the clinical feature vector and the marker-specific embedding vector to form a multi-modal fusion feature.

[0091] To realize the complementary fusion of image features and clinical features, the system splices the marker-specific embedding vector z m generated in S10312 and the clinical feature vector z clinic extracted in S1051 to form a multi-modal fusion feature representation for each gene marker m:

[0092] z m^*=Concat(z_m,z_clinic)∈R^(1×3072)

[0093] Where Concat(·) represents the vector splicing operation. This fusion feature not only retains the morphological information related to gene rearrangement in the image, but also introduces the clinical background information of the patient, thereby improving the generalization ability of the model for complex cases.

[0094] S1053, processing the fusion feature through an independent classifier to output the rearrangement state probability of each gene marker.

[0095] The system configures a new independent classification head for each gene marker m, which receives the multi-modal fusion feature as input and outputs the final rearrangement state prediction probability through fully connected layers and Sigmoid activation functions.

[0096] This classification head exists in parallel with the image-based classification head in S1041, and whether to enable multi-modal input can be selected according to actual deployment requirements.

[0097] In addition, to ensure the stability of the image feature extraction part, during the training of the multi-modal model, the image backbone network (i.e., the image modeling module constructed by S101 to S103) parameters remain in a frozen state, and only the clinical branch and the new classification head are trained. The training adopts the same binary cross-entropy loss function as the image model, and combines the early stopping mechanism to prevent overfitting.

[0098] In summary, by introducing the multi-modal fusion mechanism of clinical data, the application realizes high-precision prediction of the BCL2, MYC and BCL6 gene rearrangement state of DLBCL patients without relying on FISH detection, and further improves the applicability and reliability of the model in clinical practice.

[0099] Please refer to Figure 8 , Figure 8 The structure of a gene rearrangement state prediction device based on pathological images provided by the embodiment of the application is shown in FIG. 1. Figure 8 As shown in the figure, the prediction device 200 comprises:

[0100] An image processing module 210 is configured to acquire a HE-stained pathological section image, perform multi-scale image block processing on the image, and obtain a set of image blocks at different magnifications; a pre-trained visual coding model is used to extract features from each scale image block to obtain an initial image feature vector, and linear transformation and an activation function are used to uniformly process different scale features to the same dimension.

[0101] A multi-scale fusion module 220 is configured to construct a multi-expert structure, model different scale image features independently, and selectively fuse the features output by the expert network through a gating mechanism to form a multi-scale fusion feature representation.

[0102] A marker feature modeling module 230 is configured to construct an attention branch for each gene marker, aggregate the image blocks through a gated attention mechanism, generate a marker-specific embedding vector, and realize feature interaction modeling between markers through a Transformer structure.

[0103] A classification prediction module 240 is configured to perform classification prediction on each gene marker based on the modeling results, and output the corresponding gene rearrangement state probability.

[0104] Further, when the image processing module 210 is used to acquire a HE-stained pathological section image, perform multi-scale image block processing on the image, and obtain a set of image blocks at different magnifications; a pre-trained visual coding model is used to extract features from each scale image block to obtain an initial image feature vector, and linear transformation and an activation function are used to uniformly process different scale features to the same dimension, the image processing module 210 is configured to:

[0105] dividing the whole HE-stained slide image into first image patches at 5x magnification, dividing the whole HE-stained slide image into second image patches at 20x magnification, and aligning the first image patches and the second image patches in space;

[0106] inputting the first image patches and the second image patches into a pre-trained general visual encoder respectively, extracting scale-specific initial feature vectors, obtaining first initial feature vectors based on the first image patches and second initial feature vectors based on the second image patches;

[0107] performing linear transformation and activation function processing on the first initial feature vectors and the second initial feature vectors respectively to obtain embedded features of a unified dimension for multi-scale modeling.

[0108] Further, when the multi-scale fusion module 220 is used for constructing a multi-expert structure, independently modeling different scale image features, and selectively fusing features output by expert networks through a gating mechanism to form multi-scale fusion feature representation, the multi-scale fusion module 220 is configured to:

[0109] constructing a plurality of scale-specific expert networks, wherein the plurality of expert networks focus on processing image features of different magnifications;

[0110] independently modeling image features of each scale by corresponding expert networks, and outputting scale-specific feature representation;

[0111] dynamically calculating weighting coefficients of the expert networks according to features of the input image, and introducing a gating mechanism of an entropy regularization loss term to weight and fuse features output by the expert networks;

[0112] splicing feature vectors output by different scale expert networks to form multi-scale fusion feature representation containing macroscopic structure and microscopic details.

[0113] Further, when the marker feature modeling module 230 is used for constructing attention branches for different gene markers respectively, weighting and aggregating image patches through a gated attention mechanism to generate marker-specific embedded vectors, and realizing feature interaction modeling between markers through a Transformer structure, the marker feature modeling module 230 is configured to:

[0114] constructing an independent gated attention branch for each molecular marker MYC, BCL2, and BCL6;

[0115] for each marker, calculating attention weights of the image patches through a gated attention scoring function, wherein the attention weights are determined through calculation of feature vectors of the image patches and the gated scoring function corresponding to the marker;

[0116] The image block features are weighted and summed according to the attention weights to generate a label-specific embedding vector;

[0117] The label-specific embedding vectors generated for the BCL2, MYC and BCL6 gene labels are spliced;

[0118] The spliced label features are modeled through stacked Transformer layers, and after being processed by the Transformer layers, updated label feature representations for subsequent gene rearrangement states are output.

[0119] Further, the classification prediction module 240 is configured to:

[0120] In the classification output layer, an independent fully connected classifier is configured for each gene label, which receives the corresponding label-specific embedding vector as input and outputs the rearrangement state probability value of the corresponding label through an activation function, with the probability value ranging from 0 to 1.

[0121] During model prediction, the attention weights corresponding to each label are extracted, the key regions used for prediction are determined based on the attention weights, and the key regions are displayed in a visual manner.

[0122] Further, the prediction device 200 further comprises a multi-modal fusion module (not shown in the figure), which is configured to integrate clinical data to construct a multi-modal prediction model, specifically comprising:

[0123] The clinical data of the patient is collected, and after being standardized, the clinical data is input into a multi-layer perception machine to generate a low-dimensional clinical feature vector;

[0124] The clinical feature vector and the label-specific embedding vector are spliced to form a multi-modal fusion feature;

[0125] The fusion feature is processed by an independent classifier to output the rearrangement state probability of each gene label;

[0126] The multi-modal prediction model is trained in stages: in the first stage, the image branch model is trained and its parameters are frozen; in the second stage, the clinical data branch and the classifier head are trained separately, and are optimized through a binary cross-entropy loss function and an optimizer.

[0127] The gene rearrangement state prediction device based on pathological images provided by the embodiment of the application can obtain a HE-stained pathological section image and perform multi-scale image block processing, extract image features by using a pre-trained visual coding model and unify dimensions, construct a multi-expert structure to independently model different scale image features and fuse them through a gating mechanism to form multi-scale fusion feature representation, construct an attention branch for different gene markers, generate marker-specific embedding vectors by weighting and aggregating image blocks through a gated attention mechanism, and realize inter-marker feature interaction modeling through a Transformer structure, and finally classify and predict each gene marker based on the modeling results to output corresponding gene rearrangement state probabilities. The method improves the accuracy and interpretability of gene rearrangement state prediction through multi-scale modeling, a gated attention mechanism and a Transformer structure, can integrate clinical data to realize multi-modal prediction, and has good clinical application prospects.

[0128] The prediction device 200 uses the HE2FISH model determined by the above method, and 6 images in the test set are used to verify the performance of the HE2FISH model in different independent queues. Figures 10 to 15 The prediction device 200 uses the HE2FISH model determined by the above method, and 6 images in the test set are used to verify the performance of the HE2FISH model in different independent queues. Figure 10 The prediction results of the model in the internal test set (Beijing queue) are shown, and the robustness is evaluated by five-fold cross-validation. Figures 11 to 13 Corresponding to the three external test sets of Langfang, Shanxi and Shenzhen respectively, the generalization ability of the model in cross-center data is verified to ensure its applicability in different regions and medical institutions. Figure 14 The prediction device 200 uses the HE2FISH model determined by the above method, and 6 images in the test set are used to verify the performance of the HE2FISH model in different independent queues. Figure 15 The ROC curve of the multi-modal HE2FISH model (fusing clinical information) is mainly shown to prove the performance improvement compared with the pure image model, and further support the clinical value of the multi-modal design.

[0129] These charts intuitively present the prediction performance of the model through visualization, including key indicators such as AUROC, AUPRC, and the average AUC of the multi-center external data set is 0.867. Overall, the prediction device using the model not only verifies the accuracy and robustness of the model, but also provides empirical evidence for its promotion in clinical practice.

[0130] Please refer to Figure 9 , Figure 9 is a structural schematic diagram of an electronic device provided by the embodiment of the application. As shown in Figure 2 , the electronic device 300 includes a processor 310, a memory 320 and a bus 330.

[0131] The memory 320 stores machine readable instructions executable by the processor 310, when the electronic device 300 is running, the processor 310 and the memory 320 communicate through the bus 330, the machine readable instructions are executed by the processor 310, can execute the steps of the method embodiment as described above Figure 1 The steps of the gene rearrangement state prediction method based on pathological images in the method embodiment are not repeated here.

[0132] The computer readable storage medium provided in the embodiment of the present application stores a computer program, and the computer program can execute the steps of the gene rearrangement state prediction method based on pathological images in the method embodiment when the processor runs. Figure 1 The steps of the gene rearrangement state prediction method based on pathological images in the method embodiment are not repeated here.

[0133] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here.

[0134] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented by other ways. The device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interface, device or unit, which can be electrical, mechanical or other forms.

[0135] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0136] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0137] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application essentially or the parts of the prior art that make contributions or parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0138] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit them. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features. The modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for predicting gene rearrangement status based on pathological images, characterized in that, The prediction method includes: HE-stained pathological slide images were acquired, and the images were processed into multi-scale image blocks to obtain image block sets at different magnifications. A pre-trained visual coding model was used to extract features from the image blocks at each scale to obtain initial image feature vectors. The features at different scales were then processed to unify the dimensions through linear transformation and activation functions. A multi-expert structure is constructed to independently model image features at different scales, and the features output by the expert network are selectively fused through a gating mechanism to form a multi-scale fused feature representation. Attention branches are constructed for different gene markers, and image patches are weighted and aggregated through a gated attention mechanism to generate marker-specific embedding vectors. The feature interaction model between markers is realized through the Transformer structure. Based on the interactive modeling results, each gene marker is classified and predicted, and the corresponding gene rearrangement state probability is output.

2. The gene rearrangement state prediction method based on pathological images according to claim 1, characterized in that, The process involves acquiring HE-stained pathological slide images, performing multi-scale image block processing to obtain image block sets at different magnifications, and using a pre-trained visual coding model to extract features from image blocks at each scale to obtain initial image feature vectors. Furthermore, it includes unifying the dimensionality of features at different scales through linear transformation and activation functions. The entire HE-stained slide image is divided into a first image block at a magnification of 5, and the entire HE-stained slide image is divided into a second image block at a magnification of 15 to 25 (e.g., 20), and the first and second image blocks are spatially aligned. The first image and the second image blocks are respectively input into a pre-trained general visual encoder to extract scale-specific initial feature vectors, thereby obtaining a first initial feature vector based on the first image block and a second initial feature vector based on the second image block; Linear transformation and activation function processing are performed on the first and second initial feature vectors respectively to obtain a unified dimension of embedded features for multi-scale modeling.

3. The gene rearrangement state prediction method based on pathological images according to claim 1, characterized in that, The construction of a multi-expert structure, which independently models image features at different scales and selectively fuses the features output by the expert networks through a gating mechanism to form a multi-scale fused feature representation, includes: Construct multiple scale-specific expert networks, where each network focuses on processing image features at different magnifications; For each scale of image features, a corresponding expert network independently models and outputs scale-specific feature representations; The weighting coefficients of each expert network are dynamically calculated based on the features of the input image, and a gating mechanism of entropy regularization loss term is introduced to perform weighted fusion of the features output by the expert networks. The feature vectors output by expert networks at different scales are concatenated to form a multi-scale fused feature representation.

4. The gene rearrangement state prediction method based on pathological images according to claim 1, characterized in that, The process of constructing attention branches for different gene markers and weighting and aggregating image patches using a gated attention mechanism to generate marker-specific embedding vectors includes: For each molecular marker MYC, BCL2, and BCL6, an independent gated attention branch was constructed. For each marker, the attention weight of the image patch is calculated using a gated attention scoring function, where the attention weight is determined by the feature vector of the image patch and the gated scoring function corresponding to the marker. The image patch features are weighted and summed according to the attention weights to generate a label-specific embedding vector.

5. The gene rearrangement state prediction method based on pathological images according to claim 4, characterized in that, After generating the marker-specific embedding vector, the process further includes: The marker-specific embedding vectors generated for the BCL2, MYC, and BCL6 gene markers were spliced ​​together. The concatenated label features are modeled by stacked Transformer layers. After processing by the Transformer layers, the updated label feature representation is output for subsequent gene rearrangement states.

6. The gene rearrangement state prediction method based on pathological images according to claim 1, characterized in that, Based on the modeling results, each gene marker is classified and predicted, and the corresponding gene rearrangement state probability is output, including: In the classification output layer, an independent fully connected classifier is configured for each gene marker. The fully connected classifier receives the corresponding marker-specific embedding vector as input and outputs the rearrangement probability value of the corresponding marker through an activation function. The probability value ranges from 0 to 1. During the model prediction process, the attention weights corresponding to each label are extracted, the key regions for prediction are determined based on the attention weights, and the key regions are displayed in a visual way.

7. The method for predicting gene rearrangement status based on pathological images according to claim 1, characterized in that, This also includes integrating clinical data to build multimodal predictive models, specifically including: Collect patients' clinical data, standardize the clinical data and input it into a multilayer perceptron to generate low-dimensional clinical feature vectors; The clinical feature vector is concatenated with the label-specific embedding vector to form a multimodal fusion feature; The fusion features are processed by an independent classifier, and the rearrangement probability of each gene marker is output.

8. The gene rearrangement state prediction method based on pathological images according to claim 7, characterized in that, The multimodal prediction model is trained in stages: The first stage involves training the image branching model and freezing its parameters. In the second stage, the clinical data branch and classifier head are trained separately and optimized using a binary cross-entropy loss function and an optimizer.

9. A gene rearrangement state prediction device based on pathological images, characterized in that, The device includes: The image processing module is used to acquire HE-stained pathological slide images and perform multi-scale image block processing to obtain image block sets at different magnifications. A pre-trained visual coding model is used to extract features from image blocks at each scale to obtain initial image feature vectors, and features at different scales are processed to unify the dimensions through linear transformation and activation functions. The multi-scale fusion module is used to construct a multi-expert structure, which independently models the image features at different scales and selectively fuses the features output by the expert network through a gating mechanism to form a multi-scale fused feature representation. The marker feature modeling module is used to construct attention branches for different gene markers, weighted aggregate image patches through a gated attention mechanism to generate marker-specific embedding vectors, and realize feature interaction modeling between markers through a Transformer structure; The classification prediction module is used to classify and predict each gene marker based on the modeling results, and output the corresponding gene rearrangement state probability.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the gene rearrangement state prediction method based on pathological images as described in any one of claims 1 to 8.

Images