A Chromosomal Abnormality Detection Method Based on a Multimodal Large Model

Abstract

This invention relates to the field of chromosome abnormality recognition technology, specifically to a chromosome abnormality detection method based on a multimodal large model. The method includes: constructing an image dataset and a text dataset; fusing image features and text features to obtain multimodal fusion features; assigning anomaly scores to image blocks belonging to band regions based on anomaly scoring rules formulated from the multimodal fusion features, and comprehensively processing the anomaly scores of all image blocks corresponding to the chromosome to determine whether the chromosome image is abnormal; and decoding and generating natural language text that meets the requirements of chromosome abnormality detection based on the multimodal fusion feature representation and anomaly scoring rules. This invention achieves accurate chromosome abnormality detection and band location positioning through multimodal fusion and dynamic text generation mechanisms, generating interpretable natural language text descriptions, and improving the practicality and interpretability of the detection results.

Description

Technical Field

[0001] This invention relates to the field of chromosome abnormality recognition technology, and in particular to a chromosome abnormality detection method based on a multimodal large model. Background Technology

[0002] Chromosomal structural abnormalities are caused by one or more defects or abnormal combinations in the structure of chromosomes. Common types of chromosomal structural abnormalities include deletions, duplications, translocations, and inversions. Because these changes manifest as subtle alterations in the internal structure of chromosomes, they are difficult to identify.

[0003] In traditional techniques, chromosome abnormality detection involves taking images of chromosomes during metaphase of cell division, which are then manually arranged by the examiner into a chromosome karyotype map. This map is used to determine whether there are any structural abnormalities in the chromosomes. The evaluation results of chromosome abnormality detection vary greatly among different examiners, and the process is also quite cumbersome and time-consuming.

[0004] In existing technologies, detection models trained on large datasets of labeled chromosome images can quickly detect chromosome abnormalities. However, the output of these models is often limited to simple classification labels, such as "normal" or "abnormal." Even when the location of the abnormality is given, the method used is to select the abnormal region, but if the abnormality occurs in a banded area of ​​the chromosome, human interpretation is still required.

[0005] Furthermore, the detection model is a "black box" system, and its output is difficult for clinical experts or patients to understand. It lacks semantic explanations or decision support information that can be used for clinical reference, and fails to provide reasoning processes or auxiliary judgment logic, which limits the readability and usability of the detection results in medical scenarios.

[0006] To address the problems in existing technologies, this invention provides a method for detecting chromosome abnormalities based on a multimodal large model. Summary of the Invention

[0007] The purpose of this invention is to provide a chromosomal abnormality detection method based on a multimodal large model, in order to solve the technical problems in existing chromosomal abnormality detection technologies, which lack intuitive explanations of abnormality categories and accurate localization of abnormality locations, resulting in low readability and usability of detection results in practical applications.

[0008] The technical solution of this invention is: a method for detecting chromosomal abnormalities based on a multimodal large model, comprising:

[0009] The acquired chromosome images were labeled to construct an image dataset; based on the karyotype analysis report, the abnormality categories of chromosomes and the descriptions of the abnormality at the band positions were extracted to construct a text dataset.

[0010] The image dataset and text dataset are encoded and represented separately to obtain image features and text features; the obtained image features and text features are then fused to obtain multimodal fusion features;

[0011] Chromosomal abnormality scoring rules are formulated based on multimodal fusion features. By using the abnormality scoring rules, image blocks that belong to the encoded strip regions are scored as abnormalities. The abnormality scores of all image blocks corresponding to the chromosome strip regions are combined to determine whether the chromosome image is abnormal.

[0012] The multimodal fusion feature representation and anomaly scoring rules are transformed into the initial input representation of the decoder for text generation. The anomaly scoring probability distribution of the strip region is transformed into a feature representation usable by the decoder. The input decoding part of the decoder is set as a multi-layer decoder built based on self-attention mechanism and cross-attention mechanism. Natural language text that meets the requirements of chromosome abnormality detection is generated through decoding.

[0013] Preferably, the process of acquiring multimodal fusion features includes:

[0014] The chromosome image is encoded by an image encoder. The chromosome is divided into image blocks, and each image block is encoded into a feature vector. The feature vectors corresponding to the image blocks are merged to construct the merged image features of the chromosome image.

[0015] Text features are obtained by encoding text descriptions using a pre-trained language model;

[0016] By using a self-attention mechanism to interact with image features and text features, multimodal fusion features corresponding to each data point are obtained.

[0017] Preferably, the self-attention mechanism is used to interact with image features and text features to obtain the content of multimodal features corresponding to each data point, including:

[0018] Obtain the query matrix Q representing the corresponding chromosome image based on image features. img :

[0019] ;

[0020] Obtain the key matrix K representing the corresponding abnormal text based on text features. txt Sum matrix V txt :

[0021] ,

[0022] ;

[0023] Among them, W q W k Wv These are the weight matrices for the Query, Key, and Value in the Transformer architecture, respectively. img F represents image features. txt Representing text features;

[0024] Based on chromosome image representation Q img and abnormal text representation K txt The interaction weight is calculated using the following formula:

[0025] ;

[0026] Based on the abnormal text vector representation V txt Multimodal fusion features are obtained by combining interaction weights. The multimodal fusion features are represented as follows:

[0027] .

[0028] Preferably, based on multimodal fusion features and activation functions, anomaly scoring rules for image patches are constructed, with the following formula:

[0029] ;

[0030] Where (x,y) represents the position of the image patch, W s The weights are linearly transformed for model learning. Use the Sigmoid activation function;

[0031] For image patches belonging to the striped region, anomaly scores are obtained for each patch. The anomaly scores of all image patches belonging to the striped region are then weighted and summed to obtain the anomaly probability of the striped region. The calculation formula is as follows:

[0032] ;

[0033] Among them, R band P represents the set of image patches within the strip region. band Indicates the probability of an anomaly in the striped region;

[0034] Based on the anomaly probability P at the strip level band To determine if it is abnormal, if > Threshold If so, the band is determined to be an abnormal region.

[0035] Preferably, the process of determining whether an image patch belongs to a strip region is as follows:

[0036] By segmenting mask M i Obtain the outline of a single chromosome, and within the outline, determine the main axis of the chromosome based on the direction of the chromosome skeleton;

[0037] Using a standard human karyotype banding template, the main axis coordinate range of the chromosome is segmented, and each band region is mapped to a pixel region in the image space, that is, a rectangular or polygonal region.

[0038] The image is divided into image patches. If the center coordinates of an image patch fall within a certain strip region, the image patch is determined to belong to the corresponding strip region.

[0039] Preferably, the decoding process generates natural language text that meets the requirements of the chromosome abnormality detection task. The decoding process includes:

[0040] The input embedding part of the decoder transforms the multimodal fusion features and anomaly scoring rule formulas into the initial input representation of the decoder for text generation. The input embedding part of the decoder is divided into two parts, represented as follows:

[0041] Z init =H init +E band ;

[0042] Among them, H init H represents the embedded information of multimodal fusion features. init The multimodal fusion features are transformed into the initial hidden state of the decoder; E band This represents the embedded information of the strip region location, transforming the anomaly score probability distribution of the strip region into a feature representation usable by the decoder;

[0043] The input decoding part of the decoder is configured as a multi-layer decoder based on self-attention and cross-attention mechanisms, and the input embedding part of the decoder provides the initial hidden state: H0=Z. init ;

[0044] The hidden state is updated incrementally through a multi-layer decoder:

[0045] ;

[0046] in Indicates the first layer, It is the first The hidden state of the layer at time step t; the hidden state of the last layer of the decoder. The output hidden state H at the current time step t t ;

[0047] The decoder output is mapped to the probability distribution of the vocabulary through a fully connected layer:

[0048] ;

[0049] Where: y t For generating words at time step t, Wo To output the weight matrix, The hidden state of the Nth layer decoder at time step t is ultimately generated using an autoregressive method, based on... The target text T is generated by decoding step by step.

[0050] Preferably, during the text decoding process, the region descriptions in the generated text are dynamically adjusted by combining the coordinate information of the striped regions with the anomaly scoring rules, as follows:

[0051] Band region coordinate embedding: Mapping the band regions of chromosome images to standard chromosome banding images to obtain the boundary coordinates of abnormal band regions. Map it to a position embedding vector ;

[0052] The anomaly score S(x,y) obtained based on the anomaly scoring rules of image patches is dynamically adjusted to generate a text description, which is represented as follows:

[0053] Represents a dynamic embedding vector. ;

[0054] The input hidden state of each layer of a multilayer decoder Dynamically fused dynamic embedding vector E dynamic This allows for dynamic adjustments to the description of the strip regions when generating text.

[0055] Preferably, the annotations on the acquired chromosome images include:

[0056] Based on images of chromosome metaphase, the outline coordinates of chromosomes, chromosome number classification, and abnormality categories in the images are manually labeled.

[0057] Preferably, the content of the natural language text includes: a conclusive description of whether the chromosome is abnormal, the location of the chromosome abnormality, and the type of chromosome abnormality.

[0058] Preferably, it also includes model training and optimization methods, including:

[0059] Define a comprehensive loss function L for jointly training the chromosome abnormality detection and text generation tasks:

[0060] ;

[0061] Where: L cls For the classification loss of whether chromosomes are abnormal, L loc For the loss of strip localization, L gen For the loss of text generation, L align For multimodal feature alignment loss, , , These are the loss weighting coefficients;

[0062] in,

[0063] ;

[0064] y represents the probability of chromosomal abnormality predicted by the model. m These are the actual category labels, where 0 indicates normal and 1 indicates abnormal.

[0065] ;

[0066] B m Let m represent the set of stripes in sample m. For the anomaly score of band b predicted by the model, These are the actual category labels, where 0 indicates normal and 1 indicates abnormal.

[0067] ;

[0068] A m This represents the target text length of sample m. This represents the actual words in the target text of sample m at time step t. The probability of the word predicted by the model;

[0069] ;

[0070] , representing the cosine similarity between image and text features; Indicates the temperature coefficient;

[0071] Using a multimodal training set from a multimodal dataset, the model is optimized until convergence by minimizing the comprehensive loss function L of the formula through backpropagation. The trained model is then used for anomaly detection at the chromosome band level and text description generation.

[0072] Compared with the prior art, the advantages of the present invention are:

[0073] (1) This invention uses the information obtained by fusing image features and text strip information to construct multimodal fusion features, and fully learns the semantic association between the morphological features of chromosome images and the text information of strips. The model can capture subtle differences in chromosome strip regions; through an anomaly scoring mechanism, it can achieve high-precision strip localization and anomaly detection, especially in structural anomaly detection, it can clearly point out the chromosome band regions where anomalies occur (such as p13.1, q22.3, etc.), and finally output natural language text to describe the chromosome anomaly category, explain the specific location of the anomaly and the semantic interpretation of its biological significance.

[0074] (2) This invention supports the detection and localization of various anomaly types (such as missing, duplicate, inversion, translocation, etc.) and dynamically describes and adjusts the anomaly scores of strip positions. Through the dynamic strip position embedding mechanism, the model can adapt to complex anomaly patterns without the need for explicit modeling of each anomaly type.

[0075] (3) The natural language text description generated by this invention is directly combined with the image detection results, eliminating the need for manual analysis of chromosome images or input of detection results one by one, thus significantly reducing manual intervention and analysis time. The detection results are automatically output, highly interpretable, and can accelerate the genetic analysis process and improve clinical efficiency. Attached Figure Description

[0076] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0077] Figure 1 This is a flowchart of the chromosome abnormality detection method described in this invention;

[0078] Figure 2 This is a schematic diagram of the chromosome multimodal feature interaction described in this invention;

[0079] Figure 3 This is a schematic diagram of the output results of the chromosome multimodal features described in this invention. Detailed Implementation

[0080] The present invention will be further described in detail below with reference to specific embodiments:

[0081] like Figure 1 As shown, a method for detecting chromosomal abnormalities based on a multimodal large model is proposed, which includes the following steps:

[0082] Step 1: Preparation of multimodal dataset.

[0083] Numerous images of metaphase chromosomes from human amniotic fluid, peripheral blood, chorionic villi, or bone marrow samples were acquired using a 63x optical microscope. For each image, the outline coordinates of the chromosomes, their numbering (e.g., chr1 represents chromosome 1), and abnormality categories (e.g., deletion, inversion, duplication, translocation, or normal category) were manually labeled. The purpose of this labeling was to obtain a chromosome image dataset D corresponding to each chromosome image. img :

[0084] ;

[0085] in, The image shows chromosomes (the subscript i represents the image number), M i As a segmentation mask for chromosomes, y iThe corresponding abnormal category label (such as missing, inversion, duplication, translocation, or normal category).

[0086] Based on the karyotype analysis report, the abnormality categories of chromosomes and descriptions such as the location of the abnormal bands are extracted to construct a text dataset D. txt :

[0087] ;

[0088] Among them, T i To describe the text information of the chromosomal banding region and the type of abnormality (e.g., "q22.1 region deletion"), y i The labels are for anomaly categories and are consistent with the image dataset.

[0089] Multimodal data alignment was performed between chromosome image data and corresponding band text descriptions to construct a multimodal dataset:

[0090] ;

[0091] in, T i They are from D respectively img and D txt , This means that multimodal samples composed of images, text, and corresponding labels are used to ensure that chromosome images are matched one-to-one with their corresponding band text descriptions, which are then used for joint training of the model.

[0092] Chromosome Image Dataset D img and text dataset D txt 85% of the data is used for training, 7% for validation, and 8% for testing.

[0093] Step 2: Construction of a multimodal large model.

[0094] For each data point i in the multimodal dataset D, firstly, the chromosome image is encoded using an image encoder, for example, the input chromosome image. Chromosome images based on a pre-trained MAE (Masked Autoencoder) model. The image is divided into P fixed-size patches, forming a grid of P = H × W. Each image patch is then encoded as a feature vector.

[0095] ;

[0096] in, : Indicates the horizontal position of the image patch within the grid. : Indicates the vertical position of the image patch in the grid.

[0097] The MAE model is trained by randomly encoding chromosome images, enabling it to run even in the presence of external interference and enhancing model performance.

[0098] The extracted chromosome image features are represented as follows:

[0099] ;

[0100] Where MAE represents the MAE model. The image represents a chromosome (the subscript i indicates which image it is). d represents the number of image patches and d represents the feature dimension.

[0101] Optionally, in practice, P is set to 196 and d is set to 512.

[0102] For text encoders, pre-trained language models, such as BERT, are used.

[0103] Bidirectional Encoder Representations from Transformers is a pre-trained language model based on the Transformer architecture. It uses a Transformer encoder to capture the left and right contextual information of the input text simultaneously, generating dynamic word vectors.

[0104] Text information T i Encoded as text features, represented as:

[0105] ;

[0106] Where L is the length of the text sequence, d is the feature dimension, and BERT represents the BERT pre-trained model.

[0107] Optionally, in specific implementations, L is set to 512 and d is set to 512.

[0108] Based on chromosome image features, a self-attention mechanism is used for interaction to obtain multimodal features F for each data point. multi Combined with the appendix Figure 2 The provided diagram illustrates the interaction between chromosome multimodal features. The specific calculation process, i.e., the interaction between the chromosome image and abnormal features, is as follows:

[0109] First, based on image features F img Obtain the query matrix Q representing the corresponding chromosome image. img :

[0110] ;

[0111] Secondly, based on text feature F txt Obtain the key matrix K representing the corresponding abnormal text. txt Sum matrix V txt :

[0112] ;

[0113] ;

[0114] Among them, W q W k W v These are the weight matrices for the Query, Key, and Value in the Transformer architecture, respectively.

[0115] bond matrix K txt Value matrix V represents the abnormal text content. txt This is a text vector representation, indicating the location of the text.

[0116] Then, based on chromosome image representation Q img and abnormal text representation K txt The weight of the interaction is calculated using the following formula:

[0117] ;

[0118] Finally, based on the anomalous text vector representation V txt Multimodal fusion features are obtained by combining interaction weights. The multimodal fusion features are represented as follows:

[0119] .

[0120] Step 3: Chromosomal abnormality detection and location.

[0121] First, based on the multimodal fusion feature F multi Based on the activation function of the neural network, anomaly scoring rules for image patches are formulated. The anomaly scoring formula is as follows:

[0122] ;

[0123] Where (x,y) represents the position of the image patch, W s The weights are linearly transformed for model learning. The Sigmoid activation function. .

[0124] Secondly, band division and anomaly localization are carried out.

[0125] In chromosome mask M iWith the help of [the relevant technology], image patches (x, y) can be labeled as belonging to a certain (e.g., p13.1) or not belonging to any strip region.

[0126] The specific process for determining whether an image patch belongs to a strip region is as follows:

[0127] The segmentation mask M from step 1 i The precise outline of a single chromosome is obtained. Within the outline, the main axis of the chromosome is determined based on the orientation of the chromosome skeleton.

[0128] Using a standard human karyotype banding template (banding scheme provided by ISCN), the principal axis coordinate range of the chromosome is segmented and mapped to specific band regions (e.g., p13.1, q22.3, etc.). Each band region is mapped to a pixel region in image space, i.e., a rectangular or polygonal region.

[0129] The image is divided into patches. If the center coordinates (x, y) of a patch fall within a certain strip region, the patch is marked as belonging to that strip region.

[0130] For an image patch (x, y) belonging to the strip region, the anomaly score is obtained according to the anomaly scoring formula of the image patch.

[0131] To calculate the overall anomaly score for this strip region, the anomaly scores of all image patches belonging to this strip region are weighted and summed:

[0132] ;

[0133] Among them, R band P represents the set of image patches within the strip region. band This represents the probability of an anomaly in that band region.

[0134] Finally, based on the anomaly probability P at the strip level band To determine if it is abnormal, if > Threshold If the threshold is not met, the affected area is considered an abnormal region. In actual detection tasks, a threshold is set. It is 0.6.

[0135] Step 4: Generate a text description of the detection results.

[0136] The test results are output as a brief text description, similar to a doctor's analysis and commentary. For example:

[0137] "Chromosome 1 is normal."

[0138] “There is an abnormality on chromosome 22, located in the q22.1 region, which is a deletion type.”

[0139] That is, through multimodal fusion of features F multi and abnormal results in the strip region P band Generate a natural language text that meets the requirements of the chromosome abnormality detection task.

[0140] The entire text generation process consists of the following modules:

[0141] Input embedding module: First, the multimodal fusion feature F multi Anomalies P in the strip region band This is transformed into the initial input representation for the decoder used for text generation.

[0142] The input to the entire decoder is divided into two parts: Z init =H init +E band ;

[0143] Among them, H init For multimodal fusion feature embedding, i.e. H init =MLP(F multi ), F multi This is transformed into the initial hidden state of the decoder; MLP stands for Multilayer Perceptron.

[0144] E band Embedding for strip position This is used to transform the probability distribution of the striped region into a feature representation that can be used by the decoder.

[0145] A multi-layer Transformer decoder, stacking N layers of Transformers (e.g., N is set to 6), generates text sequences step by step based on self-attention and cross-attention mechanisms.

[0146] Decoder input embedding Z init The initial hidden state is provided: H0=Z init ;

[0147] The hidden state is updated incrementally through N layers of Transformers:

[0148] ;

[0149] in, Indicates the first layer, It is the first The hidden state of the layer at time step t.

[0150] The last hidden state of the decoder The output hidden state H at the current time step t t .

[0151] Furthermore, due to the irregular variations in the size and structure of different chromosomes, this invention dynamically adjusts the region description in the generated text by combining the coordinate information of the band region with the anomaly scoring formula of the image block.

[0152] First, perform strip coordinate embedding:

[0153] Based on the mapping of image patches and strip regions in step 3, for all patch or mask pixels belonging to a certain strip region, calculate their minimum and maximum coordinates:

[0154] ;

[0155] ;

[0156] Obtain the bounding rectangle of the strip region. (i.e., boundary coordinates, such as the start and end coordinates on page 13.1), which are mapped to position embedding vectors:

[0157] .

[0158] Secondly, a dynamic adjustment mechanism is designed: the generated text description is dynamically adjusted based on the anomaly score S(x,y) of the image patch.

[0159] ;

[0160] This represents a dynamic embedding vector that can capture the importance and location information of strip anomalies.

[0161] If S(x,y) represents a region with a high anomaly score, dynamic embedding will increase the attention given to that region, thus prioritizing the description of this anomaly region; if E coord The indicator area is small, and the generated text will accurately describe the corresponding chromosome bands.

[0162] E dynamic This will be added to each layer of the decoder to dynamically adjust the description of the striped regions when generating text; that is, in a multi-layer Transformer decoder, the input hidden state of each layer. Dynamically integrate E dynamic :

[0163] ;

[0164] CrossAttention is a cross-attention mechanism.

[0165] Text generation module: Generates words for each time step.

[0166] The decoder output is mapped to the probability distribution of the vocabulary through a fully connected layer:

[0167] ;

[0168] Where: y t For generating words at time step t, W o To output the weight matrix, Let be the hidden state of the Nth layer decoder at time step t.

[0169] Finally, using an autoregressive generation method, based on The target text T is generated step by step.

[0170] Step 5: Model training and optimization.

[0171] Define a comprehensive loss function L for jointly training the chromosome abnormality detection and text generation tasks:

[0172] ;

[0173] Where: L cls For the classification loss of whether chromosomes are abnormal, L loc For the loss of strip localization, L gen For the loss of text generation, L align For multimodal feature alignment loss, , , These are the loss weighting coefficients, which are set to 0.1, 0.01, and 0.5 in practical applications.

[0174] in,

[0175] ;

[0176] y represents the probability of chromosomal abnormality predicted by the model. m This indicates the actual category label, with 0 representing normal and 1 representing abnormal.

[0177] ;

[0178] B m Let m represent the set of stripes in sample m. For the anomaly score of band b predicted by the model, These are the actual category labels, where 0 indicates normal and 1 indicates abnormal.

[0179] ;

[0180] A m This represents the target text length of sample m. This represents the actual words in the target text of sample m at time step t. The probability of the word predicted by the model;

[0181] ;

[0182] , representing the cosine similarity between image and text features; This represents the temperature coefficient (hyperparameter, set to 0.08).

[0183] Using a multimodal training set Initialize the model and optimize it until convergence using the comprehensive loss function L. Use the trained model to make predictions; the output of the abnormal text will look like this. Figure 3 As shown.

[0184] Because chromosome images are unique, chromosomes may be obscured during image acquisition, and the banded areas may differ from their normal orientation. By scoring anomalies in a single image patch, obscured chromosome images can easily be misidentified as abnormal images, leading to misjudgments.

[0185] By using a weighted summation of all image patches, the overall situation can be comprehensively judged. This can reduce the weighted abnormality score of chromosome images with occlusion, thereby reducing the score and minimizing misjudgments.

[0186] Conventional detection models and methods for chromosomal abnormalities can only output a conclusion about whether a chromosome is abnormal or identify and select abnormal regions, but still require manual analysis of the specific abnormality type and location. This application, however, can directly provide a textual description of the abnormality when detecting chromosomal abnormalities, such as: "Chromosome 22 has a deletion located in the q22.1 band region, which may affect the expression of the key gene A"; or, for example, the output textual description of the abnormality is: "Chromosome 1 has an abnormality located in the p13 to q23 region, which is an inversion type."

[0187] This invention provides a prior knowledge fusion mechanism for training detection models by effectively integrating chromosome images and text information, thereby improving the system's comprehensive reasoning and localization capabilities in complex scenarios.

[0188] When faced with complex structural anomalies (such as translocations, deletions, and insertions), the model performs fine-grained segmentation of abnormal regions at the chromosome banding level, achieving high recognition accuracy and improving the clinical applicability of the model. It also has the ability to generate structured natural language karyotype analysis reports, transforming the test results into understandable text for doctors and patients, avoiding a single form of expression, and improving user decision-making efficiency and the system's intelligence level.

[0189] The above embodiments are merely illustrative of the technical concept and features of the present invention, intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly, and should not be construed as limiting the scope of protection of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects. The scope of the present invention is defined by the appended claims rather than the foregoing description, and thus all changes falling within the meaning and scope of the equivalents of the claims are intended to be included within the present invention.

Claims

1. A method for detecting chromosomal abnormalities based on a multimodal large model, characterized in that, include: The acquired chromosome images are labeled to construct an image dataset; Based on the karyotype analysis report, descriptions of chromosome abnormality categories and the locations of abnormal bands were extracted to construct a text dataset; the image dataset and text dataset were then aligned using multimodal data to construct a multimodal dataset. For each data point in the multimodal dataset, images and text are encoded to obtain image features and text features; Multimodal fusion features are obtained by fusing image features and text features through an attention mechanism; Based on multimodal fusion features and activation functions, anomaly scores for image patches belonging to strip regions are calculated; The anomaly scores of all image blocks belonging to the strip region are weighted and summed to obtain the anomaly probability of the strip region, and the strip region is determined to be abnormal. Multimodal fusion features are transformed into the anomaly probability of the strip region using MLP, which is then used as the initial input of the decoder. The decoder generates natural language text that meets the requirements of chromosome abnormality detection. The decoder is a multi-layer decoder built based on self-attention and cross-attention mechanisms. During the text decoding process, the region descriptions in the generated text are dynamically adjusted by combining the coordinate information of the strip regions with the anomaly score.

2. The method for detecting chromosomal abnormalities based on a multimodal large model according to claim 1, characterized in that, The process of obtaining multimodal fusion features includes: The chromosome image is encoded by an image encoder. The chromosome is divided into image blocks, and each image block is encoded into a feature vector. The feature vectors corresponding to the image blocks are merged to construct the merged image features of the chromosome image. Text features are obtained by encoding text descriptions using a pre-trained language model; Multimodal fusion features are obtained by interacting with image and text features through an attention mechanism.

3. The method for detecting chromosomal abnormalities based on a multimodal large model according to claim 2, characterized in that, Multimodal fusion features are obtained by fusing image and text features through an attention mechanism, including: Obtain the query matrix based on image features ; Obtain the key matrix based on text features Sum matrix ; Based on query matrix Bond matrix Get interaction weight ; Based on interaction weights Log-value matrix Perform weighted summation to obtain multimodal fusion features. .

4. The method for detecting chromosomal abnormalities based on a multimodal large model according to claim 3, characterized in that, The anomaly scoring formula for image patches is: ； in,( ) indicates the position of the image patch. The weights are linearly transformed for model learning. Use the Sigmoid activation function; The probability of an anomaly in a striped region is calculated using the following formula: ； in, Represents a set of image patches within a stripe region; Based on the anomaly probability of the strip region To determine if it is abnormal, if > Threshold If so, the strip region is determined to be an abnormal region.

5. The method for detecting chromosomal abnormalities based on a multimodal large model according to claim 4, characterized in that, Before calculating the anomaly score for image patches belonging to the striped region, the process also includes determining whether an image patch belongs to the striped region, specifically: By segmenting the mask Obtain the outline of a single chromosome, and determine the main axis of the chromosome within the outline based on the direction of the chromosome skeleton; Using a standard human karyotype banding template, the main axis coordinate range of the chromosome is segmented, and each band region is mapped to a pixel region in the image space; The chromosome image is divided into image blocks. If the center coordinates of an image block fall within a certain band region, the image block is determined to belong to that band region.

6. The method for detecting chromosomal abnormalities based on a multimodal large model according to claim 5, characterized in that, The decoder generates natural language text that meets the requirements for chromosome abnormality detection. During the text decoding process, the region descriptions in the generated text are dynamically adjusted by combining the coordinate information of the striped regions with anomaly scores, including: The MLP is used to transform the multimodal fusion features into the anomaly probabilities of the strip regions, thus obtaining the initial input representation of the text generation decoder. ; A multi-layer decoder is constructed based on self-attention and cross-attention mechanisms. As the initial hidden state, the hidden state is gradually updated through multiple layers of decoders: ； in, It is the first The hidden state of the layer decoder at time step t; Represents a dynamic embedding vector; by mapping the banded regions of a chromosome image to a standard chromosome banding image, the boundary coordinates of the abnormal banded regions are obtained. ], mapping it to a position embedding vector , ; ; The decoder output is mapped to the probability distribution of the vocabulary through a fully connected layer: ； in: For time step The generated words, To output the weight matrix, This represents the hidden state of the last layer of the decoder at time step t; it is generated using an autoregressive method, based on... Step-by-step decoding to generate target text .

7. The method for detecting chromosomal abnormalities based on a multimodal large model according to claim 1, characterized in that, The acquired chromosome images are annotated, including: Based on images of chromosome metaphase, the outline coordinates of chromosomes, chromosome number classification, and abnormality categories in the images are manually labeled.

8. The method for detecting chromosomal abnormalities based on a multimodal large model according to claim 1, characterized in that, The natural language text includes: whether the chromosome is abnormal, the location of the chromosome abnormality, and the type of chromosome abnormality.

9. The method for detecting chromosomal abnormalities based on a multimodal large model according to claim 1, characterized in that, It also includes model training and optimization methods, including: Define the comprehensive loss function , ; in: The loss in classification of whether chromosomes are abnormal. For the loss of strip positioning, Loss for text generation, The loss is for multimodal feature alignment. , , These are the loss weighting coefficients; Using the multimodal training set of a multimodal dataset, the comprehensive loss function of the formula is minimized through backpropagation. The model is optimized until convergence, and the trained model is used to detect chromosome abnormalities and generate text descriptions.

Images