Chromosome detection method and device for multi-species organisms

Abstract

The invention discloses a chromosome detection method for multi-species organisms, which comprises the following steps: inputting a to-be-detected image and prompt text information corresponding to species and tissue sources of the species into a trained visual language joint detection model, generating a structured detection result through reasoning of the trained visual language joint detection model, and determining the chromosome detection result according to the structured detection result. The structured detection result at least comprises target image position information, target category information and target text description information; the visual language joint detection model comprises a visual detection model used for carrying out target detection based on an image, and a multi-modal large language model used for generating target text description information based on prompt information and visual features from the visual detection model; the structured detection result is a fusion result of a target detection result from the visual detection model and target text description information from the multi-modal large language model. According to the invention, the accuracy of chromosome detection of multi-species organisms is improved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and biological microscopic image analysis, specifically to a chromosome detection method for multi-species organisms. Background Technology

[0002] Chromosomal testing is a crucial component of radiation damage assessment, chemical toxicity testing, and genetic research. Traditional methods rely on manual identification of abnormal chromosomal structures, such as dicentric mitochondria, acentric fragments, and circular chromosomes, in microscopic images. This process is labor-intensive, highly subjective, and struggles to generalize to multiple species, various staining methods, and complex aberration types.

[0003] Related deep learning-based object detection methods (such as Faster R-CNN and YOLO) perform well on fixed species biological datasets, but they suffer from difficulties in class expansion, insufficient semantic generalization, and low cross-domain performance on multi-species and multi-scene datasets. Summary of the Invention

[0004] This application provides a chromosome detection method for multi-species organisms to solve the problem of chromosome detection in multi-species organisms.

[0005] The first aspect of this application provides a chromosome detection method for multi-species organisms, the method comprising:

[0006] The image to be detected and the prompt text information corresponding to the species and the tissue origin of the species are input into the trained visual-language joint detection model. After the reasoning of the trained visual-language joint detection model, a structured detection result is generated. The structured detection result includes at least: target image location information, target category information, and target text description information.

[0007] in,

[0008] The joint visual-language detection model includes: a visual detection model for image-based object detection, and a multimodal large language model for generating target text descriptions based on cue information and visual features from the visual detection model.

[0009] The structured detection result is the fusion result of the target detection result from the visual detection model and the target text description information from the multimodal large language model. The target detection result includes at least the target image location information and the target category information.

[0010] As one possible implementation, the prompt text information is obtained in the following manner:

[0011] Based on the species and its tissue origin information, the prefix text field and chromosome number field of the corresponding prompt text information are retrieved from the prompt template library.

[0012] According to the set prompt text information format, the prefix text field, chromosome number field, species information, tissue origin information, and general malformation type information are used to construct the prompt text information.

[0013] in,

[0014] The prompt template library stores structured prompt text information for preset species. The data structure of the prompt template library is represented in key-value pair form. The key information is used to characterize the species and its tissue source information, and the value information is used to characterize the prompt text information. The prompt text information includes: chromosome number field, prefix text field of prompt text information, and rare aberration category information used to avoid hallucination output;

[0015] As one possible implementation, the visual detection model includes: a backbone network layer, a feature extraction network layer, a chromosome number prior embedding layer, and a detection head.

[0016] in,

[0017] The image to be detected is input into the backbone network layer. The scale features from the backbone network layer are input into the feature extraction network layer. The P2 to P5 level features from the feature extraction network layer are input into the detection head. The P4 and P5 level features from the feature extraction network layer are input into the multimodal large language model.

[0018] The chromosome number prior embedding layer maps the chromosome number in the prompt information into a multi-dimensional one-dimensional effective encoding vector, which is then compressed into a learnable vector by a linear layer in the chromosome number prior embedding layer. This learnable vector is then concatenated pixel-by-pixel with the P3 level features, and the concatenated features are input to the detection head.

[0019] The detection head outputs target image location information, target category information, and confidence level based on the input features.

[0020] As one possible implementation, the method further includes:

[0021] When the image to be detected is an image of a newly added species to be detected, and the number of such images is less than a set first threshold,

[0022] Instead of retraining the joint visual-language detection model, we update the cue template library and dynamic vocabulary mapping table, while maintaining constraints in the detection feature space.

[0023] If the feature space preservation constraints are not met, the model parameters of the visual detection model are adjusted until the feature space preservation constraints are satisfied.

[0024] in,

[0025] Dynamic vocabulary mapping tables are used in the process of generating target text description information in multimodal large language models to map general distortion description information in prompt text information to species-specific description information.

[0026] The feature space preservation constraint is used to characterize the distance metric between the visual feature vector of the image of the new species to be detected output by the visual detection model and the existing visual feature vector output by the visual detection model in the common subspace.

[0027] As one possible implementation, the updated prompt template library and dynamic vocabulary mapping table include:

[0028] The newly added species and tissue origin in the image to be detected are used as key information. The prefix text, chromosome number field, and rare aberration category information of the prompt text information of the newly added species are used as value information. The key information and value information of the newly added species are stored in the prompt template library in key-value pairs. The prefix text includes: species information and tissue origin information.

[0029] Configure the user-defined dictionary in the dynamic vocabulary mapping table and the system default mapping table;

[0030] The detection feature space preservation constraints include:

[0031] A feature vector is randomly selected from the visual feature distribution of the new species output by the visual detection model as the first feature vector. A feature vector is randomly selected from the existing visual feature distribution output by the visual detection model as the second feature vector. The shortest distance among the distances between the second feature vectors that are closest to the first feature vector is determined, and the shortest distance of the first feature vector is obtained.

[0032] The average of the shortest distances among the first feature vectors selected in the feature distribution of the new species output by the visual detection model is used to obtain the feature space preservation constraint for the new species.

[0033] If the feature space of the new species is less than or equal to the set distance threshold, the feature space is determined to be subject to the set constraint condition.

[0034] As one possible implementation, the method further includes:

[0035] If the image to be detected is an image of a newly added species to be detected, and the number of such images is greater than a first threshold but less than a second threshold, the transfer of the visual-language joint detection model is triggered, wherein the first threshold is less than the second threshold.

[0036] When the image to be detected is an image of a newly added species to be detected, and the number of such images is greater than or equal to the second quantity threshold, the online learning of the visual-language joint detection model is triggered.

[0037] As one possible implementation, the visual-language joint detection model performs the transfer in the following manner:

[0038] Generate prompt text information indicating the addition of a new species in the image to be detected.

[0039] Dynamically construct the prototype feature vectors of each target image of the newly added species.

[0040] The first confidence threshold, used to characterize the general category confidence threshold, is calibrated to obtain the second confidence threshold used to recall rare distortion types.

[0041] The image of the newly added species to be detected and the generated prompt text information are input into the trained visual-language joint detection model.

[0042] The visual features of the target image location information from the visual detection model in the joint visual-language detection model are matched with the prototype feature vector of the new species.

[0043] If a match is successful, the category information corresponding to the prototype feature vector is used as the category information of the target image.

[0044] If the match fails, the detection head in the visual detection model outputs the target image location information, target category information, and confidence score.

[0045] Based on the confidence threshold, target detection results with a confidence level greater than the confidence threshold are selected.

[0046] As one possible implementation, the prompt text information for the newly added species in the image to be detected includes:

[0047] Based on the species information and tissue origin information of the newly added species, the prefix text of the prompt text information and the chromosome number information are determined in the template database.

[0048] The determined prefix text, chromosome number information, and category information from the existing target detection results output by the visual detection model are concatenated to form the prompt text information;

[0049] The dynamic construction of the prototype feature vectors for each target image of the newly added species includes:

[0050] The visual detection model outputs the target image's location information, category information, confidence level, and visual features during the inference process of the image of the newly added species to be detected.

[0051] When the cumulative number of images of newly inferred species reaches a set inference threshold,

[0052] Based on the various distortion types of the cumulative inferred new species images, the cumulative inferred new species images are divided into support sets for each distortion type.

[0053] For each distortion type, average pooling is performed on the visual features of the target images of each newly added species in the support set of that distortion type to obtain the prototype vector of that distortion type.

[0054] The second confidence threshold is determined as follows:

[0055] For any rare aberration type, the prior weight is determined based on the proportion of occurrence of that rare aberration type.

[0056] The first confidence threshold is weighted by the prior weights to obtain the second confidence threshold;

[0057] Following the transfer of the triggering visual-language joint detection model, the following further includes:

[0058] When the visual-language joint detection model fails to meet expectations in detecting new species after transfer, online learning of the visual-language joint detection model is triggered.

[0059] As one possible implementation, the training of the visual-language joint detection model includes: a cross-species pre-training stage for constructing a general chromosome detection model, a species-specific fine-tuning stage for achieving specific aberration category recognition, and an online learning stage for hot updating of model parameters.

[0060] in,

[0061] Cross-species pre-training is conducted in the following manner:

[0062] The model parameters of the frozen visual detection model's backbone network layers, including the block embedding layer and Stage-1, and the unfrozen backbone network layers, including Stage-2 to Stage-4, the feature extraction network, and the detection head, are as follows:

[0063] Multi-species biological sample data is input into a visual-language joint detection model to be trained. The sample data includes: sample images and their annotation data, sample prompt text information, and desired target text description information. The annotation data includes: corrected target image location information and the annotation mask image at that location. The sample images and their annotation data are input into the backbone network layer of the visual detection model. The chromosome number in the sample prompt text information is input into the chromosome number prior embedding layer. The sample prompt text information is then input into a multimodal large language model.

[0064] The parameters of the unfrozen model are adjusted according to the total loss function value until convergence, resulting in the visual-language joint detection model obtained in the cross-species pre-training stage. The total loss function value includes: the fusion detection loss value of target image location information regression and open word classification, the loss function value of target image location region-cause text information comparison, the loss function value of sample target text description information and expected target text description information output by the multimodal large language model, and the cross-species structural consistency loss function value.

[0065] Species-specific fine-tuning is performed in the following manner:

[0066] The sample data is input into the visual-language joint detection model obtained during the cross-species pre-training stage.

[0067] The model parameters of the block embedding layer, Stage-1 layer, and Stage-2 layer in the visual detection model of the visual-language joint detection model obtained in the cross-species pre-training stage are frozen, and the model parameters of Stage-3 to Stage-4 layers, the detection head, and the description head in the multimodal large language model are fine-tuned.

[0068] Online learning will be conducted in the following ways:

[0069] The image of the newly discovered species to be detected is labeled as a sample to obtain sample data. This sample data is then input into the current visual-language joint detection model. Through inference by the visual detection model in the current visual-language joint detection model, the visual features of the target image of the newly discovered species to be detected are obtained.

[0070] The feature space is preserved when calculating the visual features of the target image.

[0071] While ensuring that the constraints in the calculated feature space are satisfied, update the model parameters of the current prompt template library, prototype vector, detection head, and descriptor head. Freeze the model parameters of the backbone network layer and feature extraction network layer. When the cumulative number of training steps reaches the set threshold for the first step, replay training is performed by mixing in sample images of the initial species.

[0072] Using the next image of a newly discovered species to be detected as a sample, the process returns to the step of labeling the current image of the newly discovered species to be detected as a sample, until the total number of training steps reaches a predetermined second threshold, at which point online learning ends. The second threshold is greater than the first threshold.

[0073] As one possible implementation, the cross-species structural consistency loss function value is determined in the following manner:

[0074] Canny edge contour extraction is performed on both the labeled mask image and the sample mask image to obtain the labeled edge point set of the labeled mask image and the sample edge point set of the sample mask image. The sample mask image is the mask image of the sample target image position detected by the detection head.

[0075] For each sample edge point, determine its nearest labeled edge point, calculate the Chamfer distance between the sample edge point and each of the determined nearest labeled edge points, and average them to obtain the first average value.

[0076] For each labeled edge point, determine its nearest sample edge point, calculate the Chamfer distance between the labeled edge point and all the determined nearest sample edge points, and average them to obtain a second average value.

[0077] Calculate the average of the first average and the second average, and apply dynamic weights to this average to obtain the cross-species structural consistency loss function value, where the dynamic weights increase linearly with the increase of the number of species;

[0078] The cross-species pre-training also includes:

[0079] For the visual-language joint detection model obtained in the cross-species pre-training stage, the model parameters of the visual detection model are frozen, and the model parameters of the multimodal large language model are fine-tuned.

[0080] The fine-tuning is carried out in the following manner:

[0081] Cross-species cytogenetics terms will be added as new tags to the vocabulary.

[0082] The sample data is input into the visual-language joint detection model obtained in the cross-species pre-training stage.

[0083] Based on the cross-entropy loss function value of standard language modeling, the parameters of the embedding vector corresponding to the new label and the output projection layer in the multimodal large language model are updated, so that the multimodal large language model can learn to establish the association mapping between the new label and the corresponding visual features in the sample image.

[0084] The online learning further includes:

[0085] If any specific term is not correctly invoked by the visual-language joint detection model, a key-value pair for that specific term is added to the dynamic vocabulary mapping table, and the specific term is explicitly embedded in the current prompt text information. Here, the key information is used to represent the abbreviation and / or alias of the specific term, and the value information is used to represent the specific term.

[0086] A second aspect of this application provides a chromosome detection device for multi-species organisms, the device comprising:

[0087] The detection module is used to input the image to be detected and the prompt text information corresponding to the species and the tissue origin of the species into the trained visual-language joint detection model. After the reasoning of the trained visual-language joint detection model, a structured detection result is generated. The structured detection result includes at least: target image location information, target category information, and target text description information.

[0088] in,

[0089] The joint visual-language detection model includes: a visual detection model for image-based object detection, and a multimodal large language model for generating target text descriptions based on cue information and visual features from the visual detection model.

[0090] The structured detection result is the fusion result of the target detection result from the visual detection model and the target text description information from the multimodal large language model. The target detection result includes at least the target image location information and the target category information.

[0091] The chromosome detection method for multi-species organisms provided in this application utilizes a visual-language joint detection model composed of a visual detection model and a multimodal large language model. This enables cross-species image chromosome detection based on prompt text information. The prompt text information allows the visual-language joint detection model to support chromosome detection for multiple species organisms without changing the model parameters or hardware of the visual-language joint detection model. Scene transfer can be achieved by replacing the prompt text information, and the model parameters of the entire model do not need to be retrained. Furthermore, the transfer through the visual-language joint detection model facilitates rapid adaptation to new species, improving the detection accuracy of chromosome detection for cross-species organisms. Attached Figure Description

[0092] Figure 1 This is a schematic diagram of a chromosome detection method for multiple species in this application.

[0093] Figure 2 This is a schematic diagram of a visual-language joint detection model structure for chromosome detection in multi-species organisms, as described in an embodiment of this application.

[0094] Figure 3 This is a flowchart illustrating a visual-language joint detection model transfer for chromosome detection of multi-species organisms when the number of images of newly added species to be detected is less than a second threshold but greater than a first threshold.

[0095] Figure 4 This is a schematic diagram of a process for training a visual-language joint detection model according to an embodiment of this application.

[0096] Figure 5 This is a schematic diagram of an online learning process in this embodiment.

[0097] Figure 6 This is a schematic diagram of a chromosome detection device for multiple species in an embodiment of this application.

[0098] Figure 7 This is another schematic diagram of a chromosome detection device for multiple species in this application. Detailed Implementation

[0099] To make the objectives, technical means, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0100] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0101] See Figure 1 As shown, Figure 1 This is a schematic diagram of a chromosome detection method for multiple species of organisms according to an embodiment of this application. The method includes:

[0102] The image to be detected and the prompt text information corresponding to the species and the tissue origin of the species are input into the trained visual-language joint detection model. After the reasoning of the trained visual-language joint detection model, a structured detection result is generated. The structured detection result includes at least: target image location information, target category information, and target text description information.

[0103] in,

[0104] The joint visual-language detection model includes: a visual detection model for image-based object detection, and a multimodal large language model for generating target text descriptions based on cue information and visual features from the visual detection model.

[0105] The structured detection result is the fusion result of the target detection result from the visual detection model and the target text description information from the multimodal large language model. The target detection result includes at least the target image location information and the target category information.

[0106] As an example, the visual detection model includes: a backbone network layer, a feature extraction network layer, a chromosome number prior embedding layer, and a detection head.

[0107] in,

[0108] The image to be detected is input into the backbone network layer. The scale features from the backbone network layer are input into the feature extraction network layer. The P2 to P5 level features from the feature extraction network layer are input into the detection head. The P4 and P5 level features from the feature extraction network layer are input into the multimodal large language model.

[0109] The chromosome number prior embedding layer maps the chromosome number in the prompt information into a multi-dimensional one-dimensional effective encoding vector, which is then compressed into a learnable vector by a linear layer in the chromosome number prior embedding layer. This learnable vector is then concatenated pixel-by-pixel with the P3 level features, and the concatenated features are input to the detection head.

[0110] The detection head outputs target image location information, target category information, and confidence level based on the input features.

[0111] Preferably, when the image to be detected is an image of a newly added species to be detected, and the number of such images is less than a set first quantity threshold,

[0112] Instead of retraining the joint visual-language detection model, we update the cue template library and dynamic vocabulary mapping table, while maintaining constraints in the detection feature space.

[0113] If the feature space preservation constraints are not met, the model parameters of the visual detection model are adjusted until the feature space preservation constraints are satisfied.

[0114] in,

[0115] Dynamic vocabulary mapping tables are used in the process of generating target text description information in multimodal large language models to map general distortion description information in prompt text information to species-specific description information.

[0116] The feature space preservation constraint is used to characterize the distance metric between the feature vector of the image of the new species to be detected output by the visual detection model and the existing visual feature vector output by the visual detection model in the common subspace.

[0117] This application embodiment uses a visual-language joint detection model composed of a visual detection model and a multimodal large language model to achieve cross-species image chromosome detection based on prompt text information.

[0118] Preferably, when the image to be detected is an image of a newly added species to be detected, and the number of such images is greater than a set first threshold and less than a set second threshold, the transfer of the visual-language joint detection model is triggered, wherein the first threshold is less than the second threshold, which is conducive to quickly completing cross-species adaptation.

[0119] When the image to be detected is an image of a newly added species to be detected, and the number of such images is greater than or equal to the second quantity threshold, the online learning of the visual-language joint detection model is triggered, which is beneficial to the updating of online incremental information.

[0120] To facilitate understanding of the embodiments of this application, the following detailed description is provided in conjunction with model structure examples. It should be understood that the embodiments of this application are not limited.

[0121] Given the strong potential of multimodal large language models (LLMs) in cross-modal understanding and open-vocabulary detection tasks, visual detection models for object detection, such as Grounding DINO and Deformable DETR, have achieved open-vocabulary object detection based on text prompts. Furthermore, joint training of the output of the visual detection model with LLMs can enable the model to simultaneously possess object localization and semantic description capabilities. This makes it possible to design a chromosome detection method that can be adapted to multiple species of organisms.

[0122] See Figure 2 As shown, Figure 2 This is a schematic diagram of a visual-language joint detection model structure for chromosome detection in multi-species organisms, as described in an embodiment of this application. The visual-language joint detection model includes:

[0123] A visual detection model for object detection in images based on open-vocabulary information from text prompts.

[0124] A multimodal large language model is used to generate target text description information Tc based on visual features from a visual detection model and cue text information Tg, in order to enhance the descriptive power of cytogenetics.

[0125] in,

[0126] The target image location information and category information from the visual detection model are fused with the target text description information Tc from the multimodal large language model to obtain the structured detection result, which includes: target image location information, target category information, and Tc.

[0127] As an example, the image to be detected, along with information on the number of chromosomes, is input into a trained visual detection model. The visual detection model then infers and outputs the target image location information and target category information. Tg, along with visual features from the visual detection model, is input into a trained multimodal large language model. The multimodal large language model then infers and generates Tc.

[0128] The visual detection model includes: a backbone network layer, such as a grounding DINO layer; a feature extraction network layer, such as a feature pyramid; a chromosome number prior embedding layer; and a detection head.

[0129] in,

[0130] The backbone network consists of: a patch embedding layer, stage-1, stage-2, stage-3, and stage-4 connected sequentially. The scale features output from stage-2, stage-3, and stage-4 are input to the feature extraction network layer.

[0131] The feature extraction network layer extracts scale features from the outputs of stage-2, stage-3, and stage-4 layers to output p2-p5 level features, such as p2-p5 feature maps. The high-resolution p2 feature map is used to capture the edges of small fragments. This feature extraction network layer has species-independent parameters, which allows multiple species to share the same convolution weights, ensuring cross-species universality. Species differences are left to be handled by the prior embedding layer for prompt text information or chromosome number.

[0132] The chromosome number prior embedding layer maps the chromosome number to a multidimensional one-hot vector (one effective encoding), the dimension of which is the same as the chromosome number. The multidimensional one-hot vector is compressed into a learnable vector by a linear layer in the chromosome number prior embedding layer. For all spatial locations of each feature map, the learnable vector is copied and concatenated pixel by pixel with the p3 level features. The concatenated features are input to the detector head, so that the detector head can simultaneously acquire visual morphology and chromosome number prior, in order to suppress the error of misclassifying proximal centromeres as microsomes.

[0133] The detection head outputs target location information, category information, and confidence level.

[0134] If different species are mixed in the same batch, simply replace the one-hot index with the corresponding chromosome number, and the vector will be automatically updated without modifying the network structure, thus achieving plug-and-play functionality with number hints.

[0135] Furthermore, to improve the image quality input to the visual detection model, color perturbation and multi-staining mixing can be applied to the input image. Color perturbation involves intensity shifting of the S and V channels in the HSV space to simulate differences in the concentration of different staining solutions. Multi-staining mixing involves saving copies of the Giemsa, G-band, and C-band channels for the same field of view image.

[0136] To facilitate the adaptation of the detection model to multiple species and to enable rapid switching between different species and tissue sources without retraining the visual-language joint detection model (e.g., switching between human → mouse → CHO cells), this application further constructs a prompt template library, a dynamic vocabulary mapping table, and feature space preservation constraints at the text level. The prompt template library stores preset prompt text information for multiple species; the dynamic vocabulary mapping table maps general distortion description information to species-specific description information during LLM-generated Tc; and the feature space preservation constraints ensure the distance between the visual feature vector output by the visual detection model and the existing visual feature vector in a common subspace when adapting to a new species. The existing visual feature vector refers to the visual feature vector output by the visual detection model before adaptation, preferably the original visual feature vector of the initial species being detected. For example, when switching between human → mouse → CHO cells, the existing feature vector is the human visual feature vector.

[0137] As an example, the cue template library stores structured cue text information for preset species. The data structure of the cue template library is represented as key-value pairs. The key information represents the species and its tissue origin information, while the value information represents the object, which includes: a chromosome number field, a prefix text field for the cue text information Tg, and a rare aberration category information field for the species represented by the key information. The rare aberration category information can be used for LLM cueing to reduce hallucinations. When a new species is added, only one key-value pair needs to be added to the cue template library; the model parameters of the visual-language joint detection model do not need to be changed, which is equivalent to hot-swapping.

[0138] As an example, the prompt text information Tg can be obtained as follows: Based on the species and tissue origin information input by the user, the key information in the prompt template library is read. The prefix text and chromosome number of the Tg corresponding to that key information can be obtained through key-value pairs. The confidence score of the current image detection is filtered according to a general label list to obtain the aberration type label. The prefix text and aberration type label of the Tg are concatenated according to the format information to obtain the Tg. For example, the prompt text information format is "{species}+{tissue origin}+{chromosome number}, which indicates {aberration type}".

[0139] The dynamic vocabulary mapping table includes a user-defined dictionary and a system default mapping table. The user-defined dictionary has higher priority than the system default mapping table. The path of the user-defined dictionary is configurable. The system default mapping table maintains a one-to-one or one-to-many correspondence between general tags and official species terms.

[0140] As an example, before generating a Tc, the LLM replaces common tags appearing in the Tg with official terms according to a dynamic vocabulary mapping table to ensure that the description is consistent with clinical reporting standards and avoids confusion of terms from multiple species.

[0141] The feature space preservation constraint ensures that the visual features of newly added species are close enough to the existing visual features in the common subspace to prevent the decrease in detection accuracy caused by species switching on the text side or visual drift on the visual side.

[0142] Taking existing visual features as an example of human characteristics. Let the distribution of human visual features be... Distribution of visual features of new species ,in, For the visual features output by the visual detection model, a distance metric is defined to represent the feature space preservation constraint:

[0143]

[0144] in, This measures the distance between the visual features of a new species and existing visual features within a common subspace. This represents a D-dimensional feature vector randomly selected from the feature distribution of the new species. This means arbitrarily selecting a D-dimensional feature vector from the distribution of human characteristics. This represents the human features that are closest to the feature vector z of the new species. The shortest distance between them. This represents the average of all the shortest distances for the new species.

[0145] When the image to be detected is an image of a newly added species and the number of images of newly added species to be detected is not greater than the first threshold, the retraining of the visual-language joint detection model is abandoned, and only the prompt template library and dynamic vocabulary mapping table are updated. The detection feature space is kept constrained. When the distance metric is greater than the set distance threshold, the model parameters of the visual detection model are adjusted until the distance metric is less than or equal to the distance threshold, so as to maintain the feature distinguishability of the new and old species by using contrastive learning loss.

[0146] To improve adaptability to multiple species, when the number of images of newly added species to be detected is less than a set second threshold but greater than a first threshold, for example, when 50 images ≤ the number of images of newly added species to be detected ≤ 1000 images, the new prompt text information Tg generated by the target image location information and category information from the visual detection model, as well as the prefix text of Tg in the template library, is provided to the LLM to quickly complete the multimodal large language model transfer.

[0147] See Figure 3 As shown, Figure 3 This is a flowchart illustrating a visual-language joint detection model transfer for chromosome detection in multi-species organisms when the number of images of newly added species to be detected is less than a second threshold but greater than a first threshold. It includes:

[0148] Step 301: Generate the prefix text of the Tg for the new species and construct the Tg for the new species.

[0149] Based on the species and tissue origin, a prefix text is generated. This prefix text includes: species information, cell type information, and chromosome number information. For example, the prefix text is "mouse spermatocyte chromosomes, 2n=40, visible".

[0150] Subsequently, the list of category logical values ​​output by the visual detection model is filtered, and the filtered list of category logical values ​​is concatenated to the prefix text to form Tg. For example, Tg is "mouse spermatocyte chromosome, 2n=40, dic, f, min are visible".

[0151] The constructed Tg input is fed into the visual-language joint detection model as an open vocabulary query condition, so that the instant switching of new species can be completed without updating any model weights.

[0152] Step 302: Construct prototype vectors to represent the visual features of the new species, so as to achieve rapid feature adaptation in the future.

[0153] First, a visual detection model is used to infer the image of the newly added species to be detected, and to obtain the target image location information, category, confidence level and target visual features.

[0154] Subsequently, based on various distortion types, the input images of the inferred new species are divided into support sets for each distortion type. Average pooling is then performed on the visual features of the inferred new species images within each support set to obtain prototype vectors for each distortion type, representing the visual "average fingerprint" of each distortion type in the new species. For example, if the distortion types include dic, f, min, and normal, then prototype vectors for each of these four types are obtained.

[0155] In this way, when performing object detection on an image, the visual detection model extracts the visual features of the object and calculates the cosine similarity between it and each prototype vector. If the highest similarity exceeds the set similarity threshold, such as 0.85, the corresponding category label of the prototype vector is directly assigned. Otherwise, the detection head outputs the object detection result.

[0156] As an example, the prototype vector can be updated online. For instance, after every 100 images of new species to be detected, the prototype vector is incrementally updated using a moving average based on the confirmed results, thereby achieving continuous optimization of the species feature space.

[0157] Step 303, adaptive first confidence threshold calibration to improve recall for rare aberration types.

[0158] For rare distortion types with extremely low frequency (e.g., about 0.3% in historical statistics), prior weights are introduced to calibrate the first confidence threshold of the detection results output by the visual detection model.

[0159] For example, for any rare aberration type, a prior weight is determined based on the occurrence ratio of the rare aberration type, and it is used to adjust the first confidence threshold. That is, the prior weight is weighted by the first confidence threshold of the general category to obtain the calibrated confidence threshold, which is used as the second confidence threshold. In this way, the second confidence threshold of the rare category is lower than the first confidence threshold of the general category, thereby effectively improving the recall capability.

[0160] To suppress false positives that may be introduced by confidence threshold calibration, a species-shape consistency check is further performed on the target image location information to ensure that only true positives are added.

[0161] Throughout the transfer process, the network weights of the visual detection model and the LLM model are not updated. Only the prototype vector and confidence threshold are dynamically maintained, enabling rapid transfer of the detection model in zero-shot or small-shot scenarios without retraining the entire detection model.

[0162] Step 304: When the number of images of newly added species to be detected exceeds the second threshold, the full fine-tuning mode of the detection model is triggered, and all trainable parameters except for the prompt template library are updated, that is, online learning is performed.

[0163] The following describes the training process of the visual-language joint detection model in the embodiments of this application.

[0164] In this embodiment, the training process may include three stages: cross-species pre-training stage, species-specific fine-tuning stage, and online learning stage. Cross-species pre-training is used to build a general chromosome detection model, species-specific fine-tuning is used to achieve specific aberration category recognition, and online learning is used to update the model parameters of the visual-language joint detection model online.

[0165] See Figure 4 As shown, Figure 4 This is a schematic diagram of a process for training a visual-language joint detection model according to an embodiment of this application.

[0166] Step 401: Collect chromosome images of multiple species to obtain chromosome sample images of multiple species.

[0167] Step 402: Label the obtained chromosome sample images from multiple species with information on species, tissue origin, and staining method.

[0168] As an example,

[0169] Step 4021: Perform semi-automatic annotation.

[0170] Initial screening: Image preprocessing was performed using OpenCV adaptive thresholding to achieve preliminary segmentation of chromosome contours.

[0171] Weakly supervised pre-annotation: Invokes existing general vision-language models to perform inference on images and automatically generates bounding boxes and annotation mask images of common distortion types such as dic, f, and min.

[0172] Correction: Adjusted the target bounding box, adjusted the annotation mask image, and added special anomalies that were difficult to identify.

[0173] Medical description generation: The corrected bounding box, labeled mask image and species hint information are input into a multimodal large language model to generate a structured medical description.

[0174] Hallucination filtering: The rule engine removes uncertain descriptions such as "possible" and "hint"; if the length of the generated hint text information is less than the set number threshold of tokens, a regeneration mechanism is triggered to ensure the completeness and professionalism of the medical description.

[0175] Step 4022: To ensure data standardization and uniformity, the medical description generated in step 4021 is subjected to schema verification to ensure that the medical description conforms to preset rules. For example, the species name exists in a preset whitelist, the morphology field is between 50 and 300 characters long, and speculative words such as "maybe" and "hint" are prohibited.

[0176] Step 4023: To ensure that rare samples are not overwhelmed during data collection, a species-stratified sampling strategy is adopted.

[0177] Step 403: Input multi-species biological sample data into the visual-language joint detection model to be trained for cross-species pre-training. The sample data includes: sample images and labeled data, expected target text description information, and sample prompt text information. The sample images include multiple species, and the expected target text description information includes: medical description information.

[0178] As an example, the sample images are uniformly scaled to the same resolution, and the tinting channels such as Giemsa, G-band, and C-band are randomly combined so that the visual detection model relies on morphology rather than tinting tone, thereby enhancing its generalization ability.

[0179] The sample image, the corrected target location in the labeled data, and the labeled mask image are input into the block embedding layer in the visual detection model. The chromosome number in the sample prompt text information is input into the chromosome prior embedding layer in the visual detection model. The sample prompt text information is input into the LLM model.

[0180] In the cross-species pre-training phase, the model parameters of the block embedding layer and Stage-1 layer are frozen to preserve general edge and texture features. The model parameters of Stage-2 to Stage-4 layers, the feature extraction network, and the detection head are not frozen. The detection and description generation tasks are jointly optimized; that is, the model parameters of the visual-language joint detection model are adjusted according to the total loss function value until the visual-language joint detection model converges, thus obtaining a general chromosome detection base with cross-species recognition capabilities. The total loss function value includes: the fusion detection loss value of target image location information regression and open word classification; the target image location region-hint text information comparison loss function value; the loss function value of the sample target text description information output by the multimodal large language model and the expected target text description information; and the cross-species structural consistency loss function value.

[0181] As an example, the cross-species structural consistency loss function value is calculated as follows:

[0182] Canny edge contour extraction is performed on the labeled mask image and the sample mask image at the location of the sample target image detected by the detection head, respectively, to obtain the labeled edge point set of the labeled mask image and the sample edge point set of the sample mask image;

[0183] For each sample edge point, determine the nearest labeled edge point, calculate the Chamfer distance between the sample edge point and each of the determined nearest labeled edge points, and average them to obtain the first average value;

[0184] For each labeled edge point, determine its nearest sample edge point, calculate the Chamfer distance between the labeled edge point and all the determined nearest sample edge points, and average them to obtain a second average value.

[0185] Calculate the average of the first average and the second average, and then apply dynamic weights to this average to obtain the cross-species structural consistency loss function value;

[0186] Among them, the dynamic weight increases linearly with the increase of the number of species to ensure that the more species there are, the stronger the shape constraint, the edge of the microbody is forced to closely fit the outline of the real fragment, reduce the circular false positive tail and ensure that the outlines of different chromosomes remain consistent.

[0187] Step 404: For the visual-language joint detection model obtained in the cross-species pre-training stage, freeze the model parameters of the visual detection model and fine-tune the LLM model parameters to enhance the visual description ability of the LLM.

[0188] As an example, a vocabulary injection approach that does not require re-pre-training is employed. Specifically, the LLM model is based on a vocabulary and adds cross-species cytogenetics terms as new tokens to the vocabulary. Each of these new tokens corresponds to a unique token ID, and its embedding vector is randomly initialized. The LLM model learns to establish an association mapping between these new tokens and the corresponding cytogenetics visual features in the input image, thereby achieving accurate identification and description of domain-specific concepts.

[0189] During training, sample data is input into the visual-language joint detection model obtained in step 403. Only the embedding vectors corresponding to newly injected terms and the output projection layer of the LLM model are updated; the model parameters of the visual detection model and the weights related to the original vocabulary are frozen. The output projection layer uses a low-rank adaptive method, for example, setting the rank to 8. The loss function value uses the standard language modeling cross-entropy loss function, enabling the LLM model to autoregressively generate target text descriptions containing accurate professional terminology based on the input sample data.

[0190] To ensure accurate triggering of terms, species prefixes are explicitly annotated in the sample prompt text. For example, if the input sample prompt text is: "Mouse spermatocyte chromosome, 2n=40, see dic, f, please describe in detail," the abbreviations "dic" and "f" are automatically replaced with new terms before inference through a pre-set dynamic vocabulary mapping table. This ensures that the multimodal large language model naturally uses standard terms in its output without introducing additional constraints or loss functions, thus achieving a stable association between professional descriptions and new vocabulary while maintaining lightweight training.

[0191] Step 405: To improve the detection accuracy of specific distortions, the detection model is fine-tuned for species-specific fine-tuning training.

[0192] As an example, the sample data is input into the visual-language joint detection model obtained in step 403, the model parameters of the block embedding layer, Stage-1 layer, and Stage-2 layer are frozen, the model parameters of Stage-3 to Stage-4 layers, the detection head, and the description head are fine-tuned, and the model parameters of the species-specific fine-tuning stage are saved as a dedicated model for inference.

[0193] Step 406: If the visual-language joint detection model performs poorly in chromosome detection for the newly added species after the model transfer is completed, the online learning process is triggered.

[0194] See Figure 5 As shown, Figure 5 This is a schematic diagram of an online learning process in this embodiment. It includes:

[0195] Step 501: In order to expand new terms in a timely manner, a hot-plug method based on prompt text information is adopted to achieve rapid activation of new terms without updating model weights.

[0196] As an example, if any specific term, such as "mouse spermatocyte synapse complex disruption," is not correctly invoked by the visual-language joint detection model, immediate expansion can be achieved through the following process:

[0197] First, add a key-value pair to the pre-defined dynamic vocabulary mapping table to map abbreviations and / or aliases to canonical terms. The key information represents the abbreviation and / or alias of the specific term, and the value information represents the specific term itself. For example, add "synaptonemal_break": "mouse synaptonemal break".

[0198] Secondly, explicitly embed the specific term in the input prompt text, for example, in the format: When the image shows a synaptonemal break, please describe it as "mouse synaptonemal break"; Current: mouse spermatocyte chromosome, 2n=40, see synaptonemal_break.

[0199] In this way, when generating target text description information, the LLM model can immediately respond and accurately output the newly added term "mouse synaptic complex rupture". The entire process does not involve any adjustment or fine-tuning of model parameters, realizing the ability to instantly expand and hot-swap terms at the prompt information level, significantly improving the flexibility and maintainability in professional domain descriptions.

[0200] Step 502: The images of the newly added species to be detected are stored in the support set as samples in real time, and after automatic annotation, quadruple samples are generated.

[0201] For example, a weakly supervised model is used to label target location information and category information, and the labeled target location information is corrected to obtain corrected target location information; based on the category information, prompt text information and medical description text are generated, and the image, the generated prompt text information, medical description text, and corrected target location information and its mask image are combined into a quadruple, and the generated medical description text Tc is subjected to description pattern (Schema) verification and hallucination filtering.

[0202] Step 503: Input the current quadruple sample into the current visual-language joint detection model. Through inference by the current visual detection model, obtain the visual features of the image of the newly added species to be detected.

[0203] Step 504: Calculate the feature space preservation constraints for the visual features of the image of the newly added species to be detected.

[0204] Step 505: If the feature space constraint remains greater than the distance threshold, update the model parameters of the last layer in the cue template library, prototype vector, detection head, and description head; freeze the model parameters of the backbone network layer and feature extraction network layer; and, if the training steps reach the set threshold for the first step, mix in the initial species samples for replay training to prevent forgetting; otherwise, abandon the update of model parameters.

[0205] Step 506: Obtain the next quaternion sample, return to step 503, and continue until the number of training steps reaches the set second step threshold, at which point the current online learning process ends, where the second step threshold is greater than the first step threshold.

[0206] Through the above three-stage training process, a general detection base, species-specific recognition, and online hot updates of the visual-language joint detection model are gradually constructed, forming a continuous learning closed loop to meet the high efficiency and accuracy requirements of on-site cytogenetics analysis.

[0207] See Figure 6 As shown, Figure 6 This is a schematic diagram of a chromosome detection device for multiple species according to an embodiment of this application. The device includes:

[0208] The detection module is used to input the image to be detected and the prompt text information corresponding to the species and the tissue origin of the species into the trained visual-language joint detection model. After the reasoning of the trained visual-language joint detection model, a structured detection result is generated. The structured detection result includes at least: target image location information, target category information, and target text description information.

[0209] in,

[0210] The joint visual-language detection model includes: a visual detection model for image-based object detection, and a multimodal large language model for generating target text descriptions based on cue information and visual features from the visual detection model.

[0211] The structured detection result is the fusion result of the target detection result from the visual detection model and the target text description information from the multimodal large language model. The target detection result includes at least the target image location information and the target category information.

[0212] As an example, the device also includes at least one of the following modules:

[0213] The adaptation module is used to abandon the retraining of the visual-language joint detection model when the image to be detected is an image of a newly added species to be detected, and the number of such images is less than a set first threshold. Instead, it updates the prompt template library and dynamic vocabulary mapping table, maintains the feature space preservation constraint, and triggers model parameter adjustment of the visual detection model when the feature space preservation constraint does not meet the set constraint conditions, until the feature space preservation constraint meets the set constraint conditions.

[0214] The transfer module is used to trigger the transfer of the visual-language joint detection model when the image to be detected is an image of a newly added species to be detected, and the number of such images is greater than a set first threshold and less than a set second threshold.

[0215] The online learning module is used to trigger the online learning of the visual-language joint detection model when the image to be detected is an image of a newly added species to be detected, and the number of such images is greater than or equal to a second quantity threshold.

[0216] See Figure 7 As shown, Figure 7 This is another schematic diagram of a chromosome detection device for multiple species according to an embodiment of this application. The device includes a memory and a processor. The memory stores a computer program, and the processor is configured to execute the computer program to implement the steps of the chromosome detection method for multiple species according to an embodiment of this application.

[0217] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0218] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0219] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the chromosome detection method for multiple species of organisms according to the embodiments of this application.

[0220] For the device / network-side equipment / storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments.

[0221] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0222] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A chromosome detection method for multi-species organisms, characterized in that, The method includes: The image to be detected and the prompt text information corresponding to the species and the tissue origin of the species are input into the trained visual-language joint detection model. After the reasoning of the trained visual-language joint detection model, a structured detection result is generated. The structured detection result includes at least: target image location information, target category information, and target text description information. in, The joint visual-language detection model includes: a visual detection model for image-based object detection, and a multimodal large language model for generating target text descriptions based on cue information and visual features from the visual detection model. The structured detection result is the fusion result of the target detection result from the visual detection model and the target text description information from the multimodal large language model. The target detection result includes at least the target image location information and the target category information.

2. The method according to claim 1, characterized in that, The prompt text information is obtained in the following manner: Based on the species and its tissue origin information, the prefix text field and chromosome number field of the corresponding prompt text information are retrieved from the prompt template library. According to the set prompt text information format, the prefix text field, chromosome number field, species information, tissue origin information, and general malformation type information are used to construct the prompt text information. in, The prompt template library stores structured prompt text information for preset species. The data structure of the prompt template library is represented in key-value pair form. The key information is used to characterize the species and its tissue source information, and the value information is used to characterize the prompt text information. The prompt text information includes: chromosome number field, prefix text field of prompt text information, and rare aberration category information used to avoid hallucination output; The visual detection model includes: a backbone network layer, a feature extraction network layer, a chromosome number prior embedding layer, and a detection head. in, The image to be detected is input into the backbone network layer. The scale features from the backbone network layer are input into the feature extraction network layer. The P2 to P5 level features from the feature extraction network layer are input into the detection head. The P4 and P5 level features from the feature extraction network layer are input into the multimodal large language model. The chromosome number prior embedding layer maps the chromosome number in the prompt information into a multi-dimensional one-dimensional effective encoding vector, which is then compressed into a learnable vector by a linear layer in the chromosome number prior embedding layer. This learnable vector is then concatenated pixel-by-pixel with the P3 level features, and the concatenated features are input to the detection head. The detection head outputs target image location information, target category information, and confidence level based on the input features.

3. The method according to claim 2, characterized in that, The method further includes: When the image to be detected is an image of a newly added species to be detected, and the number of such images is less than a set first threshold, Instead of retraining the joint visual-language detection model, we update the cue template library and dynamic vocabulary mapping table, while maintaining constraints in the detection feature space. If the feature space preservation constraints are not met, the model parameters of the visual detection model are adjusted until the feature space preservation constraints are satisfied. in, Dynamic vocabulary mapping tables are used in the process of generating target text description information in multimodal large language models to map general distortion description information in prompt text information to species-specific description information. The feature space preservation constraint is used to characterize the distance metric between the visual feature vector of the image of the new species to be detected output by the visual detection model and the existing visual feature vector output by the visual detection model in the common subspace.

4. The method according to claim 3, characterized in that, The update prompt template library and dynamic vocabulary mapping table include: The newly added species and tissue origin in the image to be detected are used as key information. The prefix text, chromosome number field, and rare aberration category information of the prompt text information of the newly added species are used as value information. The key information and value information of the newly added species are stored in the prompt template library in key-value pairs. The prefix text includes: species information and tissue origin information. Configure the user-defined dictionary in the dynamic vocabulary mapping table and the system default mapping table; The detection feature space preservation constraints include: A feature vector is randomly selected from the visual feature distribution of the new species output by the visual detection model as the first feature vector. A feature vector is randomly selected from the existing visual feature distribution output by the visual detection model as the second feature vector. The shortest distance among the distances between the second feature vectors that are closest to the first feature vector is determined, and the shortest distance of the first feature vector is obtained. The average of the shortest distances among the first feature vectors selected in the feature distribution of the new species output by the visual detection model is used to obtain the feature space preservation constraint for the new species. If the feature space of the new species is less than or equal to the set distance threshold, the feature space is determined to be subject to the set constraint condition.

5. The method according to claim 4, characterized in that, The method further includes: If the image to be detected is an image of a newly added species to be detected, and the number of such images is greater than a first threshold but less than a second threshold, the transfer of the visual-language joint detection model is triggered, wherein the first threshold is less than the second threshold. When the image to be detected is an image of a newly added species to be detected, and the number of such images is greater than or equal to the second quantity threshold, the online learning of the visual-language joint detection model is triggered.

6. The method according to claim 5, characterized in that, The visual-language joint detection model is transferred in the following manner: Generate prompt text information indicating the addition of a new species in the image to be detected. Dynamically construct the prototype feature vectors of each target image of the newly added species. The first confidence threshold, used to characterize the general category confidence threshold, is calibrated to obtain the second confidence threshold used to recall rare distortion types. The image of the newly added species to be detected and the generated prompt text information are input into the trained visual-language joint detection model. The visual features of the target image location information from the visual detection model in the joint visual-language detection model are matched with the prototype feature vector of the new species. If a match is successful, the category information corresponding to the prototype feature vector is used as the category information of the target image. If the match fails, the detection head in the visual detection model outputs the target image location information, target category information, and confidence score. Based on the confidence threshold, target detection results with a confidence level greater than the confidence threshold are selected.

7. The method according to claim 6, characterized in that, The prompt text information for generating the newly added species in the image to be detected includes: Based on the species information and tissue origin information of the newly added species, the prefix text of the prompt text information and the chromosome number information are determined in the template database. The determined prefix text, chromosome number information, and category information from the existing target detection results output by the visual detection model are concatenated to form the prompt text information; The dynamic construction of the prototype feature vectors for each target image of the newly added species includes: The visual detection model outputs the target image's location information, category information, confidence level, and visual features during the inference process of the image of the newly added species to be detected. When the cumulative number of images of newly inferred species reaches a set inference threshold, Based on the various distortion types of the cumulative inferred new species images, the cumulative inferred new species images are divided into support sets for each distortion type. For each distortion type, average pooling is performed on the visual features of the target images of each newly added species in the support set of that distortion type to obtain the prototype vector of that distortion type. The second confidence threshold is determined as follows: For any rare aberration type, the prior weight is determined based on the proportion of occurrence of that rare aberration type. The first confidence threshold is weighted by the prior weights to obtain the second confidence threshold; Following the transfer of the triggering visual-language joint detection model, the following further includes: When the visual-language joint detection model fails to meet expectations in detecting new species after transfer, online learning of the visual-language joint detection model is triggered.

8. The method according to claim 7, characterized in that, The training of the visual-language joint detection model includes: a cross-species pre-training stage for constructing a general chromosome detection model, a species-specific fine-tuning stage for achieving specific aberration category recognition, and an online learning stage for hot updating of model parameters. in, Cross-species pre-training is conducted in the following manner: The model parameters of the frozen visual detection model's backbone network layers, including the block embedding layer and Stage-1, and the unfrozen backbone network layers, including Stage-2 to Stage-4, the feature extraction network, and the detection head, are as follows: Multi-species biological sample data is input into a visual-language joint detection model to be trained. The sample data includes: sample images and their annotation data, sample prompt text information, and desired target text description information. The annotation data includes: corrected target image location information and the annotation mask image at that location. The sample images and their annotation data are input into the backbone network layer of the visual detection model. The chromosome number in the sample prompt text information is input into the chromosome number prior embedding layer. The sample prompt text information is then input into a multimodal large language model. The parameters of the unfrozen model are adjusted according to the total loss function value until convergence, resulting in the visual-language joint detection model obtained in the cross-species pre-training stage. The total loss function value includes: the fusion detection loss value of target image location information regression and open word classification, the loss function value of target image location region-cause text information comparison, the loss function value of sample target text description information and expected target text description information output by the multimodal large language model, and the cross-species structural consistency loss function value. Species-specific fine-tuning is performed in the following manner: The sample data is input into the visual-language joint detection model obtained during the cross-species pre-training stage. The model parameters of the block embedding layer, Stage-1 layer, and Stage-2 layer in the visual detection model of the visual-language joint detection model obtained in the cross-species pre-training stage are frozen, and the model parameters of Stage-3 to Stage-4 layers, the detection head, and the description head in the multimodal large language model are fine-tuned. Online learning will be conducted in the following ways: The image of the newly discovered species to be detected is labeled as a sample to obtain sample data. This sample data is then input into the current visual-language joint detection model. Through inference by the visual detection model in the current visual-language joint detection model, the visual features of the target image of the newly discovered species to be detected are obtained. The feature space is preserved when calculating the visual features of the target image. While ensuring that the constraints in the calculated feature space are satisfied, update the model parameters of the current prompt template library, prototype vector, detection head, and descriptor head. Freeze the model parameters of the backbone network layer and feature extraction network layer. When the cumulative number of training steps reaches the set threshold for the first step, replay training is performed by mixing in sample images of the initial species. The next image of a newly added species to be detected is used as a sample, and the process of annotating the current image of a newly added species to be detected is repeated until the total number of training steps reaches a certain second threshold, at which point online learning ends. The second threshold is greater than the first threshold.

9. The method according to claim 8, characterized in that, The cross-species structural consistency loss function value is determined in the following manner: Canny edge contour extraction is performed on both the labeled mask image and the sample mask image to obtain the labeled edge point set of the labeled mask image and the sample edge point set of the sample mask image. The sample mask image is the mask image of the sample target image position detected by the detection head. For each sample edge point, determine its nearest labeled edge point, calculate the Chamfer distance between the sample edge point and each of the determined nearest labeled edge points, and average them to obtain the first average value. For each labeled edge point, determine its nearest sample edge point, calculate the Chamfer distance between the labeled edge point and all the determined nearest sample edge points, and average them to obtain a second average value. Calculate the average of the first average and the second average, and apply dynamic weights to this average to obtain the cross-species structural consistency loss function value, where the dynamic weights increase linearly with the increase of the number of species; The cross-species pre-training also includes: For the visual-language joint detection model obtained in the cross-species pre-training stage, the model parameters of the visual detection model are frozen, and the model parameters of the multimodal large language model are fine-tuned. The fine-tuning is carried out in the following manner: Cross-species cytogenetics terms will be added as new tags to the vocabulary. The sample data is input into the visual-language joint detection model obtained in the cross-species pre-training stage. Based on the cross-entropy loss function value of standard language modeling, the parameters of the embedding vector corresponding to the new label and the output projection layer in the multimodal large language model are updated, so that the multimodal large language model can learn to establish the association mapping between the new label and the corresponding visual features in the sample image. The online learning further includes: If any specific term is not correctly invoked by the visual-language joint detection model, a key-value pair for that specific term is added to the dynamic vocabulary mapping table, and the specific term is explicitly embedded in the current prompt text information. Here, the key information is used to represent the abbreviation and / or alias of the specific term, and the value information is used to represent the specific term.

10. A chromosome detection device for multiple species of organisms, characterized in that, The device includes: The detection module is used to input the image to be detected and the prompt text information corresponding to the species and the tissue origin of the species into the trained visual-language joint detection model. After the reasoning of the trained visual-language joint detection model, a structured detection result is generated. The structured detection result includes at least: target image location information, target category information, and target text description information. in, The joint visual-language detection model includes: a visual detection model for image-based object detection, and a multimodal large language model for generating target text descriptions based on cue information and visual features from the visual detection model. The structured detection result is the fusion result of the target detection result from the visual detection model and the target text description information from the multimodal large language model. The target detection result includes at least the target image location information and the target category information.

Images