Retriever model training method, birt-hogg-dube syndrome identification method and system based on retrieval enhancement generation

Abstract

This application discloses a retrieval model training method, a retrieval-enhanced generation-based Bert-Hogg-Dubbs syndrome (BHD) identification method and system, relating to the field of rare disease identification. The training method includes acquiring positive and negative sample pairs; inputting the positive and negative sample pairs into an initial retrieval model; calculating the loss function of the initial retrieval model; and dynamically adjusting the angle margin of the loss function in real time according to a metric variance adaptive mechanism. The initial retrieval model is then optimized based on the loss function calculation results. This application forcibly expands the angle interval between BHD and non-BHD by using the angle margin of the loss function, and uses a metric variance adaptive mechanism to dynamically adjust the angle margin based on the statistical variance of the cosine similarity among all positive sample pairs in the current training batch. This solves the problem of weak image differences and blurred category decision boundaries in DCLDs caused by the highly similar imaging features of various rare diseases, thus improving the recognition accuracy of large models for query information.

Description

Technical Field

[0001] This application relates to the field of rare disease identification, and in particular to a retrieval model training method, a retrieval-enhanced generation-based Bert-Hogg-Dub syndrome identification method and system. Background Technology

[0002] CT imaging is the most commonly used imaging tool for diagnosing Birt-Hogg-Dubé syndrome (BHD), and the imaging diagnosis of this disease is often based on cystic lesions and pneumothorax on chest CT. However, diffuse cystic lung diseases (DCLDs) have similar imaging features, and BHD can easily be confused with other DCLDs, such as lymphangioleiomyomatosis (LAM), pulmonary Langerhans cell histiocytosis (PLCH), and lymphocytic interstitial pneumonia (LIP). Differentiating BHD from other diffuse cystic lung diseases on CT images is quite difficult.

[0003] In recent years, deep learning methods have achieved remarkable results in various challenging disease diagnosis tasks, surpassing the performance of human experts. However, due to BHD being a typical rare disease, obtaining clinically relevant training samples is difficult, and the data scale is limited. Furthermore, the similar typical image features of DCLDs make it difficult for traditional discriminative methods to optimize complex decision boundaries and achieve good BHD classification results. Currently, extensive research has been conducted on using large models to solve problems in the medical field; for example, GPT-4 can outperform the medical-specific model Med-PaLM2 on medical question-answering benchmarks without targeted training. Therefore, utilizing the general medical and pathological prior knowledge of multimodal large models to address the difficulty in obtaining BHD samples is a realistic and feasible approach.

[0004] However, large models suffer from a serious hallucination problem, producing outputs that appear reasonable but are actually incorrect. Easily confused DCLD image features can exacerbate the hallucination phenomenon. Summary of the Invention

[0005] The purpose of this application is to provide a retrieval model training method, a retrieval-enhanced generation-based Bert-Hogg-Dub syndrome identification method and system, which can solve the problem of weak differences in DCLD images and blurred category decision boundaries caused by the very similar imaging features of various DCLD rare diseases, improve the recognition accuracy of large models for query information, and enable large models to output reasonable and accurate recognition results.

[0006] To achieve the above objectives, this application provides the following solution:

[0007] Firstly, this application provides a method for training a retrieval model, the method comprising:

[0008] According to the preset sampling strategy, positive sample pairs and negative sample pairs of the current training batch are obtained. The positive sample pair includes two positive samples, and the negative sample pair includes one positive sample and one negative sample. The positive sample is a CT image of the target disease, and the negative sample is a CT image of a non-target disease.

[0009] The positive sample pairs and the negative sample pairs are input into the initial retrieval model, and the loss function of the initial retrieval model is calculated. The loss function adjusts the size of the angle margin in real time according to the metric variance adaptive mechanism, where the angle margin is the feature interval between positive and negative samples. The method for the metric variance adaptive mechanism to adjust the size of the angle margin in real time includes:

[0010] Calculate the variance of the current training batch, which is the variance of the cosine similarity of all positive samples in the current training batch; calculate the dynamic margin based on the variance of the current training batch and the global variance; update the global variance using the variance of the current training batch; wherein, the global variance is the variance of the cosine similarity of all positive samples in all training batches.

[0011] Update the angle boundary using the dynamic boundary;

[0012] The initial retrieval model is optimized based on the loss function calculation results to obtain the retrieval model.

[0013] In one embodiment, obtaining positive and negative sample pairs for the current training batch according to a preset sampling strategy includes:

[0014] Select all negative sample pairs within all training batches;

[0015] Calculate the cosine similarity between positive and negative samples in all negative sample pairs;

[0016] Based on the calculation results of the cosine similarity, all negative samples in all training batches are sorted in descending order, and the top T negative samples in the descending order are selected as high similarity negative samples.

[0017] Highly similar negative samples are classified according to the subtypes of non-target diseases to obtain at least two categories of highly similar negative sample groups, with each category of the highly similar negative sample group corresponding to a subtype of a non-target disease.

[0018] A preset number of negative sample pairs are selected from all negative sample pairs in all training batches as negative sample pairs for the current training batch. The negative samples in the current batch include high similarity negative sample groups of all categories, and the number of negative samples in each high similarity negative sample group is set according to a preset ratio.

[0019] In one embodiment, inputting the positive sample pairs and the negative sample pairs into the initial retrieval model includes:

[0020] The positive sample pairs and the negative sample pairs are input into the initial retrieval model, and the initial retrieval model performs multi-stage downsampling operations on the positive sample pairs and the negative sample pairs sequentially, wherein:

[0021] The initial retrieval model includes multiple sequentially connected convolutional modules, an attention module connected in parallel with each convolutional module, and an output layer. The first feature output by the previous convolutional module is processed by the corresponding parallel attention module to output a second feature. The second feature is returned to the convolutional module and concatenated with the first feature. The concatenated feature is used as the input of the next convolutional module. The concatenated feature output by the last convolutional module is used as the input of the output layer.

[0022] In one embodiment, the loss function is the CosFace margin loss function, and the expression of the CosFacemargin loss function is:

[0023] ;

[0024] In the formula: Indicates positive sample pairs. Indicates negative sample pairs; This represents the total number of positive and negative sample pairs in the current training batch. The cosine similarity between sample features is represented by... Indicates the scale factor. Indicates the angular boundary.

[0025] In one embodiment, the formula for calculating the dynamic margin based on the current training batch variance, the global variance, and the preset adjustment sensitivity coefficient is as follows:

[0026] ;

[0027] In the formula: Indicates dynamic margin, Indicates the boundary of the reference angle. This represents the variance of the current training batch. Represents the global variance. This represents the activation function.

[0028] Secondly, this application provides a method for identifying Bert-Hogg-Dubb syndrome based on retrieval enhancement, the method comprising:

[0029] Obtain the user's query information, which includes the original query image and the original query text;

[0030] The retrieval model is used to retrieve the K most relevant image-text pairs that are most relevant to the target query image from a pre-built knowledge base. The relevant image-text pairs include candidate images and candidate descriptive texts corresponding to the candidate images.

[0031] Based on the query information, rearrange the K relevant image-text pairs to obtain the optimal relevant image-text pair;

[0032] The optimal relevance image-text pair and the user's query information are input into the multimodal large model to obtain the recognition result;

[0033] The retrieval model is trained using the method described above.

[0034] In one implementation, rearranging K relevant image-text pairs based on query information to obtain the optimal relevant image-text pair includes:

[0035] The original query image in the query information is used as the query sample, and the candidate images in the relevant image-text pairs are used as candidate samples.

[0036] Project the feature vectors of the candidate samples onto M randomly generated feature subspaces. ;

[0037] Within each feature subspace, evaluation data for query samples and candidate samples is calculated. The evaluation data includes the cosine similarity between query samples and candidate samples, the local angular variance between query samples and candidate samples, and the cluster density of candidate samples.

[0038] Calculate the local consistency score of the candidate samples based on the evaluation data of the query samples and the candidate samples;

[0039] The rearrangement score of the candidate sample in the M feature subspaces is calculated based on the local consistency score;

[0040] Based on the rearrangement score, all candidate images in the K related image-text pairs are sorted in descending order, and the related image-text pair corresponding to the first candidate image in the descending sort is selected as the optimal related image-text pair.

[0041] In one embodiment, the formula for calculating the local consistency score is:

[0042] ;

[0043] In the formula: Indicates the first In the feature subspace, the ... Local consistency score of each candidate sample; Indicates the first In each feature subspace, the query sample With the candidate samples Cosine similarity between them; Indicates query sample With the candidate samples Local angular variance between; Indicates the first candidate samples The local cluster density.

[0044] In one embodiment, the formula for calculating the rearrangement fraction is:

[0045] ;

[0046] In the formula, Indicates for the query sample Candidate samples Rearranged fractions in M ​​feature subspaces; Indicates the first In the feature subspace, the ... The local consistency score of each candidate sample.

[0047] Thirdly, this application provides a retrieval-enhanced generation-based Bert-Hogg-Dubb syndrome identification system, characterized in that the retrieval-enhanced generation-based Bert-Hogg-Dubb syndrome identification system includes:

[0048] A data acquisition unit is used to acquire user query information, which includes original query image and original query text.

[0049] A retrieval model is used to retrieve the K most relevant image-text pairs that are most relevant to the target query image from a pre-built knowledge base. The relevant image-text pairs include candidate images and candidate descriptive texts corresponding to the candidate images.

[0050] The data rearrangement unit is used to rearrange K relevant image-text pairs based on query information to obtain the optimal relevant image-text pair;

[0051] The model processing unit is used to input the optimal relevant image-text pair and target query information into the multimodal large model to obtain the recognition result;

[0052] The retrieval model is trained using the method described above.

[0053] Fourthly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the steps of the retrieval-enhanced generation-based Burt-Hogg-Dub syndrome identification method described in any one of the above applications.

[0054] Fifthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the retrieval-enhanced generation-based Bert-Hogg-Dub syndrome identification method described above.

[0055] Sixthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the retrieval-enhanced generation-based Burt-Hogg-Dub syndrome identification method described above.

[0056] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0057] This application provides a retrieval model training method, a retrieval-enhanced generation-based Bert-Hogg-Dubbs syndrome (BHD) identification method and system. First, it distinguishes between BHD and non-BHD by constructing positive and negative sample pairs. Then, it forcibly expands the angular interval between BHD and non-BHD through the angular margin of the loss function. Next, it uses a metric variance adaptive mechanism to dynamically adjust the angular margin based on the statistical variance of the cosine similarity among all positive sample pairs in the current training batch. This addresses the problem of weak differences in DCLD images and blurred category decision boundaries caused by the highly similar imaging features of various DCLD rare diseases, thereby improving the large model's recognition accuracy of query information and enabling the large model to output reasonable and accurate recognition results. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 A flowchart illustrating a retrieval model training method provided in the first embodiment of this application;

[0060] Figure 2A flowchart illustrating the method for obtaining positive and negative sample pairs of the current training batch according to a preset sampling strategy provided in the first embodiment of this application;

[0061] Figure 3 A flowchart illustrating the method for real-time adjustment of the size of the angular margin using the adaptive mechanism for measuring variance provided in the first embodiment of this application;

[0062] Figure 4 This is a schematic diagram of the retrieval model framework provided in the first embodiment of this application;

[0063] Figure 5 A flowchart illustrating a method for identifying Bert-Hogg-Dubbs syndrome based on retrieval enhancement generation, provided for the second embodiment of this application;

[0064] Figure 6 A flowchart illustrating the method for constructing a knowledge base as provided in the second embodiment of this application;

[0065] Figure 7 This is a flowchart illustrating the method for rearranging K relevant image-text pairs based on query information to obtain the optimal relevant image-text pair, as provided in the second embodiment of this application.

[0066] Figure 8 This is a schematic diagram illustrating the process of constructing a knowledge base and processing query information in a retrieval-enhanced generation-based Bert-Hogg-Dub syndrome identification method provided in the second embodiment of this application. Detailed Implementation

[0067] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0068] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] First Embodiment

[0070] like Figures 1-3 As shown, this application embodiment provides a retrieval model training method, including the following steps S101 to S103. Wherein:

[0071] Step S101: Obtain positive sample pairs and negative sample pairs for the current training batch according to a preset sampling strategy. The positive sample pair includes two positive samples, and the negative sample pair includes one positive sample and one negative sample. The positive sample is a CT image of the target disease, and the negative sample is a CT image of a non-target disease.

[0072] It should be noted that, in this embodiment, the target disease is Birt-Hogg-Dubé syndrome (BHD), which belongs to Diffuse Cystic Lung Diseases (DCLDs). Diffuse Cystic Lung Diseases (DCLDs) have similar imaging features, and BHD is easily confused with other non-BHDs. In order to make the retrieval model more accurately distinguish between BHD and non-BHD, in this embodiment, the non-target disease is mainly a non-BHD disease of DCLDs, such as lymphangioleiomyomatosis (LAM), pulmonary Langerhans cell histiocytosis (PLCH), and lymphocytic interstitial pneumonia (LIP).

[0073] Furthermore, before training the retrieval model, in order to obtain standard samples that the retrieval model can read, it is necessary to preprocess the CT images of the target disease and the CT images of non-target diseases. The processing methods for both are the same, specifically including CT image acquisition steps and data preprocessing steps. Among them:

[0074] CT image acquisition steps: Use CT imaging equipment and sensors to acquire CT image data of patients with DCLDs and record their DICOM sequences and patient clinical information; convert the CT images to .nii format, and divide the CT images into 2D slices along the coronal, sagittal and transverse views, retaining the slices with typical DCLDs lesions, with about 5 frames of 2D slices retained in each view.

[0075] Data preprocessing steps: All slices are first subjected to lung window normalization, with the following processing parameters: window width 1500HU, window level -600HU; then, multi-scale filtering regularization and skeleton-guided contrast enhancement adjustment are applied; finally, the slices are scaled to 256×256 (retrieval trainer) and 369×369 (knowledge base construction).

[0076] To reduce the training error of the retrieval model, the positive sample pairs are constructed using two CT image slices of BHD from different patients but in the same view, such as coronal CT image slices of BHD patient A and BHD patient B; the negative sample pairs are constructed using two CT image slices of different patients but in the same view, one CT image slice belonging to BHD and the other CT image slice belonging to non-BHD, such as sagittal CT image slices of BHD patient C and non-BHD patient D.

[0077] In a preferred embodiment, step S101, obtaining the positive and negative sample pairs of the current training batch according to a preset sampling strategy, includes the following steps S201-S205. Wherein:

[0078] Step S201: Select all negative sample pairs within all training batches. Before training the retrieval model, a certain number of CT images are selected as the training sample set; during model training, a certain number of samples are selected from the training sample set as samples for the current training batch; the samples from all training batches together constitute the training sample set, and the set of all training batches is the total training batch.

[0079] Step S202: Calculate the cosine similarity between positive and negative samples in all negative sample pairs.

[0080] Step S203: Sort all negative samples in all training batches in descending order according to the calculation results of the cosine similarity, and select the top T negative samples in the descending order as high similarity negative samples. It can be understood that the high similarity negative samples are the "extremely difficult negative samples" that are the most difficult to distinguish in all training batches. Selecting the "extremely difficult negative samples" and inputting them into the retrieval model can reduce the training bias of the retrieval model and improve the recognition accuracy of the retrieval model.

[0081] Step S204: Classify the high similarity negative samples according to the subtype of the non-target disease to obtain at least two categories of high similarity negative sample groups, and each category of the high similarity negative sample group corresponds to a subtype of a non-target disease.

[0082] Step S205: Select a preset number of negative sample pairs from all negative sample pairs in all training batches as negative sample pairs for the current training batch. The negative samples in the current batch include high similarity negative sample groups of all categories, and the number of negative samples in each high similarity negative sample group is set according to a preset ratio.

[0083] By implementing steps S201-S205 above, the "extremely difficult negative samples" are stratified a second time according to non-BHD disease subtypes (such as LAM, PLCH, LIP), ensuring that difficult samples from different confusing subgroups are sampled in a preset proportion in each batch of training. This two-level sampling mechanism forces the model to continuously and selectively learn the most subtle differences between BHD features and the most easily confused non-BHD subgroup features, thereby significantly enhancing the model's ability to distinguish between very similar samples near the decision boundary, directly improving the accuracy of clinical differential diagnosis. It is worth noting that negative sample pairs are obtained using steps S201-S205 above, while positive sample pairs can be obtained using common methods in existing technologies.

[0084] Step S102: Input the positive sample pairs and the negative sample pairs into the initial retrieval model, calculate the loss function of the initial retrieval model, and adjust the size of the angle margin in real time according to the metric variance adaptive mechanism. The angle margin is the feature interval between positive and negative samples. The angle margin is used to forcibly expand the angle interval between BHD and non-BHD, and constrain the similarity of samples of the same class to be higher than that of samples of different classes by at least a margin value (angle margin) in the cosine space, significantly improving the feature separation degree between BHD and non-BHD classes in easily confused DCLDs; wherein, the method of adjusting the size of the angle margin in real time by the metric variance adaptive mechanism includes the following steps S301 to S304. Wherein:

[0085] Step S301: Calculate the variance of the current training batch, where the variance of the current training batch is the variance of the cosine similarity of all positive samples in the current training batch;

[0086] Step S302: Calculate the dynamic margin based on the current training batch variance and the global variance, wherein:

[0087] In step S302 of this application embodiment, the global variance is the variance of the cosine similarity of all positive samples in all training batches. The calculation formula for the dynamic margin based on the current training batch variance, the global variance, and the preset adjustment sensitivity coefficient is as follows:

[0088] ;

[0089] In the formula: Indicates dynamic margin, Indicates the boundary of the reference angle. This represents the variance of the current training batch. Represents the global variance. This represents the activation function. It is the sensitivity coefficient that is adjusted.

[0090] Step S303: Update the global variance using the current training batch variance.

[0091] Step S304: Update the angle boundary using the dynamic boundary.

[0092] By implementing steps S301-S304 above, when the feature distribution of samples within a batch is dispersed (large variance), the metric variance adaptive mechanism automatically increases the angle margin to widen the inter-class distance in the broad feature space; when the feature distribution of samples within a batch is compact (small variance), the angle margin is appropriately reduced for finer adjustments. The metric variance adaptive mechanism extends the safe interval between BHD and non-BHD samples in the cosine space from the traditional 3° to 7°-10°, enabling the retrieval model to adaptively adjust the strictness of the decision boundary when facing changes in the data flow. This resolves the interference of fluctuations in the feature distribution of samples within different training batches on the fixed angle margin, thus demonstrating excellent robustness to fluctuations in feature distribution.

[0093] In step S102 of this embodiment, inputting the positive sample pair and the negative sample pair into the initial retrieval model includes: inputting the positive sample pair and the negative sample pair into the initial retrieval model, and performing multi-stage downsampling operations on the positive sample pair and the negative sample pair sequentially through the initial retrieval model, wherein:

[0094] The initial retrieval model includes multiple sequentially connected convolutional modules, an attention module connected in parallel with each convolutional module, and an output layer. The first feature output by the previous convolutional module is processed by the corresponding parallel attention module to output a second feature. The second feature is returned to the convolutional module and concatenated with the first feature. The concatenated feature is used as the input of the next convolutional module. The concatenated feature output by the last convolutional module is used as the input of the output layer.

[0095] It should be noted that, as Figure 4As shown, the backbone of the initial retrieval model is based on MobileViT-XS, and view-aware fusion is introduced in parallel at each stage. Attention (VAA) can automatically switch between multiple resolutions to balance detail and speed. Specifically, the initial retrieval model includes four sequentially connected convolutional modules: ConvBlock1, ConvBlock2, ConvBlock3, and ConvBlock4. The attention modules connected in parallel with the convolutional modules are VAA1, VAA2, VAA3, and VAA4. ConvBlock1 is connected to VAA1, ConvBlock2 to VAA2, ConvBlock3 to VAA3, and ConvBlock4 to VAA4. The four sequentially connected convolutional modules output feature maps at multiple resolution stages, such as feature maps with resolutions of 128×128, 64×64, 32×32, and 16×16. These feature maps of different resolutions form the basis of multi-resolution fusion: high-resolution features retain more lesion details (such as cyst wall thickness and morphology), while low-resolution features encode more discriminative global contextual information (such as cyst distribution patterns).

[0096] The method for processing feature maps using the aforementioned attention module includes a feature projection step, an attention calculation step, and a feature fusion and output step. Among these:

[0097] Feature projection steps: First, the input feature map is projected independently to generate Query, Key, and Value vectors respectively; during this process, the typical radiographic features of different views are encoded into a set of learnable embedding vectors as additional Keys in the attention mechanism.

[0098] Attention calculation steps: Calculate the attention score between the query vector generated by the feature map itself and the two types of key vectors. One type of key vector is the key vector generated by the feature map itself (representing image content), and the other type is the preset prior view key vector (representing expert knowledge). The association weight between each position of the feature map and these two types of key vectors is obtained through the Softmax function.

[0099] Feature fusion and output steps: The final output is a weighted sum of the image content Value vector and the prior view Value vector, with the weights being the attention scores calculated in the previous step.

[0100] By implementing the above attention module for feature map processing, the retrieval model can adaptively integrate external, view-specific medical knowledge into image features, enhance diagnostic-related feature responses, and suppress irrelevant noise.

[0101] In step S102 of this embodiment, the loss function is the CosFace margin loss function, and the expression of the CosFace margin loss function is:

[0102] ;

[0103] In the formula: This represents a pair of positive samples constructed from different CT slices belonging to the same class, obtained by random sampling. This represents a negative sample pair constructed from randomly sampled CT slices belonging to different classes, where... This indicates the first randomly selected segment from the BHD slice. i and the j One sample, This indicates the first segment randomly selected from non-BHD slices. j One sample; This represents the total number of positive and negative sample pairs in the current training batch. The cosine similarity between sample features is represented by... Indicates the scale factor. This represents the angular boundary. In this embodiment, the scale factor is 40, and the initial value of the angular boundary is 0.4, which is the reference angular boundary. .

[0104] It should be noted that in step S102, the CosFace margin loss function, which calculates the model's loss by comparing the cosine similarity between sample feature vectors, is a common retrieval strategy. However, the imaging features of various DCLD rare diseases are very similar, and the retrieval machine with Softmax loss cannot accurately classify the query samples in the cosine space. To address the problem of weak differences in DCLD images and ambiguous category decision boundaries, steps S301-S304 define a decision boundary in the cosine space (which automatically increases the angle margin through a variance adaptive mechanism) to increase the angle margin between the BHD class and non-BHD classes. The loss value of this loss function directly drives the retrieval model to extract image features with high discriminative power.

[0105] Step S103: Optimize the initial retrieval model using an optimizer based on the loss function calculation results. During training, the Ranger optimizer (a combination of RAdam and Lookahead) is used to run for 400 epochs with an initial learning rate of 2×10⁻⁶. -5 The cosine annealing-cyclic warmup strategy is adopted, and it is automatically restarted every 100 epochs; the weight decay is set to 1×10. -6 The batch size is 48.

[0106] The retrieval model training method implemented in steps S101-S103 first distinguishes between BHD and non-BHD by constructing positive sample pairs and negative sample pairs. Then, it forcibly expands the angular interval between BHD and non-BHD by using the angular margin of the loss function. Next, it uses a metric variance adaptive mechanism to dynamically adjust the angular margin based on the statistical variance of the cosine similarity between all positive sample pairs in the current training batch. This solves the problem of weak differences in DCLD images and blurred category decision boundaries caused by the very similar imaging features of various DCLD rare diseases. In this way, it improves the recognition accuracy of the large model for query information and enables the large model to output reasonable and accurate recognition results.

[0107] Second Embodiment

[0108] like Figures 5-8 As shown, a method for identifying Bert-Hogg-Dubbs syndrome based on retrieval enhancement includes the following steps S401 to S404. Wherein:

[0109] Step S401: Obtain the user's query information, which includes the original query image and the original query text; the original query image is a slice of the patient's CT image, and the original query text is the descriptive text corresponding to the original query image.

[0110] Step S402: Use the retrieval model to retrieve the K most relevant image-text pairs that are most relevant to the target query image from the pre-built knowledge base. The relevant image-text pairs include candidate images and candidate descriptive texts corresponding to the candidate images.

[0111] Please see Figure 8 In the (a) DCLD-external corpus construction part of this application embodiment, step S402, the knowledge base construction method includes the following steps S501-S503. Wherein:

[0112] S501. Obtain CT slices for knowledge base construction and initial image descriptions corresponding to the CT slices. The CT slices are obtained by the CT image acquisition steps in the first embodiment. The initial image descriptions can be obtained by processing the CT slices using a multimodal large model (GPT-4-turbo).

[0113] S502. The initial image description is standardized and corrected according to diagnostic criteria and imaging definition data to obtain an accurate image description that conforms to the diagnostic logic of human experts. Figure 8 The diagnostic criteria and imaging definitions include expert knowledge; wherein the diagnostic criteria and imaging definitions include Fleischner guidelines, BHD international consensus or other publicly published literature.

[0114] S503. Construct an image-text structure by matching the CT slices and the standardized, corrected initial image description text according to their correspondence, and store the image-text structure in an external corpus to obtain the knowledge base, which is the... Figure 8 The DCLD-external corpus is used in the knowledge base construction, which employs 369×369 high-resolution slices and dynamic compression caching to improve retrieval throughput.

[0115] Step S403: Rearrange the K relevant image-text pairs according to the query information to obtain the optimal relevant image-text pair;

[0116] In step S403 of this application embodiment, rearranging K related image-text pairs according to the query information to obtain the optimal related image-text pair includes the following steps S601-S606. Wherein:

[0117] Step S601: Use the original query image in the query information as the query sample, and use the candidate images in the relevant image-text pairs as candidate samples.

[0118] Step S602: Project the feature vectors of the candidate samples onto M randomly generated feature subspaces. ;

[0119] Step S603: Within each feature subspace, calculate the evaluation data of the query sample and the candidate sample. The evaluation data includes the cosine similarity between the query sample and the candidate sample, the local angular variance between the query sample and the candidate sample, and the cluster density of the candidate sample.

[0120] Step S604: Calculate the local consistency score of the candidate samples based on the evaluation data of the query samples and the candidate samples. The formula for calculating the local consistency score is as follows:

[0121] ;

[0122] In the formula: Indicates the first In the feature subspace, the ... Local consistency score of each candidate sample; Indicates the first In each feature subspace, the query sample With the candidate samples Cosine similarity between them; Indicates query sample With the candidate samples Local angular variance between; Indicates the first candidate samples The local cluster density.

[0123] Step S605: Calculate the rearrangement score of the candidate sample in the M feature subspaces based on the local consistency score. The formula for calculating the rearrangement score is as follows:

[0124] ;

[0125] In the formula, Indicates for the query sample Candidate samples Rearranged fractions in M ​​feature subspaces; Indicates the first In the feature subspace, the ... The local consistency score of each candidate sample.

[0126] Step S606: Sort all candidate images in the K related image-text pairs in descending order according to the rearrangement score, and select the related image-text pair corresponding to the first candidate image in the descending sort as the optimal related image-text pair.

[0127] By implementing steps S601-S606, the candidate images in the K related image-text pairs are rearranged by rearranging the scores. This significantly improves the ranking of candidate samples that maintain a small angle with the query sample in multiple subspaces and are located in high-density stable regions. It effectively eliminates some "pseudo-similar" candidate samples that are similar in angle in the main feature space but inconsistent in other subspaces, thereby significantly improving the reliability and confidence of the final returned results.

[0128] Step S404: Input the optimal relevant image-text pair and the user's query information into the multimodal large model to obtain the recognition result; In this embodiment, the multimodal large model adopts the GPT-4-turbo model, and the recognition result is the image recognition result corresponding to the original query image in the query information.

[0129] The retrieval model is trained using the retrieval model training method described in the first embodiment.

[0130] The process of implementing steps S401-S404 above can be found in [reference needed]. Figure 8 In part (b) BHD-RAG, steps S401-S404 are based on retrieval enhancement generation technology (RAG) to retrieve information related to the user's query from a pre-built knowledge base, and then use the retrieved information to guide the GPT-4-turbo model to generate image recognition results, thereby improving the quality and accuracy of the GPT-4-turbo model's predictions.

[0131] Third Embodiment

[0132] A Bert-Hogg-Dubbs syndrome identification system based on retrieval enhancement includes a data acquisition unit, a retrieval model, a data rearrangement unit, and a model processing unit. Wherein:

[0133] The data acquisition unit acquires the user's query information, which includes the original query image and the original query text. The original query image is a slice of the patient's CT image, and the original query text is the descriptive text corresponding to the original query image.

[0134] A retrieval model is used to retrieve the K most relevant image-text pairs to the target query image from a pre-built knowledge base. The relevant image-text pairs include candidate images and candidate descriptive texts corresponding to the candidate images.

[0135] The data rearrangement unit is used to rearrange K relevant image-text pairs based on query information to obtain the optimal relevant image-text pair;

[0136] The model processing unit inputs the optimal relevant image-text pair and the user's query information into the multimodal large model to obtain the recognition result. In this embodiment, the multimodal large model adopts the GPT-4-turbo model, and the recognition result is the image recognition result corresponding to the original query image in the query information.

[0137] The retrieval model is trained using the retrieval model training method described in the first embodiment.

[0138] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0139] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0140] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0141] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0142] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0143] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0144] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for training a retrieval model, characterized in that, The retrieval model training method includes: According to the preset sampling strategy, positive sample pairs and negative sample pairs of the current training batch are obtained. The positive sample pair includes two positive samples, and the negative sample pair includes one positive sample and one negative sample. The positive sample is a CT image of the target disease, and the negative sample is a CT image of a non-target disease. The positive sample pairs and the negative sample pairs are input into the initial retrieval model, and the loss function of the initial retrieval model is calculated. The loss function adjusts the size of the angle margin in real time according to the metric variance adaptive mechanism, where the angle margin is the feature interval between positive and negative samples. The method for the metric variance adaptive mechanism to adjust the size of the angle margin in real time includes: Calculate the variance of the current training batch, which is the variance of the cosine similarity of all positive samples in the current training batch; calculate the dynamic margin based on the variance of the current training batch and the global variance; update the global variance using the variance of the current training batch; wherein, the global variance is the variance of the cosine similarity of all positive samples in all training batches. Update the angle boundary using the dynamic boundary; The initial retrieval model is optimized based on the loss function calculation results to obtain the retrieval model.

2. The retrieval model training method according to claim 1, characterized in that, The positive and negative sample pairs for the current training batch are obtained according to the preset sampling strategy, including: Select all negative sample pairs within all training batches; Calculate the cosine similarity between positive and negative samples in all negative sample pairs; Based on the calculation results of the cosine similarity, all negative samples in all training batches are sorted in descending order, and the top T negative samples in the descending order are selected as high similarity negative samples. Highly similar negative samples are classified according to the subtypes of non-target diseases to obtain at least two categories of highly similar negative sample groups, with each category of the highly similar negative sample group corresponding to a subtype of a non-target disease. A preset number of negative sample pairs are selected from all negative sample pairs in all training batches as negative sample pairs for the current training batch. The negative samples in the current batch include high similarity negative sample groups of all categories, and the number of negative samples in each high similarity negative sample group is set according to a preset ratio.

3. The retrieval model training method according to claim 1, characterized in that, Inputting the positive sample pairs and the negative sample pairs into the initial retrieval model includes: The positive sample pairs and the negative sample pairs are input into the initial retrieval model, and the initial retrieval model performs multi-stage downsampling operations on the positive sample pairs and the negative sample pairs sequentially, wherein: The initial retrieval model includes multiple sequentially connected convolutional modules, an attention module connected in parallel with each convolutional module, and an output layer. The first feature output by the previous convolutional module is processed by the corresponding parallel attention module to output a second feature. The second feature is returned to the convolutional module and concatenated with the first feature. The concatenated feature is used as the input of the next convolutional module. The concatenated feature output by the last convolutional module is used as the input of the output layer.

4. The retrieval model training method according to claim 1, characterized in that, The loss function is the CosFace margin loss function, and the expression of the CosFace margin loss function is: ； In the formula: Indicates positive sample pairs. Indicates negative sample pairs; This represents the total number of positive and negative sample pairs in the current training batch. The cosine similarity between sample features is represented by... Indicates the scale factor. Indicates the angular boundary.

5. The retrieval model training method according to claim 1, characterized in that, The formula for calculating the dynamic margin based on the current training batch variance, global variance, and preset adjustment sensitivity coefficient is as follows: ； In the formula: Indicates dynamic margin, Indicates the boundary of the reference angle. This represents the variance of the current training batch. Represents the global variance. This represents the activation function.

6. A method for identifying Bert-Hogg-Dubb syndrome based on retrieval-enhanced generation, characterized in that, The Bert-Hogg-Dubb syndrome identification method based on retrieval enhancement includes: Obtain the user's query information, which includes the original query image and the original query text; The retrieval model is used to retrieve the K most relevant image-text pairs that are most relevant to the target query image from a pre-built knowledge base. The relevant image-text pairs include candidate images and candidate descriptive texts corresponding to the candidate images. Based on the query information, rearrange the K relevant image-text pairs to obtain the optimal relevant image-text pair; The optimal relevance image-text pair and the user's query information are input into the multimodal large model to obtain the recognition result; The retrieval model is trained using the method described in any one of claims 1-5.

7. The method for identifying Bert-Hogg-Dubbs syndrome based on retrieval enhancement generation according to claim 6, characterized in that, Based on the query information, the K relevant image-text pairs are rearranged to obtain the optimal relevant image-text pairs, including: The original query image in the query information is used as the query sample, and the candidate images in the relevant image-text pairs are used as candidate samples. Project the feature vectors of the candidate samples onto M randomly generated feature subspaces. ; Within each feature subspace, evaluation data for query samples and candidate samples is calculated. The evaluation data includes the cosine similarity between query samples and candidate samples, the local angular variance between query samples and candidate samples, and the cluster density of candidate samples. Calculate the local consistency score of the candidate samples based on the evaluation data of the query samples and the candidate samples; The rearrangement score of the candidate sample in the M feature subspaces is calculated based on the local consistency score; Based on the rearrangement score, all candidate images in the K related image-text pairs are sorted in descending order, and the related image-text pair corresponding to the first candidate image in the descending sort is selected as the optimal related image-text pair.

8. The method for identifying Bert-Hogg-Dubbs syndrome based on retrieval enhancement generation according to claim 7, characterized in that, The formula for calculating the local consistency score is as follows: ； In the formula: Indicates the first In the feature subspace, the ... Local consistency score of each candidate sample; Indicates the first In each feature subspace, the query sample With the candidate samples Cosine similarity between them; Indicates query sample With the candidate samples Local angular variance between; Indicates the first candidate samples The local cluster density.

9. The method for identifying Bert-Hogg-Dubbs syndrome based on retrieval enhancement generation according to claim 8, characterized in that, The formula for calculating the rearranged fraction is: ； In the formula, Indicates for the query sample Candidate samples Rearranged fractions in M ​​feature subspaces; Indicates the first In the feature subspace, the ... The local consistency score of each candidate sample.

10. A Bert-Hogg-Dubbs syndrome identification system based on retrieval-enhanced generation, characterized in that, The Bert-Hogg-Dubb syndrome identification system based on retrieval enhancement includes: A data acquisition unit is used to acquire user query information, which includes original query image and original query text. A retrieval model is used to retrieve the K most relevant image-text pairs that are most relevant to the target query image from a pre-built knowledge base. The relevant image-text pairs include candidate images and candidate descriptive texts corresponding to the candidate images. The data rearrangement unit is used to rearrange K relevant image-text pairs based on query information to obtain the optimal relevant image-text pair; The model processing unit is used to input the optimal relevant image-text pair and target query information into the multimodal large model to obtain the recognition result; The retrieval model is trained using the method described in any one of claims 1-5.

Images