Medical image data management method and system based on large language model

Abstract

The application relates to the technical field of medical image management, and discloses a medical image data management method and system based on a large language model. The method comprises the following steps: firstly, performing hierarchical feature extraction on medical images in a database to obtain a feature set containing multiple dimensions such as global, regional and local dimensions; when a query containing a natural language description is received, performing deep semantic analysis on the natural language description by using a large language model to dynamically generate a weight vector capable of quantifying the query intention of a user; then, based on the dynamic weight vector, performing weighted matching calculation on the hierarchical features of the query image and each image in the database to obtain a comprehensive matching degree; finally, sorting the cases according to the matching degree and presenting the results. In this way, subtle differences in the user query text can be accurately responded to, the accuracy and relevance of similar case retrieval are significantly improved, and more intelligent and personalized medical image data management is realized.

Description

Technical Field

[0001] This application relates to the field of medical image management technology, and more specifically, to a medical image data management method and system based on a large language model. Background Technology

[0002] With the rapid development of medical imaging technology, technologies such as computed tomography (CT) and magnetic resonance imaging (MRI) are becoming increasingly prevalent in clinical diagnosis, leading to an explosive growth in medical imaging data. This massive and complex volume of imaging data presents opportunities for early disease detection and accurate diagnosis, but it also poses significant challenges to data management, retrieval, and utilization. Traditional medical imaging data management systems, such as Picture Archiving and Communication Systems (PACS), primarily focus on data storage and basic queries, failing to meet the needs of clinicians for advanced, precise, and semantic-based searches of similar cases. Doctors often need to combine imaging features with natural language descriptions (such as "blurred ground-glass nodules") to find similar cases during diagnosis, but traditional systems cannot understand this complex query intent, resulting in low retrieval efficiency and poor result relevance.

[0003] In recent years, artificial intelligence technologies, represented by Large Language Models (LLM), have made breakthroughs in the field of natural language processing. Their powerful semantic understanding capabilities offer new insights into addressing the aforementioned challenges. Currently, the application of LLM in medical imaging mainly focuses on automatic image report generation and visual question answering. These applications primarily analyze individual images and do not fully leverage the advantages of LLM in understanding user intent to optimize the retrieval process for massive datasets. Existing retrieval technologies typically employ fixed, pre-defined feature matching logic, failing to dynamically adjust retrieval strategies based on subtle differences in the user's query text (such as whether the focus is on the "macroscopic morphology" or "internal microscopic texture" of lesions). When users submit complex queries, existing systems cannot assign reasonable weights to features at different levels (such as global layout, regional morphology, and local texture), limiting the accuracy of retrieval results. Therefore, existing technologies still have shortcomings: a lack of intelligent data management and retrieval methods that can dynamically analyze user query intent using LLM and guide multi-level feature matching based on this analysis. Summary of the Invention

[0004] Existing medical image data management systems suffer from a semantic gap when handling complex, semantic queries, failing to fully understand the true diagnostic intent behind clinicians' natural language descriptions. This application proposes a medical image data management method and system based on a large language model, cleverly combining the superior semantic understanding capabilities of large language models with multi-level visual feature analysis.

[0005] According to one aspect of this application, a medical image data management method based on a large language model is provided, comprising: performing hierarchical feature extraction on medical images in a database to obtain a hierarchical feature set of medical images; receiving user queries and performing query parsing and featureization on the user queries to obtain query intent text and query feature set; performing dynamic intent analysis on the query intent text based on a large language model to obtain a weight vector; calculating the matching degree between the query feature set and each hierarchical feature set of medical images in the database based on the weight vector to obtain a sorted case list; and presenting interpretable results based on the sorted case list to obtain an interactive result interface.

[0006] In one possible implementation, hierarchical feature extraction is performed on medical images within a database to obtain a hierarchical feature set for the medical images. This includes: automatically segmenting and locating lesion regions in the medical images to obtain a region mask set; inputting the medical images into a pre-trained 3D convolutional neural network model to obtain a global visual feature vector for the medical images; extracting a set of regional sub-images from the medical images based on the region mask set; inputting each regional sub-image in the set of regional sub-images into a pre-trained region feature extractor to obtain a set of regional feature vectors for the medical images; performing image block processing on the regional sub-images to obtain a set of image blocks; and inputting each image block in the set of image blocks into a lightweight feature extractor to obtain a set of local feature vectors for the medical images.

[0007] In one possible implementation, the automatic segmentation and localization of lesion regions in medical images to obtain a region mask set includes: inputting the medical images into a pre-trained deep learning segmentation network model to obtain a probability map of lesion regions; binarizing the probability map of lesion regions based on a confidence threshold to obtain a binary map of lesion regions; and performing three-dimensional connected component analysis on the binary map of lesion regions to obtain the region mask set.

[0008] In one possible implementation, the user query includes, but is not limited to: a natural language description of the query intent, a query image, and a region of interest circled on the query image.

[0009] In one possible implementation, a user query is received, and the user query is parsed and characterized to obtain a query intent text and a query feature set, including: assigning a natural language description of the query intent in the user query to the query intent text; and extracting visual features from the query image in the user query and / or the query interest region circled on the query image to obtain the query feature set.

[0010] In one possible implementation, dynamic intent analysis based on a large language model is performed on the query intent text to obtain a weight vector, including: constructing a query suggestion word template based on the query intent text; and inputting the query suggestion word template into the large language model to obtain the weight vector.

[0011] In one possible implementation, based on the weight vector, the matching degree between the query feature set and each hierarchical feature set of medical images in the database is calculated to obtain a sorted case list. This includes: calculating the matching degree between the query feature set and each hierarchical feature set of medical images in the database based on the weight vector using the following formula: ;in, The weights represent the global feature vectors. The weights represent the feature vector set of the region. The weights represent the local feature vector set. This represents the similarity between global feature vectors. The similarity between sets of regional feature vectors. The similarity between local feature vector sets This represents the degree of matching.

[0012] In one possible implementation, calculating the matching degree between the query feature set and each hierarchical feature set of medical images in the database to obtain a sorted case list includes: sharpening the weight vector using a preset temperature coefficient to increase the difference between the dominant weight value and other weight values ​​in the weight vector, thereby obtaining a sharpened weight vector; and based on the sharpened weight vector, calculating the matching degree by performing a geometrical weighted average of the similarity between the query feature set and the hierarchical feature set of medical images in three feature dimensions: global, regional, and local.

[0013] In one possible implementation, interpretable results are presented based on a sorted list of cases to obtain an interactive results interface, including: extracting the first case from the sorted list of cases; and displaying the medical images of the first case on the screen.

[0014] According to another aspect of this application, a medical image data management system based on a large language model is provided, comprising: a hierarchical image feature extraction module for extracting hierarchical features from medical images in a database to obtain a hierarchical feature set of medical images; a user query parsing module for receiving user queries and parsing and characterizing the user queries to obtain query intent text and a query feature set; a dynamic intent weight generation module for performing dynamic intent analysis on the query intent text based on a large language model to obtain a weight vector; a multi-granularity feature matching module for calculating the matching degree between the query feature set and each hierarchical feature set of medical images in the database based on the weight vector to obtain a sorted case list; and an interpretable interactive presentation module for presenting interpretable results based on the sorted case list to obtain an interactive result interface.

[0015] Compared with existing technologies, the medical image data management method and system based on a large language model provided in this application successfully bridges the gap between users' abstract semantic needs and the concrete visual features of images by introducing a large language model to dynamically analyze user intent and guide the weighted matching of multi-level features. Ultimately, this empowers clinicians and researchers to quickly and accurately discover valuable similar cases from massive amounts of data, thereby improving diagnostic accuracy, promoting medical research, and ultimately improving patient treatment outcomes. Attached Figure Description

[0016] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0017] Figure 1 The illustration shows a schematic flowchart of a medical image data management method based on a large language model according to an embodiment of this application.

[0018] Figure 2 The illustration shows a schematic data flow diagram of a medical image data management method based on a large language model according to an embodiment of this application.

[0019] Figure 3 The illustration shows a schematic flowchart of step S1 in the medical image data management method based on a large language model according to an embodiment of this application.

[0020] Figure 4 The illustration shows a schematic flowchart of step S2 in the medical image data management method based on a large language model according to an embodiment of this application.

[0021] Figure 5 The illustration shows a schematic flowchart of step S5 in the medical image data management method based on a large language model according to an embodiment of this application.

[0022] Figure 6 The illustration shows a schematic block diagram of a medical image data management system based on a large language model according to an embodiment of this application. Detailed Implementation

[0023] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0024] Figure 1 The illustration shows a schematic flowchart of a medical image data management method and system based on a large language model according to an embodiment of this application. Figure 2 The illustration shows a schematic data flow diagram of a medical image data management method based on a large language model according to an embodiment of this application. Figure 1 and Figure 2 As shown, this application provides a medical image data management method based on a large language model, including: S1, extracting hierarchical features from medical images in the database to obtain a hierarchical feature set of medical images; S2, receiving user queries and parsing and characterizing the user queries to obtain query intent text and query feature set; S3, performing dynamic intent analysis on the query intent text based on a large language model to obtain a weight vector; S4, calculating the matching degree between the query feature set and each hierarchical feature set of medical images in the database based on the weight vector to obtain a sorted case list; S5, presenting interpretable results based on the sorted case list to obtain an interactive result interface.

[0025] For example, in step S1, hierarchical feature extraction is performed on medical images in the database to obtain a hierarchical feature set for medical images. It should be understood that single-level visual features, such as feature vectors representing only global information, are often insufficient to comprehensively and meticulously depict the rich diagnostic information contained in medical images. When clinicians perform diagnoses and case comparisons, their focus switches between different dimensions. They consider the overall layout and structure of the image (global features), focus on the shape, size, and location of specific lesions (regional features), and also observe subtle changes in texture, density, etc., within the lesions (local features). Therefore, this application constructs a hierarchical feature set that can simultaneously cover these three dimensions to accurately respond to complex user query intentions, thereby achieving highly relevant case matching.

[0026] In one embodiment, such as Figure 3 As shown, hierarchical feature extraction is performed on medical images in the database to obtain a hierarchical feature set of medical images, including: S11, automatic segmentation and localization of lesion regions in medical images to obtain a region mask set; S12, inputting the medical images into a pre-trained three-dimensional convolutional neural network model to obtain a global visual feature vector of the medical images; S13, based on the region mask set, extracting a set of regional sub-images from the medical images; S14, inputting each regional sub-image in the set of regional sub-images into a pre-trained region feature extractor to obtain a set of regional feature vectors of the medical images; S15, performing image block processing on the regional sub-images to obtain a set of image blocks; S16, inputting each image block in the set of image blocks into a lightweight feature extractor to obtain a set of local feature vectors of the medical images.

[0027] Specifically, the process first involves automatically segmenting and locating lesion regions in medical images to obtain a region mask set. This includes: inputting the medical images into a pre-trained deep learning segmentation network model to obtain a probability map of lesion regions; binarizing the probability map of lesion regions based on a confidence threshold to obtain a binary map of lesion regions; and performing three-dimensional connected component analysis on the binary map of lesion regions to obtain the region mask set.

[0028] In one specific embodiment, the deep learning segmentation network model can adopt a U-Net architecture, which includes an encoder path for capturing contextual information and a symmetric decoder path for precise localization. High-resolution features from the encoder path are combined with the output of the decoder path via skip connections, ensuring both sufficient feature perception and accurate boundary recognition in the segmentation task. The model's pre-training process is typically performed on a large-scale medical image dataset annotated by professional physicians, using a loss function such as Dice or its variants as the optimization objective, and employing a gradient descent algorithm (such as the Adam optimizer) for end-to-end training. After obtaining the lesion region probability map, the probability map is binarized based on a confidence threshold to obtain a binary lesion region map. The determination of this confidence threshold is an empirical process, selected by evaluating segmentation performance (e.g., Dice coefficient, sensitivity, specificity) at different thresholds on independent validation datasets, aiming to find an optimal balance between false positives and false negatives for specific clinical applications; for example, a confidence threshold of 0.7 is used. Finally, three-dimensional connected component analysis is performed on the binary lesion region map to obtain the region mask set. 3D connected component analysis is an image processing algorithm that iterates through every voxel in a 3D binary image, grouping spatially adjacent foreground voxels (with a value of 1) into the same group and assigning a unique label to each independent connected group. This method effectively separates multiple unconnected lesions in an image, ultimately outputting a set of masks, each corresponding to an independent lesion region.

[0029] After obtaining the region mask set, features at three levels are extracted in parallel. First, the complete medical image is input into a pre-trained 3D convolutional neural network model to obtain the global visual feature vector of the medical image. In a specific embodiment, the 3D convolutional neural network model can adopt an architecture such as ResNet3D, which is constructed by stacking residual blocks containing 3D convolutional kernels, and can effectively learn the spatial context information in the 3D medical image data. The pre-training of this model can be performed on a large-scale 3D medical image dataset, performing supervised learning tasks such as disease classification, or using self-supervised learning methods (such as contrastive learning) for training. After training, the output of the fully connected layers before the final classification layer is extracted as the global visual feature vector of the medical image.

[0030] Secondly, based on the previously generated region mask set, a set of region sub-images is extracted from the medical image. Each region sub-image in the set is then input into a pre-trained region feature extractor to obtain a set of medical image region feature vectors. This region feature extractor can employ a Vision Transformer (ViT) model architecture. This architecture segments the input region sub-image into a series of image patches and inputs the linear embedding sequence of these image patches into a standard Transformer encoder, thereby effectively capturing the morphological and structural relationships within the lesion region. The model can be pre-trained on a dedicated dataset containing various types of lesion samples, specifically for tasks such as lesion classification or attribute recognition.

[0031] Finally, the regional sub-images are segmented into image patches to obtain a set of image patches. Each image patch in the set is then input into a lightweight feature extractor to obtain a set of local feature vectors for the medical image. This lightweight feature extractor can employ efficient model architectures such as MobileNet, with depthwise separable convolutions at its core. This significantly reduces the number of model parameters and computational complexity, making it well-suited for processing a large number of small image patches generated from a single lesion. The model can be pre-trained on general natural image datasets (such as ImageNet) and then fine-tuned on datasets containing rich medical texture features.

[0032] For example, in step S2, a user query is received, and the query is parsed and characterized to obtain the query intent text and query feature set. It should be understood that in clinical practice, doctors' search needs are often complex, including both descriptive language (such as lesion features) and specific image instances. Therefore, a dedicated process is needed to receive such complex queries and use query parsing and characterization to precisely decompose them into a text portion for semantic understanding and a feature portion for visual comparison.

[0033] Specifically, the input for step S2 is the user query, which is a multimodal input information. Its specific forms include, but are not limited to: a natural language description of the user's query intent, one or more query images uploaded by the user, and regions of interest specified by the user on the query images through interactive methods (such as mouse selection). Obtaining this information allows for a comprehensive capture of the user's explicit and implicit intent. The output of step S2 consists of two core parts: the query intent text and the query feature set. The query intent text is the purely natural language description of the user query, directly reflecting the user's semantic needs; while the query feature set is a digital and structured representation of the image portion of the user query. Its data structure is consistent with the hierarchical feature set of medical images stored in the aforementioned database to ensure that subsequent comparisons can be made on the same level.

[0034] In one embodiment, such as Figure 4 As shown, the system receives user queries and performs query parsing and characterization on the user queries to obtain query intent text and query feature set, including: S21, assigning the natural language description of the query intent in the user query to the query intent text; S22, extracting visual features from the query image in the user query and / or the query interest region circled on the query image to obtain the query feature set.

[0035] Specifically, firstly, upon receiving a user's complex query, the system parses the query content and directly assigns the natural language description of the user's query intent to the query intent text. In one specific embodiment, the system is configured to receive text input in its user interface, for example, through a designated text input box. When the user types their natural language description in the input box and submits the query, the system captures the complete character data stream associated with that input box. Subsequently, the system assigns this character data stream—the raw text entered by the user—word-for-word, without any deletion, translation, or summarization, to a predefined variable or data field used to store the query intent text. This direct assignment method is crucial because it ensures that the text input into the subsequent large language model is the most authentic and lossless expression of the user's intent, thus providing a solid foundation for accurate intent analysis by the large language model.

[0036] Simultaneously, visual features are extracted from the query image and / or the region of interest circled on the query image to generate the query feature set. Crucially, this visual feature extraction process is identical to the hierarchical feature extraction process for medical images in the database in the first step. This means that the query image will also undergo a series of processes, including automatic lesion region segmentation and localization, global feature extraction, regional feature extraction, and local feature extraction, ultimately generating a query feature set containing global, regional, and local feature vectors. This consistency in processing is the fundamental guarantee for ensuring the comparability of features between the query end and the database end.

[0037] In one specific embodiment, when a user uploads a query image and may define one or more regions of interest on the image through interactive operations (such as mouse dragging), this visual data is first received. Subsequently, a visual feature extraction process is initiated, which is completely identical to the process of hierarchical feature extraction of medical images in the database in step S1 in terms of the models, algorithms, parameters, and processing steps used.

[0038] Specifically, the process unfolds as follows: If the user specifies a region of interest (ROI), subsequent regional and local feature extraction will prioritize this region. If the user does not specify a ROI, the entire query image will first undergo the same automatic lesion region segmentation and localization procedure as during database processing to automatically identify key regions. Next, the complete query image is input into the same pre-trained 3D convolutional neural network model to extract global visual feature vectors representing the overall macroscopic information of the medical image. Subsequently, based on the user-specified or automatically segmented region mask, corresponding regional sub-images are extracted from the query image, and these sub-images are input into the same pre-trained region feature extractor to obtain a set of medical image region feature vectors describing the macroscopic morphology of these key regions. Finally, these regional sub-images are processed into image blocks, and the resulting set of image blocks is input into the same lightweight feature extractor (such as the MobileNet model) to generate a set of local medical image feature vectors capable of finely depicting the microscopic texture within lesions. Through this series of operations that are completely mirrored with the database, the user's visual query is transformed into a query feature set containing global, regional, and local features, and its data structure is compatible with the feature set stored in the database.

[0039] For example, in step S3, dynamic intent analysis based on a large language model is performed on the query intent text to obtain a weight vector. It should be understood that different clinical query needs conceal differentiated focuses on different levels of imaging features. For instance, one query might focus more on the internal texture details of a lesion, while another might focus more on the overall outline of the lesion or its relative position within an organ. Traditional retrieval methods typically employ fixed, uniform feature matching weights, failing to capture these subtle changes in intent present in natural language descriptions, thus leading to discrepancies between the retrieval results and the doctor's actual needs. This application, by introducing a large language model, aims to dynamically and intelligently interpret the doctor's language, transforming this subjective, descriptive focus into an objective, quantifiable weight vector.

[0040] In one embodiment, performing dynamic intent analysis on the query intent text based on a large language model to obtain a weight vector includes: constructing a query suggestion word template based on the query intent text; and inputting the query suggestion word template into the large language model to obtain the weight vector.

[0041] Specifically, firstly, a query suggestion template is constructed based on the query intent text. This process doesn't simply send the query intent text directly to the large language model; instead, it embeds it into a carefully designed, structured suggestion template. The specific format of this query suggestion template is: "You are a senior doctor. Based on the following search requirement: '{QueryIntentText}', please determine whether, when performing image similarity matching, you should focus more on the overall image layout, the macroscopic morphology of key lesions, or the microscopic texture within the lesions. Please output three weight values ​​in JSON format:" In this template, '{QueryIntentText}' is a placeholder used to fill in the actual query intent text obtained in the previous step. This template establishes a clear context and output specification for the large language model through role-playing ("You are a senior doctor," which can be further specified, such as a senior radiologist), task descriptions, and formatting requirements, ensuring that it can stably and accurately generate the required weight information.

[0042] Next, the completed query suggestion template is input into the large language model to obtain the aforementioned weight vector. Once this complete suggestion, containing specific user needs, is sent to the large language model, the model leverages its powerful natural language understanding and reasoning capabilities to perform deep semantic analysis on the embedded query intent text. The model identifies keywords in the text (such as "ground glass" and "blurred edges") and their inherent relationships with image features at different scales (texture, morphology, layout), and based on this understanding, determines the user's relative attention to the global, regional, and local feature levels in the current query. Finally, the model generates and outputs a JSON format specified in the suggestion template, containing... A weight vector with three weight values.

[0043] For example, in step S4, based on the weight vector, the matching degree between the query feature set and each hierarchical feature set of medical images in the database is calculated to obtain a sorted list of cases. That is, by calculating the matching degree, a comprehensive score is calculated for each hierarchical feature set of medical images in the database. This score can accurately reflect its similarity to the query feature set under the guidance of the weight vector, and finally, the most valuable candidate cases are ranked according to the score.

[0044] In one embodiment, based on a weight vector, the matching degree between the query feature set and each hierarchical feature set of medical images in the database is calculated to obtain a sorted list of cases, including:

[0045] Based on the weight vector, the matching degree between the query feature set and each hierarchical feature set of medical images in the database is calculated using the following formula: ;in, The weights represent the global feature vectors. The weights represent the feature vector set of the region. The weights represent the local feature vector set. This represents the similarity between global feature vectors, such as cosine similarity. This refers to the similarity between sets of regional feature vectors, such as the maximum matching similarity. This refers to the similarity between local feature vector sets, for example, the similarity calculated based on the distance generated by optimal transmission. This represents the degree of matching.

[0046] Specifically, the system uses cosine similarity for global feature comparison. This method treats the global feature vectors of the query image and the database image as two directional arrows in a high-dimensional space, and measures their similarity by calculating the cosine of the angle between these two vectors. The result ranges from -1 to 1. The closer the value is to 1, the more consistent the overall layout, texture, and structure of the two images are in terms of direction, meaning their overall visual styles are more similar. A value close to 0 indicates that the two images are unrelated, while a value close to -1 indicates completely opposite styles. The core of this method is that it only cares about the direction of the features rather than their magnitude, effectively capturing the overall essence of the image.

[0047] Next, at the regional feature level, since an image may contain multiple lesion regions, the comparison is made between two sets of feature vectors, thus employing the maximum matching similarity. The computational logic of this method is as follows: First, a similarity matrix is ​​constructed, where each element represents the similarity between a lesion in the query image and a lesion in the database image. Then, an optimization algorithm (such as the Hungarian algorithm) is used to find the optimal matching scheme in this matrix, that is, to find a pairing method that maximizes the sum of similarities for all matched lesion pairs. Finally, this maximum total similarity is normalized to obtain a final score between 0 and 1. This method cleverly solves the problem of how to compare two sets of lesions that may have different numbers of lesions, and the result reflects the optimal correspondence similarity between two sets of key lesions.

[0048] Finally, at the finest level of local features, the optimal transmission distance (often referred to as the "bulldozer distance") is used for measurement. This method treats the local feature sets of two images as two probability distributions, like a sandpile and a pit. It calculates the minimum "cost" or "effort" required to transform one distribution (sandpile) into the other (pit), taking into account both the amount of features to be "transported" and the distance traveled. The smaller the calculated distance value, the closer the structure and content of the two local feature distributions are, meaning the more similar the microscopic textures, cell morphology, and other details within the lesions. Since it outputs a distance value, it can be converted into a similarity score between 0 and 1 using methods such as a Gaussian kernel function, facilitating fusion with similarity scores from other levels.

[0049] Furthermore, considering the... and When performing linear weighted sum calculations, extremely low scores in one dimension (e.g., lesion morphology completely inconsistent) It can score highly in other dimensions (e.g., similarity in overall lung texture, etc.). While compensation might be used, this is unreasonable in clinical practice. That is, even if the overall background is similar, if the key lesions are completely different, then the two cases are not similar, and the mismatch of key features should not be simply averaged out.

[0050] Therefore, in another embodiment, based on the weight vector, the matching degree between the query feature set and each hierarchical feature set of medical images in the database is calculated to obtain a sorted case list. This includes: sharpening the weight vector using a preset temperature coefficient to increase the difference between the dominant weight value and other weight values ​​in the weight vector, thereby obtaining a sharpened weight vector; and based on the sharpened weight vector, calculating the matching degree by performing a geometrical weighted average calculation on the similarity between the query feature set and the hierarchical feature set of medical images in three feature dimensions: global, regional, and local.

[0051] Among them, by using geometric mean, a logical negation mechanism can be naturally introduced. That is, in the interval [0,1], any term close to 0 will pull the entire product toward 0, thereby simulating the multifactor matching principle in clinical diagnosis, that is, all key aspects must reach a certain degree of similarity for the whole to be similar.

[0052] Furthermore, when using multiplication, the weights should no longer be used as multipliers, but rather as exponents. weight The form implies that when the weight w is high (e.g., w=0.9), the score has a very sensitive effect on the final result (close to its original value); when the weight w is low (e.g., w=0.1), the score's effect on the final result is greatly suppressed (the score tends to 1, reducing the weakening effect on the product), which is more effective than linear multipliers in amplifying or reducing the effect of a specific dimension.

[0053] Therefore, it is essential to first ensure that all similarity scores are strictly normalized to the [0,1] interval, where 1 represents perfect similarity and 0 represents complete dissimilarity. Based on the fact that the value range of cosine similarity is [-1, 1], through Mapped to [0,1] It represents the maximum matching similarity, which is already within the [0,1] interval. The distance d generated based on optimal transmission needs to be converted into a similarity score, for example, using a Gaussian kernel function. ,in This is a hyperparameter that controls sensitivity. Therefore, to calculate stability, the comprehensive similarity score is calculated in logarithmic space as follows: ;in, This represents the cosine similarity between global feature vectors. This represents the maximum matching similarity between sets of regional feature vectors. This represents the similarity score between local feature vector sets, calculated based on the distance generated by optimal transmission. This represents the overall similarity score.

[0054] In this way, the system becomes sensitive to mismatched feature dimensions. For example, if a case scores very low on the dimension that the large language model considers most important, its total score will drop sharply, ranking far below those cases that perform well on all dimensions, and more in line with clinical intuition.

[0055] In addition, in weight and In the case of large language model generation, the weights generated by the large language model are essentially softly distributed, meaning they are used to express preferences. However, the user's intent may be explicit and focused. For example, when a user queries a gap, they are almost 100% concerned only with regional features, while their attention to global and local features should be close to zero. In other words, the weights of the large language model may not be sharp enough to reflect focused intent.

[0056] Therefore, a Softmax function and a temperature coefficient are introduced. The Softmax function transforms any real number into a probability distribution, and the temperature parameter T controls the sharpness of the output distribution. Here, as T→∞, the distribution tends to be uniform, with all weights being roughly equal; as T→0, the distribution tends to be heterothermic, with the largest weight approaching 1 and the rest approaching 0. Therefore, by setting T < 1.0 (e.g., T = 0.5), the original preferences of the large language model can be sharpened, making them more decisive.

[0057] Therefore, the Softmax function with temperature T is used to calculate the calibrated weights: ;in represent , and They represent the first The and the first Each weight, temperature This is a hyperparameter, which can be set to a value between (0,1] based on experience, for example, 0.75. Indicates the first The optimized weights.

[0058] In this way, when a user's query intent is clear (e.g., querying a very specific morphological term), the sharpened weights will cause the system to consider almost only the most relevant feature dimensions. However, if the user queries, for example, diffuse lesions, the calibrated weights might be {w'g: 0.95, w'r: 0.03, w'l: 0.02}, making global features the absolute dominant factor in ranking. At the same time, sharpening the weights can effectively suppress the interference of irrelevant feature dimensions on the overall score. Even if there are accidental high similarities in these dimensions, it will not affect the final ranking, thereby improving the signal-to-noise ratio of the search results.

[0059] Finally, after calculating a matching degree for each case in the database, the cases are sorted in descending order based on all the calculated matching degrees to generate the final sorted list of cases.

[0060] For example, in step S5, interpretable results are presented based on the sorted case list to obtain an interactive results interface. It should be understood that in the high-risk field of medical diagnosis, a "black box" search result is unacceptable. Doctors need to understand why the system considers a case to be similar in order to make professional secondary judgments and decisions. Therefore, providing an interpretable presentation method and encapsulating it in an interactive interface that supports active user exploration builds user trust and ensures the system maximizes its value in clinical practice. Interpretable results presentation refers to a result presentation method that goes beyond a simple list display. It reveals the logic and evidence behind the search results to the user through visualization and annotation of key data. The interactive results interface is the user interface that carries this interpretable presentation. It is not a static image or text, but a dynamic environment that allows users to explore and compare image details through operations (such as zooming and panning).

[0061] In one embodiment, such as Figure 5As shown, the system presents interpretable results based on a sorted case list to obtain an interactive results interface, including: S51, extracting the first case from the sorted case list; S52, displaying the medical images from the first case on the screen. This display is not isolated but organized within a carefully designed interactive results interface. This interface typically uses a side-by-side layout; for example, one side displays the user-submitted query images, while the other side displays the medical images of the first case retrieved from the database that best matches them. To achieve interpretability, the system automatically highlights the corresponding region on the matching case's image that most closely resembles the lesion area the user was interested in during the query, and clearly marks the final matching score calculated by the system on the interface. Furthermore, to support interactivity, the interface provides a series of tools that allow users to synchronously or independently zoom, pan, and perform other operations on the two side-by-side images, enabling users to compare and analyze the key details of the two images in depth and detail.

[0062] In summary, the medical image data management method based on a large language model provided in this application first extracts hierarchical features from medical images in the database to obtain a feature set containing multiple dimensions such as global, regional, and local features. When a user query containing natural language description is received, the large language model is used to perform deep semantic analysis on the natural language description, dynamically generating a weight vector that quantifies the user's query intent. Subsequently, based on this dynamic weight vector, a weighted matching calculation is performed between the query image and the hierarchical features of each image in the database to obtain a comprehensive matching degree. Finally, the cases are sorted according to the matching degree and the results are presented. In this way, subtle differences in the user's query text can be accurately responded to, significantly improving the accuracy and relevance of similar case retrieval, and realizing more intelligent and user-friendly medical image data management.

[0063] This application also provides a medical image data management system based on a large language model, used to execute the aforementioned medical image data management method based on a large language model, such as... Figure 6As shown, the medical image data management system 600 based on a large language model includes: a hierarchical image feature extraction module 610, used to extract hierarchical features from medical images in the database to obtain a hierarchical feature set of medical images; a user query parsing module 620, used to receive user queries and perform query parsing and feature generation on the user queries to obtain query intent text and query feature set; a dynamic intent weight generation module 630, used to perform dynamic intent analysis on the query intent text based on a large language model to obtain a weight vector; a multi-granularity feature matching module 640, used to calculate the matching degree between the query feature set and each hierarchical feature set of medical images in the database based on the weight vector to obtain a sorted case list; and an interpretable interactive presentation module 650, used to present interpretable results based on the sorted case list to obtain an interactive result interface.

[0064] This application also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the above-described related method steps to implement the medical image data management method based on a large language model provided in the above embodiments.

[0065] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the medical image data management method based on a large language model provided in the above embodiments.

[0066] In this application, the system, computer-readable storage medium, or computer program product provided in the embodiments are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0067] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments.

[0068] The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous. The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A medical image data management method based on a large language model, characterized in that, include: Hierarchical feature extraction is performed on medical images within a database to obtain a hierarchical feature set for medical images. This includes: automatically segmenting and locating lesion regions in medical images to obtain a region mask set; inputting the medical images into a pre-trained 3D convolutional neural network model to obtain a global visual feature vector for the medical images; extracting a set of regional sub-images from the medical images based on the region mask set; inputting each regional sub-image from the set of regional sub-images into a pre-trained region feature extractor to obtain a set of regional feature vectors for the medical images; performing image block processing on the regional sub-images to obtain a set of image blocks; and inputting each image block from the set of image blocks into a lightweight feature extractor to obtain a set of local feature vectors for the medical images. Receive user queries and perform query parsing and characterization to obtain query intent text and query feature set; Dynamic intent analysis based on a large language model is performed on the query intent text to obtain a weight vector; Based on the weight vector, the matching degree between the query feature set and the hierarchical feature sets of various medical images in the database is calculated to obtain the sorted case list; Interpretable results are presented based on the sorted list of cases to obtain an interactive results interface.

2. The medical image data management method based on a large language model according to claim 1, characterized in that, Automatic segmentation and localization of lesion regions in medical images to obtain a region mask set, including: Medical images are input into a pre-trained deep learning segmentation network model to obtain a probability map of lesion regions; Binarization of the probability map of the lesion region is performed based on the confidence threshold to obtain a binary map of the lesion region. A three-dimensional connected component analysis is performed on the binary map of the lesion region to obtain the region mask set.

3. The medical image data management method based on a large language model according to claim 1, characterized in that, The user query includes, but is not limited to: a natural language description of the query intent, a query image, and a region of interest circled on the query image.

4. The medical image data management method based on a large language model according to claim 3, characterized in that, Receive user queries and perform query parsing and characterization to obtain the query intent text and query feature set, including: Assign the natural language description of the user's query intent to the query intent text; Visual features are extracted from the query image in the user query and / or the query interest region circled on the query image to obtain the query feature set.

5. The medical image data management method based on a large language model according to claim 1, characterized in that, Dynamic intent analysis based on a large language model is performed on the query intent text to obtain a weight vector, including: Based on the query intent text, construct a query suggestion template; Input the query suggestion template into the large language model to obtain the weight vector.

6. The medical image data management method based on a large language model according to claim 1, characterized in that, Based on the weight vector, the matching degree between the query feature set and the hierarchical feature sets of various medical images in the database is calculated to obtain a sorted list of cases, including: Based on the weight vector, the matching degree between the query feature set and each hierarchical feature set of medical images in the database is calculated using the following formula: ； in, The weights represent the global feature vectors. The weights represent the feature vector set of the region. The weights represent the local feature vector set. This represents the similarity between global feature vectors. The similarity between sets of regional feature vectors. The similarity between local feature vector sets This represents the degree of matching.

7. The medical image data management method based on a large language model according to claim 1, characterized in that, Based on the weight vector, the matching degree between the query feature set and the hierarchical feature sets of various medical images in the database is calculated to obtain a sorted list of cases, including: The weight vector is sharpened using a preset temperature coefficient to increase the difference between the dominant weight value and other weight values ​​in the weight vector, thereby obtaining a sharpened weight vector. Based on the sharpened weight vector, the matching degree is obtained by performing a geometric average weighted calculation on the similarity between the query feature set and the hierarchical feature set of medical images in three feature dimensions: global, regional, and local.

8. The medical image data management method based on a large language model according to claim 1, characterized in that, Interpretable results are presented based on the sorted case list to obtain an interactive results interface, including: Extract the first case from the sorted case list; The medical images from the first case are displayed on the screen.

9. A medical image data management system based on a large language model, characterized in that, include: The hierarchical image feature extraction module is used to extract hierarchical features from medical images in the database to obtain a hierarchical feature set of medical images. This includes: automatically segmenting and locating lesion regions in the medical images to obtain a region mask set; inputting the medical images into a pre-trained 3D convolutional neural network model to obtain a global visual feature vector of the medical images; extracting a set of sub-image regions from the medical images based on the region mask set; inputting each sub-image region in the sub-image region set into a pre-trained region feature extractor to obtain a set of regional feature vectors of the medical images; performing image block processing on the sub-image regions to obtain a set of image blocks; and inputting each image block in the image block set into a lightweight feature extractor to obtain a set of local feature vectors of the medical images. The user query parsing module is used to receive user queries and perform query parsing and characterization on the user queries to obtain the query intent text and query feature set; The dynamic intent weight generation module is used to perform dynamic intent analysis on the query intent text based on a large language model to obtain a weight vector; The multi-granularity feature matching module is used to calculate the matching degree between the query feature set and the hierarchical feature sets of various medical images in the database based on the weight vector to obtain a sorted list of cases. The interpretable interactive presentation module is used to present interpretable results based on the sorted list of cases to obtain an interactive results interface.

Images