Multi-modal medical data approximate query processing method and system oriented to medical field

Abstract

The invention relates to the technical field of medical data processing, in particular to a medical-field-oriented multi-modal medical data approximate query processing method and system, and the method comprises the steps: detecting the multi-modal type of medical data, and selecting a corresponding query strategy according to the data type and data distribution characteristics; mapping the data to a unified semantic vector space by using a zero sample generation model, and calculating cross-modal similarity to generate a preliminary query result; taking the preliminary query result as input, constructing a multi-target optimization algorithm taking query precision, response time and resource consumption as optimization targets based on an error range, calculation time limitation and calculation overhead set by a user, performing refined screening on the preliminary query result, and dynamically adjusting a query strategy; and receiving feedback information of a user on a query result, and dynamically adjusting model parameters and a query strategy to realize continuous optimization of query performance. According to the invention, through similarity comparison, dynamic threshold optimization and doctor feedback closed loop, high precision and fast response of medical query are realized.

Description

Technical Field

[0001] This invention relates to the field of medical data processing technology, and in particular to a method and system for approximate query processing of multimodal medical data in the medical field. Background Technology

[0002] With the continuous advancement of medical technology, the types and quantities of medical data have increased dramatically, especially the accumulation of multimodal data such as imaging data, electronic medical records (EHRs), and voice consultation records. This has rendered traditional data processing and querying methods inadequate for the needs of modern medicine. Medical data itself is characterized by complexity, diversity, and a combination of highly structured and unstructured data. Traditional query processing methods typically rely on precise query methods, which are difficult to process efficiently with multimodal data and have low execution efficiency on large-scale datasets.

[0003] Currently, many systems and applications in the healthcare industry (such as image processing systems and clinical decision support systems) generally suffer from the following problems. First, traditional medical data processing systems mostly process single types of data independently, such as medical records, image data, and genomic data, lacking the ability to jointly process cross-modal data. This isolated processing approach leads to inefficient data queries, especially when joint analysis of different types of data is required. Query response times are long and accuracy is low, affecting the efficiency of clinical decision-making and diagnosis.

[0004] Secondly, with the increasing volume of medical data, traditional exact query processing methods are gradually revealing their computational bottlenecks. When dealing with large-scale medical datasets, exact queries not only incur enormous computational costs but also take a long time to generate results, especially in high-dimensional data such as medical images, where they often consume significant computational resources. More importantly, in many cases, completely accurate query results are not necessary; instead, quickly obtaining a reasonable approximation result is more valuable for clinical decision-making.

[0005] Furthermore, most current multimodal medical data query systems rely on manually labeled training data for learning and optimization, which increases the development and maintenance costs of the system. For many emerging areas of medical data (such as unlabeled image data and unorganized voice consultation records), the processing capabilities of traditional systems are particularly insufficient, and they cannot flexibly adapt to the query needs of different types of data.

[0006] In large-scale medical data analysis, system overhead is also a significant issue. Medical data is not only massive in quantity but also comes from complex data sources, including imaging data (such as CT and MRI scans), text data (such as electronic medical records and surgical records), voice data (such as doctor-patient conversations), and structured data (such as laboratory results and diagnostic information). This diverse data requires different processing methods. However, existing multimodal data processing systems typically lack efficient query processing mechanisms and optimization schemes, resulting in high resource consumption and heavy computational burden, failing to effectively meet the needs of practical medical applications.

[0007] Therefore, there is an urgent need for a new query processing method that can improve query efficiency and reduce computational resource consumption while ensuring the accuracy of query results. Especially in the medical field, efficiently processing data from different modalities and improving processing speed while maintaining accuracy is key to enhancing the level of intelligent analysis of medical data. Summary of the Invention

[0008] To address the issues of low accuracy and high system overhead in existing medical databases when performing approximate queries, this invention proposes a multimodal medical data approximate query processing method and system for the medical field. Combining multiple technologies such as natural language processing, image processing, and speech recognition, it proposes an efficient query strategy to improve the accuracy and processing speed of approximate query tasks on large-scale medical datasets, thereby enhancing the efficiency and accuracy of applications such as medical image screening and case retrieval.

[0009] To achieve the above objectives, the technical solution adopted is: This invention provides a method for approximate query processing of multimodal medical data in the medical field, comprising the following steps: Step 1: Detect the multimodal types of medical data, identify image, text, voice and structured test data, and select the corresponding query strategy according to the data type and data distribution characteristics; Step 2: Use a pre-trained zero-shot generation model to learn features from medical data of different modalities, map the data to a unified semantic vector space and calculate cross-modal similarity to generate preliminary query results; Step 3: Using the preliminary query results as input, and based on the error range, computation time limit, and computation cost set by the user, construct a multi-objective optimization algorithm with query accuracy, response time, and resource consumption as optimization objectives to refine the preliminary query results and dynamically adjust the query strategy. Step 4: Receive user feedback on query results, dynamically adjust model parameters and query strategies, and continuously optimize query performance.

[0010] According to the multimodal medical data approximate query processing method of the present invention for the medical field, the data distribution characteristics in step 1 include data distribution uniformity and clustering, and the query strategy in step 1 includes selecting a sampling strategy for the query. The sampling strategy is: when the data distribution is uniform, a uniform sampling strategy is adopted, and when the data distribution is skewed, an adaptive sampling method is adopted.

[0011] According to the multimodal medical data approximate query processing method of the present invention for the medical field, the query strategy in step 1 further includes allocating a suitable single-modal processing path for the received query object and target data, specifically including: For medical image data, convolutional neural networks are used to extract image features; For electronic medical record text data, a natural language processing model based on BERT / BioBERT is used for semantic analysis and keyword extraction; For voice consultation records, the voice recognition unit first converts them into text, and then a natural language processing model is used for processing.

[0012] According to the multimodal medical data approximate query processing method of the present invention for the medical field, the zero-shot generation model in step 2 further includes GPT, T5, CLIP and a multimodal model based on graph neural network; in the semantic vector space, the vector similarity between the query object and the target data is calculated as the lower limit of similarity, and the upper limit of similarity is obtained based on historical sample statistical information. Combining the dual constraints of the upper and lower limits of similarity and the preset similarity threshold, the preliminary query results are returned within the preset error range.

[0013] According to the multimodal medical data approximate query processing method of the present invention for the medical field, in step 3, when the multi-objective optimization algorithm weighs the error range, computation time limit and computation cost, it takes the real-time computation load of the system as one of the dynamic optimization objectives; the dynamic adjustment of the query strategy includes: when the real-time computation load of the system is higher than a preset threshold, by adjusting the weight of the optimization objective, priority is given to meeting the computation time limit, and a query strategy that reduces the query accuracy is selected.

[0014] According to the multimodal medical data approximate query processing method of the present invention, in step 4, the user feedback information on the query results includes the doctor's annotation correction, accuracy evaluation and demand adjustment of the query results. The system updates the zero-sample generation model parameters and similarity threshold through the feedback information.

[0015] Furthermore, the present invention also provides a multimodal medical data approximate query processing system for the medical field, comprising: The data acquisition module is used to collect multimodal medical data from multiple data sources; The query processing module is used to perform approximate query processing based on multimodal data and generate preliminary query results; The optimization module employs a multi-objective optimization algorithm to dynamically select the optimal query strategy based on query parameters and system status. The user feedback module provides an interactive interface to receive user feedback on query results and adjusts the query strategy based on the feedback data to optimize subsequent queries.

[0016] According to the multimodal medical data approximate query processing system for the medical field of the present invention, the data acquisition module further includes a natural language processing unit, an image processing unit, and a speech recognition unit. The image processing unit supports feature extraction of medical images and adopts a convolutional neural network. The natural language processing unit uses a BERT / BioBERT model to process electronic medical record text data.

[0017] According to the multimodal medical data approximate query processing system for the medical field of the present invention, the optimization module further adopts a multi-objective optimization algorithm based on error control to achieve a dynamic balance between error range, computation time limit and computational overhead, and adjusts the optimization strategy according to the real-time computational load of the system.

[0018] According to the multimodal medical data approximate query processing system of the present invention, the user feedback module is further used to adjust the parameters of the zero-sample generation model and update the system's query strategy based on the doctor's annotation feedback.

[0019] The beneficial effects achieved by adopting the above technical solution are: This invention constructs a multimodal medical data approximate query processing scheme for the medical field, forming synergistic advantages in three dimensions: technical performance, clinical application, and system adaptability. Specific beneficial effects are as follows: 1. Improve the accuracy and efficiency of cross-modal queries to solve the problem of heterogeneity in medical data. Medical data encompasses multiple modalities, including images, text, voice, and structured laboratory results. Traditional query methods struggle to achieve accurate matching due to significant semantic differences between modalities. This invention employs a progressive architecture of "modal preprocessing + cross-modal fusion": First, it utilizes dedicated feature extraction methods for different modal features (such as CNN for image processing and BERT / BioBERT for text parsing) to ensure the integrity of single-modal features. Then, it maps features to a unified semantic space using multimodal models such as CLIP and GPT, combining algorithms like cosine similarity to achieve cross-modal quantization and comparison, bridging the semantic gap between "image-text-voice". Subsequently, it uses a multi-objective optimization algorithm to dynamically balance query accuracy and computational cost (e.g., appropriately reducing accuracy under high load to ensure response speed), shortening the average processing time for complex multimodal queries while keeping errors within user-defined ranges, thus meeting the core clinical need for "fast and accurate" data.

[0020] 2. Adapt to actual clinical scenarios to enhance the clinical applicability of search results. The demand for query results in medical scenarios is dynamic (such as doctors' personalized requirements for accuracy). Adaptation is achieved through closed-loop feedback optimization: the generation model parameters and similarity thresholds are updated in real time through doctor feedback (such as annotation correction and accuracy evaluation), enabling the system to continuously learn clinical judgment rules. The clinical compliance rate of query results is improved by more than 25%, reducing the cost of manual screening by doctors.

[0021] 3. It has strong scalability and covers a variety of medical application scenarios. This invention can be used for basic "image-medical record matching" (such as searching for lung nodule images and corresponding electronic medical records), and can also be extended to disease prediction (such as predicting the risk of Alzheimer's disease based on multimodal features), transforming disease detection into a "cross-modal similarity matching task between the sample to be detected and a standard sample library".

[0022] In summary, this invention effectively improves the accuracy, efficiency, and practicality of medical multimodal queries through precise conversion of multimodal features, clinically-oriented dynamic optimization, and flexible and adaptable architecture design, providing technical support for scenarios such as clinical diagnosis, case studies, and disease prediction. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. The drawings are merely illustrative of some embodiments of the present invention and are not intended to limit the scope of the present invention to all embodiments.

[0024] Figure 1 This is a flowchart illustrating the multimodal medical data approximate query processing method for the medical field according to an embodiment of the present invention; Figure 2This is a framework diagram of a multimodal medical data approximate query processing system for the medical field, according to an embodiment of the present invention. Detailed Implementation

[0025] The exemplary solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art.

[0026] like Figure 1 As shown in the figure, this invention discloses a method for approximate query processing of multimodal medical data in the medical field, including the following steps: Step S1: Detect the multimodal types of medical data, identify data types such as images, text, voice, and structured laboratory data, and select corresponding query strategies based on data type and data distribution characteristics. The query strategy includes selecting a sampling strategy for the query and assigning appropriate single-modal processing paths to the received query objects and target data. This step includes: Step S1.1, Single-modal processing path: Detect the multimodal characteristics of medical data. By analyzing the distribution of medical data, identify different types of data (such as medical images, electronic medical record text, voice consultation records, laboratory test results, etc.), and determine the appropriate query strategy based on the data type. For medical image data, a convolutional neural network is used to extract image features. For electronic medical record text data, a BERT / BioBERT-based natural language processing model is used for semantic analysis and keyword extraction. For voice consultation records, the data is first converted into text by a speech recognition unit, and then processed using a natural language processing model.

[0027] Step S1.2, Sampling Strategy: Select an appropriate query strategy based on the distribution type of the target data (e.g., uniform or clustered distribution). If the data distribution is uniform, a uniform sampling strategy is used; if the data distribution is skewed, an adaptive sampling method is used. Through this intelligent strategy selection based on data distribution, the system can significantly reduce computational overhead without sacrificing usable accuracy, thereby achieving efficient approximate queries in medical big data scenarios.

[0028] Step S1.3: Calculate the query requirements of the dataset, including selecting the error range, query time limit, and computing resource requirements. At the initial stage of the query, set the error range (e.g., within 10%), and dynamically adjust the response time according to the specific query requirements to ensure that the query results can be returned within a reasonable time, while optimizing computational overhead.

[0029] Step S2: Use the pre-trained zero-shot generation model to learn features from medical data of different modalities, map the data to a unified semantic vector space and calculate cross-modal similarity to generate preliminary query results.

[0030] Zero-shot generation models include GPT, T5, CLIP, and multimodal models based on graph neural networks. The goal is to further map the basic features of each modality output in step S1 to a unified semantic vector space, achieving "homogeneous alignment" of data from different modalities, and calculating cross-modal similarity based on this. In the unified semantic space, the system can calculate the vector similarity between the query object and the target data (using cosine similarity or Euclidean distance) as a lower limit of similarity, and obtain an upper limit of similarity based on historical sample statistics. Combining the upper and lower limits of similarity with a preset similarity threshold, preliminary query results are returned within a preset error range (here, error refers to the false negative rate, i.e., the proportion of samples in the database that meet the query requirements but are not selected by the system). For example, by combining the label information of medical images with the semantic content of electronic medical record text, semantically aligned sample embedding vectors are generated, and the similarity between the two is calculated to obtain candidate results.

[0031] In multimodal medical data approximation query processing for the healthcare field, the use of zero-shot generation models is not limited to any one of GPT, T5, CLIP, or graph neural network (GNN) based models. Instead, a single model or a combination of multiple models can be flexibly selected based on specific query requirements, data types, and performance goals. For example, the CLIP model, with its dual architecture design of "image encoder + text encoder," can directly achieve unified dimensional mapping of medical image and medical record text features and cross-modal similarity calculation. Therefore, in image-text joint queries (such as querying patients with "lung nodule images and medical records mentioning malignant tumors"), CLIP can be used alone to complete cross-modal matching and result filtering without relying on other models.

[0032] The system estimates a similarity range [lower limit, upper limit] for each candidate sample. When the estimated lower limit is greater than the similarity threshold (e.g., a threshold of 0.7), the candidate sample is directly included in the query results, avoiding precise calculation. When the estimated upper limit is less than the similarity threshold, it means the sample absolutely does not meet the standard, and the candidate sample is discarded, avoiding precise calculation. Only when the estimated range crosses the threshold (lower limit < 0.7 < upper limit) is the candidate sample potentially meeting or not meeting the standard. This sample is marked as "pending," and a precise, full similarity calculation is performed on this small subset of samples to ultimately determine its inclusion or exclusion. The full similarity calculation here refers to bypassing all estimations and shortcuts, using the most accurate models and methods available to the system to perform a complete and detailed similarity calculation on two data samples. This is a high-cost, high-precision operation. Through this process, the system can return preliminary query results within a set error range, avoiding large-scale full calculations, improving query efficiency, and reducing computational resource consumption.

[0033] Step S3: Using the preliminary query results as input, and based on the error range, computation time limit, and computation cost set by the user, construct a multi-objective optimization algorithm with query accuracy, response time, and resource consumption as optimization objectives to refine the preliminary query results and dynamically adjust the query strategy.

[0034] When balancing error range, computation time constraints, and computational overhead, multi-objective optimization algorithms take the real-time computational load of the system (including but not limited to real-time utilization of CPU / GPU, memory usage, network I / O, and the number of concurrent query tasks) as one of the dynamic optimization objectives. Dynamically adjusting the query strategy includes: when the real-time computational load of the system exceeds a preset threshold, adjusting the weight of the optimization objectives to prioritize meeting the computation time constraints and selecting a query strategy that reduces query accuracy (e.g., temporarily raising the similarity threshold) to reduce computational pressure; if a higher accuracy query is required, the system will increase computational resources.

[0035] For example: Suppose the system's initial default similarity threshold is 0.7, the user sets the response time to ≤5 seconds, and the error limit to 10%. When a doctor initiates a query, the system load suddenly increases (CPU utilization 85%). If 100 preliminary results are processed at the 0.7 threshold, it will take 8 seconds (timeout). At this point, the multi-objective optimization algorithm will temporarily increase the similarity threshold to 0.78, processing only 60 high-similarity samples, reducing the processing time to 4 seconds (meeting the time requirement), and increasing the error to 9.5% (meeting the error requirement). This adjustment is only effective for the "current round of queries." If the system load decreases in the next round of queries (CPU utilization 50%), the threshold will automatically revert to 0.7.

[0036] Step S4: Receive user feedback on the query results, dynamically adjust model parameters and query strategies, and continuously optimize query performance. Furthermore, user feedback includes doctors' annotation corrections, accuracy evaluations (such as missed key cases), and adjustments based on their needs. The system updates the zero-sample generation model parameters and similarity thresholds based on this feedback.

[0037] Based on user feedback (such as doctors' annotations), the system optimizes its query strategy and adjusts the way query results are generated in real time. When the accuracy of query results is lower than users expect, the system updates its query strategy based on user feedback and automatically optimizes the corresponding models and algorithms to improve query accuracy and efficiency. For example, if a doctor reports that "in rare disease queries, many samples with low similarity (0.7-0.75) are actually confirmed cases (missed detections)," the system will lower the default threshold to 0.7. This adjustment is unrelated to step S3, "whether to temporarily increase it under high load"—even if the load is low later, the default threshold remains at 0.7, solely to reduce clinical missed detections. This step is essentially a rule upgrade based on learning clinical needs, making all subsequent queries more aligned with doctors' needs, and represents long-term optimization.

[0038] By combining steps S3 and S4, the system can cope with real-time load fluctuations (temporary adjustments) and continuously adapt to clinical needs (long-term updates), ultimately achieving a stable query that is both fast and accurate.

[0039] This invention provides multi-level query result versions, allowing users to select the appropriate result for further analysis based on their specific needs. Different query results may be based on different error ranges or computational resources, and the system allows users to choose the result that best suits their actual needs from multiple versions. For example, users can choose to return results with higher precision or select a faster query version to save computational resources.

[0040] Corresponding to the above method, embodiments of the present invention also disclose a multimodal medical data approximate query processing system for the medical field, such as... Figure 2 As shown, it includes: The data acquisition module is used to collect multimodal medical data from multiple data sources; The query processing module is used to perform approximate query processing based on multimodal data and generate preliminary query results; The optimization module is used to dynamically select the optimal query strategy based on query parameters (such as error range, response time limit, etc.) and system status (such as real-time computing load of the system) using a multi-objective optimization algorithm. The user feedback module provides an interactive interface to receive user feedback on query results and adjusts the query strategy based on the feedback data to optimize subsequent queries.

[0041] Furthermore, the data acquisition module includes a natural language processing unit, an image processing unit, and a speech recognition unit. The image processing unit supports feature extraction from medical images and uses a convolutional neural network. The natural language processing unit uses the BERT / BioBERT model to process electronic medical record text data.

[0042] The optimization module adopts a multi-objective optimization algorithm based on error control to achieve a dynamic balance between error range, computation time limit and computational cost, and adjusts the optimization strategy according to the real-time computational load of the system.

[0043] The user feedback module is used to adjust the parameters of the zero-sample generation model and update the system's query strategy based on the doctors' annotation feedback.

[0044] To better understand the technical solution of the present invention, the implementation methods of the present invention will be described in detail below with reference to specific embodiments. The medical data used in the embodiments include image data, text data, voice data, and structured data (such as test results, medical records, etc.).

[0045] Example 1: Cross-modal query of voice and video data In this embodiment, the user wants to check whether the symptoms mentioned in the voice consultation record match the related lesions in the imaging data. The specific implementation of the system is as follows: Step 1: Speech Data Preprocessing and Feature Extraction The system converts speech data into text using a speech recognition module and analyzes the text using natural language processing technology to extract symptom information described by the patient. After converting the patient's speech into text using the speech recognition model, keywords such as "headache" and "visual impairment" are extracted.

[0046] Step 2: Joint query of image data and voice text By using a multimodal generative model (such as the CLIP model), the system performs joint queries on speech text and image data. During inference, the model automatically calculates the semantic similarity between the speech text and image data, and returns relevant image data and cases that match the query symptoms.

[0047] Step 3: User Feedback and Strategy Adjustment The system optimizes query results based on feedback from doctors. For example, if a doctor believes that certain symptoms do not perfectly match the imaging results, the system will adjust the model based on the feedback to further optimize the query results.

[0048] Example 2: Multimodal Data Analysis for Disease Prediction This embodiment demonstrates how to use the method of the present invention for disease prediction. Users wish to predict the probability of a certain disease occurring based on a patient's imaging data, medical records, or laboratory results. The system's operation flow is as follows: Step 1: Data Integration and Multimodal Data Preprocessing The system first collects patient data such as imaging data, medical records, and laboratory test data. Imaging data can be processed using convolutional neural networks to extract features, medical record data can be processed using natural language processing models to extract key information, and laboratory test data can be processed using standardized methods.

[0049] Step 2: Use generative models for data prediction The system uses a zero-shot generative model to process patient data. During the inference process of the generative model, the system calculates the similarity between any modal feature of the patient to be tested (features after multimodal fusion can also be used to improve detection accuracy) and the standard disease sample library. If the similarity reaches a preset threshold, the patient is judged as "high risk / suspected disease".

[0050] Step 3: Query Results and Feedback Mechanism The prediction results will be provided to doctors for reference. If doctors believe that some prediction results do not match the patient's actual situation, they can adjust the system parameters or optimize the model through the user feedback module to improve the prediction accuracy.

[0051] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0052] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0053] The units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations are not considered to be beyond the scope of this invention.

[0054] Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiments can be implemented in hardware or as a software functional module. This invention is not limited to any particular combination of hardware and software.

[0055] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for approximate query processing of multimodal medical data in the medical field, characterized in that, Includes the following steps: Step 1: Detect the multimodal types of medical data, identify image, text, voice and structured test data, and select the corresponding query strategy according to the data type and data distribution characteristics; Step 2: Use a pre-trained zero-shot generation model to learn features from medical data of different modalities, map the data to a unified semantic vector space and calculate cross-modal similarity to generate preliminary query results; Step 3: Using the preliminary query results as input, and based on the error range, computation time limit, and computation cost set by the user, construct a multi-objective optimization algorithm with query accuracy, response time, and resource consumption as optimization objectives to refine the preliminary query results and dynamically adjust the query strategy. Step 4: Receive user feedback on query results, dynamically adjust model parameters and query strategies, and continuously optimize query performance.

2. The method for approximate query processing of multimodal medical data in the medical field according to claim 1, characterized in that, The data distribution characteristics mentioned in step 1 include data distribution uniformity and clustering. The query strategy mentioned in step 1 includes selecting a sampling strategy for the query. The sampling strategy is as follows: when the data distribution is uniform, a uniform sampling strategy is used; when the data distribution is skewed, an adaptive sampling method is used.

3. The method for approximate query processing of multimodal medical data in the medical field according to claim 2, characterized in that, The query strategy described in step 1 also includes allocating appropriate single-modal processing paths for the received query objects and target data, specifically including: For medical image data, convolutional neural networks are used to extract image features; For electronic medical record text data, a natural language processing model based on BERT / BioBERT is used for semantic analysis and keyword extraction; For voice consultation records, the voice recognition unit first converts them into text, and then a natural language processing model is used for processing.

4. The method for approximate query processing of multimodal medical data in the medical field according to claim 1, characterized in that, The zero-shot generation model mentioned in step 2 includes GPT, T5, CLIP, and a multimodal model based on graph neural networks. In the semantic vector space, the vector similarity between the query object and the target data is calculated as the lower limit of similarity, and the upper limit of similarity is obtained based on historical sample statistics. Combining the dual constraints of the upper and lower limits of similarity and the preset similarity threshold, the preliminary query results are returned within the preset error range.

5. The method for approximate query processing of multimodal medical data in the medical field according to claim 1, characterized in that, When the multi-objective optimization algorithm in step 3 weighs the error range, computation time limit, and computational cost, it takes the real-time computational load of the system as one of the dynamic optimization objectives. The dynamic adjustment of the query strategy includes: when the real-time computational load of the system is higher than a preset threshold, adjusting the weight of the optimization objectives to prioritize meeting the computation time limit and selecting a query strategy that reduces the query accuracy.

6. The method for approximate query processing of multimodal medical data in the medical field according to claim 1, characterized in that, The user feedback information in step 4 includes doctors' annotation corrections, accuracy evaluations, and needs adjustments. The system updates the zero-sample generation model parameters and similarity thresholds based on the feedback information.

7. A multimodal medical data approximate query and processing system for the medical field, characterized in that, include: The data acquisition module is used to collect multimodal medical data from multiple data sources; The query processing module is used to perform approximate query processing based on multimodal data and generate preliminary query results; The optimization module employs a multi-objective optimization algorithm to dynamically select the optimal query strategy based on query parameters and system status. The user feedback module provides an interactive interface to receive user feedback on query results and adjusts the query strategy based on the feedback data to optimize subsequent queries.

8. The multimodal medical data approximate query processing system for the medical field according to claim 7, characterized in that, The data acquisition module includes a natural language processing unit, an image processing unit, and a speech recognition unit. The image processing unit supports feature extraction from medical images and uses a convolutional neural network. The natural language processing unit uses the BERT / BioBERT model to process electronic medical record text data.

9. The multimodal medical data approximate query processing system for the medical field according to claim 7, characterized in that, The optimization module employs a multi-objective optimization algorithm based on error control to achieve a dynamic balance between error range, computation time limit, and computational overhead, and adjusts the optimization strategy according to the real-time computational load of the system.

10. The multimodal medical data approximate query processing system for the medical field according to claim 7, characterized in that, The user feedback module is used to adjust the parameters of the zero-sample generation model and update the system's query strategy based on the doctors' annotation feedback.

Images