Multi-modal medical data management and query method and system based on Jina and data lake architecture
By adopting Jina and data lake architecture methods in medical data management, the feature vectors of multimodal data are extracted and stored, and the problem of difficult medical data is difficult to manage and retrieve uniformly is solved, and efficient multimodal data query and analysis is achieved.
Patent Information
- Application Number
- CN202510259064.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-05
- Publication Date
- 2025-06-27
AI Technical Summary
Due to the multimodality of medical data, it is difficult to uniformly manage and retrieve medical data on the same platform, resulting in difficulty in improving query efficiency and accuracy.
The multimodal medical data management and query method based on Jina and data lake architecture is adopted, and the feature vectors of different modal data are extracted through preset machine learning models and associated with scalar attributes are stored in the data lake. The Jina architecture is used to receive and parse multimodal query requests for efficient query and analysis.
It realizes unified processing and retrieval of different types of medical data on a single platform, improves the accuracy and efficiency of medical diagnosis, and solves the complexity of multimodal data storage and query.
Smart Images
Figure CN120216564A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data management, and particularly to a multi-modal medical data management and query method and system based on Jina and a data lake architecture. Background Art
[0002] In the medical field, with the development of 5G technology, medical data not only grows rapidly in scale but also exhibits multi-modal characteristics. Multi-modal queries are common and complex operations in medical data exploration, which involve comprehensive queries and correlation analysis of multiple data modalities (such as medical images, text reports, audio records, video surveillance, etc.) (Reference 1: Zhang, Y., Sheng, M., Liu, X. et al. A heterogeneous multi-modal medical data fusion framework supporting hybrid data exploration. Health Inf Sci Syst 10, 22 (2022)). Such queries not only require efficient processing of each modality data separately but also cross-modal matching techniques to associate these heterogeneous data, so that the query results can fully reflect the multi-dimensional information of patients. However, the challenge of multi-modal queries lies in the fact that data of different modalities have completely different structures and characteristics (Reference 2: Ren, P. et al. MHDP: An Efficient Data Lake Platform for Medical Multi-source Heterogeneous Data. Springer, Cham (2021)). For example, an image can contain millions of pixel information, text data involves complex language features, and audio and video have time-series properties. All these features make the query and matching calculation cost of data huge. Summary of the Invention
[0003] Aiming at the problem that it is difficult to uniformly manage and retrieve medical data with multiple data modalities on the same platform, the present invention provides a multi-modal medical data management and query method and system based on Jina and a data lake architecture, which can realize the unified management and retrieval of medical data of different data modalities on a single platform.
[0004] In a first aspect, the present invention provides a multi-modal medical data management and query method based on Jina and a data lake architecture, including:
[0005] Use a preset machine learning model to extract features from different modal data respectively, obtain the feature vectors of each modal data, and store the feature vectors of each modal data associated with their original scalar attributes in the data lake;
[0006] Use the Jina architecture to receive and parse the multi-modal query request of the user, query in the data lake according to the parsing result, and return the query result to the user.
[0007] Furthermore, storing the feature vectors of each modal data associated with their original scalar attributes in the data lake specifically includes:
[0008] Convert the data format of the feature vectors of each modal data into a one-dimensional array format;
[0009] Take the one-dimensional array corresponding to each modal data and its original scalar attribute as a row record in the DataFrame, so as to integrate all modal data into the DataFrame, and store the DataFrame in the data lake.
[0010] Furthermore, using the Jina architecture to receive and parse the multi-modal query request of the user, query in the data lake according to the parsing result, and return the query result to the user specifically includes:
[0011] Obtain the multi-modal query statement in text form input by the user;
[0012] Analyze the query fields and data types in the multi-modal query statement, determine the query method according to the query fields and data types, and allocate the query task to the Executor component through the Flow component; the query methods include scalar query, vector range query, and vector KNN query;
[0013] Query by the Executor component according to the determined query method. After the query task is completed, return the query result to the user.
[0014] Furthermore, if the multi-modal query statement is a multi-clause query statement and the determined query method includes more than two query methods, then execute the query operations in the order of scalar query - vector range query - vector KNN query;
[0015] Correspondingly, the Executor component queries according to the determined query method. After each query task is completed, return the query result to the Flow component for the Flow component to return the query result to the user, specifically including:
[0016] The Executor component first performs a query in the scalar query mode. After completing the query task of one clause, it returns the query result to the Flow component, so that the Flow component can send the query result to the Executor component corresponding to the next clause. The next Executor component further filters in the query result according to the vector range query mode and returns the filtered result to the Flow component, so that the Flow component can send the filtered result to the Executor component corresponding to the next clause. The next Executor component performs a query in the vector KNN query mode, and repeats the above query process until the query task of the entire multimodal query statement is completed.
[0017] In a second aspect, the present invention provides a multimodal medical data management and query system based on the Jina and data lake architectures, including:
[0018] A management module, configured to use a preset machine learning model to extract features from different modal data respectively, obtain the feature vectors of each modal data, and store the feature vectors of each modal data in association with their original scalar attributes in the data lake;
[0019] A query module, configured to receive a multimodal query request from a user using the Jina architecture and parse it, perform a query in the data lake according to the parsing result, and return the query result to the user.
[0020] Further, the management module includes a format standardization unit and a data integration unit;
[0021] The standardization unit is configured to convert the data format of the feature vectors of each modal data into a one-dimensional array format;
[0022] The data integration unit is configured to use the one-dimensional array corresponding to each modal data and its original scalar attribute as a row record in the DataFrame, so as to integrate all modal data into the DataFrame and store the DataFrame in the data lake.
[0023] Further, the query module includes an acquisition unit, a parsing unit, and an execution unit;
[0024] The acquisition unit is configured to acquire a multimodal query statement in text form input by the user;
[0025] The parsing unit is configured to analyze the query fields and data types in the multimodal query statement, determine the query mode according to the query fields and data types, and allocate the query task to the Executor component through the Flow component; the query modes include scalar query, vector range query, and vector KNN query;
[0026] The execution unit is used to perform a query in accordance with a determined query method through the Executor component. After the query task is completed, the query result is returned to the user.
[0027] Further, the execution unit is further configured to, if the multimodal query statement is a multi-clause query statement and the determined query method includes two or more query methods, sequentially execute query operations in the order of scalar query - vector range query - vector KNN query;
[0028] Correspondingly, performing a query in accordance with a determined query method through the Executor component and returning the query result to the user after the query task is completed specifically includes:
[0029] The Executor component first performs a query in accordance with the scalar query method. After the query task of one clause is completed, the query result is returned to the Flow component for the Flow component to send the query result to the Executor component corresponding to the next clause. The next Executor component further filters in the query result in accordance with the vector range query method and returns the filtered result to the Flow component for the Flow component to send the filtered result to the Executor component corresponding to the next clause. The next Executor component performs a query in accordance with the vector KNN query method, and repeats the above query process until the query task of the entire multimodal query statement is completed.
[0030] In a third aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method described in the first aspect is implemented.
[0031] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in the first aspect is implemented.
[0032] The beneficial effects of the present invention are:
[0033] (1) In the medical field, there may be logical relationships and complementary information between data of different modalities. Handling the interrelationships and integration of these multimodal data is crucial for improving the accuracy and efficiency of medical diagnosis. The multimodal medical data management and query method and system based on Jina and the data lake architecture provided by the present invention uniformly store the multimodality of medical data (such as text, images, audio, video, etc.) in the data lake and utilize the neural search ability of the Jina framework to achieve efficient querying and analysis. Through the large-scale storage and flexible data management functions provided by the data lake, combined with the powerful cross-modal data processing and retrieval capabilities of the Jina framework, the system can uniformly process and retrieve different types of medical data on a single platform.
[0034] (2) The present invention extracts feature vectors of different modality data through a pre-trained machine learning model to uniformly represent complex multimodal data, solving the problem of difficult unified management and analysis of different types of data. Subsequently, these feature vectors are stored in the data lake together with scalar attributes. The data lake not only supports the storage of massive amounts of data but also enables flexible data writing and incremental querying. This addresses the dynamic requirements of continuously growing medical data, enabling the system to process newly added data in real-time and support incremental querying on the basis of existing data. Users can input queries in natural language format through a simplified query interface, avoiding the difficulty of mastering complex query syntax and greatly improving the user experience. The Jina framework can uniformly process data of different modalities. Whether it is text, images, audio, or video, they can all be indexed and retrieved through the same architecture. This unified processing method solves the problem that traditional systems need to separately construct processing flows when facing different data types. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 One of the flow schematic diagrams of the multimodal medical data management and query method based on Jina and the data lake architecture provided by the embodiments of the present invention;
[0036] Figure 2 Another flow schematic diagram of the multimodal medical data management and query method based on Jina and the data lake architecture provided by the embodiments of the present invention.
[0037] Figure 3 A sample format of the query result finally output provided by the embodiments of the present invention;
[0038] Figure 4 The structural schematic diagram of the multimodal medical data management and query system based on Jina and the data lake architecture provided by the embodiments of the present invention;
[0039] Figure 5 The structural block diagram of an electronic device provided by the embodiments of the present invention. Detailed implementation manners
[0040] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.
[0041] As Figure 1 shown, an embodiment of the present invention provides a multimodal medical data management and query method based on the Jina and data lake architectures, including the following steps:
[0042] S101: Use a preset machine learning model to extract features from different modal data respectively to obtain feature vectors of each modal data, and at the same time, the dimension information of the feature vectors can be recorded. And associate the feature vectors of each modal data with their original scalar attributes and store them in the data lake;
[0043] Specifically, the multimodal data in this embodiment includes image, text, audio, and video data. The scalar attributes mainly refer to the metadata of the modal data, such as information such as the name, gender, and age of the patient. At least one piece of information or a combination of information in these metadata should be able to ensure the uniqueness of the identification of the multimodal data. The data lake architecture is used to store and manage the multimodal data and its associated scalar attributes.
[0044] In this embodiment, the machine learning models used for feature extraction of each modal data include, but are not limited to, pre-trained machine learning models such as CLIP, Word2Vec, Whisper, and Visil.
[0045] S102: Use the Jina architecture to receive and parse the multimodal query request from the user, query in the data lake according to the parsing result, and return the query result to the user.
[0046] As an emerging data management solution, the data lake can store multimodal data in a unified storage platform to achieve flexible management and query. In addition, as a neural search framework based on deep learning, the Jina framework is designed specifically for multimodal queries and can efficiently process and manage queries of various data types (such as text, image, video, audio, etc.). Combining the storage advantages of the data lake and the search engine capabilities of Jina, users can manage, process, and analyze multimodal data more efficiently on a unified platform to meet the needs of modern data-intensive applications.
[0047] In one embodiment, for each type of modal data, due to differences in the data formats of the feature vectors output by multiple machine learning models (such as <numpy.array>, <tensor>etc.). To ensure data compatibility, in this embodiment, the data format of the feature vectors of each modality data is converted into a one-dimensional array format (such as <array.double> format) to meet the requirements of Apache Hudi and Apache Spark; Apache Hudi supports real-time updates, deduplication, and incremental queries for large-scale data sets, and is very suitable for storing and managing multi-modal medical data. After unifying the feature vector formats, the one-dimensional array corresponding to each modality data and its original scalar attributes are used as a row record in a DataFrame (or Hudi Table) to integrate all modality data into the DataFrame (or Hudi Table), and the DataFrame (or Hudi Table) is stored in the data lake.
[0048] In this embodiment, each row of the DataFrame represents a complete multi-modal medical data record, including the feature vectors of images, texts, audios, and videos, and finally forms a complete large table with all attributes of multi-modal medical data and stores it in the data lake for query. Through this structured data management method, the present invention realizes the comprehensive storage of multi-modal medical data.
[0049] In one embodiment, as Figure 2 shown, the Flow component and the Executor component of the Jina architecture are used together to cooperate to implement the processing of multi-modal queries. Specifically, it includes the following steps:
[0050] S201: Obtain a multi-modal query statement in text form input by the user;
[0051] S202: Analyze the query fields and data types in the multi-modal query statement, determine the query method according to the query fields and data types, and allocate the query task to the Executor component through the Flow component; the query methods include scalar query, vector range query, and vector KNN query;
[0052] Specifically, the scalar query directly uses the Spark SQL query statement for query; for example, "query patients over 60 years old and who have had surgery in the past 6 months", through the Spark SQL API, create a temporary view for query, and the system can avoid direct operations on the original data to ensure data consistency and integrity.
[0053] The definition of vector KNN query is shown in formula (1). Given a query vector Q and a dataset D, the vector KNN query calculates the distances between Q and all vectors in D and returns the K vectors closest to Q. For example, for the query "Find the top 5 images that are most similar to the picture Example.jpg in terms of visual features", it is necessary to convert the multimodal data Example.jpg involved into a feature vector form, perform similarity matching with the feature vectors of the corresponding modal data in the data to be queried, and return the top 5 results with the highest scores.
[0054] KNN(Q,K)=argmin X∈D,|R|＝K ∑ x∈R d(Q,X)(1)
[0055] The definition of vector range query is shown in formula (2). Given a query vector Q, a dataset D, and a distance threshold r, the vector range query returns all vectors whose distances from Q are less than or equal to r. For example, for the query "Find the texts whose semantic similarity to the text 'diabetes' is above 0.7", it is necessary to convert the multimodal data 'diabetes' involved into a feature vector form, perform similarity matching with the feature vectors of the corresponding modal data in the data to be queried, and return all results.
[0056] Range(Q,r)={X∈D|d(Q,X)≤r}(2)
[0057] When the user inputs a multimodal query statement, such as "Find the top 10 pictures that are most similar to the picture Example.jpg and were taken after 2020", the system first needs to parse this statement. Using regular expression technology, it parses that this query contains two clauses: Clause 1 is being most similar to the picture Example.jpg, and Clause 2 is being taken after 2020. The final result is the top 10 pictures, and it determines the query method based on whether the clauses involve unstructured data and the returned results. Clause 1 contains the image data Example.jpg and the returned result is the top K items with the highest scores, so it belongs to vector KNN query. Clause 2 only involves the scalar data 2020, so it belongs to scalar query.
[0058] S203: Query according to the determined query method through the Executor component. After the query task is completed, return the query results to the user.
[0059] In this embodiment, the Flow component is responsible for scheduling the query process and coordinating each execution task, and is responsible for passing the query request and query method to the Executor component. The Executor component is responsible for specific data processing and query execution. The overall architecture forms a complete closed-loop process from query parsing to result return. Moreover, since the present invention provides a query interface for users by introducing the Jina architecture, users do not need to master complex query syntax, and only need to input text queries that conform to the format. The system will automatically parse the query conditions input by the user, and the Jina framework will execute corresponding processing processes according to different data modalities. Finally, the qualified query results will be returned to the user in the form of a DataFrame. This solves the problem that the existing medical data lake lacks methods for storing and querying multi-modal medical data.
[0060] In one embodiment, if the multi-modal query statement is a multi-clause query statement and the determined query method includes more than two query methods, the query operations are sequentially executed in the order of scalar query - vector range query - vector KNN query; for example, when both scalar query and vector KNN query are included in the query, the scalar query should be executed first to filter out the qualified data, and then the vector KNN query is executed on the data filtered by the scalar query to ensure the gradual precision of the query process.
[0061] Correspondingly, the Executor component performs queries according to the determined query method. After each query task is completed, the query result is returned to the Flow component for the Flow component to return the query result to the user, specifically including:
[0062] The Executor component first performs a query according to the scalar query method. After the query task of one clause is completed, the query result is returned to the Flow component for the Flow component to send the query result to the Executor component corresponding to the next clause. The next Executor component further filters in the query result according to the vector range query method, and returns the filtered result to the Flow component for the Flow component to send the filtered result to the Executor component corresponding to the next clause. The next Executor component performs a query according to the vector KNN query method, and repeats the above query process until the query task of the entire multi-modal query statement is completed.
[0063] In this embodiment, Flow receives each parsed clause and its corresponding query type, and executes them in the order of scalar query, vector range query, and vector KNN query. First, scalar query and vector range query are performed to filter out a larger dataset that meets the conditions. Then, the most accurate top K items are selected through vector KNN query to improve the accuracy and efficiency of the query. Flow is responsible for encapsulating the parsed data and query type into a Doc document object, and passing this document to the Executor. Each query clause is specifically executed by the corresponding Executor. After receiving the Doc document, the Executor first reads the data from the data lake table and converts it into the DataFrame format. For scalar queries, Spark SQL is used for operation; for vector range queries and KNN queries, first, feature extraction is performed on the multimodal data, and then similarity calculation is performed in the vector space to return the results that meet the conditions. After each clause is executed, the result is returned to Flow, and Flow sequentially passes the result to the Executor corresponding to the next clause until all clauses are executed. Finally, the final result is returned to the user. This architecture design of sequential execution and step-by-step screening effectively improves the accuracy and processing efficiency of the query.
[0064] In one embodiment, assume that the number of query clauses provided by the user does not exceed 2. For all combined query methods that the present invention can handle, examples and Jina APIs are shown in Table 1. Compared with traditional databases or search engines, Jina API can easily handle different modal data such as images and texts. Through efficient vector indexing and distributed computing capabilities, it can quickly respond to the query requirements of large-scale data, ensuring that the system can quickly find the cases or images that meet the conditions in a large amount of medical data.
[0065] Table 1
[0066]
[0067]
[0068] After the query is completed, the query result returned by the Executor component to the user is as Figure 3 shown. The returned result is a DataFrame where each row contains the personal information of a complete case and the corresponding multimodal data. Through the DataFrame, users can clearly and comprehensively analyze and process the case data. This form facilitates users to quickly locate specific cases and perform further data mining, statistical analysis, or pattern recognition, effectively improving the efficiency and accuracy of data query.
[0069] Such as Figure 4 As shown in the figure, an embodiment of the present invention provides a multimodal medical data management and query system based on the Jina and data lake architectures, including a management module and a query module.
[0070] The management module is used to extract features from different modal data respectively by using a preset machine learning model, obtain the feature vectors of each modal data, and store the feature vectors of each modal data associated with their original scalar attributes in the data lake; the query module uses the Jina architecture to receive and parse the multimodal query request of the user, and perform a query in the data lake according to the parsing result, and return the query result to the user.
[0071] The multimodal medical data management and query system based on the Jina and data lake architectures provided by the embodiment of the present invention aims to address the storage, retrieval, and management challenges brought by the heterogeneity and complexity of medical data. This system provides efficient storage and management capabilities through the data lake, and stores large-scale multimodal medical data (such as images, texts, audios, videos, etc.) in the data lake table in the form of feature vectors for unified storage and management. For multimodal queries, using the OpenAPI of the Jina architecture, the system can receive and parse the multimodal query requests of users, and provide personalized retrieval strategies for different types of data. With the help of data lake management technologies such as Apache Hudi, Jina can directly execute queries in the data lake, ensure real-time access to the latest medical data, and quickly feedback the query results to users. In this way, the system can not only ensure the consistency and real-time nature of the data, but also efficiently process the complex query requirements of multimodal data, and users can also complete the exploration of multimodal data on a single platform.
[0072] In one embodiment, the management module includes a format standardization unit and a data integration unit. The standardization unit is used to convert the data format of the feature vectors of each modal data into a one-dimensional array format; the data integration unit is used to use the one-dimensional array corresponding to each modal data and its original scalar attribute as a row record in the DataFrame, so as to integrate all modal data into the DataFrame, and store the DataFrame in the data lake.
[0073] In one embodiment, the query module includes an acquisition unit, an analysis unit, and an execution unit. The acquisition unit is used to acquire the multimodal query statement in text form input by the user; the analysis unit is used to analyze the query fields and data types in the multimodal query statement, determine the query method according to the query fields and data types, and allocate the query task to the Executor component through the Flow component; the query methods include scalar query, vector range query, and vector KNN query; the execution unit is used to perform a query through the Executor component according to the determined query method, and after completing the query task, return the query result to the user.
[0074] In one embodiment, the execution unit is further configured to, if the multimodal query statement is a multi-clause query statement and the determined query methods include more than two query methods, sequentially execute query operations in the order of scalar query - vector range query - vector KNN query;
[0075] Correspondingly, the Executor component performs a query according to the determined query method. After the query task is completed, the query result is returned to the user, specifically including: the Executor component first performs a query according to the scalar query method. After the query task of one clause is completed, the query result is returned to the Flow component for the Flow component to send the query result to the Executor component corresponding to the next clause. The next Executor component further filters in the query result according to the vector range query method and returns the filtered result to the Flow component for the Flow component to send the filtered result to the Executor component corresponding to the next clause. The next Executor component performs a query according to the vector KNN query method, and repeats the above query process until the query task of the entire multimodal query statement is completed.
[0076] Figure 5 An example of the physical structure diagram of an electronic device is shown in Figure 5 As shown, the electronic device may include: a processor 501, a communications interface 502, a memory 503, and a communication bus 504. Among them, the processor 501, the communications interface 502, and the memory 503 communicate with each other through the communication bus 504. The processor 501 can call the logical instructions in the memory 503 to execute a multimodal medical data management and query method based on Jina and a data lake architecture. The method includes: respectively extracting features of different modal data by using a preset machine learning model to obtain feature vectors of each modal data, and associating and storing the feature vectors of each modal data with their original scalar attributes in the data lake; using the Jina architecture to receive and parse a multimodal query request from a user, query in the data lake according to the parsing result, and return a query result to the user.
[0077] In addition, when the logical instructions in the above-mentioned memory 503 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.
[0078] An embodiment of the present invention also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the multi-modal medical data management and query method based on Jina and the data lake architecture provided by the above-mentioned various method embodiments.
[0079] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the multi-modal medical data management and query method based on Jina and the data lake architecture provided by the above-mentioned various method embodiments.
[0080] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disks, optical discs, etc., and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.
[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / tensor>
Claims
1. A multimodal medical data management and query method based on Jina and data lake architecture, characterized in that: include: Use the preset machine learning model to extract features from different modal data, obtain the feature vector of each modal data, and associate the feature vector of each modal data with its original scalar attribute and store it in the data lake; The Jina architecture is used to receive and parse the user's multimodal query request, query the data lake according to the parsed result, and return the query result to the user.
2. The multimodal medical data management and query method based on Jina and data lake architecture according to claim 1 is characterized in that: The feature vector of each modality data is associated with its original scalar attributes and stored in the data lake, including: Convert the data format of the characteristic vector of each modal data into a one-dimensional array format; The one-dimensional array corresponding to each modal data and its original scalar attributes are recorded as a row in the DataFrame to integrate all modal data into the DataFrame, and the DataFrame is stored in the data lake.
3. The multimodal medical data management and query method based on Jina and data lake architecture according to claim 1 is characterized in that: The Jina architecture is used to receive and parse the user's multimodal query request, query the data lake based on the parsed result, and return the query result to the user, including: Obtain a multimodal query statement in text form input by the user; Analyze the query fields and data types in the multimodal query statement, determine the query mode according to the query fields and data types, and assign the query task to the Executor component through the Flow component; the query mode includes scalar query, vector range query and vector KNN query; The query is performed through the Executor component according to the determined query method. After the query task is completed, the query result is returned to the user.
4. The multimodal medical data management and query method based on Jina and data lake architecture according to claim 3 is characterized in that: If the multimodal query statement is a multi-clause query statement, and the determined query method includes more than two query methods, the query operation is performed in the order of scalar query-vector range query-vector KNN query; Correspondingly, the Executor component performs queries according to the determined query method. After each query task is completed, the query result is returned to the Flow component, so that the Flow component can return the query result to the user, including: The Executor component first performs a query in a scalar query mode. After completing the query task of a clause, it returns the query result to the Flow component so that the Flow component can send the query result to the Executor component corresponding to the next clause. The next Executor component further filters the query result in a vector range query mode and returns the filtered result to the Flow component so that the Flow component can send the filtered result to the Executor component corresponding to the next clause. The next Executor component queries in a vector KNN query mode. The above query process is repeated until the query task of the entire multimodal query statement is completed.
5. A multimodal medical data management and query system based on Jina and data lake architecture, characterized by: include: A management module is used to extract features from different modal data using a preset machine learning model to obtain feature vectors for each modal data, and associate the feature vectors of each modal data with its original scalar attributes and store them in the data lake; The query module uses the Jina architecture to receive and parse the user's multimodal query request, performs a query in the data lake based on the parsed result, and returns the query result to the user.
6. The multimodal medical data management and query system based on Jina and data lake architecture according to claim 5 is characterized in that: The management module includes a format standardization unit and a data integration unit; The standardization unit is used to convert the data format of the characteristic vector of each modal data into a one-dimensional array format; The data integration unit is used to record the one-dimensional array corresponding to each modal data and its original scalar attribute as a row in the DataFrame, so as to integrate all modal data into the DataFrame and store the DataFrame in the data lake.
7. The multimodal medical data management and query system based on Jina and data lake architecture according to claim 5 is characterized in that: The query module includes an acquisition unit, a parsing unit and an execution unit; The acquisition unit is used to acquire a multimodal query sentence in text form input by a user; The parsing unit is used to analyze the query fields and data types in the multimodal query statement, determine the query mode according to the query fields and data types, and distribute the query task to the Executor component through the Flow component; the query mode includes scalar query, vector range query and vector KNN query; The execution unit is used to perform a query in a determined query mode through the Executor component, and return the query result to the user after completing the query task.
8. The multimodal medical data management and query system based on Jina and data lake architecture according to claim 5 is characterized in that: The execution unit is further configured to, if the multimodal query statement is a multi-clause query statement and the determined query method includes more than two query methods, execute the query operation in the order of scalar query-vector range query-vector KNN query; Correspondingly, the query is performed by the Executor component according to the determined query method. After the query task is completed, the query results are returned to the user, including: The Executor component first performs a query in a scalar query mode. After completing the query task of a clause, it returns the query result to the Flow component so that the Flow component can send the query result to the Executor component corresponding to the next clause. The next Executor component further filters the query result in a vector range query mode and returns the filtered result to the Flow component so that the Flow component can send the filtered result to the Executor component corresponding to the next clause. The next Executor component queries in a vector KNN query mode. The above query process is repeated until the query task of the entire multimodal query statement is completed.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.