Multi-modal data search method, device and equipment based on generative AI large model, storage medium and product

Abstract

The invention relates to the technical field of computers, in particular to a multi-modal data search method, device and equipment based on a generative AI large model, a storage medium and a product. The method comprises the following steps: acquiring various types of data from a plurality of heterogeneous data sources to obtain acquired data; performing feature information extraction on the acquired data based on a preset generative AI large model to obtain feature data; constructing a multi-modal index structure based on the feature data; matching the search instruction with index data in the multi-modal index structure; and fusing and sorting the matching results to obtain a target search result. According to the method, various types of data are collected from a plurality of heterogeneous data sources, a multi-modal index structure is constructed after feature information extraction is performed on the data by adopting a generative AI large model, and the search instruction is efficiently matched with the various types of data, so that the efficiency and accuracy of data query are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a multimodal data search method, device, equipment, storage medium and product based on a generative AI big model. Background Art

[0002] In the information and data-driven era, enterprises and organizations store data in various forms, including structured data (such as database records), semi-structured data (such as XML, JSON files) and unstructured data (such as text, images, audio, video, etc.). Traditional data query methods are usually designed for a single data source and data model, relying on structured query language (SQL) or simple full-text search technology. However, in the face of growing multimodal and heterogeneous data, these methods have obvious limitations in data integration, query efficiency and accuracy, multimodal processing capabilities, and personalized support. Especially in large-scale data environments, traditional query methods are difficult to respond to user requests quickly, and cannot provide highly relevant and personalized query results. In response to these challenges, existing research attempts to use technologies such as big data analysis, machine learning, and deep learning to improve the performance of data query and search, but the efficiency and accuracy are low when processing large-scale, multi-source heterogeneous data. Therefore, how to improve the efficiency and accuracy of data query has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of this application is to provide a multimodal data search method, device, equipment, storage medium and product based on a generative AI big model, aiming to solve the technical problem of how to improve the efficiency and accuracy of data query.

[0004] To achieve the above objectives, the present application provides a multimodal data search method based on a generative AI large model, the method comprising the following steps: Collect various types of data from multiple heterogeneous data sources to obtain collected data; Extracting feature information from the collected data to obtain feature data; Based on the feature data, construct a multimodal index structure; matching a search instruction with index data in the multimodal index structure; The matching results are merged and sorted to obtain the target search results.

[0005] In one embodiment, the step of collecting multiple types of data from multiple heterogeneous data sources to obtain collected data includes: Establishing connections with the plurality of heterogeneous data sources; Obtaining tasks to be collected, and classifying and allocating the tasks to be collected; Based on the classified and allocated tasks to be collected, metadata from the plurality of heterogeneous data sources are collected to obtain preliminary collected data; The preliminary collected data is cleaned and verified to obtain the collected data.

[0006] In one embodiment, the step of extracting feature information from the collected data based on a preset generative AI big model to obtain feature data includes: Preprocessing the collected data to obtain preprocessed data, wherein the preprocessing includes one or more of denoising, standardization and format conversion; Determining an extraction method according to the data type of the preprocessed data; Based on the preset generative AI big model, feature extraction is performed on the corresponding pre-processed data to obtain preliminary extracted data; The initially extracted data are fused to obtain the feature data.

[0007] In one embodiment, the step of constructing a multimodal index structure based on the feature data includes: Classifying and sorting the characteristic data to obtain classified characteristic data; Based on the classification feature data, construct a text inverted index, a vector index and a tree index; Merging the text inverted index, the vector index and the tree index to obtain a fused index; The fusion index is compressed and optimized to obtain the multimodal index structure.

[0008] In one embodiment, the step of matching the search instruction with the index data in the multimodal index structure comprises: Parsing the search instruction and generating a query task based on the parsing result; Decomposing the query task to obtain sub-query tasks; Determine the index mapping of the sub-query task according to a preset matching strategy and the index type of the sub-query task; Based on the index mapping, the sub-query task is matched with the index data.

[0009] In one embodiment, the step of fusing and sorting the matching results to obtain target search results includes: Determine a fusion strategy according to the type of the search instruction and preset user history data; Based on the fusion strategy, evaluating the matching result; Sorting the matching results based on the evaluation results to obtain ordered results; The ordered results are deduplicated to obtain the target search results.

[0010] In addition, to achieve the above-mentioned purpose, the present application also proposes a multimodal data search device based on a generative AI big model, and the multimodal data search device based on a generative AI big model includes: A collection module, used to collect various types of data from multiple heterogeneous data sources to obtain collected data; A feature extraction module, used to extract feature information from the collected data to obtain feature data; A multimodal indexing module, used to construct a multimodal indexing structure based on the feature data; A matching module, used for matching the search instruction with the index data in the multimodal index structure; The target module is used to merge and sort the matching results to obtain the target search results.

[0011] In addition, to achieve the above-mentioned objectives, the present application also proposes a multimodal data search device based on a generative AI big model, the device comprising: a memory, a processor, and a multimodal data search program based on a generative AI big model stored on the memory and executable on the processor, the multimodal data search program based on the generative AI big model being configured to implement the steps of the multimodal data search method based on the generative AI big model as described above.

[0012] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, on which a multimodal data search program based on a generative AI big model is stored. When the multimodal data search program based on the generative AI big model is executed by a processor, the steps of the multimodal data search method based on the generative AI big model as described above are implemented.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the multimodal data search method based on the generative AI big model as described above.

[0014] This application obtains collected data by collecting various types of data from multiple heterogeneous data sources; extracts feature information from the collected data based on a preset generative AI big model to obtain feature data; constructs a multimodal index structure based on the feature data; matches search instructions with index data in the multimodal index structure; and fuses and sorts the matching results to obtain target search results. This application collects various types of data from multiple heterogeneous data sources, and uses a generative AI big model to extract feature information from the data and then construct a multimodal index structure. The multimodal data search method based on the generative AI big model can support efficient matching of search instructions with various types of data; optimizes matching results by fusing and sorting, and realizes rapid retrieval and relevance sorting of cross-modal data, thereby improving the efficiency and accuracy of data query. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flow chart of the first embodiment of the multimodal data search method based on the generative AI big model of the present application; Figure 2 This is a schematic diagram of a sub-process in the second embodiment of the multimodal data search method based on the generative AI big model of this application; Figure 3 This is a schematic diagram of a sub-process in the third embodiment of the multimodal data search method based on the generative AI big model of this application; Figure 4 This is a schematic diagram of the data search process in an embodiment of a multimodal data search method based on a generative AI big model of the present application; Figure 5 This is a schematic diagram of the module structure of a multimodal data search device based on a generative AI big model in an embodiment of the present application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the multimodal data search method based on the generative AI big model in the embodiment of the present application.

[0016] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0017] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0018] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0019] It should be noted that in the era of informatization and data-driven, enterprises and organizations store data in various forms, including structured data (such as database records), semi-structured data (such as XML, JSON files) and unstructured data (such as text, images, audio, video, etc.). Traditional data query (search) methods are usually designed for a single data source and data model, relying on structured query language (SQL) or simple full-text search technology. However, in the face of growing multimodal and heterogeneous data, these methods have obvious limitations in data integration, query efficiency and accuracy, multimodal processing capabilities, and personalized support. Especially in large-scale data environments, traditional query methods are difficult to respond to user requests quickly, and cannot provide highly relevant and personalized query results. In response to these challenges, existing research attempts to use technologies such as big data analysis, machine learning, and deep learning to improve the performance of data queries, but the efficiency and accuracy are low when processing large-scale, multi-source heterogeneous data. Therefore, how to improve the efficiency and accuracy of data queries has become a technical problem that needs to be solved urgently.

[0020] The main solution of this application is: this application obtains collected data by collecting various types of data from multiple heterogeneous data sources; extracts feature information from the collected data based on a preset generative AI big model to obtain feature data; builds a multimodal index structure based on the feature data; matches the search instructions with the index data in the multimodal index structure; and fuses and sorts the matching results to obtain the target search results.

[0021] This application collects various types of data from multiple heterogeneous data sources, and uses a generative AI big model to extract feature information and then construct a multimodal index structure. The multimodal data search method based on the generative AI big model can support efficient matching of search instructions with various types of data; by optimizing the matching results through fusion and sorting, it realizes rapid retrieval and relevance sorting of cross-modal data, thereby improving the efficiency and accuracy of data query.

[0022] It should be noted that the execution subject of the method of this embodiment can be a computing service device with data processing, network communication and program running functions, or it can be the above-mentioned multimodal data search device based on the generative AI big model with the same or similar functions. This embodiment and the following embodiments will be described by taking the multimodal data search device based on the generative AI big model as an example.

[0023] Based on this, the first embodiment of the multimodal data search method based on the generative AI big model of this application is proposed, please refer to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the multimodal data search method based on the generative AI big model of the present application.

[0024] In this embodiment, the multimodal data search method based on the generative AI big model includes the following steps: S1: Collect various types of data from multiple heterogeneous data sources to obtain collected data; It should be noted that heterogeneous data sources refer to data sources with different data types, data structures, storage formats and access methods. Heterogeneous data sources can include relational databases (such as MySQL, Oracle), NoSQL databases (such as MongoDB, Cassandra), document management systems (such as SharePoint, Confluence), cloud storage platforms (such as AWS S3, Azure Blob Storage), data files (such as CSV, JSON, XML) and other forms of internal and external data sources. Due to the differences in these data sources, there are certain technical challenges in achieving unified data collection and processing. Multiple types of data refer to different data forms collected from multiple heterogeneous data sources, covering structured data (such as table data in databases), semi-structured data (such as data in XML and JSON formats), and unstructured data (such as text, images, audio, video, etc.). These data types differ in content representation, storage form and parsing methods, requiring different collection and processing methods. Collected data refers to the original data set obtained from multiple heterogeneous data sources after data connection, extraction and preliminary processing. Collected data usually contains original information of different data types and structural forms, which is the basis for subsequent feature information extraction and data index construction.

[0025] Specifically, establish connections with multiple heterogeneous data sources and configure necessary connection parameters (such as IP address, port number, authentication information, etc.) to ensure safe and efficient access to various data sources. To this end, the system will load the corresponding data source driver or client library (such as JDBC, ODBC, RESTAPI, etc.) and perform identity authentication. After completing the connection, the system will obtain the metadata of each data source (such as data structure, field information, data type, etc.) to analyze and understand the data format and storage method of each data source.

[0026] Furthermore, based on the acquired metadata, a data collection task list is generated, and the data is classified and deployed according to data type and priority. During the execution of the data collection task, the system selects the corresponding collection method according to the characteristics of different data sources. For example, for relational databases, SQL query statements are executed to obtain data; for NoSQL databases, query interfaces or APIs are used to obtain data; for document management systems and cloud storage platforms, data is collected through file reading or API download operations. After completing data collection, the system will perform preliminary cleaning and verification of the collected data to remove duplicate, invalid or incorrectly formatted data to ensure the integrity and consistency of the collected data.

[0027] By collecting various types of data from multiple heterogeneous data sources, this step achieves unified access and integration of various data forms, solves the problems of data dispersion and inconsistent formats, and provides comprehensive and complete basic data support for subsequent data processing and analysis, thereby significantly improving the coverage of data queries and the efficiency of data utilization.

[0028] S2: Extract feature information from the collected data based on a preset generative AI big model to obtain feature data; It should be noted that the preset generative AI big model refers to a pre-trained deep learning model, especially a generative model (such as GPT, BERT, T5, etc.). These models are usually pre-trained with massive data and have strong natural language understanding, semantic analysis and feature extraction capabilities. Generative AI models can process multimodal data such as text, images, audio, etc., and generate high-quality feature representations. Feature information extraction refers to extracting key information or features that can characterize the essence of the data from the original data. The generative AI big model can automatically understand and parse the content of the data, and extract information such as embedded vectors, keywords, and topic tags that represent its semantics or features. Feature data refers to structured or semi-structured feature representations extracted from the collected data. These features can be text semantic vectors, image feature vectors, audio spectrum features, or video frame features. Feature data is the basis for subsequent index construction and query processing.

[0029] Specifically, the collected data is input into the preset generative AI big model. According to the different types of data (such as text, image, audio, etc.), the system performs necessary preprocessing on the data. For example, text data will be segmented, denoised and normalized; image data will be size normalized and color space converted; audio data may require noise reduction and sampling rate standardization. These preprocessing operations ensure that the generative AI big model can receive input data in the optimal form. The preset generative AI big model extracts feature information from the preprocessed data. For text data, generative AI big models (such as GPT, BERT) can capture the contextual semantic relationship of text and generate accurate semantic vectors or keyword representations; for image data, deep learning-based generative models (such as CLIP, DALL·E) can generate feature vectors of images and extract high-level semantic features such as color, texture, shape, etc. of images; for audio data, generative AI models can extract spectral features, rhythm and timbre features of audio. Through the highly automated and intelligent processing of these models, the system can obtain accurate and expressive feature data.

[0030] Furthermore, the extracted feature data is further normalized and integrated to form a structured feature representation that can support subsequent multimodal index construction and query processing. For example, the semantic vector of the text can be used as a keyword index, the feature vector of the image can be used for similarity matching, and the audio feature can be used for pattern recognition and retrieval. These feature data provide high-quality input for the query system and can significantly improve the efficiency and accuracy of subsequent data processing.

[0031] By extracting feature information from collected data based on a preset generative AI big model, the system can automatically and accurately capture the deep semantics and feature information in the data. The powerful capabilities of the generative AI model in multimodal data processing significantly improve the quality of feature extraction, enabling the system to generate highly expressive and unified feature data, thereby improving the efficiency and accuracy of subsequent data retrieval and query processing.

[0032] S3: constructing a multimodal index structure based on the feature data; It should be noted that the multimodal index structure refers to an index system that can uniformly store and manage data features of different types (modalities). The multimodal index structure supports the retrieval of multiple data types, including text, images, videos, audio, and structured data. It combines feature representation methods of different types of data (such as inverted index, vector index, tree index, etc.) to support cross-modal data retrieval and similarity matching.

[0033] Specifically, the feature data is classified and sorted to determine the indexing method suitable for the features of each data type. For text feature data, an inverted index is used to map each term in the text to a list of document IDs containing the term to support efficient keyword retrieval and full-text search. For embedded vector data such as images, videos, and audio, vector indexes such as HNSW, IVF, and KD trees are used to support fast similarity calculations and nearest neighbor searches. For structured data features, tree indexes such as B-trees and R-trees are used to support range queries and multidimensional data searches.

[0034] Furthermore, the computer device fuses various indexes (such as text inverted index, vector index and tree index) to build a unified multimodal index structure. The fusion process can be implemented through a variety of strategies, such as allocating storage space for each index and selecting the optimal query path according to the data type, or supporting cross-modal retrieval by building a joint index node. The fused index structure is further compressed and optimized, using sparse representation, quantization coding, clustering and other technologies to reduce storage space and computational complexity, and ultimately forming an efficient and scalable multimodal index structure.

[0035] By building a multimodal index structure based on feature data, the system can efficiently manage and retrieve feature information from multiple data types, and support cross-modal comprehensive multimodal data search and similarity calculation based on generative AI big models. This process improves the coverage and response speed of multimodal data search based on generative AI big models, ensuring that high retrieval accuracy and performance can be maintained in large-scale data environments, and significantly enhances the applicability and practicality of multimodal data search methods based on generative AI big models.

[0036] S4: matching the search instruction with the index data in the multimodal index structure; It should be noted that search instructions refer to search requests or query requirements input by users, which can be natural language descriptions, keywords, voice input, or multimodal queries containing images and videos. Search instructions are intended to express the data or information that users want to retrieve, and the system needs to parse and understand them to generate specific query tasks. Multimodal index structure refers to a comprehensive index system that contains indexes of multiple data types (modalities), which can support the retrieval of data in different modalities such as text, images, audio, and video. The multimodal index structure uses a variety of indexing methods such as inverted indexes, vector indexes, and tree indexes to effectively organize and manage data features, and supports fast and accurate cross-modal multimodal data searches based on generative AI large models. Index data refers to various feature data and their associated information stored in the multimodal index structure. These data are used to respond to user query requests and provide the most relevant results based on query conditions and index matching strategies.

[0037] Specifically, after receiving the user's search instruction, it is parsed and understood. Natural language processing (NLP) technology is used to perform operations such as word segmentation, semantic analysis, named entity recognition (NER), and intent recognition on text queries to extract keywords, topics, constraints, and logical relationships in the query; for voice input, speech recognition (ASR) is first performed to convert it into text; for image or video queries, its visual features or frame features are extracted. Based on the parsing results, the system generates a set of sub-query tasks and specifies the corresponding modal data type (such as text, image, audio, etc.) for each sub-query task.

[0038] Furthermore, according to the modal type and matching strategy of the sub-query tasks, these sub-query tasks are mapped to the corresponding index data in the multimodal index structure. For text queries, the system searches for a set of document IDs related to the query keywords in the inverted index; for image, video or audio queries, the system uses vector indexes to calculate the similarity between the query feature vector and the index feature vector, and finds the best matching result through a fast nearest neighbor search algorithm (such as HNSW, IVF, KD tree, etc.); for queries on structured data, the system performs multidimensional conditional filtering or range queries in the tree index. The matching results will be aggregated according to the specified strategy for subsequent result fusion and sorting processing.

[0039] By matching the search instructions with the index data in the multimodal index structure, this step achieves comprehensive analysis and precise mapping of user query requirements, and supports unified retrieval of cross-modal and multi-type data. By adapting to the data characteristics and matching strategies of different modalities, the system can quickly and accurately find relevant results, significantly improving the query response speed and the relevance of the results, and effectively improving the efficiency and user experience of the multimodal data search method based on the generative AI large model.

[0040] S5: Merge and sort the matching results to obtain target search results.

[0041] It should be noted that the matching results refer to the set of all potentially relevant data items obtained by matching the user's search instructions with the index data in the multimodal index structure. The matching results can include multiple data items of different data modes (such as text, images, videos, and audio), and each data item has its corresponding matching score or relevance measurement. Fusion and sorting refers to the process of comprehensively analyzing and integrating data items of different modes or sources to calculate the comprehensive score of each data item and sort all data items according to the score. The purpose of fusion and sorting is to give priority to displaying the results that are most relevant to the user's query intent, and to optimize the effect of data retrieval and user experience. The target search results refer to the final query result set generated after fusion and sorting. The target search results are arranged from high to low according to relevance or matching degree to ensure that the most relevant data items that best meet the user's query needs are at the top of the result set.

[0042] Specifically, all data matching results from different sub-query tasks and modalities are collected to form a preliminary set of matching results. Each matching result item contains its corresponding score (such as keyword matching score, vector similarity score, etc.) and related metadata (such as document ID, data modality, data source, etc.). Then, the system determines the appropriate fusion strategy based on the type of search instruction, user preferences and historical behavior. For example, for mixed text and image queries, a weighted fusion method can be used to assign weights to the scores of different modalities and calculate the comprehensive score of each data item; for compound queries involving multiple sub-queries, Boolean logic operations (such as AND, OR operations) can be used to combine the results of different sub-queries.

[0043] Furthermore, the comprehensive scores are calculated and sorted. The final relevance of each data item can be evaluated using a weighted average method, a linear regression model, or a machine learning ranking model (such as RankNet, LambdaMART). After sorting, the system deduplicates the result set to ensure that the results displayed to the user do not contain duplicate or highly similar data items. For data items with higher similarity, the system chooses to retain the version with a higher score or one that is more in line with user preferences. Finally, the system returns the sorted preferred result set to the user as the target search result, and displays it on the user interface by relevance or priority.

[0044] By fusing and sorting the matching results, this step achieves comprehensive analysis and intelligent sorting of cross-modal data, ensuring the relevance and accuracy of the query results. This process significantly optimizes the display order of data items, improves the user search experience, enables users to quickly find the result items that best meet their needs, and improves the efficiency of data query methods and the quality of query results.

[0045] This embodiment obtains collected data by collecting various types of data from multiple heterogeneous data sources; extracts feature information from the collected data based on a preset generative AI big model to obtain feature data; constructs a multimodal index structure based on the feature data; matches search instructions with index data in the multimodal index structure; and fuses and sorts the matching results to obtain target search results. This embodiment collects various types of data from multiple heterogeneous data sources, and uses a generative AI big model to extract feature information from the data and then construct a multimodal index structure. The multimodal data search method based on the generative AI big model can support efficient matching of search instructions with various types of data; optimizes matching results by fusing and sorting, and realizes rapid retrieval and relevance sorting of cross-modal data, thereby improving the efficiency and accuracy of data query.

[0046] Based on the above first embodiment, a second embodiment of the multimodal data search method based on a generative AI large model is proposed. Figure 2 , Figure 2 This is a schematic diagram of a sub-process in the second embodiment of the multimodal data search method based on the generative AI big model of this application.

[0047] like Figure 2 As shown, in this embodiment, step S1 includes: S11: Establishing connections with the multiple heterogeneous data sources; S12: Obtaining tasks to be collected, and classifying and allocating the tasks to be collected; S13: Based on the classified and allocated tasks to be collected, metadata from the plurality of heterogeneous data sources are collected to obtain preliminary collected data; S14: Cleaning and verifying the preliminary collected data to obtain the collected data.

[0048] It should be noted that heterogeneous data sources refer to multiple data sources with different data types, data structures, storage formats and access methods. Heterogeneous data sources can include relational databases (such as MySQL, Oracle), NoSQL databases (such as MongoDB, Cassandra), document management systems (such as SharePoint), cloud storage platforms (such as AWS S3, Azure Blob Storage), file storage (such as CSV, JSON, XML files), etc. The task to be collected refers to the specific task that needs to extract data from multiple heterogeneous data sources. Each task to be collected contains information such as the type, location, access rights, collection scope (such as a specific table, collection or folder) and collection method of the data source. Metadata refers to information that describes the structure, content, context and other key attributes of the data. For example, metadata in a database includes table structure, field type, index information, etc.; metadata in a document system includes file name, format, size, creation time, etc. Metadata helps to understand the characteristics and structure of data and guide subsequent data collection and processing. Preliminary collected data refers to the original data set obtained after performing data collection tasks on multiple heterogeneous data sources. It may contain some invalid, duplicate or abnormal data and needs further cleaning and verification. Cleaning and verification refers to the process of quality inspection and processing of preliminary collected data. Cleaning operations include removing duplicate data, filling missing data, format standardization and outlier processing; verification operations ensure the integrity, accuracy and consistency of the data.

[0049] Specifically, the data connection module is used to establish connections with multiple heterogeneous data sources. According to the type of different data sources, the system loads the corresponding driver or client library (such as JDBC, ODBC, RESTAPI, etc.), configures the connection parameters (such as IP address, port number, user name and password, etc.), and performs necessary identity authentication and security authentication. After the connection is established, the system will monitor the connection status and maintain the session to ensure the continuity and stability of the data collection process. Obtain a list of tasks to be collected, and classify and allocate these tasks. The classification process divides tasks according to the type of data source (such as relational database, NoSQL database, document storage, etc.), data structure (such as table structure, collection, folder, etc.) and data priority (such as timeliness, importance). The allocation process optimizes the allocation of data collection tasks according to the system resource status (such as bandwidth, processing power) and task complexity. The classified and allocated task list will be used to guide the order and strategy of data collection.

[0050] Furthermore, in the specific data collection process, the system collects metadata from various heterogeneous data sources according to the classified and deployed task list. For relational databases, the system will execute SQL query statements to extract table structure and field information; for NoSQL databases, the corresponding query interface or API is used to obtain the architecture information of the collection; for document management systems and cloud storage platforms, metadata (such as file name, size, format) is extracted through the file directory structure. The collected metadata is used to preliminarily understand the structure and content of the data source, guide the next step of data extraction operations, and form preliminary collected data. Finally, the system cleans and verifies the preliminary collected data to ensure the integrity and reliability of the data. During the cleaning process, the system removes duplicate data, fills missing values ​​(such as using mean, median, etc.), handles outliers (such as eliminating outliers that exceed a reasonable range) and unifies data formats (such as date format, encoding format). During the verification process, the system ensures the quality and accuracy of the data through data integrity checks (such as foreign key constraints, data type checks) and consistency checks (such as the matching of the same field in different data sources). The cleaned and verified data is the final collected data for subsequent feature extraction and index construction.

[0051] This step achieves unified management and flexible access to different types of data sources by establishing connections with multiple heterogeneous data sources and obtaining tasks to be collected for classification and deployment. By collecting metadata and performing cleansing and verification, the integrity, consistency and accuracy of the data are ensured, providing high-quality basic data support for subsequent data analysis and query, thereby improving the efficiency and reliability of the overall multimodal data search method based on the generative AI large model.

[0052] Based on the above first embodiment, in this embodiment, step S2 includes: S21: preprocessing the collected data to obtain preprocessed data, wherein the preprocessing includes one or more of denoising, standardization and format conversion; S22: determining an extraction method according to the data type of the preprocessed data; S23: Based on the preset generative AI big model, feature extraction is performed on the corresponding pre-processed data to obtain preliminary extracted data; S24: Fusing the preliminary extracted data to obtain the feature data.

[0053] It should be noted that preprocessing refers to a series of preliminary cleaning and standardization operations on the collected data before feature extraction to improve the accuracy and efficiency of subsequent feature extraction. Preprocessing operations include denoising (such as removing invalid information or noise), standardization (such as numerical normalization, data distribution standardization) and format conversion (such as text encoding conversion, image format conversion). Denoising refers to the process of removing or reducing invalid information, noise or interference signals in data. Denoising operations can be applied to various data types, such as removing special characters and stop words in text, removing noise in images, or filtering audio data. Standardization refers to converting data from different sources or formats into a consistent standard form for subsequent processing. For example, normalizing numerical data to a specific range (such as 0 to 1), or resizing an image to a uniform size so that different data types have the same scale and format. Format conversion refers to the process of converting data from one format to another. For example, converting text data from different encodings (such as UTF-8, GBK) to a unified encoding, or converting image data from one format (such as PNG) to another format (such as JPEG). Format conversion helps ensure data compatibility and processability. Feature extraction refers to the process of extracting information that can represent the essential characteristics of the data from preprocessed data. Generative AI big models refer to deep learning models that have been pre-trained with large-scale data and can understand, generate and extract semantic information from data. Common generative AI models include GPT, BERT, CLIP, etc., which can process data in multiple modalities such as text, images, audio, etc., and generate high-quality feature vectors.

[0054] Specifically, the data collected from multiple heterogeneous data sources are preprocessed to obtain more standardized and structured preprocessed data. Depending on the type of data, the preprocessing process includes denoising, standardization, and format conversion. For example, for text data, the system will remove stop words, correct spelling errors, and standardize text encoding; for image data, the system may perform denoising filtering, unify resolution, and perform color space conversion; for audio data, the system will remove background noise, standardize sampling rate, etc. This preprocessing process ensures that all data can be input into the subsequent feature extraction stage in a consistent form.

[0055] Furthermore, the appropriate feature extraction method is determined according to the type of preprocessed data. For text data, generative AI large models (such as GPT, BERT) can extract semantic features and keywords; for image data, models based on convolutional neural networks (CNN) (such as CLIP, VGG) can extract visual features of images; for audio data, the model can extract spectral features, tonal features, etc. of audio. The system will input each data modality into the corresponding generative AI large model to extract feature information and obtain preliminary feature data. After the preliminary extraction data is generated, the system fuses the feature data of different modalities to generate a unified feature representation. The fusion process can be achieved through a multimodal deep learning model or simple feature splicing. For example, the semantic vector of the text is spliced ​​or weightedly fused with the visual features of the image to form a unified feature representation. This process ensures that feature data from different data modalities can work together in a unified representation space, and finally generate feature data for subsequent query or index construction.

[0056] By preprocessing the collected data and extracting features based on the generative AI big model, the system can automatically and efficiently extract deep feature information from data of different modalities. After the generated preliminary feature data is fused, a unified feature representation is generated to support the comprehensive retrieval and processing of multimodal data. This process greatly improves the accuracy, efficiency and consistency of feature extraction, provides high-quality feature data support for subsequent data indexing and query, and effectively improves the overall query performance and accuracy of the system.

[0057] This embodiment obtains collected data by collecting various types of data from multiple heterogeneous data sources; extracts feature information from the collected data based on a preset generative AI big model to obtain feature data; constructs a multimodal index structure based on the feature data; matches search instructions with index data in the multimodal index structure; and fuses and sorts the matching results to obtain target search results. This embodiment collects various types of data from multiple heterogeneous data sources, and uses a generative AI big model to extract feature information from the data and then construct a multimodal index structure. The multimodal data search method based on the generative AI big model can support efficient matching of search instructions with various types of data; optimizes matching results by fusing and sorting, and realizes rapid retrieval and relevance sorting of cross-modal data, thereby improving the efficiency and accuracy of data query.

[0058] Based on the above second embodiment, a third embodiment of the multimodal data search method based on a generative AI large model is proposed. Figure 3 , Figure 3 This is a schematic diagram of a sub-process in the third embodiment of the multimodal data search method based on the generative AI big model of this application.

[0059] In this embodiment, step S3 includes: S31: classifying and sorting the feature data to obtain classified feature data; S32: constructing a text inverted index, a vector index and a tree index based on the classification feature data; S33: Merge the text inverted index, the vector index and the tree index to obtain a fused index; S34: compressing and optimizing the fusion index to obtain the multimodal index structure.

[0060] It should be noted that classification and sorting refer to the process of grouping and sorting feature data. Classification is grouping according to the type, source or modality (such as text, image, audio, etc.) of feature data; sorting is arranging data according to a certain measure of feature data (such as similarity, importance, etc.) to facilitate more efficient index construction and query processing. Text inverted index is a data structure for fast full-text retrieval, which records the mapping relationship between each term and the document collection containing the term. Inverted index is suitable for keyword search and query of text data, and efficient text retrieval is achieved through the inverted list between term and document ID. Vector index is a data structure for fast similarity retrieval, which indexes high-dimensional feature vectors (such as embedded vectors of images, videos, and audio). Vector index supports fast nearest neighbor search, and common methods include HNSW, IVF, KD tree, etc. Tree index is an index structure for efficient management and query of structured data (such as database table data). Tree indexes (such as B-trees and R-trees) support range queries, prefix matching, and multidimensional data searches, and are suitable for processing large-scale structured data. Fusion indexes refer to the combination and integration of multiple different types of index structures (such as text inverted indexes, vector indexes, and tree indexes) to form a unified index system to support cross-modal comprehensive retrieval and query of multimodal data. Compression and optimization refer to the process of data compression and structural optimization of fusion indexes to reduce storage space, reduce computational complexity, and increase query speed. Common compression techniques include sparse representation, quantization coding, etc., and optimization techniques include index structure adjustment, hierarchical storage, etc.

[0061] Specifically, the feature data is classified and sorted in order to build a suitable index structure. The classification process groups the feature data according to the modal type (such as text, image, video, audio, etc.), data source or other attributes; the sorting process arranges the feature data in each group according to specific metrics (such as data similarity, importance, frequency, etc.). Through classification and sorting, the system can determine the index type required for each type of data and its construction order, obtain classified feature data, and provide a basis for subsequent index construction. Based on the classified feature data, index structures suitable for different data types are constructed respectively. For text data, the system constructs an inverted index to map each term with a list of document IDs containing the term to support fast keyword retrieval; for image, video, audio and other data, the system constructs a vector index to support fast retrieval of feature vectors through efficient similarity search algorithms (such as HNSW, IVF, KD tree); for structured data, the system constructs a tree index (such as B-tree or R-tree) to support multi-dimensional conditional queries and range queries.

[0062] Furthermore, the constructed text inverted index, vector index and tree index are fused to form a unified fused index. The fusion process enables the collaboration of different types of indexes by defining appropriate fusion strategies. For example, the system can allocate a dedicated storage area for each index and dynamically select the optimal query path according to the query requirements; or build a joint index node to integrate indexes of different modalities into the same index framework to support cross-modal queries and comprehensive retrieval. The fused index structure is compressed and optimized to improve storage efficiency and query speed. Compression operations may include using sparse representation to reduce unnecessary storage space, or using quantized encoding to reduce the storage size of the index; optimization operations may include adjusting the index structure to reduce the query path length, or using hierarchical storage to accelerate data access. After compression and optimization, an efficient multimodal index structure is finally obtained, which can support rapid response to various complex queries and comprehensive retrieval of multimodal data.

[0063] By classifying and sorting feature data, the system can effectively select appropriate data indexing methods and optimize the index construction process; based on the construction and fusion of multiple index structures, the system realizes unified management and retrieval of multimodal data; through index compression and optimization, the system greatly reduces storage requirements and computing overhead, improves the speed and accuracy of data query (search), and significantly improves the efficiency and flexibility of overall data query.

[0064] Based on the above second embodiment, in this embodiment, step S4 includes: S41: parsing the search instruction, and generating a query task based on the parsing result; S42: Decompose the query task to obtain sub-query tasks; S43: Determine the index mapping of the sub-query task according to a preset matching strategy and the index type of the sub-query task; S44: Based on the index mapping, match the sub-query task with the index data.

[0065] It should be noted that the search instruction refers to the query request or search demand input by the user, which may take various forms, such as natural language description, keyword search, voice input, image or video query, etc. The search instruction is used to express the information that the user wants to retrieve, and the system needs to parse it to determine the specific query requirements and scope. The query task refers to one or more retrieval operation tasks generated by the system based on the parsed search instruction. The query task describes the data type, scope, conditions and corresponding operation steps to be retrieved, and is the basis for the system to perform query operations. The sub-query task refers to a smaller retrieval task unit formed by further refining or decomposing the query task. The sub-query task corresponds to the specific data type or modality (such as text, image, video, etc.) in the search instruction, and can be executed independently and produce partial query results. The preset matching strategy refers to the rules and methods pre-defined by the system for matching the sub-query task with the index data. The matching strategy selects the optimal matching method according to the characteristics of the sub-query task and the data modality. For example, the keyword retrieval strategy can be used for text matching, and the vector similarity calculation strategy can be used for image matching. Index type refers to different data indexing methods in a multimodal index structure, such as inverted index, vector index, tree index, etc. Each index type is suitable for a specific data modality or data feature. Index mapping refers to the process of mapping a subquery task to a specific index type in a multimodal index structure based on the requirements of the subquery task. Index mapping is used to determine the index data corresponding to the subquery task for efficient matching and retrieval operations.

[0066] Specifically, after receiving the user's search instruction, it is parsed using natural language processing (NLP) technology. The parsing process includes operations such as word segmentation, part-of-speech tagging, semantic analysis, named entity recognition (NER), and intent recognition to extract keywords, query conditions, target data modalities (such as text, images, videos, etc.) and logical relationships in the search instruction. At the same time, the system also needs to process non-text search instructions (such as voice, image, video, etc.), convert voice into text through voice recognition (ASR) technology, or use computer vision technology to extract visual features in images and videos. According to the parsing results, the system generates one or more query tasks, and clarifies the modality, target data and retrieval conditions involved in each task. The generated query tasks are decomposed to obtain multiple sub-query tasks. Each sub-query task corresponds to a specific data modality (such as text, image, audio, etc.) and a specific retrieval operation. For compound query tasks (such as "find pictures containing specific keywords"), the system decomposes them into text query tasks and image query tasks. Through decomposition, the system can more effectively allocate computing resources and optimize the query process.

[0067] Furthermore, according to the modality type and query requirements of each sub-query task, the system uses a preset matching strategy to select the optimal index mapping. For example, for text sub-query tasks, the system selects the text inverted index (InvertedIndex) to support keyword matching; for image or video sub-query tasks, the system selects the vector index (Vector Index) for similarity calculation; for structured data sub-query tasks, the system selects the tree index (Tree Index) to support range query and multi-dimensional data retrieval. Based on these mappings, the system determines the matching method of the sub-query task with the specific index data in the multi-modal index structure. Based on the determined index mapping, each sub-query task is matched with the corresponding index data. For text sub-query tasks, the system searches for the relevant document ID set in the inverted index; for image, video or audio sub-query tasks, the system uses the vector index to perform feature vector similarity calculation and find the result with the highest similarity; for structured data sub-query tasks, the system performs conditional filtering or range query in the tree index. The matching results of all sub-queries will be further processed and fused to form the final query result.

[0068] By parsing the search instructions and decomposing the tasks, the system can accurately understand the user's query intent and generate appropriate sub-query tasks; the optimal index mapping is selected according to the preset matching strategy to ensure efficient matching and execution of query tasks and index data. This process significantly improves the accuracy and response speed of data queries, realizes the comprehensive retrieval capability of multimodal data, and optimizes the overall performance of data queries and user experience.

[0069] Based on the above second embodiment, in this embodiment, step S5 includes: S51: Determine a fusion strategy according to the type of the search instruction and preset user history data; S52: Based on the fusion strategy, evaluating the matching result; S53: Sort the matching results based on the evaluation results to obtain ordered results; S54: De-duplicate the ordered results to obtain the target search results.

[0070] It should be noted that the search instruction type refers to the specific form or characteristics of the query request input by the user, such as keyword search, natural language query, image search, video search, voice search, etc. The search instruction type determines the retrieval method and fusion strategy that the system needs to adopt. Preset user history data refers to the user behavior data and preference information stored and analyzed by the system, such as the user's historical query records, click behavior, access frequency, stay time, etc. Preset user history data helps the system understand the user's interests and preferences, and is used to optimize the sorting and recommendation of query results. Fusion strategy refers to the method and rules for comprehensive evaluation of multiple matching results determined by the system based on the search instruction type and user history data. Fusion strategy can be used in a variety of ways, such as weighted average, linear regression, machine learning model, etc., to generate a result ranking that is more in line with the user's query intention. Matching result evaluation refers to the process of calculating and evaluating the relevance and priority of different matching results according to the fusion strategy. The evaluation process may involve a variety of factors, including the matching score of the result, user preferences, the contextual relationship of the result, etc., to determine the comprehensive score of each result. Sorting refers to arranging the result set from high to low according to the evaluation score of the matching result. The purpose of sorting is to ensure that the most relevant and user-friendly results are displayed to the user first. Deduplication refers to identifying and removing duplicate or highly similar results in the sorted result set to ensure that the query result set seen by the user is accurate and non-redundant. Deduplication can be based on unique identifiers, similarity metrics, or content comparison methods.

[0071] Specifically, the appropriate fusion strategy is determined according to the type of search instruction (such as text search, image search, video search, etc.) and the user's preset historical data (such as query history, click preference, etc.). For example, if the search instruction is a natural language text query and the user has an obvious preference for a specific topic, the system may adopt a weighted fusion strategy based on a topic model; if the user is more interested in image search results, the system may give priority to image matching scores and use visual feature similarity as the main fusion basis. The fusion strategy maximizes the relevance and personalization of query results by combining the characteristics of the search instruction and the user's historical data. All matching results are comprehensively evaluated based on the determined fusion strategy. Each matching result item is weighted summed or comprehensively scored through a machine learning model (such as RankNet, LambdaMART) based on its score in different subqueries, user preference weight, relevance measurement, etc. The evaluation process ensures that the query results can more accurately reflect the user's intentions and interests and provide a basis for subsequent sorting.

[0072] Furthermore, the matching results are sorted according to their comprehensive evaluation scores. The sorting algorithm can be a quick sort, a heap sort, or a model-based sorting algorithm to ensure that the results with higher scores are ranked first, thereby improving the user's search experience. The sorted result set is arranged from high to low in terms of relevance. The sorted ordered result set is deduplicated. By comparing the unique identifiers of the result items (such as document ID, image ID, etc.) or calculating the similarity between the result items (such as text similarity, image feature similarity), the system identifies and deletes duplicate or highly similar result items, ensuring that the target search result set ultimately displayed to the user is both accurate and non-redundant.

[0073] This step ensures high relevance and personalization of query results by determining the fusion strategy based on the search instruction type and user historical data, and comprehensively evaluating, sorting and deduplicating the matching results. Users can quickly obtain the results that best meet their needs while avoiding the interference of redundant information, significantly improving the efficiency of data query and user experience.

[0074] This embodiment obtains collected data by collecting various types of data from multiple heterogeneous data sources; extracts feature information from the collected data based on a preset generative AI big model to obtain feature data; constructs a multimodal index structure based on the feature data; matches search instructions with index data in the multimodal index structure; and fuses and sorts the matching results to obtain target search results. This embodiment collects various types of data from multiple heterogeneous data sources, and uses a generative AI big model to extract feature information from the data and then construct a multimodal index structure. The multimodal data search method based on the generative AI big model can support efficient matching of search instructions with various types of data; optimizes matching results by fusing and sorting, and realizes rapid retrieval and relevance sorting of cross-modal data, thereby improving the efficiency and accuracy of data query.

[0075] For example, in order to help understand the technical concept or technical principle of the multimodal data search method based on the generative AI big model in the above embodiment, please refer to Figure 4 , Figure 4 This is a schematic diagram of the data search process in one embodiment of the multimodal data search method based on the generative AI big model of the present application.

[0076] like Figure 4As shown, in this embodiment, the data connector collects and aggregates data from multiple sources, including knowledge bases, business data, third-party cloud service data, etc.; the data type can be text, image, video, audio, and structured data. The collected data is processed by the data parsing module to extract useful features and information, including extracting text, extracting text from pictures, semantic description of pictures, video semantic extraction, etc.; this step involves large model technology, which is used to understand and parse unstructured data. The information after data parsing is used to establish a search index for different types of data through LLM (large language model); the index module creates a data retrieval structure, including text embedding vectors, picture vectors, video batches, structured data tables, and databases. The user makes a query request, and the query content is semantically understood by the LLM model service and converted into different query task requests to understand from the user's question whether to find documents, pictures, or videos; the processed query content will be matched with the index data and return relevant results; there are several search methods, including keyword search, full-text search, semantic search, and SQL query. According to the query type (keyword query, semantic query, SQL query), the preliminary search results of the user's query are obtained; then the different types of query results are combined with the user's questions to determine the matching degree and re-rank them; the re-ranking will comprehensively consider the relevance of the query, user intent, business logic and other factors, and finally determine the priority and order of the results. The final processed query results are returned to the user, who can interact with the system through the LLM model service for further query or adjustment.

[0077] The present application also provides a multimodal data search device based on a generative AI large model. Please refer to Figure 5 , Figure 5 The module structure diagram of the multimodal data search device based on the generative AI big model in the embodiment of the present application is as follows: The collection module 501 is used to collect various types of data from multiple heterogeneous data sources to obtain collected data; A feature extraction module 502 is used to extract feature information from the collected data based on a preset generative AI large model to obtain feature data; A multimodal indexing module 503, configured to construct a multimodal indexing structure based on the feature data; A matching module 504, configured to match the search instruction with the index data in the multimodal index structure; The target module 505 is used to merge and sort the matching results to obtain target search results.

[0078] The multimodal data search device based on the generative AI big model provided in the embodiment of the present application adopts the multimodal data search method based on the generative AI big model in the above embodiment, which can solve the technical problem of how to improve the efficiency and accuracy of data query. Compared with the prior art, the beneficial effects of the multimodal data search device based on the generative AI big model provided in the embodiment of the present application are the same as the beneficial effects of the multimodal data search method based on the generative AI big model provided in the above embodiment, and the other technical features in the multimodal data search device based on the generative AI big model are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0079] The present application provides a multimodal data search device based on a generative AI big model, and the multimodal data search device based on a generative AI big model includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal data search method based on the generative AI big model in the above-mentioned embodiment.

[0080] Reference below Figure 6 , Figure 6 The schematic diagram of the device structure of the hardware operating environment involved in the multimodal data search method based on the generative AI big model in the embodiment of the present application shows a schematic diagram of the structure of the multimodal data search device based on the generative AI big model suitable for implementing the embodiment of the present application. The multimodal data search device based on the generative AI big model in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions: tablet computers), PMPs (Portable Media Players: portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The multimodal data search device based on the generative AI big model shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0081] like Figure 6As shown, a multimodal data search device based on a generative AI large model may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. Various programs and data required for the operation of a multimodal data search device based on a generative AI large model are also stored in RAM1004. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the multimodal data search device based on the generative AI big model to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a multimodal data search device based on the generative AI big model with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0082] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0083] The multimodal data search device based on the generative AI big model provided by the present application adopts the multimodal data search method based on the generative AI big model in the above-mentioned embodiment, which can solve the technical problem of how to improve the efficiency and accuracy of data query. Compared with the prior art, the beneficial effects of the multimodal data search device based on the generative AI big model provided by the present application are the same as the beneficial effects of the multimodal data search method based on the generative AI big model provided by the above-mentioned embodiment, and the other technical features in the multimodal data search device based on the generative AI big model are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0084] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0085] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0086] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the multimodal data search method based on the generative AI big model in the above-mentioned embodiment.

[0087] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0088] The above-mentioned computer-readable storage medium may be included in a multimodal data search device based on a generative AI big model; or it may exist independently without being assembled into a multimodal data search device based on a generative AI big model.

[0089] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a multimodal data search device based on a generative AI big model, the multimodal data search device based on the generative AI big model: collects multiple types of data from multiple heterogeneous data sources to obtain collected data; extracts feature information from the collected data based on a preset generative AI big model to obtain feature data; constructs a multimodal index structure based on the feature data; matches the search instruction with the index data in the multimodal index structure; merges and sorts the matching results to obtain the target search results. The computer program code for performing the operations of the present application can be written in one or more programming languages ​​or a combination thereof, and the programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or it can be connected to an external computer (for example, through the Internet using an Internet service provider).

[0090] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0091] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0092] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned multimodal data search method based on the generative AI big model, and can solve the technical problem of how to improve the efficiency and accuracy of data query. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the multimodal data search method based on the generative AI big model provided in the above-mentioned embodiment, and will not be repeated here.

[0093] An embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the multimodal data search method based on the generative AI big model as described above.

[0094] The computer program product provided in this application can solve the technical problem of how to improve the efficiency and accuracy of data query. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiment of this application are the same as the beneficial effects of the multimodal data search method based on the generative AI big model provided in the above embodiment, which will not be repeated here.

[0095] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A multimodal data search method based on a generative AI large model, characterized in that: The method comprises: Collect various types of data from multiple heterogeneous data sources to obtain collected data; Extract feature information from the collected data based on a preset generative AI big model to obtain feature data; Based on the feature data, construct a multimodal index structure; matching a search instruction with index data in the multimodal index structure; The matching results are merged and sorted to obtain the target search results.

2. The method according to claim 1, characterized in that The step of collecting multiple types of data from multiple heterogeneous data sources to obtain collected data includes: Establishing connections with the plurality of heterogeneous data sources; Obtaining tasks to be collected, and classifying and allocating the tasks to be collected; Based on the classified and allocated tasks to be collected, metadata from the plurality of heterogeneous data sources are collected to obtain preliminary collected data; The preliminary collected data is cleaned and verified to obtain the collected data.

3. The method according to claim 1, characterized in that The step of extracting feature information from the collected data based on the preset generative AI big model to obtain feature data includes: Preprocessing the collected data to obtain preprocessed data, wherein the preprocessing includes one or more of denoising, standardization and format conversion; Determining an extraction method according to the data type of the preprocessed data; Based on the preset generative AI big model, feature extraction is performed on the corresponding pre-processed data to obtain preliminary extracted data; The initially extracted data are fused to obtain the feature data.

4. The method according to claim 1, characterized in that The step of constructing a multimodal index structure based on the feature data comprises: Classifying and sorting the characteristic data to obtain classified characteristic data; Based on the classification feature data, construct a text inverted index, a vector index and a tree index; Merging the text inverted index, the vector index and the tree index to obtain a fused index; The fusion index is compressed and optimized to obtain the multimodal index structure.

5. The method according to claim 1, characterized in that The step of matching the search instruction with the index data in the multimodal index structure comprises: Parsing the search instruction and generating a query task based on the parsing result; Decomposing the query task to obtain sub-query tasks; Determine the index mapping of the sub-query task according to a preset matching strategy and the index type of the sub-query task; Based on the index mapping, the sub-query task is matched with the index data in the multimodal index structure.

6. The method according to any one of claims 1 to 5, characterized in that The step of fusing and sorting the matching results to obtain target search results includes: Determining a fusion strategy according to the type of the search instruction and preset user history data; Based on the fusion strategy, evaluating the matching result; Sorting the matching results based on the evaluation results to obtain ordered results; The ordered results are deduplicated to obtain the target search results.

7. A multimodal data search device based on a generative AI large model, characterized in that: The device comprises: A collection module, used to collect various types of data from multiple heterogeneous data sources to obtain collected data; A feature extraction module is used to extract feature information from the collected data based on a preset generative AI large model to obtain feature data; A multimodal indexing module, used to construct a multimodal indexing structure based on the feature data; A matching module, used for matching the search instruction with the index data in the multimodal index structure; The target module is used to merge and sort the matching results to obtain the target search results.

8. A computer device, characterized in that: The device includes: a memory, a processor, and a multimodal data search program based on a generative AI big model stored in the memory and executable on the processor, wherein the multimodal data search program based on a generative AI big model is configured to implement the steps of a multimodal data search method based on a generative AI big model as described in any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium stores a multimodal data search program based on a generative AI big model. When the multimodal data search program based on a generative AI big model is executed by a processor, the steps of the multimodal data search method based on a generative AI big model as described in any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product includes a computer program, which, when executed by a processor, implements the steps of the multimodal data search method based on a generative AI big model as described in any one of claims 1 to 6.

Images