Data processing method and device, electronic equipment, storage medium and program product

Abstract

The present disclosure provides a data processing method and device, electronic equipment, storage medium and program product. The method comprises: obtaining a data query request, the data query request comprising query data of at least one type of modality; performing feature extraction on the query data of at least one type of modality to obtain at least one query vector; matching the at least one query vector with data vectors of the corresponding modality type in a plurality of vector groups pre-stored in a first database to obtain at least one candidate vector group; based on the at least one candidate vector group, obtaining at least one expanded vector group associated with the at least one candidate vector group according to at least one of a first association relationship and a second association relationship; and generating a data query result corresponding to the data query request according to at least one of the at least one candidate vector group and the at least one expanded vector group. The present disclosure can realize the associated retrieval of multi-modal data and provide users with deeper retrieval results.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] Large Language Models (LLMs) can achieve strong generalization capabilities on cloud servers with hundreds of billions of parameters. However, in low-computing-power scenarios such as industrial equipment and mobile terminals, the model parameters usually need to be compressed to a smaller scale, such as less than 100 million parameters. When such models are not sufficiently trained in a domain or when reasoning about long-tail entities such as specific technical terms, they are prone to knowledge illusions, leading to a decrease in the accuracy of their answers.

[0003] Retrieval-Augmented Generation (RAG) technology is an effective means to alleviate the above problems, which introduces external knowledge bases to assist model generation. However, existing RAG technologies have the following limitations: mainstream RAG frameworks such as LangChain and Llamalndex mainly rely on text embedding for retrieval, which makes it difficult to effectively process and associate multimodal data such as images, voiceprints, and videos, resulting in insufficient information utilization when faced with complex queries of multimodal data; traditional vector databases usually store data with a single vector corresponding to a single data point, which cannot effectively express and utilize the implicit semantic relationships and hierarchical structures between cross-modal data, resulting in a lack of depth and relevance in retrieval results. Summary of the Invention

[0004] In view of this, the purpose of this disclosure is to provide a data processing method, apparatus, electronic device, storage medium and program product.

[0005] For the purposes described above, the first aspect of this disclosure provides a data processing method, comprising:

[0006] Obtain a data query request, wherein the data query request includes query data of at least one modal type; Feature extraction is performed on the query data of the at least one type of modality to obtain at least one corresponding query vector; The at least one query vector is matched with data vectors of corresponding modal types in a plurality of vector groups pre-stored in the first database to obtain at least one candidate vector group; wherein, each vector group includes a plurality of data vectors, and the first database is used to store the plurality of vector groups, a first association relationship between the plurality of vector groups, and a second association relationship between the plurality of data vectors in each vector group; Based on the at least one candidate vector group, at least one extended vector group associated with the at least one candidate vector group is obtained according to at least one of the first association relationship and the second association relationship; Based on at least one of the at least one candidate vector group and the at least one extended vector group, generate a data query result corresponding to the data query request.

[0007] In some embodiments, the at least one modality type includes a first modality type and a second modality type, and the at least one query vector includes a first query vector corresponding to the first modality type and a second query vector corresponding to the second modality type; The step of matching the at least one query vector with data vectors of corresponding modal types from multiple vector groups pre-stored in the first database to obtain at least one candidate vector group includes: The first query vector is matched with the data vectors of the corresponding modality type in the plurality of vector groups to obtain at least one first vector group with the highest similarity. Based on the at least one first vector group, the second query vector is matched with the data vectors of the corresponding modality type in the plurality of vector groups to obtain at least one second vector group with the highest similarity. The at least one candidate vector group is obtained based on the at least one first vector group and the at least one second vector group.

[0008] In some embodiments, the at least one modality type includes a third modality type and a fourth modality type, and the at least one query vector includes a third query vector corresponding to the third modality type and a fourth query vector corresponding to the fourth modality type; The step of matching the at least one query vector with data vectors of corresponding modal types from multiple vector groups pre-stored in the first database to obtain at least one candidate vector group includes: The third query vector is matched with the data vectors of the corresponding modality type in the plurality of vector groups to obtain at least one third vector group with the highest similarity. The fourth query vector is matched with the data vectors of the at least one third vector group to obtain at least one fourth vector group with the highest similarity. The at least one candidate vector group is obtained based on the at least one fourth vector group.

[0009] In some embodiments, matching the at least one query vector with data vectors of corresponding modality types from a plurality of vector groups pre-stored in a first database to obtain at least one candidate vector group includes: Calculate the similarity between the at least one query vector and the data vectors of the corresponding modality type in the plurality of vector groups; Obtain the weight that matches the modality type in the data query request; The fusion similarity is determined based on the similarity, the weight, and the preset harmonic weighting formula; The at least one vector group with the highest fusion similarity is determined as the at least one candidate vector group.

[0010] In some embodiments, the method further includes: Acquire multimodal data to be stored, wherein the multimodal data contains data units of at least two different modal types; Feature extraction is performed on the data units of the at least two different modal types to obtain their respective corresponding data vectors; Associate the data vectors corresponding to the data units of the at least two different modal types to obtain at least one fifth vector group; Establish the first association relationship between the at least one fifth vector group and the plurality of vector groups pre-stored in the first database, and store the at least one fifth vector group in the first database.

[0011] In some embodiments, multiple data vectors of the same vector group form multiple semantic clusters, each semantic cluster being used to represent an abstract concept hierarchy; the multiple semantic clusters form a tree structure.

[0012] In some embodiments, the plurality of semantic clusters are stored using an index structure based on a hierarchical navigation graph, and the plurality of data vectors are subspaced using vector segmentation quantization compression technology.

[0013] In some embodiments, matching the at least one query vector with data vectors of corresponding modality types from a plurality of vector groups pre-stored in a first database to obtain at least one candidate vector group includes: An approximate nearest neighbor search is performed on the semantic cluster that is most relevant to the at least one query vector to obtain the at least one candidate vector group.

[0014] In some embodiments, the plurality of vector groups includes a fifth vector group and a sixth vector group; The first association relationship includes the adjacency relationship between the fifth vector group and the sixth vector group, and the first parent-child relationship between any of the semantic clusters of the fifth vector group and the sixth vector group; The second association relationship includes a second parent-child relationship between the plurality of semantic clusters.

[0015] In some embodiments, obtaining at least one extended vector group associated with the at least one candidate vector group based on at least one of the first association relationship and the second association relationship includes at least one of the following: Based on the adjacency relationship, obtain at least one extended vector group that is adjacent to the at least one candidate vector group; Based on the second parent-child relationship, at least one of the parent semantic cluster and child semantic cluster of the first semantic cluster in the at least one candidate vector group is obtained. Based on the first parent-child relationship, the parent vector node of the parent semantic cluster and the child semantic cluster is obtained. The vector group to which the parent vector node belongs is determined as the at least one extended vector group.

[0016] A second aspect of this disclosure provides a data processing method, comprising: Obtain question and answer information, and generate a data query request based on the question and answer information; Obtain the data query results corresponding to the data query request; Input the question-and-answer information and the data query results into the first model; Based on the output information of the first model, generate response information corresponding to the question and answer information.

[0017] In some embodiments, the data query results are obtained based on the data processing method described in the first aspect.

[0018] A third aspect of this disclosure provides a data processing apparatus, comprising: The acquisition module is configured to: acquire a data query request, wherein the data query request includes query data of at least one modal type; The feature extraction module is configured to: extract features from the query data of the at least one type of modality to obtain at least one corresponding query vector; The matching module is configured to: match the at least one query vector with data vectors of corresponding modal types in a plurality of vector groups pre-stored in a first database to obtain at least one candidate vector group; wherein each vector group includes a plurality of data vectors, and the first database is used to store the plurality of vector groups, a first association relationship between the plurality of vector groups, and a second association relationship between the plurality of data vectors in each vector group; The association acquisition module is configured to: based on the at least one candidate vector group, and according to at least one of the first association relationship and the second association relationship, acquire at least one extended vector group associated with the at least one candidate vector group; The generation module is configured to generate a data query result corresponding to the data query request based on at least one of the at least one candidate vector group and the at least one extended vector group.

[0019] A fourth aspect of this disclosure provides a data processing apparatus, comprising: The first acquisition module is configured to: acquire question and answer information, and generate a data query request based on the question and answer information; The second acquisition module is configured to: acquire the data query result corresponding to the data query request; The input module is configured to input the question-and-answer information and the data query results into the first model; The generation module is configured to generate response information corresponding to the question-and-answer information based on the output information of the first model.

[0020] In some embodiments, the data query results are obtained based on the data processing method described in the first aspect.

[0021] The fifth aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described in the first or second aspect.

[0022] A sixth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method described in the first or second aspect.

[0023] The seventh aspect of this disclosure provides a computer program product including computer program instructions that, when executed on a computer, cause the computer to perform the method as described in the first or second aspect.

[0024] As can be seen from the above, the data processing method, apparatus, electronic device, storage medium, and program product provided in this disclosure store data of multiple different modal types of data as vector groups. A first association relationship characterizes the association between different vector groups, and a second association relationship characterizes the association between multiple data vectors within the same vector group. This allows for the association of vector groups of different modalities during multimodal data queries, achieving a comprehensive understanding and semantic expression of multimodal data composite queries. Simultaneously, during retrieval, the first association between different vector groups and the second association within the same vector group enable related retrieval, creating a divergent thinking effect and achieving the ability to apply knowledge to new situations, providing users with deeper, richer, and more intelligent retrieval results. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in this disclosure or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic diagram of an exemplary scenario provided by an embodiment of this disclosure is shown.

[0027] Figure 2 A flowchart illustrating an exemplary method provided by an embodiment of this disclosure is shown.

[0028] Figure 3 A schematic diagram of an exemplary data structure according to an embodiment of the present disclosure is shown.

[0029] Figure 4 A schematic diagram of an exemplary data structure according to an embodiment of the present disclosure is shown.

[0030] Figure 5 A schematic diagram of an exemplary data structure provided by an embodiment of this disclosure is shown.

[0031] Figure 6 A schematic diagram of an exemplary data structure provided by an embodiment of this disclosure is shown.

[0032] Figure 7 A schematic diagram of an exemplary apparatus provided by an embodiment of the present disclosure is shown.

[0033] Figure 8 A schematic diagram of an exemplary apparatus provided by an embodiment of the present disclosure is shown.

[0034] Figure 9 A schematic diagram of the hardware structure of an exemplary computer device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0035] It is understood that before using the technical solutions of the various embodiments in this disclosure, users will be informed of the type, scope of use, and usage scenarios of the personal information involved in an appropriate manner, and user authorization will be obtained.

[0036] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media performing the operations of this disclosed technical solution.

[0037] As an optional but not limited implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0038] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0039] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0040] To make the objectives, technical solutions, and advantages of this disclosure clearer, the principles and spirit of this disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0041] It is important to understand in this article that any number of elements in the accompanying figures is for illustrative purposes and not for limitation, and that any naming is for distinction only and has no limiting meaning.

[0042] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar words used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly. The article "a" or "an" preceding an element does not exclude the existence of multiple such elements.

[0043] The principles and spirit of this disclosure will be explained in detail below with reference to several representative embodiments.

[0044] Figure 1 A schematic diagram of an exemplary system 100 provided in an embodiment of this disclosure is shown.

[0045] like Figure 1 As shown, system 100 can be used to implement data processing functions and may include terminal devices 102A and 102B, server 106, and database server 108. A medium (e.g., a network) may be included to provide a communication link between terminal devices 102A and 102B and server 106 and database server 108. This network may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0046] The terminal devices 102A and 102B can be equipped with various applications (APPs) or software, such as project management applications or software, collaborative office applications or software, image processing applications or software, video conferencing applications or software, reading applications or software, video applications or software, social applications or software, payment applications or software, web browsers, and instant messaging tools. In some embodiments, these applications or software can be used for data processing.

[0047] The terminal devices 102A and 102B here can be either hardware or software. When terminal devices 102A and 102B are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, e-book readers, MP3 players, laptops, and desktop computers (PCs). When terminal devices 102A and 102B are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., to provide distributed services) or as a single software program or software module. No specific limitations are set here.

[0048] Server 106 can be a server that provides various services, such as a backend server that supports various applications displayed on terminal devices 102A and 102B. Database server 108 can also be a database server that provides various services. It is understood that if server 106 can implement the relevant functions of database server 108, database server 108 may not need to be set up in system 100.

[0049] The server 106 and database server 108 here can be either hardware or software. When they are hardware, they can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When they are software, they can be implemented as multiple software programs or software modules (for example, to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0050] It should be noted that the information processing method provided in this embodiment can be executed by server 106. It should be understood that... Figure 1 The number of terminal devices, users, servers, and database servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, users, servers, and database servers.

[0051] Terminal device 102A can be equipped with an application or software that includes data processing functions, and user 104A can initiate data query requests through this application or software. Terminal device 102B can provide a management page for this application or software, and user 104B can manage the application or software through the management page.

[0052] The following is combined with Figure 1 The above application scenarios are used to describe the methods according to exemplary embodiments of this disclosure. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of this disclosure, and the embodiments of this disclosure are not limited in any way. Rather, the embodiments of this disclosure can be applied to any applicable scenario.

[0053] like Figure 2 As shown in the embodiments of this disclosure, a data processing method is provided, including: Step S101: Obtain a data query request, wherein the data query request includes query data of at least one modality type.

[0054] In this embodiment, the data query request includes query data of at least one modal type. The modal type may include text, images, audio, video, etc., but this embodiment does not limit this.

[0055] For example, in industrial display defect detection applications, based on screenshots of production line quality inspections and production data retrieval, the location and cause of defects can be queried or estimated; in medical diagnostic applications, retrieval is performed by combining patient complaint texts with CT images to achieve multi-dimensional pathological correlation analysis; in industrial equipment predictive maintenance applications, fault identification is performed based on a hybrid retrieval of vibration soundprint features and log text; in legal knowledge graph applications, retrieval is performed based on case precedents and legal provisions to achieve case recommendation and legal provision causal reasoning; in artificial intelligence operating systems (AIOS), a dataset generated by reinforcement learning is produced based on system logs, screenshots, and system sounds for querying or estimating the user's desired actions and possible subsequent actions, etc.

[0056] Step S103: Extract features from the query data of the at least one modality to obtain at least one corresponding query vector.

[0057] In this embodiment, after obtaining a data query request, feature extraction can be performed on the query data of each modality in the data query request to obtain at least one query vector corresponding to the query data of at least one modality. In this embodiment, for query data of different modalities, feature extraction can be performed based on different feature extraction models to obtain the corresponding query vectors.

[0058] Step S105: Match the at least one query vector with data vectors of corresponding modal types in a plurality of vector groups pre-stored in the first database to obtain at least one candidate vector group; wherein, each vector group includes a plurality of data vectors, and the first database is used to store the plurality of vector groups, the first association relationship between the plurality of vector groups, and the second association relationship between the plurality of data vectors in each vector group.

[0059] In this embodiment, the first database can be a preset knowledge base, which stores knowledge data of various modalities. Each piece of knowledge data is characterized from multiple dimensions to form multiple data vectors. Multiple data vectors form a vector group, and each vector group stores only one modal type of data vector.

[0060] There is a first association relationship between each vector group. This first association relationship can be, for example, a semantic association relationship, such as the association relationship between the vector group corresponding to diabetes treatment and the vector group corresponding to metabolic disease management. The vector group corresponding to diabetes treatment and the vector group corresponding to metabolic disease management can be used to represent knowledge data of the same modality type or to represent knowledge data of different modalities. The first association relationship can also be a structural association relationship generated based on pre-stored knowledge data, such as the association relationship between multiple vector groups generated based on text data in the same report, or the vector group generated based on text data in the same report and the vector group generated based on the image in the report. This embodiment does not limit this.

[0061] There may be a second relationship between multiple data vectors in the same vector group, such as parent-child relationship, adjacent relationship, etc. This embodiment does not limit this.

[0062] In this embodiment, when a query vector corresponding to a data query request is obtained, the query vector is matched with data vectors of the corresponding modal type in multiple vector groups pre-stored in the first database. For example, if the modal type of the query vector is text, the query vector is only matched with the vector group of the modal type of text in the first database, and is not matched with the vector group of the modal type of image, video, audio, etc. in the first database, so as to obtain at least one candidate vector group based on the matching result.

[0063] In some embodiments, at least one candidate vector group is at least one vector group in the first database that has the highest matching degree with the query vector, or at least one candidate vector group may be other vector groups in the first database, and this embodiment does not limit this.

[0064] Step S107: Based on the at least one candidate vector group, obtain at least one extended vector group associated with the at least one candidate vector group according to at least one of the first association relationship and the second association relationship.

[0065] In this embodiment, starting from at least one candidate vector group, a search is performed based on at least one of the first association relationship and the second association relationship to obtain at least one extended vector group associated with at least one candidate vector group.

[0066] For example, the candidate vector group could be the vector group corresponding to diabetes treatment, and the extended vector group could be the vector group corresponding to metabolic disease management. By retrieving diabetes treatment data, we can further extend to metabolic disease management data, thereby creating a divergent thinking effect and achieving the effect of learning by analogy when retrieving certain knowledge.

[0067] For example, the candidate vector group can be a vector group of the text modality of a certain knowledge data, and the extended vector group can be a vector group of the image modality of the knowledge data. Then, through this retrieval, data of multiple modal types of the knowledge data can be obtained.

[0068] Step S109: Generate a data query result corresponding to the data query request based on at least one of the at least one candidate vector group and the at least one extended vector group.

[0069] In this embodiment, at least one of at least one candidate vector group and at least one extended vector group can be used as the data query result corresponding to the data query request.

[0070] In this embodiment, by storing multiple different modal types of data as vector groups, and using a first association relationship to represent the association between different vector groups and a second association relationship to represent the association between multiple data vectors within the same vector group, different modal vector groups can be associated when performing multimodal data queries, achieving a comprehensive understanding and semantic expression of multimodal data composite queries. At the same time, during retrieval, associative retrieval can be achieved through the first association relationship between different vector groups and the second association relationship within the same vector group, thereby forming a divergent thinking effect and achieving the effect of applying knowledge to other situations, providing users with deeper and richer intelligent retrieval results.

[0071] In some embodiments, the at least one modality type includes a first modality type and a second modality type, and the at least one query vector includes a first query vector corresponding to the first modality type and a second query vector corresponding to the second modality type.

[0072] Then, step S105, which involves matching the at least one query vector with data vectors of corresponding modal types from multiple vector groups pre-stored in the first database to obtain at least one candidate vector group, includes: Step S201: Match the first query vector with the data vectors of the corresponding modality type in the plurality of vector groups to obtain at least one first vector group with the highest similarity.

[0073] Step S203: Based on the at least one first vector group, the second query vector is matched with the data vectors of the corresponding modality type in the plurality of vector groups to obtain at least one second vector group with the highest similarity.

[0074] Step S205: Obtain the at least one candidate vector group based on the at least one first vector group and the at least one second vector group.

[0075] In this embodiment, when the data query request includes query data of the first modality type and query data of the second modality type, the query vector generated in step S103 also includes the query vector corresponding to the first modality type, namely the first query vector, and the query vector corresponding to the second modality type, namely the second query vector.

[0076] In this embodiment, the first query vector is first matched with data vectors belonging to the first modality type in multiple vector groups to obtain at least one top_k1 most similar first vector groups, i.e., RAG knowledge, thus achieving preliminary knowledge generalization. Then, for each piece of RAG knowledge, i.e., each first vector group, a second modality type matching is performed. That is, based on at least one first vector group, the second query vector is matched with data vectors belonging to the second modality type in multiple vector groups to obtain at least one second vector group with the highest similarity. For example, for a PDF document containing images, image matching can be further performed; for PPT / WPS files with audio / video content, audio / video vector similarity matching can be performed. In this embodiment, data of each modality type can be matched according to a preset order. This process is similar to a left join in database operations, i.e., based on a primary query dimension, it is then expanded layer by layer to other modalities to achieve multimodal fusion.

[0077] In some embodiments, the matching process is not limited to static matching; it can also incorporate time-series features for dynamic association. These time-series features can be user behavior data. For example, in real-time knowledge recommendation scenarios, the matching strategy can be dynamically adjusted based on the user's recent interaction data to more accurately match the user's potential query needs.

[0078] In some embodiments, the at least one modality type includes a third modality type and a fourth modality type, and the at least one query vector includes a third query vector corresponding to the third modality type and a fourth query vector corresponding to the fourth modality type; Then, step S105, which involves matching the at least one query vector with data vectors of corresponding modal types from multiple vector groups pre-stored in the first database to obtain at least one candidate vector group, includes: Step S301: Match the third query vector with the data vectors of the corresponding modality type in the plurality of vector groups to obtain at least one third vector group with the highest similarity.

[0079] Step S303: Match the fourth query vector with the data vector of the at least one third vector group to obtain at least one fourth vector group with the highest similarity.

[0080] Step S305: Obtain the at least one candidate vector group based on the at least one fourth vector group.

[0081] In this embodiment, when the data query request includes query data of the third modality type and query data of the fourth modality type, the query vector generated in step S103 also includes the query vector corresponding to the third modality type, namely the third query vector, and the query vector corresponding to the fourth modality type, namely the fourth query vector.

[0082] In this embodiment, the third query vector is first matched with the data vectors of the vector groups belonging to the first modality type among multiple vector groups to obtain at least one of the top_k1 most similar third vector groups, thereby narrowing the scope. Then, the fourth query vector is matched with the data vectors of at least one third vector group to obtain at least one fourth vector group with the highest similarity, achieving fine matching. For example, by first narrowing the scope through text retrieval and then performing fine matching of images or audio on the top_k1 most similar at least one third vector group, the retrieval accuracy can be improved.

[0083] In some embodiments, step S105, which involves matching the at least one query vector with data vectors of corresponding modality types from multiple vector groups pre-stored in the first database to obtain at least one candidate vector group, includes: Step S401: Calculate the similarity between the at least one query vector and the data vectors of the corresponding modality type in the plurality of vector groups.

[0084] Step S403: Obtain the weight that matches the modality type in the data query request.

[0085] Step S405: Determine the fusion similarity based on the similarity, the weight, and the preset harmonic weighting formula.

[0086] Step S407: Determine at least one vector group with the highest fusion similarity as the at least one candidate vector group.

[0087] In some embodiments, when a data query request involves multiple modalities, these modalities may not be ordered sequentially but rather influence the final similarity calculation in parallel. Taking a data query request involving three modalities—text, image, and audio—as an example, a multidimensional matrix can be obtained, such as TEXT_EMBED×WAV_EMBED×VISION_EMBED, abbreviated as TE×WE×VE. Here, TEXT_EMBED and TE represent the vector group for the text modality, WAV_EMBED and WE represent the vector group for the audio modality, and VISION and VE represent the vector group for the image modality.

[0088] In this embodiment, weights are used to balance the contributions of different modalities. Let the weight of TE be Wt, the weight of WE be Ww, and the weight of VE be Wv. Let the similarity of TE be St, the similarity of WE be Sw, and the similarity of VE be Sv. Then, the fusion similarity S(T,W,V) is calculated using a preset harmonic weighting formula: S(T,W,V)=(Wt+Ww+Wv) / ((Wt / St+ +θ)+(Ww / Sw+ +θ)+(Wv / Sv+ +θ)); in, Indicates the stabilizing factor. Typically, it is 1e-6, used to prevent the denominator factor from excessively approaching 0 and causing computational anomalies; θ represents the reinforcement learning reward parameter, which can be adjusted through the reward function during the reinforcement learning process. Make dynamic adjustments, such as e←e+η Reward / e, where η is the learning rate and Reward is the performance metric of the fusion result. In this embodiment, the stabilization factor can be dynamically adjusted using a preset reinforcement learning model.

[0089] In some embodiments, weights can be optimized based on the knowledge attributes and data relationships of the knowledge base. In some embodiments, gradient descent can be used to train the database data by querying the dataset to obtain the optimal weight configuration.

[0090] In some embodiments, a uniform distribution strategy can also be used to configure the weights, that is, Wt+Ww+Wv=1 and make each weight equal.

[0091] This process is similar to a full join in database operations. The full join approach is suitable for multimodal data fusion, such as cross-modal retrieval and semantically enhanced recommendation applications. For example, in a medical imaging diagnostic system, multiple sources of information, including medical record text (TE), CT / MRI images (VE), and doctor's voice reports (WE), can be considered simultaneously, combined with weight optimization strategies, to generate more accurate diagnostic suggestions.

[0092] In some embodiments, the method further includes: Step S501: Obtain the multimodal data to be stored, wherein the multimodal data contains data units of at least two different modal types.

[0093] In this embodiment, multimodal data can be stored in a first database to form a knowledge base for the Retrieval Enhanced Generation (RAG) system. The multimodal data includes data units of at least two different modalities, such as PDF files containing both text and images, or PPT files including videos.

[0094] Step S503: Perform feature extraction on the data units of the at least two different modal types to obtain their respective corresponding data vectors.

[0095] In this embodiment, when constructing the knowledge base of the retrieval enhancement generation system, each piece of knowledge data stored in the retrieval enhancement generation table (RagTable) of the retrieval enhancement data generation system can be characterized from multiple dimensions to obtain multiple data vectors. These data vectors will become the basis for subsequent semantic retrieval and knowledge fusion.

[0096] Different feature extraction techniques are needed for knowledge carriers of different modalities, and these techniques should be optimized to improve retrieval accuracy and generalization ability.

[0097] Text-based features: For knowledge data described in pure text, we can perform deep semantic encoding using Natural Language Processing (NLP) techniques. First, the text is segmented into subword units using a text processing model's tokenizer. Then, an encoder performs context modeling, and the output or mean pooling generates semantic vectors, resulting in data vectors corresponding to the text modality data units. The text processing model can include, for example, BERT, RoBERTa, or T5; the tokenizer can be WordPiece or SentencePiece; and the encoder can be a multi-layer Transformer encoder, such as a 12-24 layer BERT or DeBERTa architecture. The Transformer architecture's multi-head attention mechanism (typically 8-16 heads) can capture semantic associations at different positions and, combined with positional encoding, enhances sequence information. Finally, the output labeled with [CLS] or mean pooling generates semantic vectors of 768 to 1024 dimensions. This encoding method preserves fine-grained semantics, long-range dependencies, and cross-sentence contextual information. Furthermore, for large-scale corpora, Faiss or Annoy can be used for vectorized indexing to accelerate searches.

[0098] Image-based features: For knowledge data containing visual information, feature extraction can be performed using contrastive learning-based multimodal models or traditional CNN models. Contrast-based multimodal models, such as CLIP's ViT (Vision Transformer) architecture, segment the image into 16x16 regions (patch), for example, generating 196 patches for a 224x224 image. After linear projection and positional encoding, global visual features are extracted through a 12-layer Transformer encoder. CLIP, for instance, uses 400 million image-text pairs for contrastive learning training, enabling image and text features to share the same semantic space, thus achieving better generalization capabilities in open-domain retrieval tasks. Traditional CNN models, such as ResNet-152 or EfficientNet, progressively extract local to global features through stacked convolutional layers with residual connections, finally generating a 2048-dimensional vector using global average pooling (GAP). Traditional CNN models still maintain high computational efficiency and stability in single-modal tasks (such as image classification and object detection). Furthermore, in large-scale image data retrieval, IVF-PQ indexes can be used to quantize and compress high-dimensional image feature vectors, thereby improving retrieval speed and reducing storage costs.

[0099] Audio features: Feature extraction from audio data can begin with extracting classic acoustic features such as Mel-frequency cepstral coefficients (MFCC, typically 13-26 dimensions), zero-crossing rate, spectral centroid, and chromatogram features, and then combining these with Hidden Markov Models (HMMs) or Gaussian Mixture Models (GMMs) for matching. Alternatively, deep learning models such as wav2vec 2.0 can be used to encode the original waveform using a convolutional neural network (CNN), then a Transformer can be used to model temporal relationships, and self-supervised pre-training can be performed using contrastive predictive coding (CPC) to generate a 768-dimensional vector per 25ms frame at a 16kHz sampling rate. Furthermore, for audio retrieval tasks in low-resource language or noisy environments, models such as HuBERT and Whisper can be combined to improve retrieval robustness through multi-level modeling.

[0100] Video Feature Extraction: Video feature extraction requires simultaneous processing of spatiotemporal information, thus it can be optimized by combining various methods. For example, 3D-CNN methods such as C3D and I3D can be used, employing 3x3x3 convolutional kernels sliding along the spatiotemporal dimension, extracting behavioral features through 5 convolutional layers and 5 pooling layers. This method is suitable for short video understanding, such as action recognition. Alternatively, TSN (Temporal Segment Network) can be used to first extract single-frame features through 2D CNN, and then use LSTM / Transformer to model temporal information, thereby extracting global features over a longer time scale. Or, ViViT (Video VisionTransformer) can be used to separate spatiotemporal attention, achieving state-of-the-art performance on datasets such as Kinetics. This method models spatiotemporal information through a Spatio-Temporal module, providing higher semantic understanding capabilities in large-scale video data retrieval. Furthermore, in efficient video retrieval, FAISS indexing and HNSW nearest neighbor search can be combined to improve the retrieval speed of large-scale video vector libraries by three orders of magnitude.

[0101] Custom Features: When domain knowledge requires specific representation, domain datasets can be collected and constructed, such as a dataset of millions of agricultural knowledge graph-text pairs, and joint modeling can be performed using different modalities. For example, a cross-modal alignment mechanism can be established between multimodal data such as speech, images, and text through a cascaded network structure of CNN+Transformer. During model training, multiple loss functions (such as classification, retrieval, and summarization generation) can be introduced through a multi-task learning framework to jointly optimize and improve feature representation capabilities. Alternatively, on the basis of a pre-trained large model, lightweight methods such as low-rank adaptation (LoRA) can be used to efficiently adjust parameters for specific tasks, thereby achieving excellent retrieval results with limited computing resources. At the same time, for specific domains, such as financial, medical, or legal documents, domain-specific word vectors (such as FinBERT and ClinicalBERT) and customized ANN indexes (such as IVF-HNSW) can be combined to improve retrieval precision and recall.

[0102] Step S505: Associate the data vectors corresponding to the data units of the at least two different modal types to obtain at least one fifth vector group.

[0103] In this embodiment, data vectors corresponding to data units of at least two different modal types can be grouped and classified, and then associated based on the grouping and classification results to obtain a fifth vector group.

[0104] like Figure 3As shown, taking the TXT_EMBED node as an example, by constructing interdependent relationships between nodes TXT_EMB_CLS1-7, nodes TXT_EMB_CLS1-7 form a fifth vector group. This interdependent relationship allows for knowledge generalization and divergent thinking, enabling the full utilization of data relationships to obtain relevant retrieved data during multimodal knowledge storage and retrieval. Simultaneously, relationships are constructed between the TXT_EMBED, VIT_EMBED, and WAV_EMBED nodes, forming fifth vector groups corresponding to each node. Each vector group can be stored using a dendritic table structure.

[0105] Step S507: Construct the first association relationship between the at least one fifth vector group and the plurality of vector groups pre-stored in the first database, and store the at least one fifth vector group in the first database.

[0106] In this embodiment, a first association relationship is established between the fifth vector group and multiple vector groups pre-stored in the first database, and the fifth vector group is stored in the first database based on the first association relationship. The first association relationship can be established based on the similarity between the fifth vector group and the multiple vector groups pre-stored in the first database. For example, the first association relationship is established when the similarity meets a preset threshold; otherwise, it is not established. This embodiment does not limit this approach.

[0107] In this embodiment, by generalizing and divergent thinking through the first association knowledge, the correlation between data can be fully utilized to obtain associated retrieval data during the process of multimodal knowledge storage and retrieval.

[0108] In some embodiments, multiple data vectors of the same vector group form multiple semantic clusters, each semantic cluster being used to represent an abstract concept hierarchy; the multiple semantic clusters form a tree structure.

[0109] like Figure 3As shown, for the vector group TXT_EMBED, this vector group is divided into 7 interconnected semantic clusters, namely TXT_EMB_CLS1-7. Each semantic cluster represents an abstract concept level. For example: TXT_EMB_CLS1, basic entities, such as mobile phones and computers; TXT_EMB_CLS3, functional attributes, such as battery life and screen resolution; TXT_EMB_CLS7, application scenarios, such as mobile office and gaming. The dependencies between semantic clusters are modeled using a hypergraph, and the edge weights represent the strength of the association between abstract concept levels. This database is a variant of a multimodal preprocessing pipeline, whose functions include: text classification, integrating a zero-shot classification model to achieve unsupervised coarse classification; image feature extraction, generating 2048-dimensional visual vectors based on ResNet-152, and reducing the dimensionality to 128 dimensions using PCA; and voiceprint encoding, using the Wav2Vec 2.0 model to generate 256-dimensional MFCC features, and generating a 64-bit fingerprint using SimHash.

[0110] In some embodiments, inverted index (IVF) technology is used to initially partition the data, dividing the entire dataset into multiple semantic clusters. This way, during queries, searches are performed only within the most relevant semantic clusters, effectively reducing unnecessary computation by more than 90%.

[0111] In some embodiments, the plurality of vector groups includes a sixth vector group and a seventh vector group; the first association relationship includes the adjacency relationship between the sixth vector group and the seventh vector group, and the first parent-child relationship between any of the semantic clusters of the sixth vector group and the seventh vector group; the second association relationship includes the second parent-child relationship between the plurality of semantic clusters.

[0112] like Figure 4 As shown, the vector group TXT_EMBED in RagTable1 and the vector group TXT_EMBED in RagTable2 are adjacent to each other because they are sub-vector groups of RagGroup.

[0113] like Figure 5 As shown, for the vector group TXT_EMBED in the enhanced generation table 1 (RagTable1) and the vector group TXT_EMBED in the enhanced generation table 2 (RagTable2), the vector group TXT_EMBED in the enhanced generation table 1 (RagTable1) is the parent vector group of TXT_EMB_CLS7 of the vector group TXT_EMBED in the enhanced generation table 2 (RagTable2), and there is a first parent-child relationship between the two.

[0114] like Figure 3 , Figure 4 , Figure 5 As shown, there is a second parent-child relationship between TXT_EMB_CLS1-7 within the vector group TXT_EMBED.

[0115] The step S107, which involves obtaining at least one extended vector group associated with the at least one candidate vector group based on at least one of the first association relationship and the second association relationship, includes: obtaining the at least one extended vector group adjacent to the at least one candidate vector group based on the adjacency relationship.

[0116] In this embodiment, when performing association retrieval, extended vector groups adjacent to the candidate vector group can be obtained based on adjacency relationships, for example: Figure 4 As shown, when the candidate vector group is the vector group TXT_EMBED in the enhanced retrieval table 1 (RagTable1), the vector group TXT_EMBED in the enhanced retrieval table 2 (RagTable2) can be obtained as the extended vector group, thereby achieving knowledge generalization. For example, the query is "diabetes treatment" → "metabolic disease management" (L1) → "chronic disease prevention system" (L2).

[0117] In some embodiments, for inductive divergent or deep thinking, when performing reverse RAG association, the similarity needs to be multiplied by a decay factor, or the divergent thinking similarity threshold needs to be increased, thereby gradually reducing the scale of divergent data as a pruning strategy to prevent the search space from becoming too large and affecting retrieval efficiency.

[0118] Step S107, which involves obtaining at least one extended vector group associated with the at least one candidate vector group based on at least one of the first association relationship and the second association relationship, further includes: obtaining at least one of the parent semantic cluster and child semantic cluster of the first semantic cluster in the at least one candidate vector group based on the second parent-child relationship; obtaining the parent vector node of the parent semantic cluster and the child semantic cluster based on the first parent-child relationship; and determining the vector group to which the parent vector node belongs as the at least one extended vector group. The first semantic cluster is any one of the multiple semantic clusters in the candidate vector group.

[0119] In this embodiment, when performing association retrieval, at least one of the parent semantic cluster and child semantic cluster of a certain semantic cluster in the candidate vector group can be obtained based on the second parent-child relationship. Then, the parent vector node of at least one of the parent semantic cluster and child semantic cluster can be obtained according to the first parent-child relationship, and the vector group to which the parent vector node belongs is determined as the at least one extended vector group.

[0120] like Figure 5As shown, when the candidate vector group is the vector group TXT_EMBED in the enhanced generation table 2 (RagTable2), the parent semantic cluster and / or child semantic cluster of its first semantic cluster can be obtained, such as TXT_EMB_CLS7. Then, based on the first parent-child relationship, the parent vector node of TXT_EMB_CLS7, namely the vector group TXT_EMBED in the enhanced generation table 1 (RagTable1), is obtained as the extended vector group, thereby realizing the generalization of knowledge.

[0121] In some embodiments, the divergent thinking path can be personalized by combining domain knowledge or user behavior data. For example, the weight of divergent thinking can be dynamically adjusted by combining users' query history, preferences, or access records to make the recommended content more targeted. In addition, knowledge graph path optimization strategies can be introduced, such as prioritizing knowledge nodes with high relevance and high confidence to improve the accuracy of retrieval.

[0122] By processing the knowledge corresponding to other dimensions and dendritic nodes using the method described above, a large association tree can be obtained. If the data at the rel_k level (within the similarity threshold range) is selected, theoretically, a maximum of (rel_k-1) x top_k x N associated data points can be obtained, where N represents the number of dendritic tables. This data represents a general retrieval of the data, or knowledge induction. If we reverse the process and find the child data of the dendritic table, we achieve a deep thinking effect; by searching backwards to the RAG knowledge on the other side of the dendritic table, we obtain a deep thinking tree. In this embodiment, the number of deep thinking layers, rel_k, can be manually set to mine the knowledge that may be needed.

[0123] In some embodiments, a dynamic path selection mechanism can be employed to dynamically adjust the depth of rel_k by analyzing the user's query objectives. For example, in the medical field, knowledge related to initial diagnosis might delve as deep as rel_k=3, while for general encyclopedic knowledge, rel_k=2 might suffice. Furthermore, reinforcement learning strategies can be combined to allow the system to automatically optimize the search scope of deep thinking, balancing retrieval quality and computational cost.

[0124] For example, input: The patient's chief complaint is "persistent thirst and frequent urination"; Divergence path: Initial diagnosis: Diabetes mellitus (Top1); Delving deeper: Blood glucose testing methods → Insulin treatment plans → Lifestyle management → Prediction of related complications; In this process, combining different data sources, such as electronic medical records (EMR), medical research papers, and clinical guidelines, can improve the credibility of search results. Furthermore, the deep thinking process can be further optimized based on user feedback data to better meet users' actual needs, thereby improving the efficiency and accuracy of knowledge retrieval.

[0125] In some embodiments, the plurality of semantic clusters are stored using an index structure based on a hierarchical navigation graph, and the plurality of data vectors are subspaced using vector segmentation quantization compression technology.

[0126] In this embodiment, multiple semantic clusters are stored using an index structure based on a Hierarchical Navigable Small World (HNSW) architecture, and the high-dimensional data vector is subspaced using Product Quantization (PQ) technology. For example, a 768-dimensional vector is divided into 8 segments (96 dimensions each), and each segment is encoded with 256 centroids. This compresses the index memory footprint to 1 / N of the original size (N being the number of segments). At a data scale of billions, this optimization strategy reduces the index memory from 704GB (HNSW) to 114GB while maintaining a search latency consistently below 117ms.

[0127] HNSW is a nearest neighbor search algorithm based on a multi-level, small-scale navigation graph structure. In the HNSW index structure, the upper levels of data consist of a sparse vector distribution, while the data distribution becomes denser as you move down the hierarchy. The highest level of a newly inserted node is generated by an exponential decay probability model, specifically calculated as follows: l = floor ln(rand(0,1)) m_L Where m_L = 1 / ln(M), M represents the number of node links. For example, if rand(0,1) = 0.05, M = 16, m_L = mL = 1 / ln(16) ≈ 0.36, then l = ln(0.05) 0.36 = 3.0 =3l= ln(0.05) 0.36 = 3.0 =3, the node appears in layers 0 to 3, and when M=16, the probability of the node appearing is P(l)=e l / m_L (1 ), that is, P(3)≈e 3 / 0.36 (1 e 1 / 0.36)≈e 8.33 0.94≈0.0002 (i.e. 0.02% 0.02%), the probability of being in the 3rd layer is only 1 / 163≈0.02% 1 / 163≈0.02%, ensuring that only a very small number of cross-regional "navigation nodes" are retained in the higher layers.

[0128] In some embodiments, the step S105 of matching the at least one query vector with data vectors of corresponding modality types in a plurality of vector groups pre-stored in the first database to obtain at least one candidate vector group includes: performing an approximate nearest neighbor search on the semantic cluster that is most relevant to the at least one query vector to obtain the at least one candidate vector group.

[0129] In this embodiment, when performing a query, an Approximate Nearest Neighbor (ANN) search is performed only within the most relevant cluster, thereby effectively reducing more than 90% of invalid computations.

[0130] like Figure 6 As shown, this is an index structure based on a hierarchical navigable small world (HNSW) structure. Approximate nearest neighbor search (ANN) is a method specifically designed for fast retrieval of high-dimensional vectors. Essentially, it sacrifices some retrieval accuracy for higher search efficiency. As a type of nearest neighbor search algorithm, it reduces the computational cost of ANNs and improves search efficiency through spatial partitioning and quantization compression.

[0131] like Figure 6 As shown, the ANN index employs a multi-level search strategy, such as Layer2, Layer1, Layer0, etc. The higher the layer (e.g., Layer2), the sparser the data distribution, while the lower the layer (e.g., Layer0), the denser the data distribution. The query process typically starts from the upper sparse data layer, finding the node closest to the query vector through nearest neighbor search, and then gradually descending to lower-level denser data regions to achieve precise matching in a more refined search space. Through unidirectional, ordered, and acyclic layer-by-layer search, the top-K nearest neighbor vectors that are most similar or meet the threshold requirements are ultimately found. This hierarchical structure effectively reduces unnecessary computation, significantly improving search efficiency in large-scale data retrieval.

[0132] In this application, the top-K nearest neighbor vector data, i.e., candidate vector groups, can be obtained first through ANN query. Then, based on the candidate vector groups, an extended vector group associated with the candidate vector groups is obtained through multiple vector groups stored in the first database, the first association relationship between the multiple vector groups, and the second association relationship between the multiple data vectors of each vector group. Then, an ANN query is performed using the extended vector group as a benchmark, and the extended vector group is obtained according to the query results. After one or more rounds of query and knowledge generalization, a more accurate and intelligent query is achieved.

[0133] In this application, an inverted file index (IVF) and an ANN are combined for joint retrieval to meet the needs of searching larger corpora.

[0134] In some embodiments, the deletion operation of ANN indexes is complex, especially when deleting multidimensional data or data with parent-child sets, resulting in high costs for reconstructing ANN nearest neighbor relationships. Therefore, this embodiment can adopt a logical deletion (softdelete) approach, marking obsolete data with the is_deleted field and periodically triggering blue-green deployment for index reconstruction to ensure atomic updates of the database while keeping the switching latency within 50ms.

[0135] In some embodiments, when inserting new data, multi-threaded image, text, and voice data classification needs to be performed, and IVF and ANN indexes need to be built. In this embodiment, data insertion efficiency can also be improved by optimizing batch updates and asynchronous index building strategies.

[0136] In some embodiments, a pruned search strategy can be employed, where nodes undergo category filtering during each navigation search layer, making the HNSW navigation process more targeted. For example, if the database contains data across multiple categories, such as sparse vectors after image internal classification, sparse data after text classification, and data after speech classification, the HNSW index can perform specialized index navigation for a specific category of data, thereby improving search accuracy, reducing retrieval nodes, and increasing efficiency.

[0137] In some embodiments, structured metadata such as feature version and model ID are stored in an SQLite table, while unstructured vectors are stored in an ANN engine, with ACID transactions implemented through foreign key associations. Alternatively, custom field types can be used to store vectors via virtual tables or SQLite plugins. In some embodiments, SQLite's Write-Ahead Logging (WAL) mode is enabled for edge adaptation, supporting multi-threaded read and write operations and avoiding lock contention during feature insertion. Simultaneously, the SQLCipher module is integrated to implement AES-256 encryption of database files, meeting edge-side data security requirements.

[0138] In this embodiment, the relationship between SQLite and ANN is similar to that of an index structure: when querying data, the nearest neighbor search is first performed through ANN to obtain Top-K candidate results, and then the specific database record is retrieved through the SQLite index.

[0139] In some embodiments, a similar SQLite-Vec plugin technology can be used to add the ANN engine as an extension function to SQLite, enabling the database to directly perform vector-related searches, thereby enhancing SQLite's vector search capabilities.

[0140] In some embodiments, PQ quantization technology is used to convert high-dimensional vectors into 8-bit encoded data to achieve fast indexing and accelerate in-database search of high-dimensional data.

[0141] This case study employs an IVF-PQ composite index architecture for ANN optimization of massive datasets. It combines an inverted index (IVF) for coarse-grained filtering with product quantization (PQ) for fine-grained optimization. Furthermore, the multi-level structure and categorical branch pruning strategy of the HNSW index effectively improve search efficiency, while logical deletion marking and blue-green deployment mechanisms ensure the atomicity of index updates and a superior user experience. At the billion-level data scale, this optimization scheme significantly reduces index memory usage, maintains search latency at around 100ms, and simultaneously achieves efficient massive data retrieval capabilities.

[0142] Based on the same inventive concept, corresponding to any of the methods in the above embodiments, this disclosure also provides a data processing method, including: Step S601: Obtain question and answer information, and generate a data query request based on the question and answer information.

[0143] In this embodiment, the question-and-answer information input by the user to the first model can be obtained, the question-and-answer information can be processed, and a data query request for the retrieval enhancement generation technology can be generated.

[0144] Step S603: Obtain the data query result corresponding to the data query request.

[0145] In some embodiments, the data query results are obtained based on the data processing method described in any of the above embodiments.

[0146] Step S605: Input the question and answer information and the data query results into the first model.

[0147] In this embodiment, prompt words can be generated based on question and answer information and the data query results, and the prompt words can be input into the first model.

[0148] In some embodiments, the first model may be a large language model or other types of question-answering models, and this embodiment does not limit this.

[0149] Step S607: Generate response information corresponding to the question-and-answer information based on the output information of the first model.

[0150] In this embodiment, response information corresponding to the question and answer information is generated based on the output information of the first model, and the response information is returned to the user.

[0151] The methods described above are used to include the corresponding methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0152] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0153] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0154] Based on the same inventive concept, corresponding to any of the above-described embodiments, this disclosure also provides a data processing apparatus.

[0155] refer to Figure 7 The device includes: The acquisition module 11 is configured to: acquire a data query request, wherein the data query request includes query data of at least one modal type; The feature extraction module 13 is configured to: extract features from the query data of the at least one type of modality to obtain at least one corresponding query vector; The matching module 15 is configured to: match the at least one query vector with data vectors of corresponding modal types in a plurality of vector groups pre-stored in a first database to obtain at least one candidate vector group; wherein, each vector group includes a plurality of data vectors, and the first database is used to store the plurality of vector groups, a first association relationship between the plurality of vector groups, and a second association relationship between the plurality of data vectors in each vector group; The association acquisition module 17 is configured to: based on the at least one candidate vector group, and according to at least one of the first association relationship and the second association relationship, acquire at least one extended vector group associated with the at least one candidate vector group; The generation module 19 is configured to generate a data query result corresponding to the data query request based on at least one of the at least one candidate vector group and the at least one extended vector group.

[0156] In some embodiments, the at least one modality type includes a first modality type and a second modality type, and the at least one query vector includes a first query vector corresponding to the first modality type and a second query vector corresponding to the second modality type; The step of matching the at least one query vector with data vectors of corresponding modal types from multiple vector groups pre-stored in the first database to obtain at least one candidate vector group includes: The first query vector is matched with the data vectors of the corresponding modality type in the plurality of vector groups to obtain at least one first vector group with the highest similarity. Based on the at least one first vector group, the second query vector is matched with the data vectors of the corresponding modality type in the plurality of vector groups to obtain at least one second vector group with the highest similarity. The at least one candidate vector group is obtained based on the at least one first vector group and the at least one second vector group.

[0157] In some embodiments, the at least one modality type includes a third modality type and a fourth modality type, and the at least one query vector includes a third query vector corresponding to the third modality type and a fourth query vector corresponding to the fourth modality type; The step of matching the at least one query vector with data vectors of corresponding modal types from multiple vector groups pre-stored in the first database to obtain at least one candidate vector group includes: The third query vector is matched with the data vectors of the corresponding modality type in the plurality of vector groups to obtain at least one third vector group with the highest similarity. The fourth query vector is matched with the data vectors of the at least one third vector group to obtain at least one fourth vector group with the highest similarity. The at least one candidate vector group is obtained based on the at least one fourth vector group.

[0158] In some embodiments, matching the at least one query vector with data vectors of corresponding modality types from a plurality of vector groups pre-stored in a first database to obtain at least one candidate vector group includes: Calculate the similarity between the at least one query vector and the data vectors of the corresponding modality type in the plurality of vector groups; Obtain the weight that matches the modality type in the data query request; The fusion similarity is determined based on the similarity, the weight, and the preset harmonic weighting formula; The at least one vector group with the highest fusion similarity is determined as the at least one candidate vector group.

[0159] In some embodiments, the method further includes: Acquire multimodal data to be stored, wherein the multimodal data contains data units of at least two different modal types; Feature extraction is performed on the data units of the at least two different modal types to obtain their respective corresponding data vectors; Associate the data vectors corresponding to the data units of the at least two different modal types to obtain at least one fifth vector group; Establish the first association relationship between the at least one fifth vector group and the plurality of vector groups pre-stored in the first database, and store the at least one fifth vector group in the first database.

[0160] In some embodiments, multiple data vectors of the same vector group form multiple semantic clusters, each semantic cluster being used to represent an abstract concept hierarchy; the multiple semantic clusters form a tree structure.

[0161] In some embodiments, the plurality of semantic clusters are stored using an index structure based on a hierarchical navigation graph, and the plurality of data vectors are subspaced using vector segmentation quantization compression technology.

[0162] In some embodiments, matching the at least one query vector with data vectors of corresponding modality types from a plurality of vector groups pre-stored in a first database to obtain at least one candidate vector group includes: An approximate nearest neighbor search is performed on the semantic cluster that is most relevant to the at least one query vector to obtain the at least one candidate vector group.

[0163] In some embodiments, the plurality of vector groups includes a fifth vector group and a sixth vector group; The first association relationship includes the adjacency relationship between the fifth vector group and the sixth vector group, and the first parent-child relationship between any of the semantic clusters of the fifth vector group and the sixth vector group; The second association relationship includes a second parent-child relationship between the plurality of semantic clusters.

[0164] In some embodiments, obtaining at least one extended vector group associated with the at least one candidate vector group based on at least one of the first association relationship and the second association relationship includes at least one of the following: Based on the adjacency relationship, obtain at least one extended vector group that is adjacent to the at least one candidate vector group; Based on the second parent-child relationship, at least one of the parent semantic cluster and child semantic cluster of the first semantic cluster in the at least one candidate vector group is obtained. Based on the first parent-child relationship, the parent vector node of the parent semantic cluster and the child semantic cluster is obtained. The vector group to which the parent vector node belongs is determined as the at least one extended vector group.

[0165] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing this disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0166] The apparatus of the above embodiments is used to implement the corresponding method in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0167] Based on the same inventive concept, corresponding to any of the above-described embodiments, this disclosure also provides a data processing apparatus.

[0168] refer to Figure 8 The device includes: The first acquisition module 21 is configured to: acquire question and answer information and generate a data query request based on the question and answer information; The second acquisition module 23 is configured to: acquire the data query result corresponding to the data query request; Input module 25 is configured to input the question-and-answer information and the data query results into the first model; The generation module 27 is configured to generate response information corresponding to the question-and-answer information based on the output information of the first model.

[0169] In some embodiments, the data query results are obtained based on the data processing method described in the foregoing embodiments.

[0170] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing this disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0171] The apparatus of the above embodiments is used to implement the corresponding method in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0172] Based on the same inventive concept, corresponding to any of the above embodiments, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any of the above embodiments.

[0173] Figure 9 This embodiment illustrates a more specific hardware structure of an electronic device. The device may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0174] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0175] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0176] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0177] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0178] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0179] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0180] The electronic devices described above are used to implement the corresponding methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0181] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to perform the methods described in any of the above embodiments.

[0182] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0183] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to perform the methods described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0184] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a computer program product, which includes a computer program. In some embodiments, the computer program is executable by one or more processors to cause the processors to perform the method. Corresponding to the execution entity for each step in each embodiment of the method, the processor performing the corresponding step may belong to the corresponding execution entity.

[0185] The computer program products of the above embodiments are used to cause a processor to execute the methods described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0186] Those skilled in the art will recognize that embodiments of this disclosure can be implemented as a system, method, or computer program product. Therefore, this disclosure can be implemented as entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this disclosure can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0187] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (not exhaustive) of a computer-readable storage medium may include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0188] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0189] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0190] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0191] It should be understood that each block of a flowchart and / or block diagram, as well as combinations of blocks in a flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine that, when executed by a computer or other programmable data processing device, creates means for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0192] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce a product comprising an instruction apparatus that implements the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0193] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions that execute on the computer or other programmable apparatus can provide a process for implementing the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0194] Furthermore, although the operations of the methods of this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowcharts may be executed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0195] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0196] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0197] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this disclosure as described above, which are not provided in detail for the sake of brevity.

[0198] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this disclosure, the provided drawings may or may not show well-known power / ground connections to integrated circuit (IC) chips and other components. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this disclosure, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this disclosure will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that the embodiments of this disclosure can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0199] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0200] This disclosure is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data processing method, comprising: Obtain a data query request, wherein the data query request includes query data of at least one modal type; Feature extraction is performed on the query data of the at least one type of modality to obtain at least one corresponding query vector; The at least one query vector is matched with data vectors of corresponding modal types in a plurality of vector groups pre-stored in the first database to obtain at least one candidate vector group; wherein, each vector group includes a plurality of data vectors, and the first database is used to store the plurality of vector groups, a first association relationship between the plurality of vector groups, and a second association relationship between the plurality of data vectors in each vector group; Based on the at least one candidate vector group, at least one extended vector group associated with the at least one candidate vector group is obtained according to at least one of the first association relationship and the second association relationship; Based on at least one of the at least one candidate vector group and the at least one extended vector group, generate a data query result corresponding to the data query request.

2. The method according to claim 1, wherein, The at least one modality type includes a first modality type and a second modality type, and the at least one query vector includes a first query vector corresponding to the first modality type and a second query vector corresponding to the second modality type; The step of matching the at least one query vector with data vectors of corresponding modal types from multiple vector groups pre-stored in the first database to obtain at least one candidate vector group includes: The first query vector is matched with the data vectors of the corresponding modality type in the plurality of vector groups to obtain at least one first vector group with the highest similarity. Based on the at least one first vector group, the second query vector is matched with the data vectors of the corresponding modality type in the plurality of vector groups to obtain at least one second vector group with the highest similarity. The at least one candidate vector group is obtained based on the at least one first vector group and the at least one second vector group.

3. The method according to claim 1, wherein, The at least one modality type includes a third modality type and a fourth modality type, and the at least one query vector includes a third query vector corresponding to the third modality type and a fourth query vector corresponding to the fourth modality type; The step of matching the at least one query vector with data vectors of corresponding modal types from multiple vector groups pre-stored in the first database to obtain at least one candidate vector group includes: The third query vector is matched with the data vectors of the corresponding modality type in the plurality of vector groups to obtain at least one third vector group with the highest similarity. The fourth query vector is matched with the data vectors of the at least one third vector group to obtain at least one fourth vector group with the highest similarity. The at least one candidate vector group is obtained based on the at least one fourth vector group.

4. The method according to claim 1, wherein, The step of matching the at least one query vector with data vectors of corresponding modal types from multiple vector groups pre-stored in the first database to obtain at least one candidate vector group includes: Calculate the similarity between the at least one query vector and the data vectors of the corresponding modality type in the plurality of vector groups; Obtain the weight that matches the modality type in the data query request; The fusion similarity is determined based on the similarity, the weight, and the preset harmonic weighting formula; The at least one vector group with the highest fusion similarity is determined as the at least one candidate vector group.

5. The method according to claim 1, further comprising: Acquire multimodal data to be stored, wherein the multimodal data contains data units of at least two different modal types; Feature extraction is performed on the data units of the at least two different modal types to obtain their respective corresponding data vectors; Associate the data vectors corresponding to the data units of the at least two different modal types to obtain at least one fifth vector group; Establish the first association relationship between the at least one fifth vector group and the plurality of vector groups pre-stored in the first database, and store the at least one fifth vector group in the first database.

6. The method according to claim 1, wherein, Multiple data vectors in the same vector group form multiple semantic clusters, each semantic cluster representing an abstract concept level; the multiple semantic clusters form a tree structure.

7. The method according to claim 6, wherein, The multiple semantic clusters are stored using an index structure based on a hierarchical navigation graph, and the multiple data vectors are subspaced using vector segmentation quantization compression technology.

8. The method according to claim 6, wherein, The step of matching the at least one query vector with data vectors of corresponding modality types from multiple vector groups pre-stored in the first database to obtain at least one candidate vector group includes: An approximate nearest neighbor search is performed on the semantic cluster that is most relevant to the at least one query vector to obtain the at least one candidate vector group.

9. The method according to claim 6, wherein, The plurality of vector groups includes a fifth vector group and a sixth vector group; The first association relationship includes the adjacency relationship between the fifth vector group and the sixth vector group, and the first parent-child relationship between any of the semantic clusters of the fifth vector group and the sixth vector group; The second association relationship includes a second parent-child relationship between the plurality of semantic clusters.

10. The method according to claim 9, wherein, Obtaining at least one extended vector group associated with the at least one candidate vector group based on at least one of the first association relationship and the second association relationship includes at least one of the following: Based on the adjacency relationship, obtain at least one extended vector group that is adjacent to the at least one candidate vector group; Based on the second parent-child relationship, at least one of the parent semantic cluster and child semantic cluster of the first semantic cluster in the at least one candidate vector group is obtained. Based on the first parent-child relationship, the parent vector node of the parent semantic cluster and the child semantic cluster is obtained. The vector group to which the parent vector node belongs is determined as the at least one extended vector group.

11. A data processing method, comprising: Obtain question and answer information, and generate a data query request based on the question and answer information; Obtain the data query results corresponding to the data query request; Input the question-and-answer information and the data query results into the first model; Based on the output information of the first model, generate response information corresponding to the question and answer information.

12. The method according to claim 11, wherein, The data query results are obtained based on the data processing method described in any one of claims 1-10.

13. A data processing apparatus, comprising: The acquisition module is configured to: acquire a data query request, wherein the data query request includes query data of at least one modal type; The feature extraction module is configured to: extract features from the query data of the at least one type of modality to obtain at least one corresponding query vector; The matching module is configured to: match the at least one query vector with data vectors of corresponding modal types in a plurality of vector groups pre-stored in a first database to obtain at least one candidate vector group; wherein each vector group includes a plurality of data vectors, and the first database is used to store the plurality of vector groups, a first association relationship between the plurality of vector groups, and a second association relationship between the plurality of data vectors in each vector group; The association acquisition module is configured to: based on the at least one candidate vector group, and according to at least one of the first association relationship and the second association relationship, acquire at least one extended vector group associated with the at least one candidate vector group; The generation module is configured to generate a data query result corresponding to the data query request based on at least one of the at least one candidate vector group and the at least one extended vector group.

14. A data processing apparatus, comprising: The first acquisition module is configured to: acquire question and answer information, and generate a data query request based on the question and answer information; The second acquisition module is configured to: acquire the data query result corresponding to the data query request; The input module is configured to input the question-and-answer information and the data query results into the first model; The generation module is configured to generate response information corresponding to the question-and-answer information based on the output information of the first model.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the data processing method as described in any one of claims 1 to 12.

16. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the data processing method according to any one of claims 1 to 12.

17. A computer program product, characterized in that, It includes computer program instructions that, when executed on a computer, cause the computer to perform the data processing method as described in any one of claims 1 to 12.

Images