A cross-modal data query method

Abstract

This invention discloses a cross-modal data query method, belonging to the field of data query processing technology. It includes: pre-extracting multiple descriptor information from a data source according to a preset triplet registration format to generate an access interface for accessing data from the data source; further including: parsing the modality type in response to the query content; selecting a data source of the corresponding type based on the modality type; selecting an access path for the data source based on historical latency data of the data source; setting a caching mode based on the number of modality types; accessing the corresponding database according to the access path; obtaining the access result of the corresponding modality type based on the correlation between the query content and the corresponding data of the modality type through a preset comparison model; and caching the result based on the caching mode. This invention achieves dynamic adaptation, low-latency access, and high-precision querying across modal data sources by automatically generating an interface using triplet descriptors, combined with modality parsing and a multi-level caching mechanism.

Description

Technical Field

[0001] This invention belongs to the field of data query and processing technology, and specifically relates to a cross-modal data query method. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially breakthroughs in Large Language Models (LLM) and multimodal learning, the demand for cross-modal data interaction is growing exponentially. In fields such as intelligent customer service, cross-modal search engines, digital content management, medical image analysis, and autonomous driving, systems need to simultaneously process heterogeneous data from multiple sources, including text, images, audio, and video. For example, media companies need to perform semantic association and retrieval on massive amounts of multimodal content (such as news photos, documentary videos, and background music audio).

[0003] Although multimodal data processing technology has made some progress, the storage formats and access protocols of different modal data vary significantly, requiring traditional methods to write independent adaptation code for each data source. For example, accessing an image database requires developing an image encoder interface, and accessing an audio database requires developing an audio decoder interface. In addition, adding a new data source (such as introducing a new video database) requires redeveloping the adaptation logic, which is time-consuming and prone to errors.

[0004] Furthermore, cross-modal queries require multiple round trips to different data sources, leading to cumulative latency, long response times, and high resource consumption. In addition, the feature vectors of multimodal queries reside in different semantic spaces, making it difficult for traditional methods to establish connections. For example, the matching accuracy between musical mood and poetic emotion is low, resulting in irrelevant query results.

[0005] In summary, existing technologies generally suffer from data heterogeneity barriers, query latency and resource waste, and insufficient accuracy of query results when querying cross-modal data. Summary of the Invention

[0006] This invention provides a cross-modal data query method to address the problems of data heterogeneity, query latency and resource waste, and insufficient accuracy of query results.

[0007] The technical solution adopted in this invention is as follows:

[0008] A cross-modal data query method includes: pre-extracting multiple descriptor information of a data source according to a preset triplet registration format to generate an access interface corresponding to the data source for accessing data from the data source;

[0009] The method further includes, in response to the query content, parsing to obtain the modality type, selecting the corresponding data source based on the modality type, selecting the access path of the data source based on the historical latency data of the data source, and setting the caching mode based on the number of modality types;

[0010] According to the access path, the corresponding database is accessed. Through a preset comparison model, based on the correlation between the query content and the data in the data source corresponding to the modality type, the access result of the corresponding modality type is obtained and cached based on the caching mode.

[0011] The cross-modal data query method disclosed in this invention also has the following additional technical features:

[0012] The specific format for the triple registration is as follows:

[0013] The triple registration format includes modality type, data format, and access protocol.

[0014] The modal types mentioned above include text, image, and audio types.

[0015] In response to the query content, the modal type is parsed and obtained, specifically:

[0016] Based on the query content, keywords are identified through syntactic analysis, and the modality type is obtained by parsing according to the mapping relationship table between the keywords and the modality type.

[0017] Based on the historical latency data of the data source, the access path of the data source is selected, specifically as follows:

[0018] Based on the latency data obtained from historical queries, combined with the load status of the current data source access path, the latency of the current data source on different paths is predicted through a learning model.

[0019] If the local GPU latency is below the threshold, the data source of the local resource path is invoked;

[0020] Otherwise, call the data source from the cloud API path.

[0021] The threshold is specifically:

[0022] According to the modality type, an initial threshold is set for each type, with the initial thresholds for text type, image type, and audio type decreasing sequentially.

[0023] The initial threshold is adjusted based on the complexity of the query content to obtain the threshold, which is negatively correlated with the complexity.

[0024] Based on the number of modal types, the caching mode is set as follows:

[0025] When the parsed modal type in response to the query content is 1, the access result caching mode is set to short-term caching, and the first pre-caching time is set accordingly; otherwise, the access result caching mode is set to long-term caching, and the second pre-caching time is set accordingly, wherein the first pre-caching time is shorter than the second pre-caching time.

[0026] Based on the parsed modality type and the hit rate of the query content, the first pre-caching time and the second pre-caching time are adjusted to obtain the corresponding first cache time and second cache time;

[0027] The first cache time and the second cache time are positively correlated with the hit rate.

[0028] Based on the correlation between the query content and the data in the data source corresponding to the modality type, the access results for the corresponding modality type are obtained, including:

[0029] When the parsed response to the query content yields only one modal type.

[0030] Based on the query content, an embedding vector is obtained. The embedding vector is then compared with the feature vector of the corresponding data source. Based on the comparison result, the access result is returned.

[0031] Based on the correlation between the query content and the data in the data source corresponding to the modality type, the access result for the corresponding modality type is obtained, which also includes:

[0032] When the response to the query content yields multiple modal types...

[0033] Based on the completeness of the description in the query content, the query modality and the target modality are obtained, wherein the completeness of the description of the query modality is greater than that of the target modality;

[0034] The query vector is obtained by comparing the embedding vector of the query modality with the feature vector of the data in the corresponding data source.

[0035] Based on the correlation between the query vector corresponding to the query modality and the feature vector corresponding to the target modality, the target vector is obtained;

[0036] The access results are obtained based on the query vector and the target vector.

[0037] The comparison model is specifically as follows:

[0038] Based on text type, image type, and audio type, a three-modal training set is obtained;

[0039] Based on the three-modal training set, the corresponding feature vectors are obtained through the corresponding encoders;

[0040] Based on the similarity of feature vectors between modalities, feature vector alignment and association are performed between modalities;

[0041] In response to user query feedback, set reward values ​​and adjust the alignment and correlation of feature vectors between modalities.

[0042] The present invention also discloses a processing apparatus, comprising:

[0043] Memory, used to store computer programs;

[0044] A processor is used to implement the steps of the cross-modal data query method when executing the computer program.

[0045] Due to the adoption of the above technical solution, the beneficial effects achieved by this invention are as follows:

[0046] 1. In this invention, multiple descriptor information from the data source is extracted in advance according to a preset triplet registration format to generate an access interface corresponding to the data source, thus solving the problem of data heterogeneity barriers in the prior art. Interfaces are automatically generated through standardized triplet descriptors, eliminating the need for manual adaptation. There is no need to write independent adapter code for each data source, reducing development costs; adding a new data source only requires registering the triplet, without modifying the core code, improving system scalability.

[0047] In response to the query content, the modality type is parsed, and a corresponding data source is selected based on the modality type, solving the fragmented adaptation problem in existing technologies and improving system flexibility. Keywords are identified through syntax analysis, and data sources are dynamically selected in conjunction with a modality mapping table. Modality parsing accuracy is improved, avoiding invalid queries caused by misjudgments; selecting the corresponding data source type based on the modality type avoids full-modality queries, thus increasing system throughput.

[0048] By selecting the access path for the data source based on its historical latency data, the problem of high query latency in existing technologies is solved, significantly reducing response time. Selecting the optimal path based on historical latency data reduces average query latency, improves throughput in high-concurrency scenarios, and utilizes a caching mechanism to avoid redundant calculations.

[0049] The caching mode is set according to the number of modal types. This invention optimizes resource utilization through a multi-level caching mechanism, solves the problem of redundant calculations in existing technologies, and significantly improves system efficiency. Short-term cache is used to store original query results (such as text search results) to avoid duplicate requests. Long-term cache is used to store cross-modal feature vectors for reuse in multiple queries. The caching mode is set according to the number of modal types (single-modal or cross-modal), with short-term caching for single-modal and long-term caching for cross-modal, improving cache hit rate and resource utilization.

[0050] Furthermore, based on the access path, the corresponding database is accessed. Through a pre-set comparison model, based on the correlation between the query content and the data in the data source corresponding to the modality type, the access result of the corresponding modality type is obtained, and cached based on the caching mode. This solves the semantic gap problem in the prior art, significantly improves the accuracy of cross-modal queries, supports complex queries, and improves the accuracy of cross-modal queries. Attached Figure Description

[0051] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0052] Figure 1 This is a flowchart illustrating the cross-modal data query method according to one embodiment of the present invention. Detailed Implementation

[0053] To more clearly illustrate the overall concept of the present invention, a detailed description will be provided below with reference to the accompanying drawings and examples.

[0054] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0055] like Figure 1 As shown, a cross-modal data query method includes:

[0056] S100: Extract multiple descriptor information from the data source in advance according to the preset triplet registration format to generate an access interface corresponding to the data source for accessing data from the data source.

[0057] The core objective of this step is to address the data heterogeneity barrier in existing technologies by uniformly describing multi-source heterogeneous data through a pre-defined triple registration format and automatically generating standardized access interfaces, thereby reducing development complexity and improving system scalability.

[0058] Data source needs to be passed<modality_type, data_format, access_protocol> The registration is performed using a triple, where modality_type represents the modality type, data_format represents the data format, and access_protocol represents the access protocol.

[0059] The data source submits descriptor information (i.e., triples) to the MCP proxy, declaring its type, format, and access protocol. The MCP proxy automatically generates the corresponding data source access interface code based on the descriptor (such as encapsulating gRPC as a REST API), eliminating the need for developers to manually write adaptation logic.

[0060] Furthermore, when adding a new data source, only the new triple descriptor needs to be submitted, and the system will automatically adapt and generate the interface without modifying the core code.

[0061] Understandably, traditional methods require writing independent adaptation code for each data source (such as text, images, and audio). For example, image databases require developing image encoder interfaces, and audio databases require developing audio decoder interfaces, resulting in high development costs and difficult maintenance.

[0062] This invention automatically generates interfaces using triplet standardized descriptors, eliminating the need for manual adaptation. For example, an image data source is registered as...<image, jpeg, http> Subsequently, the system automatically encapsulates it into a unified REST API, eliminating the need for developers to concern themselves with the underlying implementation details. This reduces development costs by eliminating the need to write separate adapter code for each data source; adding a new data source only requires registering the triplet, without modifying the core code, thus improving system scalability.

[0063] Furthermore, all data sources are uniformly described using a triplet format, and interfaces are dynamically generated by parsing triples, enabling unified access to multimodal data. Standardized descriptors unify the management of heterogeneous data from multiple sources, simplifying the system architecture; developers no longer need to write adaptation logic for different modalities, improving development efficiency.

[0064] This step systematically solves the data heterogeneity barrier in cross-modal data queries by using a triple registration format and automated interface generation. This not only reduces development costs but also improves the system's scalability and dynamic adaptability.

[0065] The method further includes, S200: in response to the query content, parsing to obtain the modality type, selecting the corresponding data source according to the modality type, selecting the access path of the data source according to the historical latency data of the data source, and setting the caching mode according to the number of modality types.

[0066] The core objective of this step is to address the issues of low query efficiency and resource waste in existing technologies. By dynamically parsing modal types, optimizing data source access paths, and setting caching modes, the latency of cross-modal queries can be significantly reduced, thereby improving system throughput and resource utilization.

[0067] The system uses syntax analysis to identify keywords based on the query content and parse out the modality type. For example, if a user enters "find documents related to this image," the system parses out two modality types: "text" (text type) and "image" (image type). Based on the parsed modality type, the system matches the corresponding type from registered data sources (e.g., image → image database, audio → audio database), and dynamically loads the adaptation code using the triple descriptor of the MCP protocol, eliminating the need for manual interface development. By automatically identifying the modality through syntax analysis and dynamically parsing the modality type, the system avoids full-modality queries.

[0068] It's important to note that data source access paths generally include both local resource access and cloud API access. Predicting the latency of the current data source on the local resource access path allows for the selection of either local resource access or cloud API access. Choosing the optimal path based on historical latency data reduces response time and lowers average query latency.

[0069] The caching mode is set according to the number of modal types. For unimodal queries (e.g., image only), a short-term cache is set. For cross-modal queries (e.g., image + text), a long-term cache is set. The short-term cache stores the original query results (e.g., text search results) to avoid duplicate requests; the long-term cache stores cross-modal feature vectors for reuse in multiple queries. It is understood that the computational cost of cross-modal queries is much greater than that of unimodal queries, therefore, a long-term cache is set for cross-modal queries, with a longer cache duration than that for unimodal queries, in order to balance redundant computation and resource utilization.

[0070] This step systematically solves the problems of high query latency and resource waste in cross-modal data queries by dynamically resolving modal types, using intelligent routing algorithms and multi-level caching mechanisms, and significantly improves query efficiency and resource utilization.

[0071] S300: According to the access path, access the corresponding database, and through a preset comparison model, obtain the access result of the corresponding modality type based on the correlation between the query content and the data in the data source corresponding to the modality type, and cache it based on the caching mode.

[0072] The core objective of this step is to address the semantic gap problem in existing technologies. By aligning the multimodal feature space with a pre-built contrast model and combining it with a caching mechanism to reduce redundant computation, the accuracy and efficiency of cross-modal queries can be significantly improved.

[0073] Based on the selected access path (such as local GPU or cloud API), the corresponding data source interface (such as REST API) is called to extract data related to the query content (such as image feature vectors, audio embeddings) from the database.

[0074] Using a pre-defined comparison model, semantic association analysis is performed on the query content and data source to obtain the access results. Complex queries are supported, and cross-modal accuracy is improved. Access results are stored according to the set caching mode (short-term cache or long-term cache).

[0075] This step systematically solves the semantic gap and resource waste problems in cross-modal data queries by aligning multimodal feature spaces through comparative learning and multi-level caching mechanisms, significantly improving query accuracy and resource utilization.

[0076] In a preferred embodiment of the present invention, the triplet registration format is specifically as follows:

[0077] The triple registration format includes modality type, data format, and access protocol.

[0078] The modal types mentioned above include text, image, and audio types.

[0079] The core objective of this implementation is to solve the fragmented adaptation problem in the prior art. It uses a standardized triplet registration format to uniformly describe multi-source heterogeneous data and automatically generate standardized access interfaces, thereby reducing development complexity and improving system scalability.

[0080] The triplet format is as follows:<modality_type, data_format, access_protocol> Here, `modality_type` represents the modality type, including text, image, and audio. `data_format` represents the data format, such as json (text), jpeg (image), and mp3 (audio). `access_protocol` represents the access protocol, such as http (HTTP interface), gRPC (gRPC interface), and s3 (object storage protocol).

[0081] For example, an image data source is registered as<image, jpeg, http> Audio data source registration as<audio, mp3,grpc> .

[0082] Understandably, the access methods for different data sources can be automatically adapted based on the content of the triple. For example,<image, jpeg, http> It will generate HTTP API call code.<audio, mp3, grpc> It will generate gRPC interface call code. Regardless of the protocol or format of the data source, the system uniformly encapsulates it into a standardized interface (such as a REST API) through triple descriptors. Developers only need to call the unified interface to access different data sources without manual adaptation. Developers do not need to care about the underlying implementation details, reducing development costs and improving system scalability.

[0083] In this embodiment, the "fragmentation adaptation" problem in cross-modal data query is systematically solved by using a standardized descriptor for the triple registration format.

[0084] In a preferred embodiment of the present invention, the modality type is parsed in response to the query content, specifically as follows:

[0085] Based on the query content, keywords are identified through syntactic analysis, and the modality type is obtained by parsing according to the mapping relationship table between the keywords and the modality type.

[0086] The core objective of this implementation is to identify keywords through syntactic analysis and parse modal types by combining them with a mapping table, thereby achieving dynamic adaptation to data sources and improving system flexibility and query efficiency.

[0087] Based on the grammatical structure of the query content, keywords (such as "image," "audio," and "document") are extracted. Specifically, Natural Language Processing (NLP) techniques (such as word segmentation and dependency parsing) are used to identify entities or function words in the query. For example, if a user enters "find documents related to this image," the system extracts the keywords "image" and "document."

[0088] Based on a pre-defined keyword-modal type mapping table (e.g., "image" → image, "audio" → audio, "document" → text), keywords are converted into their corresponding modal types. Modal identification is achieved dynamically through syntactic analysis and the mapping table, requiring no manual intervention. Furthermore, adding new data sources only requires updating the mapping table, without modifying the core code.

[0089] Based on the parsed modal type (e.g., image or text), match the corresponding type (e.g., ...) from the registered data sources.<image, jpeg, http> or<text, json, elasticsearch> (and call its interface, thereby reducing resource waste and improving system throughput by reducing full-modal queries).

[0090] This implementation method identifies keywords through syntax analysis and parses modal types using a mapping table, reducing development complexity and improving system flexibility and query efficiency.

[0091] In a preferred embodiment of the present invention, the access path of the data source is selected based on the historical latency data of the data source, specifically as follows:

[0092] Based on the latency data obtained from historical queries, combined with the load status of the current data source access path, the latency of the current data source on different paths is predicted through a learning model.

[0093] If the local GPU latency is below the threshold, the data source of the local resource path is invoked;

[0094] Otherwise, call the data source from the cloud API path.

[0095] The core objective of this implementation is to solve the problem of high query latency in the prior art. It uses a dynamic routing algorithm combined with a learning model to predict latency and selects the optimal access path (local GPU or cloud API) based on a threshold, thereby significantly reducing the response time of cross-modal queries and improving system throughput.

[0096] Collect historical latency data from data sources (such as average latency of local GPUs and network latency of cloud APIs). Combine this with the load status of the current data source access path (such as GPU utilization and network bandwidth) and use a learning model (such as linear regression or LSTM) to predict the latency of the current data source on different paths.

[0097] Specifically, historical data is used to train the model and learn the relationship between latency and load status; real-time load status is obtained through monitoring tools (such as Prometheus) and input into the model for dynamic prediction.

[0098] Based on the prediction results, a threshold-based strategy is used to select the access path. If the local GPU latency is below the threshold, the local GPU is used for processing first; if the local GPU latency is higher than or equal to the threshold, the cloud API is called. This implementation dynamically selects the optimal path based on historical latency data and real-time load status.

[0099] This implementation solves the problems of high query latency and resource waste in cross-modal data queries by fusing historical latency data with load status analysis, learning model to predict latency, and dynamic routing decisions, thus balancing query efficiency and resource utilization.

[0100] In a preferred embodiment of this implementation, the threshold is specifically:

[0101] According to the modality type, an initial threshold is set for each type, with the initial thresholds for text type, image type, and audio type decreasing sequentially.

[0102] The initial threshold is adjusted based on the complexity of the query content to obtain the threshold, which is negatively correlated with the complexity.

[0103] This embodiment further optimizes the access path selection between the local GPU and the cloud API by dynamically adjusting the threshold (based on modality type and query complexity), thereby significantly reducing the response time of cross-modality queries and improving system throughput and resource utilization.

[0104] Set initial thresholds according to modality type (text, image, audio):

[0105] For text types, the initial threshold is relatively high (e.g., 200ms) because it has a higher tolerance for latency; for image types, the initial threshold is medium (e.g., 150ms) because it needs to balance computation and latency; for audio types, the initial threshold is the lowest (e.g., 100ms) because it has higher requirements for real-time performance.

[0106] Furthermore, the initial threshold is adjusted based on the complexity of the query content, and the complexity is evaluated through feature extraction (such as query length and embedding vector dimension). For example, long text queries or high-dimensional audio features have higher complexity.

[0107] Complexity is negatively correlated with the threshold (the higher the complexity, the lower the threshold). For example:

[0108] When the complexity is 1 (low), the threshold = initial threshold × 1.0; when the complexity is 3 (high), the threshold = initial threshold × 0.5.

[0109] The access path is selected based on the adjusted threshold. If the predicted latency is lower than the adjusted threshold, the local GPU is invoked; otherwise, the cloud API is invoked.

[0110] For example, if the image query complexity is 2, the initial threshold is 150ms, the adjusted threshold is 120ms, and the prediction local GPU latency is 110ms, then the local path is selected.

[0111] It should be noted that static thresholds cannot adapt to the needs of different modalities and query complexities. For example, the static threshold for text queries may be too low, leading to unnecessary cloud calls; while the static threshold for audio queries may be too high, causing local GPU overload.

[0112] The threshold is dynamically adjusted based on modality type and query complexity to ensure that path selection better matches actual needs. For example, for high-complexity audio queries, the threshold is lowered, local GPUs are used first to reduce latency, the average query latency is reduced, and throughput is improved in high-concurrency scenarios by reducing redundant requests and optimizing path selection.

[0113] Furthermore, thresholds are dynamically adjusted based on complexity to balance the load on local GPUs and cloud APIs. For example, low-complexity text queries allow for higher thresholds to fully utilize local resources, while high-complexity audio queries have strictly limited thresholds to avoid overloading the local GPU.

[0114] This embodiment sets an initial threshold based on the modality type, dynamically adjusts the threshold based on the query complexity, and selects the access path based on the adjusted threshold. This solves the problems of high query latency and resource waste in cross-modal data queries, and significantly improves query efficiency and resource utilization.

[0115] In a preferred embodiment of the present invention, a caching mode is set according to the number of modal types, specifically as follows:

[0116] When the parsed modal type in response to the query content is 1, the access result caching mode is set to short-term caching, and the first pre-caching time is set accordingly; otherwise, the access result caching mode is set to long-term caching, and the second pre-caching time is set accordingly, wherein the first pre-caching time is shorter than the second pre-caching time.

[0117] Based on the parsed modality type and the hit rate of the query content, the first pre-caching time and the second pre-caching time are adjusted to obtain the corresponding first cache time and second cache time;

[0118] The first cache time and the second cache time are positively correlated with the hit rate.

[0119] This implementation method dynamically sets the caching mode (short-term caching or long-term caching) based on the number of modal types and adjusts the caching time in conjunction with the hit rate of the query content, which significantly improves the system resource utilization and query efficiency, and reduces redundant calculations and data access latency.

[0120] Set the caching mode based on the number of modal types obtained from parsing (single-modal or cross-modal):

[0121] For unimodal queries (such as image only), set a short-term cache to store the original query results (such as text search results);

[0122] For cross-modal queries (such as image+text), set up a long-term cache to store cross-modal feature vectors (such as CLIP image embedding).

[0123] Based on the modality type obtained from parsing and the hit rate of the query content, adjust the pre-caching time. Count the number of times the same query is repeated within a specific time period. For example, the high-frequency query "dog pictures" has a high hit rate, while the low-frequency query "rare bird audio" has a low hit rate. When the hit rate is high, extend the short-term cache time; when the hit rate is high again, extend the long-term cache time.

[0124] In addition, hit rate coefficients are set according to modality type to adjust cache time. It should be noted that the hit rate coefficients for text, image, and audio types increase sequentially. Ensure that the first and second cache times are positively correlated with the hit rate: short-term cache time = first pre-cache time × (1 + hit rate coefficient); long-term cache time = second pre-cache time × (1 + hit rate coefficient).

[0125] Specifically, if the hit rate of a cross-modal query increases from 20% to 80%, the long-term cache time is extended from 72 hours to 30 days.

[0126] Single-modal queries use short-term caching to avoid duplicate requests; cross-modal queries use long-term caching to reuse feature vectors. The cache time for high-frequency queries is extended based on hit rate, reducing redundant computation, improving cache hit rate, and increasing resource utilization. Furthermore, the cache time for high-hit-rate queries is extended, while the cache time for low-hit-rate queries is shortened, optimizing cache space allocation.

[0127] This implementation solves the problem of resource waste in cross-modal data queries by setting the caching mode (short-term caching or long-term caching) according to the number of modal types and dynamically adjusting the caching time in combination with the query hit rate, and significantly improves the cache hit rate and resource utilization.

[0128] In a preferred embodiment of the present invention, the access result of the corresponding modality type is obtained based on the correlation between the query content and the data in the data source corresponding to the modality type.

[0129] In one embodiment, when the parsed modal type in response to the query content is 1,

[0130] Based on the query content, an embedding vector is obtained. The embedding vector is then compared with the feature vector of the corresponding data source. Based on the comparison result, the access result is returned.

[0131] The core objective of this embodiment is to accurately match the query content with the data source by comparing the embedded vector and feature vector under unimodal query, thereby significantly improving the accuracy and efficiency of unimodal query.

[0132] Based on the query (e.g., "find images similar to this image"), the embedding vector is extracted, and the image is converted into an embedding vector using encoders such as CLIP and ResNet.

[0133] The query embedding vector is compared with the feature vectors stored in the data source for similarity. This embodiment does not limit the comparison method; algorithms such as cosine similarity and Euclidean distance can be used to calculate the matching degree between vectors. It is understood that the data in the data source has been pre-converted into feature vectors and stored using the same encoder.

[0134] Specifically, the embedding vector of the query image is compared with the CLIP image feature vector stored in the data source, and the similarity is calculated before returning the Top-K matching result.

[0135] Based on the comparison results (such as similarity sorting), the access results that best match the query content are returned. For unimodal matching, the data with the highest matching degree is directly returned (such as "dog pictures").

[0136] In another embodiment, obtaining the access result for the corresponding modality type based on the correlation between the query content and the data in the data source corresponding to the modality type further includes:

[0137] When the response to the query content yields multiple modal types...

[0138] Based on the completeness of the description in the query content, the query modality and the target modality are obtained, wherein the completeness of the description of the query modality is greater than that of the target modality;

[0139] The query vector is obtained by comparing the embedding vector of the query modality with the feature vector of the data in the corresponding data source.

[0140] Based on the correlation between the query vector corresponding to the query modality and the feature vector corresponding to the target modality, the target vector is obtained;

[0141] The access results are obtained based on the query vector and the target vector.

[0142] The core objective of this embodiment is to accurately match cross-modal data (such as image → text, audio → text) through description completeness assessment and feature vector association analysis under multimodal queries, thereby significantly improving the accuracy of complex queries and system scalability.

[0143] Based on the completeness of the description in the query content, the query modality and the target modality are distinguished. Description completeness is defined as follows: the query modality's description is more specific (e.g., "pictures of dogs"), while the target modality's description is more vague (e.g., "dog barks"). Specifically, Natural Language Processing (NLP) is used to analyze the entity completeness of the query content (e.g., "pictures of dogs" contains explicit entities and modality types, while "dog barks" only contains entities and vague modalities).

[0144] The query embedding vector is compared with the feature vectors of the corresponding data source. A similarity calculation (e.g., cosine similarity) is performed between the query embedding vector and the feature vectors stored in the data source (e.g., image feature vectors) to generate the query vector.

[0145] A target vector is generated based on the correlation between the query vector and the feature vector of the target modality. Based on the query vector and the target vector, the access results are returned. By weighted fusion of the query vector and the target vector, the data that best matches the query content is returned (e.g., "dog pictures" matches "dog barking audio files").

[0146] This embodiment evaluates the completeness of the query content description, prioritizing high-completeness modalities (such as images), and then generating results for low-completeness modalities (such as audio) through feature alignment. A query vector is obtained by using the embedding vector describing the complete query modality, and a target vector is derived from the query vector. Compared to directly using the embedding vector of an incomplete target modality to obtain the target vector, the accuracy of the access results is significantly improved.

[0147] This embodiment significantly improves the accuracy and resource utilization of complex queries through description completeness evaluation, feature vector comparison, and cross-modal alignment under multimodal queries.

[0148] As a preferred embodiment of the present invention, the comparative model is specifically as follows:

[0149] Based on text type, image type, and audio type, a three-modal training set is obtained;

[0150] Based on the three-modal training set, the corresponding feature vectors are obtained through the corresponding encoders;

[0151] Based on the similarity of feature vectors between modalities, feature vector alignment and association are performed between modalities;

[0152] In response to user query feedback, set reward values ​​and adjust the alignment and correlation of feature vectors between modalities.

[0153] The core objective of this implementation method is to significantly improve the accuracy and adaptability of cross-modal queries by constructing a trimodal training set, aligning feature vectors, and dynamically optimizing the alignment associations based on user feedback, while reducing the reliance on manual annotation.

[0154] Based on three types of data—text, image, and audio—a three-modal training set is constructed. By automatically matching natural language descriptions with multimodal data (such as using the CLIP model to generate text-image pairs), the cost of manual annotation is reduced.

[0155] Trimodal feature vectors are extracted using corresponding encoders. The text encoder uses BERT or Transformer model to extract semantic features; the image encoder uses CLIP or ResNet to extract visual features; and the audio encoder uses AST or Wav2Vec to extract audio features.

[0156] By contrastive learning of feature vectors between modalities, InfoNCE loss or Triplet Loss is used to maximize the similarity of matching modal pairs and minimize the similarity of non-matching pairs.

[0157] Specifically, text-image alignment (TV) uses the CLIP dual-tower structure to shorten the feature distance between semantically similar text and images; text-audio alignment (TA) uses the VALOR model to jointly optimize text and audio features; and image-audio alignment (VA) uses a cross-modal attention mechanism to align visual and auditory features.

[0158] In response to user query feedback, reward values ​​are set and alignment correlations are adjusted. User clicks on positive samples (e.g., "correctly matched audio") or mislabeling (e.g., "irrelevant image") serve as reward signals; reward values ​​are set based on feedback type (positive / negative samples) (e.g., +1 for positive samples, -0.5 for negative samples); feature alignment parameters are adjusted through reinforcement learning (e.g., Policy Gradient) or online learning (e.g., incremental model updates).

[0159] Specifically, user feedback is transformed into a driving force for model optimization through a user points incentive system and a dopamine feedback mechanism.

[0160] This implementation establishes a unified semantic space by comparing and learning aligned text, image, and audio feature vectors (such as TV, TA, VA), achieving trimodal alignment training. Alignment relationships are dynamically adjusted through a reward mechanism (e.g., reinforcement learning to update model parameters), with user feedback driving optimization.

[0161] This implementation improves query accuracy by constructing a three-modal training set, aligning feature vectors, and dynamically optimizing alignment associations based on user feedback.

[0162] The present invention also provides a processing apparatus, comprising:

[0163] Memory, used to store computer programs;

[0164] A processor is used to implement the steps of the cross-modal data query method when executing the computer program.

[0165] Therefore, it can achieve any effect in cross-modal data query methods, which will not be elaborated here.

[0166] For any parts not mentioned in this invention, existing technologies can be used or referenced.

[0167] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0168] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A cross-modal data query method, characterized in that, include: Based on a preset triplet registration format, multiple descriptor information of the data source is extracted in advance to generate an access interface corresponding to the data source for accessing data from the data source; The method further includes, in response to the query content, parsing to obtain the modality type, selecting the corresponding data source based on the modality type, selecting the access path of the data source based on the historical latency data of the data source, and setting the caching mode based on the number of modality types; According to the access path, the corresponding database is accessed. Through a preset comparison model, based on the correlation between the query content and the data in the data source corresponding to the modality type, the access result of the corresponding modality type is obtained, and cached based on the caching mode. Specifically, the triplet registration format includes modality type, data format, and access protocol, wherein the modality type includes text type, image type, and audio type; In response to the query content, the modality type is parsed to obtain the modality type. Specifically, based on the query content, keywords are identified through syntax analysis, and the modality type is parsed to obtain the modality type according to the mapping relationship table between the keywords and the modality type. Based on the historical latency data of the data source, the access path of the data source is selected. Specifically, based on the latency data obtained from historical queries and combined with the load status of the current data source access path, the latency of the current data source on different paths is predicted through a learning model. If the local GPU latency is lower than the threshold, the data source of the local resource path is called; otherwise, the data source of the cloud API path is called. Based on the number of modal types, a caching mode is set. Specifically, when only one modal type is parsed in response to a query, the access result caching mode is set to short-term caching, corresponding to a first pre-caching time; otherwise, the access result caching mode is set to long-term caching, corresponding to a second pre-caching time, wherein the first pre-caching time is shorter than the second pre-caching time. The first pre-caching time and the second pre-caching time are adjusted according to the parsed modal types and the hit rate of the query content, resulting in a first cache time and a second cache time. The first cache time and the second cache time are positively correlated with the hit rate. The comparison model specifically involves obtaining a three-modal training set based on text type, image type, and audio type; obtaining corresponding feature vectors based on the three-modal training set through corresponding encoders; aligning and associating inter-modal feature vectors based on the similarity of inter-modal feature vectors; and adjusting the alignment and association of inter-modal feature vectors in response to user query feedback by setting reward values.

2. The cross-modal data query method according to claim 1, characterized in that, The threshold is specifically: According to the modality type, an initial threshold is set for each type, with the initial thresholds for text type, image type, and audio type decreasing sequentially. The initial threshold is adjusted based on the complexity of the query content to obtain the threshold, which is negatively correlated with the complexity.

3. The cross-modal data query method according to claim 1, characterized in that, Based on the correlation between the query content and the data in the data source corresponding to the modality type, the access results for the corresponding modality type are obtained, including: When the parsed response to the query content yields only one modal type. Based on the query content, an embedding vector is obtained. The embedding vector is then compared with the feature vector of the corresponding data source. Based on the comparison result, the access result is returned.

4. The cross-modal data query method according to claim 1, characterized in that, Based on the correlation between the query content and the data in the data source corresponding to the modality type, the access result for the corresponding modality type is obtained, which also includes: When the response to the query content yields multiple modal types... Based on the completeness of the description in the query content, the query modality and the target modality are obtained, wherein the completeness of the description of the query modality is greater than that of the target modality; The query vector is obtained by comparing the embedding vector of the query modality with the feature vector of the data in the corresponding data source. Based on the correlation between the query vector corresponding to the query modality and the feature vector corresponding to the target modality, the target vector is obtained; The access results are obtained based on the query vector and the target vector.

5. A processing apparatus, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the cross-modal data query method as described in any one of claims 1 to 4 when executing the computer program.

Images