A method and system for multimodal data joint query and analysis that supports natural language interaction

Abstract

This application provides a method and system for joint query analysis of multimodal data supporting natural language interaction, relating to the technical field of data query analysis. This application collects historical operation data, audio data, and text data from an intelligent search platform supporting natural language interaction; then models the historical operation data to obtain a user preference vector; processes the audio data through speech recognition and text standardization to obtain standard query data, and integrates it with the text data to form contextual information; then uses an attention-based algorithm for semantic understanding and intent recognition to obtain an intent feature vector, and fuses it with the user preference vector to obtain a classification result; finally, based on this result, a collaborative filtering algorithm is used to query a pre-set multimodal database to obtain joint query results. This enables personalized and accurate querying of multimodal data under natural language interaction, ensuring that the results match user intent and long-term preferences.

Description

Technical Field

[0001] This application relates to the technical field of data query and analysis, and in particular to a multimodal data joint query and analysis method and system that supports natural language interaction. Background Technology

[0002] With the popularization of intelligent search and voice interaction technologies, users' demands for convenient query methods and rich results are increasing. In daily information retrieval, users not only initiate real-time queries through natural language forms such as voice and text, but also expect to obtain multimodal data results such as text, images, audio, and video. At the same time, they hope that the query results can match their long-term interests and preferences. This scenario is widely used in fields such as smart terminal search, intelligent assistant interaction, and online information services, giving rise to an urgent need for natural language interaction and multimodal data joint query technology.

[0003] Currently, existing technologies for natural language interactive queries typically collect users' text or audio data, convert the audio data into text through speech recognition, and then identify the query intent through keyword matching and simple semantic analysis. For multimodal data queries, most are based on single-modal retrieval rules, extracting corresponding data results from the database. Some technologies combine a small number of users' historical query records and use basic recommendation algorithms to return relevant data to users.

[0004] However, the main drawback of existing technologies is that relying solely on a single intent recognition or basic matching method makes it difficult for query results to accurately match users' personalized needs. They cannot simultaneously take into account real-time query goals and long-term interests and preferences. Therefore, the integrated recommendation effect of multimodal data is not good. Summary of the Invention

[0005] The purpose of this application is to provide a method and system for joint query analysis of multimodal data that supports natural language interaction, so as to solve the problem of poor recommendation effect of multimodal data integration in the prior art.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a multimodal data joint query and analysis method supporting natural language interaction, comprising:

[0007] Collect historical operation data, audio data, and text data generated by users in intelligent search platforms that support natural language interaction;

[0008] Sequence modeling is performed on the historical operation data to obtain a user preference vector. The audio data is then subjected to speech recognition and text standardization processing to obtain standard query data. The standard query data and the text data are then integrated to obtain context information.

[0009] An attention-based intent classification algorithm is used to perform semantic understanding and intent recognition on the context information to obtain an intent feature vector. The intent feature vector is then fused with the user preference vector in a multimodal manner to obtain the intent classification result.

[0010] Based on the intent classification results, a recommendation algorithm based on collaborative filtering is used to perform a joint query in a preset multimodal database to obtain the joint query results.

[0011] Optionally, the intention classification algorithm based on an attention mechanism is used to perform semantic understanding and intention recognition on the context information to obtain an intention feature vector. The intention feature vector is then fused with the user preference vector in a multimodal manner to obtain the intention classification result, including:

[0012] An attention-based intent classification algorithm is used to perform term importance analysis on the context information to determine target terms. Based on a preset allocation rule, corresponding semantic weights are assigned to the target terms.

[0013] Based on the semantic weights, the context information is reorganized to obtain weighted semantic features. The weighted semantic features are then quantized in a fixed dimension to obtain an intent feature vector.

[0014] Establish a connection path between the intent feature vector and the user preference vector, and based on the connection path, integrate the preference information in the user preference vector into the intent feature vector to obtain an enhanced intent feature vector;

[0015] Based on the distribution features corresponding to the semantic components in the enhanced intent feature vector, the intent direction of the current query is classified and determined to obtain the intent classification result.

[0016] Optionally, the intention classification algorithm based on an attention mechanism is used to perform term importance analysis on the context information to determine the target term, including:

[0017] The context information is processed by word segmentation to obtain a word set. The distribution characteristics of the word set are statistically analyzed to obtain the basic importance score of each word.

[0018] Using semantic association analysis, semantic association mining is performed on the word set to establish a semantic association network between words. Based on the connection density of each word in the semantic association network, an intent classification algorithm is used to dynamically adjust the corresponding basic importance score to obtain the target importance score for each word.

[0019] Select target terms from the set of words whose target importance score is higher than the preset screening criteria.

[0020] Optionally, the step of classifying and determining the intent direction of the current query based on the distribution features corresponding to the semantic components in the enhanced intent feature vector to obtain the intent classification result includes:

[0021] The enhanced intent feature vector is subjected to dimensionality decomposition to obtain multiple semantic components;

[0022] The semantic components are subjected to distribution statistical processing to obtain the distribution features corresponding to each semantic component;

[0023] Based on the distribution characteristics and the preset mapping relationship, the intent direction of the current query is classified and determined to obtain the intent classification result.

[0024] Optionally, based on the intent classification result, a recommendation algorithm based on collaborative filtering is used to perform a joint query in a preset multimodal database to obtain the joint query result, including:

[0025] Based on the intent direction and modal requirements of the current query in the intent classification results, the corresponding basic data set is extracted from the preset multimodal database;

[0026] Based on the preference information in the user preference vector, the basic data set is initially screened to obtain a candidate data set. Then, an association analysis mechanism is used to mine the association between the candidate data set and the historical operation data to obtain data association information.

[0027] Based on the data association information, a recommendation algorithm based on collaborative filtering is used to determine the joint query results.

[0028] Optionally, determining the joint query result using a collaborative filtering-based recommendation algorithm based on the data association information includes:

[0029] Based on the data association information, the text, image, audio, and video data corresponding to the functions are combined into data groups;

[0030] Calculate the degree of fit between each data set and the user preference vector to obtain the corresponding matching degree;

[0031] Based on the matching degree, all the data groups are sorted to obtain the joint query result.

[0032] Optionally, the step of performing sequence modeling on the historical operation data to obtain a user preference vector includes:

[0033] The historical text data and operation trajectory records contained in the historical operation data are arranged in a preset time order to obtain a time-series operation sequence;

[0034] The relationships between operation records in the time-series operation sequence are identified to obtain a set of relationships between operation records.

[0035] Based on the set of relationships, behavioral pattern features reflecting users' long-term interests are extracted from the time-series operation sequence, and the behavioral pattern features are converted into a numerical representation of user preference vectors.

[0036] Secondly, this application provides a multimodal data joint query and analysis system that supports natural language interaction, including:

[0037] The data collection module is used to collect historical operation data, audio data, and text data generated by users in intelligent search platforms that support natural language interaction.

[0038] The integration module is used to perform sequence modeling on the historical operation data to obtain a user preference vector, perform speech recognition and text standardization processing on the audio data in sequence to obtain standard query data, and integrate the standard query data with the text data to obtain context information;

[0039] The fusion module is used to perform semantic understanding and intent recognition on the context information using an intent classification algorithm based on an attention mechanism, to obtain an intent feature vector, and to perform multimodal fusion of the intent feature vector with the user preference vector to obtain the intent classification result;

[0040] The query module is used to perform a joint query in a preset multimodal database based on the intent classification result and a recommendation algorithm based on collaborative filtering to obtain the joint query result.

[0041] Thirdly, this application provides an electronic device, comprising:

[0042] Memory, used to store computer programs;

[0043] A processor, configured to execute the computer program to implement the steps of the multimodal data joint query analysis method supporting natural language interaction as described in the first aspect above.

[0044] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the multimodal data joint query and analysis method supporting natural language interaction as described in the first aspect above.

[0045] The technical solution of this application has the following beneficial effects:

[0046] This application collects user historical operation data, audio data, and text data, covering long-term user behavior, real-time voice needs, and text needs, providing a complete and diverse data foundation for subsequent personalized processing, intent recognition, and multimodal queries. Through sequence modeling, historical operation data is transformed into user preference vectors, accurately extracting long-term user interest features. Next, audio data undergoes speech recognition and text standardization processing to ensure that voice needs are transformed into effective query data in a unified format. Then, standard query data and text data are integrated to form complete contextual information, thereby ensuring the comprehensiveness of intent recognition. Finally, an attention-based intent classification mechanism is employed. The algorithm enables deep semantic understanding and accurate intent recognition of contextual information to generate structured intent feature vectors. Then, through multimodal fusion, user preference vectors are integrated into the intent feature vectors, ensuring that the intent classification results not only align with real-time query targets and match long-term user preferences, but also improve the personalization and accuracy of intent classification. Based on the accurate intent classification results and combined with a collaborative filtering-based recommendation algorithm, cross-modal collaborative queries can be achieved in multimodal databases. This allows for efficient filtering and integration of text, image, audio, and video data, ultimately outputting joint query results tailored to user needs, ensuring both the richness and personalized accuracy of the query results.

[0047] Furthermore, this application enables query semantic features to focus on core needs and improve the relevance of feature expression through term importance analysis and semantic weight allocation; after feature recombination and quantification, a structured intent feature vector can be generated, which can provide reliable support for intent recognition; subsequently, by integrating user preference information, the personalized attributes of query intent features are strengthened, and by combining semantic dimension decomposition and distribution feature analysis, the query intent can be accurately classified, ultimately improving the accuracy, relevance and personalization of query intent classification, and providing accurate intent basis for subsequent multimodal queries. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart illustrating a multimodal data joint query and analysis method supporting natural language interaction, provided as an embodiment of this application;

[0050] Figure 2 This is a schematic diagram illustrating a specific implementation of a multimodal data joint query and analysis method supporting natural language interaction, provided in an embodiment of this application.

[0051] Figure 3 A schematic diagram of the structure of a multimodal data joint query and analysis system supporting natural language interaction is provided in an embodiment of this application;

[0052] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0053] Current natural language interaction and multimodal data query technologies generally suffer from insufficient depth in understanding user needs: relying solely on single-dimensional intent recognition or basic matching methods results in query results that fail to accurately match users' personalized needs and cannot balance real-time query goals with long-term interests.

[0054] To address the aforementioned issues, this application proposes a multimodal data joint query analysis method supporting natural language interaction. This method first comprehensively collects users' historical operation trajectories, real-time audio queries, and text data, and then extracts and integrates users' long-term preferences through modeling to form a complete query context. Next, it uses an attention mechanism to accurately identify real-time query intent and deeply integrates it with user preferences. Finally, it achieves cross-modal collaborative querying in a multimodal database through a collaborative filtering algorithm. This approach not only overcomes the shortcomings of existing technologies in integrating user preferences and real-time intent but also enhances the collaborative recommendation capabilities of multimodal data, enabling query results to accurately match real-time needs and align with long-term interests, thereby improving the personalization and accuracy of queries.

[0055] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] The core of this application is to provide a multimodal data joint query and analysis method that supports natural language interaction. A flowchart of one specific implementation is shown below. Figure 1 As shown, the method includes:

[0057] S101. Collect historical operation data, audio data, and text data generated by users in intelligent search platforms that support natural language interaction.

[0058] In the above scheme, historical operation data refers to the past behavioral data generated by users during their use of the intelligent search platform. Historical operation data includes click operation records, collection operation records, viewing operation records, and historical query keyword records, which are used to extract users' long-term interests and preferences. Audio data refers to the voice information input by users through the audio acquisition module of the intelligent search platform when initiating a query, which is used to capture users' current instant query needs. Text data refers to the text query content submitted by users through the text input module of the intelligent search platform. Text data includes the query statement entered in a single instance and historical text query records, which are used to help clarify the user's real-time query intent.

[0059] In this embodiment, the intelligent search platform first continuously captures various historical user actions within the platform through its behavior log recording module. These historical actions include user clicks, favorites, and viewing durations of different modalities in the query results. It also records the corresponding query keywords and timestamps for each action to form an initial historical action dataset. Next, the intelligent search platform's audio acquisition module receives the user's voice input in real time when initiating a query. This module supports real-time capture and storage of mainstream audio formats, ensuring the integrity of the voice information. Finally, the intelligent search platform's text input module receives the user's query content submitted in text form, while simultaneously retrieving text query records submitted by the user in the current session and historical usage, integrating them to form complete text data.

[0060] In this embodiment of the invention, by collecting users' historical operation behavior data, real-time voice query data, and text query data, the system covers the behavioral trajectory formed by users over a long period of use and captures users' current instant query needs. The collected data provides a diverse foundation for subsequent extraction of user preferences, integration of query context, and accurate identification of query intent. This ensures that subsequent processing steps can fully combine users' long-term interests and real-time needs, thereby laying a data foundation for ultimately achieving personalization and precision.

[0061] S102. Perform sequence modeling on the historical operation data to obtain a user preference vector. Perform speech recognition and text standardization processing on the audio data in sequence to obtain standard query data. Integrate the standard query data with the text data to obtain context information.

[0062] In one specific implementation, step S102 includes:

[0063] Step 1021: Arrange the historical text data and operation trajectory records contained in the historical operation data according to a preset time order to obtain a time-series operation sequence.

[0064] Among them, historical text data refers to the collection of query texts submitted by users through the text input module of the intelligent search platform in the past, which is used to reflect the user's historical query needs; operation trajectory records refer to the records of user actions such as clicking, collecting, and watching the text, image, audio, and video query results returned by the platform, which are used to capture the user's preferences for different modal data; time-series operation sequence refers to the ordered data collection formed by organizing historical text data and operation trajectory records in chronological order, which is used to present the time logic of user operations; the preset time order can be chronological order.

[0065] For example, if the user of the intelligent search platform is an outdoor gear enthusiast, the platform first collects three types of user data: historical operation data includes records of 6 queries related to outdoor gear for age group C within the past 3 months; 12 text results, 9 image results, and 7 video results clicked; 4 image results and 3 text results saved; and the viewing time of 5 video results was 8 minutes, 12 minutes, 15 minutes, 10 minutes, and 9 minutes respectively. The total viewing time is calculated to be 8+12+15+10+9=54 minutes. The audio data is the user's current voice saying "Recommended lightweight outdoor gear for age group". The text data includes the "budget range X to Y" entered in the current conversation and the historical text query "waterproof performance of outdoor gear".

[0066] Step 1022: Identify the relationships between operation records in the time-series operation sequence to obtain a set of relationships between operation records. Based on the set of relationships, extract behavioral pattern features reflecting the user's long-term interests from the time-series operation sequence and convert the behavioral pattern features into a numerical representation of the user preference vector.

[0067] Among them, behavioral pattern features refer to the core features extracted from the time-series operation sequence that can reflect the user's long-term interest tendencies and are used to quantify user preferences; user preference vectors refer to the numerical vector form converted from behavioral pattern features, which are used to facilitate subsequent algorithm calculations.

[0068] In step 1022, an association rule mining algorithm can be used to analyze the operation records and identify the inherent associations between query keywords and modal operations, as well as between different modal operations, to obtain a set of association relationships. Then, based on this set, a feature extraction algorithm is used to extract behavioral pattern features that reflect the user's long-term interests, and the behavioral pattern features are converted into a numerical user preference vector through a feature quantification algorithm.

[0069] For example, sequence modeling can be performed on historical operational data, and behavioral weight calculation formulas can be used. Where W represents the operation weight, w represents the basic weight of the operation (e.g., the basic weight for clicking is set to 1, the basic weight for adding to favorites is set to 2, and the basic weight for watching is set to 1.5), and f represents the operation frequency. Substituting the data, the total weight of the click operation is calculated to be 1×(12+9+7)=28, the total weight of the favorite operation is 2×(4+3)=14, the total weight of the watch operation is 1.5×5=7.5, and the total weight of all operations is 28+14+7.5=49.5. Combined with the operation sequence sorted by time, the behavioral pattern features of "preferring image + video modal and focusing on the practicality of outdoor equipment" are extracted and converted into a user preference vector.

[0070] Step 1023: Perform speech segment separation on the audio data to obtain a clean speech signal, perform amplitude optimization and amplification processing on the clean speech signal to obtain an enhanced speech signal, and use speech conversion technology to convert the enhanced speech signal into intermediate text data.

[0071] Among them, the pure speech signal refers to the speech signal after removing noise interference by performing frequency band separation processing on the audio data, which is used to improve the accuracy of speech conversion; the intermediate text data refers to the text data obtained by converting the enhanced speech signal through speech conversion technology, which is the preliminary text form of the speech query.

[0072] For example, after frequency band separation to remove environmental noise from audio data, and then amplitude amplification to optimize signal strength, the intermediate text "Recommended lightweight outdoor gear for the recommended age group" is obtained through speech-to-text technology.

[0073] Step 1024: Based on preset text normalization rules, the intermediate text data is sequentially processed to eliminate redundant components and regularize the structure to obtain standard query data. Using a time-series integration method, the standard query data is combined with the text data in the form of text input by the user in the current session according to the receiving time to obtain context information.

[0074] Among them, text normalization rules refer to the platform's preset standard rules for optimizing text data. These rules include redundancy judgment standards and text structure regularization standards to improve text data quality. The redundancy judgment standards are used to identify and define meaningless particles, repeated words, modal particles, and other invalid content. The text structure regularization standards are used to standardize the word order and expression format of query text to ensure that the text meets the query parsing requirements of the intelligent search platform. Standard query data refers to the standardized text obtained after redundancy elimination and structure regularization of intermediate text data, which is used for integration with text data. Contextual information refers to the complete set of query information formed by combining standard query data with the text data in the current session according to the reception time, which is used to support accurate intent recognition.

[0075] For example, based on the preset text normalization rules, the redundancy judgment criteria in these rules define "ya" as a meaningless modal particle, and the text structure regularity standard requires the query text to start with the core demand keyword and be concise. Subsequently, the platform deletes "ya" from the middle text and adjusts the word order to "Recommendation of Lightweight Outdoor Equipment for Age Groups" to obtain standard query data. This data is then integrated with text data such as "Budget X to Y range" to form contextual information.

[0076] This application successfully transforms long-term user behavior into quantifiable user preference vectors through sequence modeling of historical operation data, enabling precise extraction of users' long-term interest characteristics. Subsequently, speech recognition and text standardization processing of audio data effectively converts voice requests into standardized text forms, thereby ensuring the effectiveness of voice queries. Furthermore, by integrating standard query data with text data, complete contextual information is formed, comprehensively covering users' real-time query needs.

[0077] S103. An attention-based intent classification algorithm is used to perform semantic understanding and intent recognition on the context information to obtain an intent feature vector. The intent feature vector is then fused with the user preference vector in a multimodal manner to obtain the intent classification result.

[0078] In one specific implementation, such as Figure 2 As shown, step S103 includes:

[0079] Step 1031: Use an attention-based intent classification algorithm to perform term importance analysis on the context information to determine target terms, and assign corresponding semantic weights to the target terms based on preset allocation rules.

[0080] Among them, the attention-based intent classification algorithm is an algorithm that focuses on the core information of the query and is used to perform deep semantic analysis on the context information; term importance analysis is the process of evaluating the keyness of words in the query text and is used to distinguish between words related to core needs and auxiliary needs; semantic weight is a numerical indicator that measures the importance of words in expressing query intent and is used to strengthen core information and weaken secondary information; target terms include core query terms and modifying terms, and modifying terms are words that help explain core query terms and supplement the details of needs and are used to improve the description of query intent;

[0081] The preset allocation rule combines the core logic of the query intent with the priority of user needs, and is a semantic weight allocation standard formulated for core query terms and modifier terms. It takes the contribution of terms to the query intent as the core basis, first assigns a fixed weight with a higher basic proportion to core query terms, and then allocates the remaining weight to modifier terms according to the proportion of the target importance score. At the same time, it supplements the position of the term in the query context, the degree of matching with the user's historical preference terms, and the semantic association strength. This ensures that the weight allocation highlights the core needs and accurately reflects the differences in the importance of auxiliary needs, and can provide a scientific basis for subsequent semantic feature reorganization.

[0082] Step 1031 may specifically include the following steps: performing word segmentation on the context information to obtain a word set; performing distribution feature statistics on the word set to obtain a basic importance score for each word; using semantic association analysis methods to mine semantic associations in the word set to establish a semantic association network between words; dynamically adjusting the corresponding basic importance scores using an intent classification algorithm based on the connection density of each word in the semantic association network to obtain a target importance score for each word; and selecting target terms from the word set whose target importance scores are higher than a preset screening standard.

[0083] Among them, the word set refers to the set of words obtained after segmenting the context information, which is used to analyze the importance of each word individually; the distribution feature statistics refer to the statistical process of features such as word frequency and position, which is used to initially assess the importance of words; the basic importance score refers to the initial importance value of words obtained based on the distribution feature statistics, which is used as the basis for weight adjustment; the semantic association analysis method refers to the technique of mining semantic connections between words, which is used to discover the inherent logic between words; the semantic association network refers to the network structure that presents the semantic association relationship between words, which is used to reflect the degree of connection between words.

[0084] The connection density of a word refers to the degree of semantic association between the word and all other words in the word set. The connection objects are other words obtained after splitting the same context information. The process of generating the connection density is as follows: count the number of effective association edges of the target word in the semantic association network, and combine the association degree value corresponding to each edge to perform a weighted sum. The comprehensive result is the connection density of the word. The more association edges and the higher the association degree value, the greater the connection density.

[0085] Furthermore, the preset screening criteria refer to the threshold standards for defining core terms and modifier terms. For example, the criteria include preset core term screening criteria and preset modifier term screening criteria, which are used to clarify the boundary between the two types of terms.

[0086] For example, an attention-based intent classification algorithm is used to segment contextual information to obtain words such as "age group," "lightweight," "outdoor gear," "recommendation," "budget," and "X to Y range," and a basic importance score calculation formula is used. ,in, Let 'a' represent the basic importance score, 'f' represent the frequency weight coefficient (set to 0.6), 'f' represent the word frequency (f = 1 since all words appear once), 'b' represent the position weight coefficient (set to 0.4), and 'p' represent the position weight (3 for core positions and 2 for secondary core positions). Since "outdoor equipment" is in the core position, its corresponding p value is 3; the other words are in the secondary core positions, so their corresponding p values ​​are 2. Substituting the data, the basic importance score for "outdoor equipment" is 0.6 × 1 + 0.4 × 3 = 1.8, and the basic importance score for the other words is 0.6 × 1 + 0.4 × 2 = 1.4. Then, the target importance score calculation formula is used. Calculate the target importance score, where, Let d represent the target importance score and d represent the connection density. The platform then uses semantic association analysis to mine the associations between words to establish a semantic association network. It was found that the connection density between "outdoor equipment" and "lightweight, age group, budget, X to Y range" is 1, while the connection density between other words is 0. Substituting the data, the target importance score of outdoor equipment is calculated to be 1.8×(1+1)=3.6, the target importance scores of "lightweight, age group, budget, X to Y range" are 1.4×(1+1)=2.8, and the target importance score of "recommendation" is 1.4×(1+0)=1.4.

[0087] Subsequently, the preset core term selection criteria were set as target importance score ≥ 3.0, and the preset modifier term selection criteria were set as 1.5 ≤ target importance score < 3.0. The core query term "outdoor equipment" and the modifier terms "lightweight, age group, budget, X to Y range" were selected. Based on the preset allocation rules, the semantic weight of the core term was set to 0.5, and the remaining 0.5 weight of the modifier terms was allocated according to the proportion of the target importance score. The total target importance score of the modifier terms was 2.8 × 4 = 11.2, and the weight of each modifier term was 0.5 × (2.8 ÷ 11.2) = 0.125.

[0088] Step 1032: Based on the semantic weights, perform feature reorganization processing on the context information to obtain weighted semantic features, and perform fixed-dimensional quantization processing on the weighted semantic features to obtain the intent feature vector.

[0089] Among them, the intent feature vector refers to the quantized query semantic feature vector, which is used to intuitively express the query intent;

[0090] In step 1032, a feature reorganization algorithm is used to integrate the semantic weights of each word into the corresponding semantic features based on semantic weights. Then, the context information is reorganized to obtain weighted semantic features. Finally, a vector quantization algorithm is used to quantize the weighted semantic features in a fixed dimension and convert them into numerical vector form to obtain the intent feature vector.

[0091] Step 1033: Establish the association path between the intent feature vector and the user preference vector. Based on the association path, integrate the preference information in the user preference vector into the intent feature vector to obtain an enhanced intent feature vector.

[0092] The association path refers to the mapping path that connects the intent feature vector and the user preference vector, which is used to integrate preference information.

[0093] In step 1033, an association path construction algorithm is used to establish an association path between the two based on the association logic between the content and to analyze the matching point between the intent feature vector and the user preference vector. Then, a feature fusion algorithm is used to integrate the preference information in the user preference vector into the intent feature vector based on the association path to obtain an enhanced intent feature vector.

[0094] The content association logic refers to the inherent fit between the core features of the intent feature vector and the user preference vector. It serves as a semantic and logical bridge connecting the two types of vectors, specifically manifested as the correspondence and association between the core needs and potential modal tendencies of the query intent and the long-term interest themes and high-frequency interaction modal types extracted from user preferences. Obtaining this association logic requires first extracting the core dimensions of the intent feature vector and the key features of the user preference vector using a feature parsing algorithm. Then, a feature matching algorithm is used to analyze the correlation between the two types of features, identifying overlapping or highly correlated feature dimensions. Simultaneously, this is verified and reinforced by combining the association records of query intent and preference features in the user's historical operations. Finally, these feature matching results and historical association patterns are integrated to form a stable and accurate content association logic.

[0095] Step 1034: Based on the distribution features corresponding to the semantic components in the enhanced intent feature vector, classify and determine the intent direction of the current query to obtain the intent classification result.

[0096] Among them, semantic components refer to intent feature fragments corresponding to different semantic dimensions, which are used to analyze intent details one by one; distribution features refer to the proportion of semantic components in the enhanced intent feature vector, the strength of association, and other features, which are used to support intent classification; intent classification results refer to the final determined user query intent category and related information, which are used to guide subsequent multimodal queries.

[0097] In one specific implementation, step 1034 may specifically include the following steps: performing dimensionality decomposition processing on the enhanced intent feature vector to obtain multiple semantic components; performing distribution statistical processing on the semantic components to obtain the distribution features corresponding to each semantic component; and performing classification and determination processing on the intent direction of the current query based on the distribution features and a preset mapping relationship to obtain the intent classification result.

[0098] In step 1034, a dimensionality decomposition algorithm is used to decompose the enhanced intent feature vector to obtain multiple semantic components such as topic demand, modal demand, and budget demand. Then, a criticality quantification algorithm is used to quantify the criticality of each semantic component to obtain the corresponding intensity value. Based on the intensity value, a dominant dimensionality filtering algorithm is used to select the semantic dimension with the highest intensity value as the dominant semantic dimension information. Next, a feature splitting algorithm is used to perform targeted splitting of the enhanced intent feature vector based on the dominant semantic dimension information to obtain multiple semantic components corresponding to the dominant semantic dimension. Subsequently, a distribution statistics algorithm is used to perform distribution statistics processing on each semantic component to obtain the corresponding distribution features. Finally, a mapping algorithm is used to establish a mapping relationship between the semantic components and the preset intent categories based on the distribution features, and a classification judgment algorithm is used to classify and judge the current query intent direction based on the mapping relationship to obtain the intent classification result.

[0099] For example, a dimensionality decomposition algorithm is used to split the enhanced intent feature vector into three semantic components: topic demand, modality demand, and budget demand, and an intensity numerical calculation formula is used. Calculate the intensity value of each semantic component;

[0100] Where I represents the intensity value, c represents the demand association weight set to 0.7, r represents the degree of association with the core demand (for example, r is 1 for the topic demand, 0.8 for the modal demand, and 0.7 for the budget demand), e represents the user importance weight (which can be set to 0.3), and s represents the user's historical importance (s is 1 for the topic demand, 0.9 for the modal demand, and 0.8 for the budget demand). Substituting the data, the intensity value of the topic demand is calculated to be 0.7×1+0.3×1=1.0, the intensity value of the modal demand is 0.7×0.8+0.3×0.9=0.83, and the intensity value of the budget demand is 0.7×0.7+0.3×0.8=0.73.

[0101] Next, the topic requirements are selected as the dominant semantic dimension information. The corresponding semantic components are obtained according to this dimension, and the distribution characteristics are statistically analyzed. Then, a mapping relationship with the preset "resource search" intent category is established, and the intent classification result is finally obtained, namely "outdoor equipment resource search category, preference for image and video modalities, budget range X to Y".

[0102] This application uses an attention-based intent classification algorithm to accurately distinguish core and modifier terms in query text and assign reasonable semantic weights, enabling semantic features to focus more on core needs. It then obtains structured intent feature vectors, providing standardized data support for intent analysis. Furthermore, by multimodally fusing user preference vectors and intent feature vectors, it integrates personalized interest attributes into intent features. Finally, through semantic dimension decomposition and dominant dimension filtering, it clarifies the core query direction and achieves accurate classification of query intent by combining distribution features with the mapping of preset intent categories.

[0103] S104. Based on the intent classification result, a recommendation algorithm based on collaborative filtering is used to perform a joint query in a preset multimodal database to obtain the joint query result.

[0104] Among them, collaborative filtering-based recommendation algorithms refer to algorithms that combine user preferences and data association features to achieve accurate recommendations of multimodal data, and are used to filter and integrate various types of data that fit user needs from multimodal databases;

[0105] In one specific implementation, step S104 includes:

[0106] Step 1041: Based on the intent direction and modal requirements of the current query in the intent classification results, extract the corresponding basic data set from the preset multimodal database.

[0107] Modal requirements refer to the user's preferences or explicit requirements for the presentation of query results extracted from the intent classification results. Specifically, they are reflected in the multimodal data types that users want to obtain, as well as the priority ranking or combination of different types of data. They also reflect which information carriers users prefer to receive query results through. They are the core basis for generating intent-adaptive retrieval strategies and targeted extraction of corresponding types of data, ensuring that the presentation of query results is highly consistent with user habits or demand scenarios.

[0108] Furthermore, the pre-defined multimodal database refers to a comprehensive database that stores text, image, audio, and video data, providing multiple data sources for queries; the basic data set refers to the initial data set extracted from the multimodal database, which covers various modal data related to the query intent.

[0109] In step 1041, the intent direction and modal requirements in the intent classification results are extracted, and an intent adaptation retrieval strategy containing data type priority and topic keyword matching threshold is generated in combination with the multimodal database index rules. Then, based on the strategy, a targeted retrieval algorithm is used to extract data from the preset multimodal database to filter out relevant data that meet the keywords and modal requirements to obtain the basic data set.

[0110] For example, based on the above classification results, a collaborative filtering-based recommendation algorithm is used to generate an intent-adaptive retrieval strategy, and the topic keywords are set as "outdoor equipment, lightweight, age group", the data type priority is "image > video > text", and the budget matching criterion is "the price marked in the data is within the range of X to Y". Then, based on this strategy, a targeted search is performed on the preset multimodal database to extract 160 data that meet the requirements, including 60 images, 45 videos, 50 texts, and 5 audios, forming the basic data set.

[0111] Step 1042: Based on the preference information in the user preference vector, perform preliminary screening on the basic data set to obtain a candidate data set. Then, use an association analysis mechanism to mine the association between the candidate data set and the historical operation data to obtain data association information.

[0112] Among them, the correlation analysis mechanism refers to the technology of mining the intrinsic relationship between data and users' historical behavior, which is used to obtain data correlation information;

[0113] For example, a preference filtering algorithm is used to filter the basic dataset based on the user preference vector, which includes "preferring image + video modalities and focusing on equipment practicality". This process removes 5 audio data items and 20 text data items that emphasize decoration, resulting in a candidate dataset of 135 items. Then, a collaborative filtering algorithm is used to associate the features of the candidate dataset with the behavioral features of user A in the historical operation data to identify 30 data items that are similar to the features of the historical collection data, thus forming data association information.

[0114] Step 1043: Based on the data association information, a recommendation algorithm based on collaborative filtering is used to determine the joint query results.

[0115] As a specific implementation method, step 1043 can perform the following process: Step a1, based on the data association information, combine the text, image, audio and video data corresponding to the function into data groups; Step a2, calculate the degree of fit between each data group and the user preference vector to obtain the corresponding matching degree; Step a3, based on the matching degree, perform final sorting processing on all the data groups to obtain the joint query result.

[0116] Among them, the degree of fit refers to the degree of matching between data items and user preferences, which is calculated through multi-dimensional feature comparison; the data group refers to the collection formed by combining different modal data related to functions, used to provide comprehensive query results; the matching degree refers to the quantitative value of the degree of fit between the data group as a whole and user preferences, used for the final ranking of the data group; the joint query result refers to the structured data group collection sorted according to the matching degree, used to present to the user.

[0117] As another specific implementation, step 1043 can perform the following process: based on data association information, calculate the matching degree score between each data item in the candidate data set and the user preference vector, and perform preliminary sorting based on the score; perform content feature analysis on the sorted multimodal data to obtain feature composition information, identify complementary feature combinations through cross-comparison, and obtain data complementary relationship information through association mapping; based on the data complementary relationship information, combine functionally related multimodal data into data groups; calculate the matching degree between each data group and the user preference vector, and perform final sorting of the data groups based on the matching degree to obtain the joint query result.

[0118] Among them, feature composition information refers to the core feature details of multimodal data. Text data includes keywords and topic features, image data includes visual elements and color features, audio data includes timbre and rhythm features, and video data includes visual content and narrative features.

[0119] Complementary feature combination refers to the pairing of features that can complement each other between different modal data, used for data association pairing; complementary data relationship information refers to the association information of complementary functions between multimodal data, used for data combination.

[0120] For example, using the matching score calculation formula Calculate the matching score between each data item and the user preference vector, where M represents the matching score. This represents the modality preference weight, which can be set to 0.4 here. T represents the matching value between the data type and the preference. Here, T is 1 for image and video classes, and 0.6 for text classes. This represents the matching weight of the topic, which can be set to 0.3 here. P represents the matching value between the data topic and the core requirement. For example, P is 1 if all keywords are included, and P is 0.7 if some keywords are included. The value represents the weight of similar user preferences, which is set to 0.3 here. U represents the matching coefficient of similar user preferences. Here, the value of U for similar user preference data mined by collaborative filtering is 1, and the value of U for no matching is 0.5.

[0121] Taking image-based equipment data as an example, with T valued at 1, P valued at 1, and U valued at 1, the matching score M is calculated as 0.4×1+0.3×1+0.3×1=1.0. Taking text-based data as an example, with T valued at 0.6, P valued at 1, and U valued at 0.8, the matching score M is calculated as 0.4×0.6+0.3×1+0.3×0.8=0.78.

[0122] Based on the matching score, a priority ranking method is used to obtain a preliminary ranking result in which image and video data occupy the top positions. Then, a content feature analysis method is used to decompose the feature composition information of the data in the preliminary ranking result to extract material keywords of text data, appearance features of image data, and usage scenario features of video data. Combined with collaborative filtering for item complementarity analysis, complementary feature combinations of "text description, image display, and video demonstration" are identified. Finally, an association mapping method is used to pair multimodal data with complementary features to obtain data complementarity relationship information.

[0123] Based on this information, the function-related data were then grouped into 28 data sets, and a matching degree calculation formula was used. Where G represents the matching degree, and n represents the number of data items in the data group. This represents the matching score of the i-th data item within the group. This represents the data coordination coefficient, where complementary features are used. A value of 1 generally indicates association. The value is 0.9;

[0124] Taking a data set containing "text data with a matching score of 0.78 and a synergy coefficient of 1, image data with a matching score of 1.0 and a synergy coefficient of 1, and video data with a matching score of 0.95 and a synergy coefficient of 1" as an example, the matching score G is calculated to be approximately 0.91 (0.78×1+1.0×1+0.95×1)÷3. The 28 data sets are then sorted based on the matching score to obtain the joint query results. These results will be presented to users through the platform's results display module. Subsequently, the platform will further optimize the weight parameters and matching criteria in the algorithm based on user feedback such as clicks and favorites on these results.

[0125] This application achieves targeted extraction of relevant data from a multimodal database through an intent-adaptive retrieval strategy, ensuring the relevance of initial data to query requirements. Further enhancement of personalized data adaptation is achieved through preliminary screening using user preference vectors, while the correlation mining between candidate data and historical operation data strengthens the targeting of data recommendations. Subsequently, based on the matching degree calculation and ranking using collaborative filtering algorithms, and the feature decomposition and complementary combination of multimodal data, scattered single-modal data are integrated into a comprehensive data set. This not only highlights the modality types preferred by users but also satisfies diverse query needs through data complementarity. The final joint query results accurately match the user's real-time query intent and long-term preferences while providing users with a rich and comprehensive information presentation format.

[0126] Figure 3 This is a schematic diagram illustrating a specific implementation of a multimodal data joint query and analysis system supporting natural language interaction, as provided in this application embodiment. (Refer to...) Figure 3 The system may include:

[0127] The data acquisition module 31 is used to collect historical operation data, audio data, and text data generated by users in the intelligent search platform that supports natural language interaction.

[0128] The integration module 32 is used to perform sequence modeling on the historical operation data to obtain a user preference vector, perform speech recognition and text standardization processing on the audio data in sequence to obtain standard query data, and integrate the standard query data with the text data to obtain context information.

[0129] The fusion module 33 is used to perform semantic understanding and intent recognition on the context information using an intent classification algorithm based on an attention mechanism, to obtain an intent feature vector, and to perform multimodal fusion of the intent feature vector and the user preference vector to obtain an intent classification result.

[0130] The query module 34 is used to perform a joint query in a preset multimodal database based on the intent classification result and a recommendation algorithm based on collaborative filtering to obtain the joint query result.

[0131] The multimodal data joint query and analysis system supporting natural language interaction in this application is used to implement the aforementioned multimodal data joint query and analysis method supporting natural language interaction. Therefore, the specific implementation of the multimodal data joint query and analysis system supporting natural language interaction can be found in the embodiment section of the multimodal data joint query and analysis method supporting natural language interaction above. The specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0132] like Figure 4As shown, this application also provides an electronic device, including: a memory 41 for storing a computer program; and a processor 42 for executing the computer program to implement the steps of any of the above-described multimodal data joint query analysis methods supporting natural language interaction.

[0133] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described multimodal data joint query and analysis methods supporting natural language interaction.

[0134] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0135] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the embodiments of the multimodal data joint query analysis method supporting natural language interaction.

[0136] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0137] The foregoing has provided a detailed description of a multimodal data joint query and analysis method and system supporting natural language interaction provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for multi-modal data joint query analysis supporting natural language interaction, characterized in that, include: Collect historical operation data, audio data, and text data generated by users in intelligent search platforms that support natural language interaction; Sequence modeling is performed on the historical operation data to obtain a user preference vector. The audio data is then subjected to speech recognition and text standardization processing to obtain standard query data. The standard query data and the text data are then integrated to obtain context information. An attention-based intent classification algorithm is used to perform semantic understanding and intent recognition on the context information to obtain an intent feature vector. The intent feature vector is then fused with the user preference vector using a multimodal method to obtain the intent classification result, including: An attention-based intent classification algorithm is used to perform term importance analysis on the context information to determine target terms. Based on a preset allocation rule, corresponding semantic weights are assigned to the target terms. Based on the semantic weights, the context information is reorganized to obtain weighted semantic features. The weighted semantic features are then quantized in a fixed dimension to obtain an intent feature vector. Establish an association path between the intent feature vector and the user preference vector. Based on the association path, integrate the preference information in the user preference vector into the intent feature vector to obtain an enhanced intent feature vector. The association path is a mapping path connecting the intent feature vector and the user preference vector, used to integrate preference information. Based on the distribution features corresponding to the semantic components in the enhanced intent feature vector, the intent direction of the current query is classified and determined to obtain the intent classification result; The intent classification algorithm based on an attention mechanism is used to perform term importance analysis on the context information to determine the target term, including: The context information is processed by word segmentation to obtain a word set. The distribution characteristics of the word set are statistically analyzed to obtain the basic importance score of each word. Using semantic association analysis, semantic association mining is performed on the word set to establish a semantic association network between words. Based on the connection density of each word in the semantic association network, an intent classification algorithm is used to dynamically adjust the corresponding basic importance score to obtain the target importance score for each word. Select target terms from the word set whose target importance score is higher than the preset selection criteria; Based on the intent classification results, a recommendation algorithm based on collaborative filtering is used to perform a joint query in a preset multimodal database to obtain the joint query results.

2. The method of claim 1, wherein, The step of classifying and determining the intent direction of the current query based on the distribution features corresponding to the semantic components in the enhanced intent feature vector to obtain the intent classification result includes: The enhanced intent feature vector is subjected to dimensionality decomposition to obtain multiple semantic components; The semantic components are subjected to distribution statistical processing to obtain the distribution features corresponding to each semantic component; Based on the distribution characteristics and the preset mapping relationship, the intent direction of the current query is classified and determined to obtain the intent classification result.

3. The method of claim 1, wherein, Based on the intent classification result, a recommendation algorithm based on collaborative filtering is used to perform a joint query in a preset multimodal database to obtain the joint query result, including: Based on the intent direction and modal requirements of the current query in the intent classification results, the corresponding basic data set is extracted from the preset multimodal database; Based on the preference information in the user preference vector, the basic data set is initially screened to obtain a candidate data set. Then, an association analysis mechanism is used to mine the association between the candidate data set and the historical operation data to obtain data association information. Based on the data association information, a recommendation algorithm based on collaborative filtering is used to determine the joint query results.

4. The method of claim 3, wherein, The step of determining the joint query result using a collaborative filtering-based recommendation algorithm based on the data association information includes: Based on the data association information, the text, image, audio, and video data corresponding to the functions are combined into data groups; Calculate the degree of fit between each data set and the user preference vector to obtain the corresponding matching degree; Based on the matching degree, all the data groups are sorted to obtain the joint query result.

5. The method of claim 1, wherein, The step of performing sequence modeling on the historical operation data to obtain a user preference vector includes: The historical text data and operation trajectory records contained in the historical operation data are arranged in a preset time order to obtain a time-series operation sequence; The relationships between operation records in the time-series operation sequence are identified to obtain a set of relationships between operation records. Based on the set of relationships, behavioral pattern features reflecting users' long-term interests are extracted from the time-series operation sequence, and the behavioral pattern features are converted into a numerical representation of user preference vectors.

6. A multi-modal data joint query analysis system supporting natural language interaction, characterized in that, A method for implementing a multimodal data joint query and analysis method supporting natural language interaction as described in claim 1 includes: The data collection module is used to collect historical operation data, audio data, and text data generated by users in intelligent search platforms that support natural language interaction. The integration module is used to perform sequence modeling on the historical operation data to obtain a user preference vector, perform speech recognition and text standardization processing on the audio data in sequence to obtain standard query data, and integrate the standard query data with the text data to obtain context information; The fusion module is used to perform semantic understanding and intent recognition on the context information using an intent classification algorithm based on an attention mechanism, to obtain an intent feature vector, and to perform multimodal fusion of the intent feature vector with the user preference vector to obtain the intent classification result; The query module is used to perform a joint query in a preset multimodal database based on the intent classification result and a recommendation algorithm based on collaborative filtering to obtain the joint query result.

7. An electronic device, comprising: include: Memory, used to store computer programs; A processor, configured to implement the steps of the multimodal data joint query analysis method supporting natural language interaction as described in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the multimodal data joint query and analysis method supporting natural language interaction as described in any one of claims 1 to 5.

Images