Keyword extraction method and system based on knowledge graph and user intent perception
By combining knowledge graph and user intent perception methods with large language models and multimodal data processing technology, the keyword extraction strategy is dynamically adjusted, which solves the problem of lack of dynamic adjustment and multimodal data processing capabilities in existing technologies. This enables accurate identification of user intent and personalized keyword extraction, thereby improving the accuracy and efficiency of information retrieval.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SICHUAN SOSSET TECHNOLOGY CO LTD
- Filing Date
- 2026-04-07
- Publication Date
- 2026-07-03
Smart Images

Figure CN122332585A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of natural language processing and multimodal artificial intelligence, specifically to a keyword extraction method and approach based on knowledge graphs and user intent perception. Background Technology
[0002] Keyword extraction is a core task in Natural Language Processing (NLP), aiming to automatically identify words or phrases that best represent the theme of a text. This technology is widely used in information retrieval, text mining, recommendation systems, search engines, and many other fields. Traditional keyword extraction methods primarily rely on algorithms based on statistics and graph theory, such as:
[0003] TF-IDF (Term Frequency-Inverse Document Frequency): This method determines the weight of each word by calculating its frequency in a document and its rarity in the entire corpus, thereby selecting the most representative words in the text. Although the TF-IDF method is simple to implement, it ignores the semantic relationships between words, and its accuracy and effectiveness are limited, especially when dealing with long texts or complex topics.
[0004] TextRank, based on graph theory, treats words in text as nodes in a graph. It evaluates the importance of each node by calculating the relationships and similarities between nodes, thereby extracting keywords. While TextRank can extract relatively accurate keywords from text through unsupervised learning, it still cannot solve the problems of semantic information and contextual understanding in text.
[0005] In recent years, with the development of deep learning technology, keyword extraction methods based on neural networks have gradually become mainstream. These methods typically utilize deep learning architectures such as Convolutional Neural Networks (CNNs), Long Short-Term Memory Networks (LSTMs), or Transformers to automatically learn features from text and extract keywords based on contextual information. However, these methods still have certain limitations, such as a lack of flexible user intent awareness and dynamic adjustment mechanisms.
[0006] With the development of multimodal technologies, researchers have begun to explore the fusion of data from multiple modalities, such as text, images, and audio, for more in-depth analysis and processing. The core objective of multimodal data processing technology is to extract useful knowledge from data of different modalities and perform comprehensive analysis, thereby providing more comprehensive input for downstream tasks. Multimodal data fusion not only helps improve data understanding but also provides more contextual information for artificial intelligence systems.
[0007] Current multimodal data processing methods can be broadly categorized into two types: early fusion and late fusion. Early fusion methods fuse data from different modalities at the feature layer, processing the multimodal data through a unified feature representation. Late fusion, on the other hand, fuses the results after processing data from each modality. Each method has its advantages and disadvantages. Early fusion can more tightly integrate information, but it requires more computation and struggles to handle the differences between different modalities. Late fusion methods are simpler to implement, but the fused information is less comprehensive than that obtained through early fusion.
[0008] In recent years, Transformer-based multimodal models have demonstrated great potential in processing multimodal data. These models can jointly learn data from various modalities, such as text and images, establishing cross-modal relationships to better understand and generate multimodal data outputs. However, existing multimodal models still face some challenges in practical applications, particularly in user intent perception and dynamic adjustment. Many models still rely on static patterns or fixed rules, lacking the ability to flexibly adapt to user needs.
[0009] In many practical applications, keyword extraction and information retrieval are not simple tasks; they often involve specific user needs and goals. To improve system accuracy and user experience, increasing research is focusing on user intent perception. User intent perception technology aims to analyze user input to extract the user's actual needs or purposes, allowing for better adjustment of the system's response. Current user intent perception methods largely rely on natural language processing techniques, especially deep learning-based models. For example, through sentiment analysis and text classification, the system can understand the user's query intent and adjust the output according to the user's needs. However, existing intent perception technologies are typically based on fixed rules or models, lacking the ability to flexibly handle different contexts, scenarios, and personalized user needs. Furthermore, intent perception technologies have poor adaptability to multimodal input, making it difficult to effectively capture the user's true intent from data across different modalities.
[0010] Despite significant progress in keyword extraction and multimodal data processing, existing technologies still have some important shortcomings, particularly in the following areas:
[0011] Lack of dynamic adjustment capability: Most existing keyword extraction methods are static and cannot be adjusted in real time according to different user needs or data context. This results in a significant reduction in the accuracy and relevance of keyword extraction in scenarios with strong personalized requirements.
[0012] Insufficient multimodal data processing capabilities: Most existing keyword extraction methods primarily target single modalities (such as text), failing to effectively integrate information from multiple modalities. Multimodal data typically contains richer and more complex semantic information, making it impossible to fully uncover its inherent relationships using traditional methods.
[0013] Insufficient understanding of user intent: Existing methods often rely on simple keyword matching, lacking a deep understanding of user intent. Especially in multimodal data scenarios, they cannot fully perceive the user's actual needs and goals, resulting in a discrepancy between the extracted results and the user's expectations.
[0014] Computational efficiency issues: Some existing deep learning-based keyword extraction methods are computationally expensive and have limited processing capabilities for large-scale data, especially in application scenarios that require real-time response, where they present significant performance bottlenecks.
[0015] To address the shortcomings of existing technologies, this invention proposes a keyword extraction method and system based on a large language model that is user intent-aware and dynamically adjusted. By introducing a large language model and multimodal data processing technology, this invention can dynamically adjust the keyword extraction strategy while processing user input data in real time, and fully understand the user's intent, thereby improving the accuracy and relevance of keyword extraction. Summary of the Invention
[0016] The purpose of this invention is to provide a keyword extraction method based on knowledge graphs and user intent awareness, in order to solve the problem mentioned in the background art that the existing technology lacks solutions for intelligent retrieval and analysis tasks in engineering fields and other complex information environments.
[0017] To achieve the above objectives, the present invention provides the following technical solution:
[0018] The keyword extraction method based on knowledge graphs and user intent awareness includes the following steps:
[0019] Step S1: Receive multimodal data input by the user, wherein the multimodal data includes at least one of text, image, and audio;
[0020] Step S2: Use a large language model to perform intent perception on the multimodal data, parse and extract the user's search intent;
[0021] Step S3: Dynamically adjust the keyword extraction strategy based on the user intent;
[0022] Step S4: Extract keywords from the multimodal data based on the dynamically adjusted strategy;
[0023] Step S5: Output and display the extracted keywords to the user.
[0024] Further steps in dynamically adjusting the keyword extraction strategy include:
[0025] Step S31: Calculate the adjustment weight α using the user intent vector I and the multimodal feature vector F. The formula for calculating the adjustment weight is:
[0026]
[0027] Where I represents the user intent vector, F represents the multimodal feature vector, and α represents the adjustment weight, which is used to dynamically adjust the weight allocation of the keyword extraction strategy.
[0028] Step S32: Calculate the personalized adjustment coefficient β using the user's historical behavior vector H and current intent vector I. The formula for calculating the personalized adjustment coefficient is:
[0029]
[0030] Where H represents the user's historical behavior vector, γ is an adjustment parameter, and β represents the personalized adjustment coefficient, which is used to adjust the keyword extraction strategy based on the user's historical behavior.
[0031] Furthermore, the steps for extracting keywords from multimodal data include:
[0032] Utilizing the feature vector after multimodal fusion The keyword extraction model M calculates the keyword extraction probability. The formula for calculating the keyword extraction probability is:
[0033]
[0034] in, This represents the fused multimodal feature vector. This represents the score of the keyword extraction model for keyword k, where K represents the set of candidate keywords. This represents the extraction probability of keyword k.
[0035] Furthermore, the following processing methods are adopted for various types of multimodal data:
[0036] For text data, semantic analysis is performed using a large language model to extract key information and determine user intent.
[0037] For image data, computer vision technology is used to extract image features and convert them into descriptive text, which is then combined with text information for intent analysis.
[0038] For audio data, speech recognition technology is used to convert the audio into text, and then a large language model is used for intent parsing.
[0039] Furthermore, the method also includes the steps of feature extraction and fusion of the multimodal data, specifically including:
[0040] Natural language processing techniques are used to extract features from text data;
[0041] Computer vision techniques are used to extract features from image data and then transform them into descriptive text.
[0042] Speech recognition technology is used to extract text information from audio data;
[0043] By fusing features from different modalities through deep learning models, comprehensive semantic understanding can be achieved.
[0044] A keyword extraction system based on a large-scale language model with engineering domain knowledge graphs and user intent perception and dynamic adjustment includes:
[0045] The user input module is used to receive multimodal data input by the user;
[0046] The intent-aware module is used to use a large language model to perceive the intent of the multimodal data, parse and extract the user's search intent;
[0047] The dynamic adjustment module is used to dynamically adjust the keyword extraction strategy based on the user's intent.
[0048] The keyword extraction module is used to extract keywords from the multimodal data based on a dynamically adjusted strategy;
[0049] The output module is used to display the extracted keywords to the user.
[0050] Furthermore, the dynamic adjustment module includes:
[0051] The weight calculation unit is used to calculate the adjustment weight α using the user intent vector I and the multimodal feature vector F. The formula for calculating the adjustment weight is as follows:
[0052] The historical behavior adjustment unit is used to calculate the personalized adjustment coefficient β using the user's historical behavior vector H and current intent vector I. The formula for calculating the personalized adjustment coefficient is as follows: .
[0053] Furthermore, the keyword extraction module includes a probability calculation unit, used to calculate the keyword extraction probability P(k) using the multimodal fusion-derived feature vector Ffused and the keyword extraction model M. The formula for calculating the keyword extraction probability is: .
[0054] Furthermore, the intent-aware module includes:
[0055] The text intent parsing unit is used to perform semantic analysis on text data, extract key information, and determine user intent;
[0056] The multimodal intent parsing unit is used to perform intent parsing by combining image descriptions and text information, as well as by combining audio-transcribed text.
[0057] Furthermore, the system also includes a multimodal data processing and fusion module, used for feature extraction and fusion of the multimodal data, specifically including:
[0058] The feature extraction unit is used to extract features from text, image, and audio data respectively.
[0059] The feature fusion unit is used to fuse features from different modalities through a deep learning model to achieve comprehensive semantic understanding.
[0060] Compared with the prior art, the beneficial effects of the present invention are:
[0061] (1) Realize user intent perception and dynamic adjustment: By parsing user multimodal input through a large language model, accurately identify query intent, and dynamically adjust keyword extraction strategy based on intent, significantly improving the relevance of keywords to user needs.
[0062] (2) Enhance multimodal data processing capabilities: Support the fusion processing and feature extraction of multimodal data such as text, images, and audio, and achieve unified semantic understanding through cross-modal semantic matching to overcome the limitations of single-modal processing.
[0063] (3) Enhance the accuracy and adaptability of keyword extraction: Combine contextual information and user history behavior to optimize extraction parameters and methods in real time, and accurately extract representative keywords in professional fields and complex contexts.
[0064] (4) Provide personalized keyword extraction service: Adaptively adjust the output results according to user preferences and usage history to achieve personalized customization and effectively improve the user experience in information retrieval and analysis.
[0065] (5) Optimize system computation efficiency: By optimizing the model structure and algorithm, the processing speed is improved while ensuring accuracy, which is suitable for large-scale data and real-time application scenarios. Attached Figure Description
[0066] Figure 1 This is a diagram showing the overall architecture of the system of the present invention. Detailed Implementation
[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0068] Please see Figure 1 This demonstrates the overall architecture of the system of the present invention. The system adopts a modular design and mainly includes a user input module, an intent perception module, a dynamic adjustment module, a keyword extraction module, a multimodal data fusion module, and an output module. These modules work together to form a complete processing pipeline from multimodal data input to accurate keyword output.
[0069] Specifically, the implementation steps of the keyword extraction method based on knowledge graphs and user intent awareness are as follows:
[0070] Step S101: Receive and preprocess multimodal data
[0071] Users input multimodal data through the system interface, which can be one or more combinations of text, images, and audio.
[0072] Text data: Received directly and subjected to basic preprocessing, such as word segmentation and stop word removal.
[0073] Image data: Visual features are extracted using convolutional neural networks (CNNs, such as ResNet), and the features are converted into descriptive text using image description generation models (such as VLP models based on Transformer).
[0074] Audio data: Transcribed into text using an Automatic Speech Recognition (ASR) engine.
[0075] All modal data are ultimately represented uniformly as text or embedded vectors that can be processed by large language models.
[0076] Step S102: User Intent Perception and Analysis
[0077] This step is performed by the intent-aware module, which at its core utilizes large language models (such as GPT-4, BERT, etc.) to understand the deeper purpose of the user's input.
[0078] For plain text input, the large language model directly performs semantic analysis to identify the core query entities and intent. For example, if a user inputs "concrete curing standards for plateau railway construction", the model will identify the domain ("plateau railway construction"), the core object ("concrete"), and the requirement ("curing standards").
[0079] For multimodal inputs, such as when a user uploads a bridge construction drawing with the text "risk points of this structure", the system first converts the image into descriptive text (e.g., "The image shows a large cable-stayed bridge undergoing deck hoisting operations"). Then, this descriptive text is concatenated with the original query text and input into a large language model for joint intent parsing. Finally, the system identifies that the user's intent is to query "safety risks of cable-stayed bridge deck hoisting construction".
[0080] Step S103: Dynamically adjust keyword extraction strategy
[0081] This step is performed by the dynamic adjustment module, which performs quantitative dynamic adjustments based on the user intent parsed in step S102. Specifically, it includes the following calculations:
[0082] Calculation Intent - Feature Adjustment Weights (α):
[0083] The user intent vector I (output by the intent perception module) and the multimodal feature vector F (output by the data fusion module) are compared using a cosine similarity calculation, with the following formula:
[0084]
[0085] The weight α is used to measure the degree of matching between the current multimodal data features and the user's intent. The higher the α value, the more the subsequent keyword extraction will focus on features that are highly related to the intent.
[0086] Calculate the personalized adjustment factor (β):
[0087] The system retrieves the user's historical behavior vector H (constructed based on historical queries, clicks, etc.), performs a dot product operation with the current intent vector I, and then normalizes it using the Sigmoid function. The formula is as follows:
[0088]
[0089] Here, γ is an adjustment parameter used to control the strength of the personalized influence. The closer the β value is to 1, the more the system tends to adjust keyword extraction according to the user's personalized preferences.
[0090] Step S104: Multimodal feature fusion and keyword extraction
[0091] This step is completed jointly by the multimodal data fusion module and the keyword extraction module.
[0092] Feature fusion: Feature vectors from text, image descriptions, and audio transcribed text are deeply fused using an attention-based Transformer model to generate a unified, semantically rich, multimodal fused feature vector. .
[0093] Probabilistic keyword extraction:
[0094] Keyword extraction model M (a pre-trained classification or sequence labeling model) is based on fused features. Calculate the extraction probability of each keyword k in the candidate keyword set K. The calculation formula is:
[0095]
[0096] This formula uses the Softmax function to transform the model's keyword scoring into a probability distribution. Simultaneously, during the calculation process, the adjusted weights α and personalization coefficients β obtained in step S103 will affect the scoring process or final probability ranking of model M, thereby achieving dynamic and personalized adjustment of the extraction strategy.
[0097] Step S105: Output keywords
[0098] The output module presents the final extracted, probability-sorted keyword list to the user in a clear and visual manner. For example, for the query about bridge construction mentioned above, the system may output a series of highly relevant keywords that match the user's intent, such as "bridge deck hoisting, cable tensioning, high-altitude operations, temporary support, and wind load impact."
[0099] Corresponding to the above method, a specific system implementation scheme is provided. The functions and interaction relationships of each module of the system are as follows: Figure 1 As shown.
[0100] User input module: Implemented in the form of a graphical user interface (GUI) or application programming interface (API), responsible for receiving various types of data uploaded by users.
[0101] Intent awareness module: Deployed with large language models (such as ChatGLM, ERNIE, etc.), it completes the intent parsing of input data by calling the model API or local inference service.
[0102] Dynamic adjustment module: As a microservice, it has the built-in weight calculation unit and historical behavior adjustment unit to calculate the α and β values in real time based on the input vector.
[0103] Multimodal data fusion module: Integrates computer vision, speech recognition and natural language processing models, and achieves cross-modal feature alignment and fusion through a feature fusion network (such as a multilayer perceptron or Transformer encoder).
[0104] Keyword extraction module: Loads a keyword extraction model M, receives fused feature vectors and adjustment parameters, performs probability calculations and outputs candidate keywords.
[0105] Output module: Returns the results to the user via a web interface, mobile application, or API response.
[0106] The system can be deployed on cloud server clusters, using Docker containerization technology to encapsulate each module as an independent service, and orchestrating and managing it through Kubernetes to ensure high availability and scalability.
[0107] The following is an example of a specific practical application:
[0108] To make the technical solution of this invention clearer and easier to understand, a specific scenario of bridge construction safety risk analysis is used as an example to illustrate the implementation process of this method.
[0109] Suppose a bridge engineer needs to prepare a report on the "Construction Risks of Cantilever Cast-in-Place Methods". He has a photo of the construction site and a short audio recording of the site. He uses this system for auxiliary analysis, inputting the following multimodal data:
[0110] Text query: "Analyze the main safety risks of cantilever casting construction".
[0111] An image shows bridge piers undergoing cantilever casting, with formwork already erected and workers operating equipment.
[0112] An audio recording: a voice record of the on-site supervisor mentioning, "...the wind is quite strong today, pay attention to the stability of the hanging basket...".
[0113] The system processes the data according to the following steps:
[0114] Step 1: Multimodal data reception and preprocessing
[0115] The system receives text, image, and audio data.
[0116] The image is processed and a descriptive text is generated using a computer vision model: "The image shows the pier section of a large bridge, where concrete cantilever pouring is underway. The formwork system is in place, and there are construction workers on site."
[0117] The audio is processed and transcribed into text using a speech recognition model: "The on-site supervisor reminds us that the wind is strong today, so we need to pay special attention to the stability of the hanging basket."
[0118] Step Two: User Intent Analysis
[0119] The intent-aware module inputs the original text query, image description text, and audio transcribed text into the large language model.
[0120] After comprehensive understanding by the large language model, the user's complex intent is accurately parsed: the user wants to identify the key safety risk factors related to the "hanging basket" under the specific construction environment (cantilever casting, windy weather).
[0121] Step 3: Dynamically adjust keyword extraction strategy
[0122] The dynamic adjustment module performs quantitative calculations based on the above intentions.
[0123] Calculations revealed that concepts such as "hanging basket" and "wind power" are highly correlated with current multimodal features, and therefore are given a high intent-feature adjustment weight (α).
[0124] Meanwhile, the system found that the engineer had repeatedly searched for questions related to "high-altitude operations" and "temporary structures" in his history, so it enhanced the weight of such keywords by adjusting the coefficient (β) in a personalized manner.
[0125] Step 4: Multimodal Feature Fusion and Keyword Extraction
[0126] The system integrates the text "cantilever casting construction", the image description "hanging basket system" and the audio transcription "strong wind, hanging basket stability" to form a unified semantic representation that includes environmental information (wind), core construction method (cantilever casting) and key equipment (hanging basket).
[0127] Based on this fusion feature, the keyword extraction model calculates the probability of each candidate keyword using a dynamically adjusted strategy. Ultimately, the system prioritizes extracting the keywords that best match the current scenario and user intent.
[0128] Step 5: Output Results
[0129] The system outputs the final keyword list to the engineer, for example:
[0130] Hanging basket stability, wind load impact, anti-overturning measures, construction load, temporary anchoring, deflection control
[0131] Case Study Results Analysis:
[0132] The advantages of this invention are fully demonstrated in this case compared to traditional methods:
[0133] Traditional text-based queries may only return general risk keywords such as "fall from height" and "object strike." This invention, by fusing image information, precisely links to the core device, the "hanging basket"; and by fusing audio information, it captures the key environmental factor of "wind force." The resulting extracted keywords are more targeted, contextualized, and actionable, directly assisting engineers in conducting accurate risk assessments and writing reports, significantly improving the efficiency and accuracy of information retrieval.
[0134] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A keyword extraction method based on a knowledge graph and user intent perception, characterized by, Includes the following steps: Step S1: Receive multimodal data input by the user, wherein the multimodal data includes at least one of text, image, and audio; Step S2: Use a large language model to perform intent perception on the multimodal data, parse and extract the user's search intent; Step S3: Dynamically adjust the keyword extraction strategy based on the user intent; Step S4: Extract keywords from the multimodal data based on the dynamically adjusted strategy; Step S5: Output and display the extracted keywords to the user. 2.The knowledge graph and user intent perception based keyword extraction method according to claim 1, characterized in that, The steps for dynamically adjusting the keyword extraction strategy include: Step S31: Calculate the adjustment weight α using the user intent vector I and the multimodal feature vector F. The formula for calculating the adjustment weight is: Where I represents the user intent vector, F represents the multimodal feature vector, and α represents the adjustment weight, which is used to dynamically adjust the weight allocation of the keyword extraction strategy. Step S32: Calculate the personalized adjustment coefficient β using the user's historical behavior vector H and current intent vector I. The formula for calculating the personalized adjustment coefficient is: Where H represents the user's historical behavior vector, γ is an adjustment parameter, and β represents the personalized adjustment coefficient, which is used to adjust the keyword extraction strategy based on the user's historical behavior. 3.The knowledge graph and user intent perception based keyword extraction method of claim 1, wherein, The step of extracting keywords from the multimodal data includes: Utilize the feature vector after multimodal fusion And the keyword extraction model M calculates the keyword extraction probability The calculation formula of the keyword extraction probability is: in, This represents the fused multimodal feature vector. This represents the score of the keyword extraction model for keyword k, where K represents the set of candidate keywords. This represents the extraction probability of keyword k.
4. The keyword extraction method based on knowledge graph and user intent perception according to claim 1, characterized in that, The following processing methods are used for various types of multimodal data: For text data, semantic analysis is performed using a large language model to extract key information and determine user intent. For image data, computer vision technology is used to extract image features and convert them into descriptive text, which is then combined with text information for intent analysis. For audio data, speech recognition technology is used to convert the audio into text, and then a large language model is used for intent parsing.
5. The keyword extraction method based on knowledge graph and user intent perception according to claim 1, characterized in that, The method further includes a step of feature extraction and fusion of the multimodal data, specifically including: Natural language processing techniques are used to extract features from text data; Computer vision techniques are used to extract features from image data and then transform them into descriptive text. Speech recognition technology is used to extract text information from audio data; By fusing features from different modalities through deep learning models, comprehensive semantic understanding can be achieved.
6. A keyword extraction system based on a large-scale language model of engineering domain knowledge graph and user intent perception and dynamic adjustment, characterized in that, include: The user input module is used to receive multimodal data input by the user; The intent-aware module is used to use a large language model to perceive the intent of the multimodal data, parse and extract the user's search intent; The dynamic adjustment module is used to dynamically adjust the keyword extraction strategy based on the user's intent. The keyword extraction module is used to extract keywords from the multimodal data based on a dynamically adjusted strategy; The output module is used to display the extracted keywords to the user.
7. The keyword extraction system based on engineering domain knowledge graph and user intent perception and dynamic adjustment according to claim 6, characterized in that, The dynamic adjustment module includes: The weight calculation unit is used to calculate the adjustment weight α using the user intent vector I and the multimodal feature vector F. The formula for calculating the adjustment weight is as follows: , The historical behavior adjustment unit is used to calculate the personalized adjustment coefficient β using the user's historical behavior vector H and current intent vector I. The formula for calculating the personalized adjustment coefficient is as follows: .
8. The keyword extraction system based on engineering domain knowledge graph and user intent perception and dynamic adjustment according to claim 6, characterized in that, The keyword extraction module includes a probability calculation unit, used to calculate the keyword extraction probability P(k) using the multimodal fusion feature vector Ffused and the keyword extraction model M. The formula for calculating the keyword extraction probability is: .
9. The keyword extraction system based on engineering domain knowledge graph and user intent perception and dynamic adjustment according to claim 6, characterized in that, The intent-aware module includes: The text intent parsing unit is used to perform semantic analysis on text data, extract key information, and determine user intent; The multimodal intent parsing unit is used to perform intent parsing by combining image descriptions and text information, as well as by combining audio-transcribed text.
10. The system according to claim 6, characterized in that, The system also includes a multimodal data processing and fusion module, used for feature extraction and fusion of the multimodal data, specifically including: The feature extraction unit is used to extract features from text, image, and audio data respectively. The feature fusion unit is used to fuse features from different modalities through a deep learning model to achieve comprehensive semantic understanding.