Multimodal dialogue intent recognition method, system, device and medium

Abstract

The application discloses a multi-modal dialogue intention recognition method, system, device and medium, and belongs to the technical field of artificial intelligence and natural language processing, and aims to solve the technical problem of how to overcome the defects of single mode, limited deployment resources and difficult domain migration in the prior art, and improve the service capability and user experience of an intelligent dialogue system in a real scene, and the technical scheme is as follows: acquiring and preprocessing multi-modal data: acquiring query data input by a user, the query data comprising text modal data, speech modal data and / or image modal data, preprocessing the text modal data, the speech modal data and the image modal data by using a pre-trained Transformer encoder, a pre-trained speech encoder and a pre-trained visual encoder respectively, and extracting a text semantic embedding vector, a speech semantic embedding vector and an image semantic embedding vector; cross-modal alignment; lightweight intention recognition based on knowledge distillation; domain adaptive fine-tuning; and acquiring an intention category.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing technology, specifically to a method, system, device, and medium for multimodal dialogue intent recognition. Background Technology

[0002] User intent recognition is the core foundation of intelligent dialogue systems, widely used in scenarios such as intelligent customer service and virtual assistants. However, existing technologies still face numerous challenges in open-domain real-world dialogue scenarios, including limited modality, semantic ambiguity, data scarcity, and deployment constraints. Traditional intent recognition methods rely on rule templates or traditional machine learning classifiers, using keywords, sentence structure rules, and shallow semantic features such as TF-IDF and n-grams, combined with SVM and Naive Bayes models to complete intent matching, which has significant shortcomings. This method has weak generalization ability, heavily relies on manual rules and feature engineering, cannot adapt to diverse user expressions, performs poorly in recognizing novel and low-frequency intents, and has high operation and maintenance expansion costs; it can only achieve shallow semantic understanding, making it difficult to capture deep semantics and contextual relationships, and cannot accurately identify ambiguous or complex intents with long contexts; moreover, it only supports single-modal text input and cannot use multimodal information such as speech and images to eliminate semantic ambiguity. While end-to-end deep learning methods based on pre-trained language models have improved semantic understanding capabilities, they still have inherent defects. Most models are designed in a single modality, which has an information bottleneck and makes it difficult to resolve semantic ambiguity by relying on multimodal information. Large-scale pre-trained models have a large number of parameters, and when deployed on edge devices or in real-time scenarios, they suffer from high inference latency, large memory and energy consumption. General pre-trained models have insufficient domain transfer capabilities, and their accuracy drops significantly when adapted to vertical domains. Moreover, the cost of obtaining high-quality domain-labeled data is extremely high.

[0003] While knowledge distillation can transfer the capabilities of large models to lightweight models and alleviate deployment pressure, it still has shortcomings. Existing distillation solutions are suitable for pure text scenarios, but are prone to information loss in multimodal scenarios and cannot transfer cross-modal semantic alignment capabilities; distilled models lack domain adaptation mechanisms, have low fine-tuning efficiency, and are prone to overfitting; most multimodal fusion methods are simple and cannot achieve deep semantic alignment and complementarity across multiple modalities, resulting in limited information gain.

[0004] Therefore, how to overcome the shortcomings of existing technologies, such as single modality, limited deployment resources, and difficulty in domain migration, and improve the service capabilities and user experience of intelligent dialogue systems in real-world scenarios is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] The technical objective of this invention is to provide a method, system, device, and medium based on multimodal dialogue intent recognition to address the shortcomings of existing technologies, such as single modality, limited deployment resources, and difficulty in domain migration, thereby improving the service capabilities and user experience of intelligent dialogue systems in real-world scenarios.

[0006] The technical objective of this invention is achieved as follows: a method for multimodal dialogue intent recognition, the specific details of which are as follows: Acquire and preprocess multimodal data: Acquire query data input by the user, including text modal data, speech modal data and / or image modal data, and preprocess the text modal data, speech modal data and image modal data using a pre-trained Transformer encoder, a pre-trained speech encoder and a pre-trained visual encoder respectively, and extract text semantic embedding vectors, speech semantic embedding vectors and image semantic embedding vectors; Cross-modal alignment: By using cross-modal contrastive learning, text semantic embedding vectors, speech semantic embedding vectors, and image semantic embedding vectors are mapped to a unified semantic space to obtain a unified semantic representation after each modality is aligned. The unified semantic representations after each modality is aligned are then concatenated and weighted and fused through an attention mechanism to generate multimodal fusion features, thereby achieving multimodal semantic alignment. Lightweight Intent Recognition Based on Knowledge Distillation: The teacher's large language model is used to understand the intent of the fused features after multimodal semantic alignment, generate soft label prediction distribution, and the intent understanding ability of the teacher's large language model is distilled into the student intent recognition model through the knowledge distillation loss function. Domain-adaptive fine-tuning: Freeze all original weights of the student intent recognition model, use the LoRA method to perform domain-adaptive fine-tuning of the student intent recognition model based on labeled samples of the target application domain, update the parameters of the student intent recognition model, and obtain the fine-tuned student intent recognition model to adapt it to the intent recognition task of the target domain. Obtain Intent Category: Input the user query data into the fine-tuned student intent recognition model to obtain the intent category label corresponding to the user query.

[0007] As a preferred method, the preprocessing of multimodal data is as follows: Preprocessing text modal data: The user-input text modal data (text) is segmented and stop word removed to obtain preprocessed text modal data (text). Then, the preprocessed text modal data (text) is converted into text embedding vectors using the pre-trained word embedding model BERT-base-Chinese. ;in, Indicates the length of the text sequence; Indicates the dimension of the text embedding vector; Represents text modal data; Preprocessing of speech modal data: When user query data contains speech modal data, the speech modal data is framed, windowed, and subjected to Fourier transform. Mel-frequency cepstral coefficient features are extracted to obtain preprocessed speech modal data. The preprocessed speech modal data is then converted into speech semantic embedding vectors using the Hubert-base speech encoder. ;in, Indicates the number of audio frames; The dimension of the speech semantic embedding vector; Represents speech modal data; Preprocessing image modal data: When a user query contains image modal data, the image modal data is normalized in size and converted in color space to obtain preprocessed image modal data. This preprocessed image modal data is then converted into image semantic embedding vectors using the ViT-B / 16 image encoder. ;in, The number of spatial locations representing image features; The dimension of the image semantic embedding vector; This represents image modal data.

[0008] More preferably, cross-modal alignment is as follows: Unified Semantic Space Mapping: Through text mapping networks Voice mapping network and image mapping network Map the text semantic embedding vector, speech semantic embedding vector, and image semantic embedding vector to dimensions respectively. d A unified semantic space is used to obtain the unified semantic representation after modality alignment; the formula is as follows: ; ; ; in, , and These represent the aligned unified semantic representation of text, the aligned unified semantic representation of speech, and the aligned unified semantic representation of image, respectively. This indicates that the pre-trained Transformer encoder (BERT-base) is followed by a projection head. ; This indicates that a pre-trained speech encoder (wav2vec 2.0) is connected to a projection head. ; This indicates that the pre-trained visual encoder (ViT-B / 16) is followed by the projection head ViT ( ); Constructing positive and negative sample pairs: Aligned unified semantic representations of text, speech, and images from the same user query are paired across modalities to construct a set of positive sample pairs. The aligned unified semantic representations of text, speech, and images from different user queries are paired to construct a set of negative sample pairs. ; Calculate the cross-modal contrastive loss: Use the cross-modal NCE loss function for contrastive learning optimization, as shown in the following formula: ; in, , and Both represent local index variables, used to distinguish semantic representations of different modalities and different elements in sample pairs; Modal semantic representation and Cosine similarity between them; Indicates temperature hyperparameter; Represents the set of positive sample pairs; Represents the set of negative sample pairs; Generating multimodal fusion features: The aligned unified semantic representations of text, speech, and image are concatenated and then fused using attention-weighted methods to generate multimodal fusion features, as shown in the following formula: ; Among them, attention weight ; Representing modes The learnable attention parameter; T represents transpose; This represents the multimodal fusion feature.

[0009] More specifically, lightweight intent recognition based on knowledge distillation is as follows: Generating soft labels for the teacher's large language model: Multimodal feature fusion using the teacher's large language model DeepSeek-V4-Pro Process to obtain intent category Soft label probability distribution (including but not limited to any one or more of the following: fact lookup, causal analysis, methodological guidance, comparative evaluation, sentiment expression, operation request, and information confirmation) ;in, and These represent the weight matrix and bias vector of the classification layer in the teacher's large language model, respectively. This represents the high-level semantic features extracted by the teacher's large language model; The hidden layer dimension represents the teacher's large language model; K The total number of intent categories, ranging from 10 to 100; the Softmax function is defined as follows: ; The original score vector (logits) representing the output of the classification layer is a K-dimensional real number vector; The original score vector representing the output of the classification layer. The element corresponding to the category index c is the original score of the classification layer predicting that the input sample belongs to the c-th intention category; This represents the k-th element in the original score vector x output by the classification layer, where k is the summation index variable. It iterates through all intent categories and is used for normalization calculation of the denominator. c Indicates a category index; Training and distilling the student intent recognition model: The student intent recognition model is a lightweight neural network with fewer than 1% of the parameters of the teacher model, employing a 6-layer Transformer encoder architecture (hidden layer dimension...). =384, number of attention heads =6, Feedforward Neural Network (This refers to the use of a student intent recognition model to fuse multimodal features.) Perform encoding processing to obtain the hard label prediction distribution. ;in, and These represent the weight matrix and bias vector of the classification layer in the student intent recognition model, respectively. This represents the high-level semantic features extracted by the student intent recognition model; The hidden layer dimension represents the dimension of the student intent recognition model, i.e., the dimension of the feature vector obtained after encoding each word vector in the Transformer encoder of the student intent recognition model; student represents the student intent recognition model; the total loss function of knowledge distillation is defined as: ;in, ; This represents the hard-label cross-entropy loss. Intent category in one-hot encoding of true intent label c The value corresponding to dimension (the c-th category in C); The student intent recognition model predicts the first... c The probability of each category; ; This represents the soft label distillation loss (i.e., KL divergence). The teacher's large language model predicts the first c The probability of each category; Balance coefficient (default) ).

[0010] More preferably, the domain-adaptive fine-tuning is as follows: Obtain domain-specific labeled samples: Obtain a small set of labeled samples from the target application domain. ;in, Indicates the first Text queries for each sample; This represents the corresponding multimodal auxiliary data, namely, speech modal data or image modal data; ∈{1,2,…,K} represents the true intent label; Indicates the number of labeled samples ( N ≥100); Initialize low-rank adaptation parameters: Freeze all original weight parameters of the student intent recognition model; for each weight matrix in the student intent recognition model that needs fine-tuning. (including multi-head attention layers) , , , and feedforward network layer , Introducing trainable low-rank decomposition matrices and Where, rank r ≪ min(d, k) (default) r =8), and the low-rank decomposition matrix A is initialized using a random Gaussian distribution. ; Initialize to a zero matrix; Forward propagation and fine-tuning optimization: During fine-tuning, the original weight matrix W remains frozen, and only the low-rank matrices A and B are updated. The modified weight matrix is ​​as follows: Label the sample set in the target application domain. To minimize the cross-entropy loss, the formula is as follows: ; in, This indicates an indicator function that takes a value of 1 when the condition is true and 0 otherwise; and it uses the Adam optimizer with a learning rate of... Update the low-rank parameters A and B, set the batch size to 32, and the number of training epochs to 50.

[0011] More preferably, the intent categories are as follows: Student intent recognition model intent inference: Input user query data into the penultimate layer of the fine-tuned student intent recognition model to obtain the high-level semantic feature vector extracted by the student intent recognition model. ;in, =384 is the hidden layer dimension of the student intent recognition model; then the high-level semantic feature vector Input the student intent recognition model's classification layer, and calculate the probability distribution of user queries belonging to each intent category using the Softmax function. The formula is as follows: ;in, and These represent the weight matrix and bias vector of the classification layer in the fine-tuned student intent recognition model, respectively; K represents the total number of intent categories; the Softmax function is defined as... ; c∈{1,2,…,K} is the category index, exp( ) represents an exponential function. Let be a probability distribution vector, satisfying ; Calculate prediction confidence: Calculate the confidence score Conf of the student intent recognition model for the predicted results of the user query, using the following formula: , that is, the maximum value in the probability distribution, which takes the range [1 / K, 1]; Intent category decision: Decisions are made using a confidence threshold δ∈[0.5,0.9]. Intent category mapping: Mapping intent category labels Mapped to a set of preset intent categories The corresponding intent name It is output in a structured data format; wherein the structured data format contains at least an intent category identifier. Intent Category Name The confidence score (Conf) and the original content of the query text.

[0012] More preferably, the decision is made using a confidence threshold δ∈[0.5,0.9] as follows: If the confidence score Conf ≥ δ, then output the intent category label. It also returns the corresponding intent category name and confidence score; If the confidence score Conf < δ, it is determined to be an unrecognizable intent. The "Uncertain Intent" flag and the top T candidate intents with the highest probability distribution (T is a preset positive integer, T≤3) are returned for the user to confirm or ask clarification questions.

[0013] A multimodal dialogue intent recognition system is provided, which implements the multimodal dialogue intent recognition method described above; the system includes: The acquisition and preprocessing module is used to acquire query data input by the user, including text modal data, speech modal data and / or image modal data. It uses a pre-trained Transformer encoder, a pre-trained speech encoder and a pre-trained visual encoder to preprocess the text modal data, speech modal data and image modal data respectively, and extracts text semantic embedding vectors, speech semantic embedding vectors and image semantic embedding vectors. The cross-modal alignment module is used to map text semantic embedding vectors, speech semantic embedding vectors and image semantic embedding vectors to a unified semantic space through cross-modal contrastive learning, obtain a unified semantic representation after each modality alignment, and then concatenate the unified semantic representations after each modality alignment and then perform weighted fusion through an attention mechanism to generate multimodal fusion features, thereby achieving multimodal semantic alignment. The lightweight intent recognition module is used to understand the intent of the fused features after multimodal semantic alignment by the teacher's big language model, generate soft label prediction distribution, and distill the intent understanding ability of the teacher's big language model into the student intent recognition model through the knowledge distillation loss function. The domain adaptive fine-tuning module is used to freeze all the original weights of the student intent recognition model, and use the LoRA method to perform domain adaptive fine-tuning of the student intent recognition model based on the labeled samples of the target application domain, update the parameters of the student intent recognition model, and obtain the fine-tuned student intent recognition model to adapt it to the intent recognition task of the target domain. The intent category acquisition module is used to input user query data into the fine-tuned student intent recognition model and obtain the intent category label corresponding to the user query.

[0014] An electronic device includes: a memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the multimodal dialogue intent recognition method as described above.

[0015] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the multimodal dialogue intent recognition method described above.

[0016] The multimodal dialogue intent recognition method, system, device, and medium of the present invention have the following advantages: (i) This invention introduces multimodal data (text, speech, image) and a cross-modal contrastive learning mechanism to map the semantic features of different modalities to a unified semantic space, thereby achieving deep semantic alignment and complementarity of multimodal information. Compared with traditional methods that rely on a single text modality, it significantly improves the accuracy and robustness of intent recognition in cases of semantic ambiguity and incomplete information. (ii) This invention designs a knowledge distillation framework from a large teacher language model (>10B parameters) to a lightweight student intent recognition model (<1% of the teacher model's parameters), which efficiently transfers the powerful semantic understanding capabilities of the large model to a lightweight network. While maintaining high recognition accuracy, it significantly reduces inference latency and memory usage, solving the practical problem of difficult deployment of large models in resource-constrained scenarios. (III) By introducing a LoRA-based efficient parameter fine-tuning mechanism, this invention can achieve rapid adaptation of the student model to the target application domain by only updating a very small number of low-rank adaptation parameters. Compared with full parameter fine-tuning, it greatly reduces computational overhead and storage requirements, while effectively avoiding overfitting problems in small sample scenarios and significantly improving the model's generalization ability in vertical domains. (iv) This invention constructs a complete technical chain from multimodal coding, cross-modal alignment, knowledge distillation to efficient parameter fine-tuning. Experimental results show that this invention achieves significant performance improvement on standard intent recognition benchmark datasets compared to baseline models (such as plain text BERT and plain text distillation schemes). At the same time, it achieves significant optimization in terms of inference speed and model size, providing an intent recognition solution with both high accuracy and high deployment efficiency for high-requirement scenarios such as intelligent customer service and multimodal interaction. (V) This invention can integrate multimodal information, utilize large model knowledge distillation to achieve lightweight deployment, and possess robust intent recognition with rapid domain adaptation capabilities. It solves the core problems of single modality, limited deployment resources, and difficulty in domain migration in the prior art, improves the service capabilities and user experience of intelligent dialogue systems in real and complex scenarios, and has the advantages of strong practicality and high accuracy. It is a comprehensive technical solution that integrates multimodal semantic encoding, cross-modal comparison and alignment, large model knowledge distillation, and efficient domain adaptation of parameters. It supports high-performance multimodal dialogue intent recognition that can be quickly adapted to new domains under limited computing resources. Attached Figure Description

[0017] The invention will be further described below with reference to the accompanying drawings.

[0018] Appendix Figure 1 This is a flowchart of a multimodal dialogue intent recognition method. Detailed Implementation

[0019] The following detailed description of the multimodal dialogue intent recognition method, system, device, and medium of the present invention is provided with reference to the accompanying drawings and specific embodiments.

[0020] Example 1: As shown in the attached document Figure 1 As shown in the figure, this embodiment provides a method for multimodal dialogue intent recognition, which is as follows: S1. Acquire and preprocess multimodal data: Acquire query data input by the user, including text modal data, speech modal data and / or image modal data, and preprocess the text modal data, speech modal data and image modal data using a pre-trained Transformer encoder, a pre-trained speech encoder and a pre-trained visual encoder respectively, and extract text semantic embedding vectors, speech semantic embedding vectors and image semantic embedding vectors. S2, Cross-modal alignment: By using cross-modal contrastive learning, text semantic embedding vectors, speech semantic embedding vectors, and image semantic embedding vectors are mapped to a unified semantic space to obtain a unified semantic representation after each modality is aligned. The unified semantic representations after each modality is aligned are then concatenated and weighted and fused through an attention mechanism to generate multimodal fusion features, thereby achieving multimodal semantic alignment. S3. Lightweight Intent Recognition Based on Knowledge Distillation: The teacher's large language model is used to understand the intent of the fused features after multimodal semantic alignment, generate soft label prediction distribution, and the intent understanding ability of the teacher's large language model is distilled into the student intent recognition model through the knowledge distillation loss function. S4. Domain Adaptive Fine-tuning: Freeze all original weights of the student intent recognition model, use the LoRA method to perform domain adaptive fine-tuning of the student intent recognition model based on labeled samples of the target application domain, update the parameters of the student intent recognition model, and obtain the fine-tuned student intent recognition model to adapt it to the intent recognition task of the target domain. S5. Obtain Intent Category: Input the user query data into the fine-tuned student intent recognition model to obtain the intent category label corresponding to the user query.

[0021] The preprocessing of multimodal data in step S1 of this embodiment is as follows: S101. Preprocessing Text Modality Data: The user-input text modality data (text) is segmented and stop word removed to obtain preprocessed text modality data (text). The preprocessed text modality data (text) is then converted into text embedding vectors using the pre-trained word embedding model BERT-base-Chinese. ;in, Indicates the length of the text sequence; Indicates the dimension of the text embedding vector; Represents text modal data; S102. Preprocessing Speech Modal Data: When the user query data contains speech modal data, the speech modal data is framed, windowed, and subjected to Fourier transform. Mel-frequency cepstral coefficient features are extracted to obtain preprocessed speech modal data. The preprocessed speech modal data is then converted into speech semantic embedding vectors using the Hubert-base speech encoder. ;in, Indicates the number of audio frames; The dimension of the speech semantic embedding vector; Represents speech modal data; S103. Preprocessing Image Modal Data: When the user query data contains image modal data, the image modal data is normalized in size and converted in color space to obtain preprocessed image modal data. The preprocessed image modal data is then converted into image semantic embedding vectors by the image encoder ViT-B / 16. ;in, The number of spatial locations representing image features; The dimension of the image semantic embedding vector; This represents image modal data.

[0022] The cross-modal alignment in step S2 of this embodiment is as follows: S201, Unified Semantic Space Mapping: This is achieved through text mapping networks. Voice mapping network and image mapping network Map the text semantic embedding vector, speech semantic embedding vector, and image semantic embedding vector to dimensions respectively. d A unified semantic space is used to obtain the unified semantic representation after modality alignment; the formula is as follows: ; ; ; in, , and These represent the aligned unified semantic representation of text, the aligned unified semantic representation of speech, and the aligned unified semantic representation of image, respectively. This indicates that the pre-trained Transformer encoder (BERT-base) is followed by a projection head. ; This indicates that a pre-trained speech encoder (wav2vec 2.0) is connected to a projection head. ; This indicates that the pre-trained visual encoder (ViT-B / 16) is followed by the projection head ViT ( ); S202. Constructing positive and negative sample pairs: Pairing aligned text unified semantic representations, aligned speech unified semantic representations, and aligned image unified semantic representations of the same user query across modalities to construct a set of positive sample pairs. The aligned unified semantic representations of text, speech, and images from different user queries are paired to construct a set of negative sample pairs. ; S203. Calculate the cross-modal contrastive loss: Use the cross-modal NCE loss function for contrastive learning optimization, as shown in the following formula: ; in, , and Both represent local index variables, used to distinguish semantic representations of different modalities and different elements in sample pairs; Modal semantic representation and Cosine similarity between them; Indicates temperature hyperparameter; Represents the set of positive sample pairs; Represents the set of negative sample pairs; S204. Generating Multimodal Fusion Features: The aligned unified semantic representations of text, speech, and image are concatenated and then fused using attention weighting to generate multimodal fusion features, as shown in the following formula: ; Among them, attention weight ; Representing modes The learnable attention parameter; T represents transpose; This represents the multimodal fusion feature.

[0023] The lightweight intent recognition based on knowledge distillation in step S3 of this embodiment is as follows: S301. Generate soft tags for the teacher's large language model: Use the teacher's large language model DeepSeek-V4-Pro to fuse multimodal features. Process to obtain intent category Soft label probability distribution (including but not limited to any one or more of the following: fact lookup, causal analysis, methodological guidance, comparative evaluation, sentiment expression, operation request, and information confirmation) ;in, and These represent the weight matrix and bias vector of the classification layer in the teacher's large language model, respectively. This represents the high-level semantic features extracted by the teacher's large language model; The hidden layer dimension represents the teacher's large language model; K The total number of intent categories, ranging from 10 to 100; the Softmax function is defined as follows: ; The original score vector (logits) representing the output of the classification layer is a K-dimensional real number vector; The original score vector representing the output of the classification layer. The element corresponding to the category index c is the original score of the classification layer predicting that the input sample belongs to the c-th intention category; This represents the k-th element in the original score vector x output by the classification layer, where k is the summation index variable. It iterates through all intent categories and is used for normalization calculation of the denominator. c Indicates a category index; S302. Training and Distilling the Student Intent Recognition Model: The student intent recognition model is a lightweight neural network with fewer than 1% of the parameters of the teacher model, employing a 6-layer Transformer encoder architecture (hidden layer dimension...). =384, number of attention heads =6, Feedforward Neural Network (This refers to the use of a student intent recognition model to fuse multimodal features.) Perform encoding processing to obtain the hard label prediction distribution. ;in, and These represent the weight matrix and bias vector of the classification layer in the student intent recognition model, respectively. This represents the high-level semantic features extracted by the student intent recognition model; The hidden layer dimension represents the dimension of the student intent recognition model, i.e., the dimension of the feature vector obtained after encoding each word vector in the Transformer encoder of the student intent recognition model; student represents the student intent recognition model; the total loss function of knowledge distillation is defined as: ;in, ; This represents the hard-label cross-entropy loss. Intent category in one-hot encoding of true intent label c The value corresponding to dimension (the c-th category in C); The student intent recognition model predicts the first... c The probability of each category; ; This represents the soft label distillation loss (i.e., KL divergence). The teacher's large language model predicts the first c The probability of each category; Balance coefficient (default) ).

[0024] The specific details of the neighborhood adaptive fine-tuning in step S4 of this embodiment are as follows: S401. Obtain Domain Labeled Samples: Obtain a small set of labeled samples from the target application domain. ;in, Indicates the first Text queries for each sample; This represents the corresponding multimodal auxiliary data, namely, speech modal data or image modal data; ∈{1,2,…,K} represents the true intent label; Indicates the number of labeled samples ( N ≥100); S402. Initialize low-rank adaptation parameters: Freeze all original weight parameters of the student intent recognition model; for each weight matrix in the student intent recognition model that needs fine-tuning... (including multi-head attention layers) , , , and feedforward network layer , Introducing trainable low-rank decomposition matrices and Where, rank r ≪ min(d, k) (default) r =8), and the low-rank decomposition matrix A is initialized using a random Gaussian distribution. ; Initialize to a zero matrix; S403. Forward Propagation and Fine-Tuning Optimization: During fine-tuning, the original weight matrix W remains frozen, and only the low-rank matrices A and B are updated. The modified weight matrix is: Label the sample set in the target application domain. To minimize the cross-entropy loss, the formula is as follows: ; in, This indicates an indicator function that takes a value of 1 when the condition is true and 0 otherwise; and it uses the Adam optimizer with a learning rate of... Update the low-rank parameters A and B, set the batch size to 32, and the number of training epochs to 50.

[0025] The intent categories in step S5 of this embodiment are as follows: S501, Student Intent Recognition Model Intent Reasoning: Input user query data into the penultimate layer of the fine-tuned student intent recognition model to obtain the high-level semantic feature vector extracted by the student intent recognition model. ;in, =384 is the hidden layer dimension of the student intent recognition model; then the high-level semantic feature vector Input the student intent recognition model's classification layer, and calculate the probability distribution of user queries belonging to each intent category using the Softmax function. The formula is as follows: ;in, and These represent the weight matrix and bias vector of the classification layer in the fine-tuned student intent recognition model, respectively; K represents the total number of intent categories; the Softmax function is defined as... ; c∈{1,2,…,K} is the category index, exp( ) represents an exponential function. Let be a probability distribution vector, satisfying ; S502. Calculate the prediction confidence score: Calculate the confidence score Conf of the student intent recognition model for the prediction results of the user query, using the following formula: That is, the maximum value in the probability distribution, which takes the value range of [1 / K, 1]. S503, Intent Category Decision: Decision is made using a confidence threshold δ∈[0.5,0.9]. S504, Intent Category Mapping: Mapping intent category labels Mapped to a set of preset intent categories The corresponding intent name It is output in a structured data format; wherein the structured data format contains at least an intent category identifier. Intent Category Name The confidence score (Conf) and the original content of the query text.

[0026] In this embodiment, step S503, which uses a confidence threshold δ∈[0.5,0.9] for decision-making, is as follows: S50301. If the confidence score Conf≥δ, then output the intent category label. It also returns the corresponding intent category name and confidence score; S50302. If the confidence score Conf < δ, it is determined to be an unrecognizable intent. The "Uncertain Intent" flag and the top T candidate intents with the highest probability distribution (T is a preset positive integer, T≤3) are returned for the user to confirm or ask clarification questions.

[0027] This embodiment solves the problem of single-modal semantic ambiguity by introducing multimodal contrastive learning to align multiple modalities such as text, speech, and images into a unified semantic space. It also transfers the intent understanding capability of the teacher's large language model to a lightweight student intent recognition model through large-model knowledge distillation, significantly reducing inference latency and memory usage. Furthermore, it achieves rapid adaptation of the student model to the target domain through efficient LoRA parameter fine-tuning. Compared with existing technologies, this invention significantly improves intent recognition accuracy while achieving lightweight model deployment, combining high accuracy, strong robustness, and efficient inference performance. It is suitable for intelligent application scenarios requiring accurate and robust recognition of user multimodal query intent, such as intelligent customer service, virtual assistants, multimedia information retrieval, and multimodal human-computer interaction. It is particularly suitable for practical application environments with limited training data and computational resources but requiring the deployment of high-performance intent recognition models.

[0028] Example 2: This example provides a multimodal dialogue intent recognition system, which implements the multimodal dialogue intent recognition method as described in Example 1; the system includes: The acquisition and preprocessing module is used to acquire query data input by the user, including text modal data, speech modal data and / or image modal data. It uses a pre-trained Transformer encoder, a pre-trained speech encoder and a pre-trained visual encoder to preprocess the text modal data, speech modal data and image modal data respectively, and extracts text semantic embedding vectors, speech semantic embedding vectors and image semantic embedding vectors. The cross-modal alignment module is used to map text semantic embedding vectors, speech semantic embedding vectors and image semantic embedding vectors to a unified semantic space through cross-modal contrastive learning, obtain a unified semantic representation after each modality alignment, and then concatenate the unified semantic representations after each modality alignment and then perform weighted fusion through an attention mechanism to generate multimodal fusion features, thereby achieving multimodal semantic alignment. The lightweight intent recognition module is used to understand the intent of the fused features after multimodal semantic alignment by the teacher's big language model, generate soft label prediction distribution, and distill the intent understanding ability of the teacher's big language model into the student intent recognition model through the knowledge distillation loss function. The domain adaptive fine-tuning module is used to freeze all the original weights of the student intent recognition model, and use the LoRA method to perform domain adaptive fine-tuning of the student intent recognition model based on the labeled samples of the target application domain, update the parameters of the student intent recognition model, and obtain the fine-tuned student intent recognition model to adapt it to the intent recognition task of the target domain. The intent category acquisition module is used to input user query data into the fine-tuned student intent recognition model and obtain the intent category label corresponding to the user query.

[0029] Example 3: This example provides specific steps for multimodal dialogue intent recognition in an intelligent customer service system, based on Example 1: (1) Multimodal data acquisition and preprocessing: The user submits a query in the intelligent customer service system, and the intelligent customer service system acquires the query data input by the user; in this embodiment, the user uploaded a photo of a rash on his hand, along with the text query "What is this? Do I need to go to the hospital?", as follows: ① Text Modal Data Preprocessing: The intelligent customer service system performs word segmentation and stop word removal on the user-input text query "What is this? Do I need to go to the hospital?"; using a pre-trained BERT-base-Chinese model, the text is encoded into a 768-dimensional text embedding vector sequence, with a maximum sequence length set to 512 (the actual text length is much smaller than this value), resulting in the text embedding vector. ; ② Image modal data preprocessing: The intelligent customer service system normalizes the size of user-uploaded photos of hand rashes, adjusting them to 224×224 pixels, and performs RGB color space conversion. Then, it uses a pre-trained VisionTransformer model (ViT-B / 16) to encode the image, extracting the output vector at the category label position to obtain a 768-dimensional image semantic embedding vector. ; ③ Voice modal data processing: In this embodiment, the user did not provide voice data, so the voice modal data input is empty. During multimodal fusion, only the information of two modalities, text modal data and image modal data, are fused.

[0030] (2) Multimodal semantic encoding and cross-modal alignment: The intelligent customer service system adopts the CLIP-style cross-modal contrastive learning framework for multimodal semantic alignment, as follows: ① Unified semantic space mapping: The text embedding vector obtained in step (1) is mapped to the text embedding vector. Average pooling is performed along the sequence dimension to obtain a 768-dimensional sentence-level representation, which is then mapped to a 256-dimensional unified semantic space through a linear projection layer to obtain the aligned unified semantic representation of the text. Embedding the image into a vector By applying a linear projection mapping to a 256-dimensional unified semantic space, an aligned unified semantic representation of the image is obtained. Furthermore, the parameters of the two linear projection layers have been pre-trained during the offline comparative learning phase. ② Construction of positive and negative sample pairs: In the offline pre-training stage before actual operation in this embodiment, the intelligent customer service system constructs training samples for contrastive learning from a large-scale multimodal corpus; for each training data, the aligned text from the same user query is given a unified semantic representation. Unified semantic representation with aligned images Combining to construct positive sample pairs ( , Negative sample pairs are constructed by cross-combining aligned text unified semantic representations from different user queries in the same training batch with aligned image unified semantic representations. , The aligned image has a unified semantic representation. The samples come from different query samples; in this embodiment, the training batch size is 256, and each positive sample in each batch corresponds to 255 negative sample pairs; ③ Cross-modal contrastive loss calculation: The intelligent customer service system uses the cross-modal NCE contrastive loss function for optimization. The cross-modal NCE contrastive loss function encourages the maximization of the cosine similarity of positive sample pairs in the unified semantic space, while minimizing the similarity of negative sample pairs. Specifically, for each positive sample pair in the batch, the logarithm ratio of it to all negative sample pairs is calculated, and the negative logarithms are summed and averaged. The temperature parameter is set to 0.07 to adjust the model's sensitivity to similarity. Through the optimization of the cross-modal NCE contrastive loss function, the unified semantic representation of text and the unified semantic representation of images are effectively aligned in the unified space, making semantically related cross-modal representations closer to each other and semantically unrelated representations further apart.

[0031] (3) Lightweight intent recognition based on knowledge distillation, as follows: ① Multimodal fusion feature generation: unify the semantic representation of the aligned text obtained in step (2). Unified semantic representation of aligned images By performing attention-weighted fusion, the intelligent customer service system calculates the learnable attention weights for text modal data and image modal data respectively, and then sums the weighted representations of each modality to obtain a 256-dimensional multimodal fusion feature. ; ② Soft label generation of teacher's big language model: fusing features from multiple modalities Input the teacher's large language model Qwen-14B. After passing through multiple layers of Transformer encoding, the teacher's large language model extracts high-level semantic features and outputs the soft label probability distribution of the intent category through a classification layer. In this embodiment, the soft label distribution output by the teacher's big language model is as follows: symptom identification 0.75, medical guidance 0.18, and other 0.07, indicating that the teacher's big language model is highly confident that the core intent of the query is symptom identification. ③ Student Intent Recognition Model Training and Distillation: The student intent recognition model is a lightweight network with a 6-layer Transformer encoder, 384 hidden layer dimensions, 6 attention heads, and approximately 20M parameters, far less than the 14B parameters of the teacher's large language model. During the offline distillation training phase, the intelligent customer service system inputs a large number of multimodal query samples into the teacher's large language model and the student intent recognition model respectively, and optimizes them through the total loss of knowledge distillation. The total distillation loss is obtained by weighted summation of hard label cross-entropy loss (the user's true intent label) and soft label KL divergence loss (the soft label probability distribution output by the teacher's large language model), with a balance coefficient set to 0.3. After distillation training, the student intent recognition model can perform intent recognition with an accuracy close to that of the teacher's large language model, but with a significant improvement in inference speed and a significant reduction in memory usage. (4) Domain-adaptive fine-tuning, as detailed below: ① Domain-labeled sample acquisition: The intelligent customer service system is designed for online dermatology consultation scenarios. The intelligent customer service system has acquired 200 labeled samples in the target domain. Each sample contains text modal and image modal data of user queries and corresponding real intent labels (such as symptom identification, medical guidance, medication consultation, etc.). The number of labeled samples is far less than the amount of data required to train the model from scratch. ② Low-rank adaptation parameter initialization: Freeze all original weight parameters of the student intent recognition model obtained from distillation in step (3), including the weight matrices of all Transformer encoding and classification layers; for the attention layer in the student intent recognition model , , , and feedforward network layer , There are a total of 6 weight matrices, and a trainable low-rank decomposition matrix is ​​introduced. The rank r = 8, which is much smaller than the dimension of the original weight matrix (384 × 384). During initialization, A is distributed using a random Gaussian distribution, and B is initialized as a zero matrix, resulting in an initial state of AB = 0, representing the modified weights. =W+AB is consistent with the original weights; in this embodiment, only 6×(384×8+8×384)≈36,864 parameters need to be trained, which is far less than the millions of parameters required for full fine-tuning; ③ Fine-tuning and optimization: The low-rank parameters were trained on 200 domain-labeled samples, while the original weights remained frozen. The Adam optimizer was used with a learning rate of 1e-4 and a batch size of 16 for 20 training epochs. The loss function during fine-tuning was the cross-entropy loss between the predicted intent and the actual label. After fine-tuning, the student intent recognition model was able to accurately identify professional intents such as symptom identification and medical guidance in dermatology scenarios. The recognition accuracy on the domain test set was improved by about 8 percentage points compared to the distilled general model. (5) Intent category output, as follows: ① Student Intent Recognition Model Intent Reasoning: After processing the user's query "What is this? Do I need to go to the hospital?" through the above steps, the fine-tuned student intent recognition model outputs the probability distribution of each intent category. In this embodiment, the output is: symptom identification 0.82, medical guidance 0.13, and other categories 0.05. ② Confidence Calculation and Decision: The maximum probability value of 0.82 is calculated as the confidence score; the confidence threshold is set to 0.7. Since 0.82 ≥ 0.7, it indicates that there is high confidence in the prediction result, and there is no need to return to the candidate intent for clarification. ③ Intent Category Mapping Output: The intent category label "Symptom Identification" is mapped to the corresponding intent name in the preset intent category set and output in a structured data format; the final output recognition result includes: intent label (symptom identification), intent name (symptom identification and preliminary judgment), confidence score (0.82) and original query content ("What is this? Do I need to go to the hospital?"); this recognition result is passed to the downstream question answering system to retrieve matching medical knowledge and generate accurate response content; ④ Example of handling low confidence: If a user's query confidence score is lower than the threshold of 0.7, it is determined to be an unidentifiable intent. The system returns an "uncertain intent" label and the top 3 candidate intents with the highest probability for the user to confirm or ask clarifying follow-up questions. For example, if a user queries "How about that?", it lacks a clear referent and the model outputs a confidence score of only 0.45. The system returns candidate intents: product consultation (0.45), treatment suggestions (0.28), and effect feedback (0.18). The user selects the true intent and continues the dialogue process.

[0032] Example 4: This example also provides an electronic device, including: a memory and a processor; The memory stores the instructions executed by the computer. The processor executes computer execution instructions stored in the memory, causing the processor to perform the multimodal dialogue intent recognition method in any embodiment of the present invention.

[0033] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0034] Memory can be used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0035] Example 5: This embodiment also provides a computer-readable storage medium storing multiple instructions, which are loaded by a processor to cause the processor to execute the multimodal dialogue intent recognition method according to any embodiment of the present invention. Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0036] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0037] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0038] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0039] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for multi-modal dialogue intent recognition, characterized in that, The method is as follows: Acquire and preprocess multimodal data: Acquire query data input by the user, including text modal data, speech modal data and / or image modal data, and preprocess the text modal data, speech modal data and image modal data using a pre-trained Transformer encoder, a pre-trained speech encoder and a pre-trained visual encoder respectively, and extract text semantic embedding vectors, speech semantic embedding vectors and image semantic embedding vectors; Cross-modal alignment: By using cross-modal contrastive learning, text semantic embedding vectors, speech semantic embedding vectors, and image semantic embedding vectors are mapped to a unified semantic space to obtain a unified semantic representation after each modality is aligned. The unified semantic representations after each modality is aligned are then concatenated and weighted and fused through an attention mechanism to generate multimodal fusion features. Lightweight Intent Recognition Based on Knowledge Distillation: The teacher's large language model is used to understand the intent of the fused features after multimodal semantic alignment, generate soft label prediction distribution, and the intent understanding ability of the teacher's large language model is distilled into the student intent recognition model through the knowledge distillation loss function. Domain-adaptive fine-tuning: Freeze all original weights of the student intent recognition model, use the LoRA method to perform domain-adaptive fine-tuning of the student intent recognition model based on labeled samples of the target application domain, update the parameters of the student intent recognition model, and obtain the fine-tuned student intent recognition model to adapt it to the intent recognition task of the target domain. Obtain Intent Category: Input the user query data into the fine-tuned student intent recognition model to obtain the intent category label corresponding to the user query. 2.The multi-modal dialogue intent recognition method of claim 1, wherein, The preprocessing of multimodal data is as follows: Preprocessing text modal data: The user-input text modal data (text) is segmented and stop word removed to obtain preprocessed text modal data (text). Then, the preprocessed text modal data (text) is converted into text embedding vectors using the pre-trained word embedding model BERT-base-Chinese. ;in, Indicates the length of the text sequence; Indicates the dimension of the text embedding vector; Represents text modal data; Preprocessing of speech modal data: When user query data contains speech modal data, the speech modal data is framed, windowed, and subjected to Fourier transform. Mel-frequency cepstral coefficient features are extracted to obtain preprocessed speech modal data. The preprocessed speech modal data is then converted into speech semantic embedding vectors using the Hubert-base speech encoder. ;in, Indicates the number of audio frames; The dimension of the speech semantic embedding vector; Represents speech modal data; Preprocessing image modal data: When a user query contains image modal data, the image modal data is normalized in size and converted in color space to obtain preprocessed image modal data. This preprocessed image modal data is then converted into image semantic embedding vectors using the ViT-B / 16 image encoder. ;in, The number of spatial locations representing image features; The dimension of the image semantic embedding vector; This represents image modal data. 3.The method of claim 2, wherein, Cross-modal alignment is as follows: Mapping the unified semantic space: respectively mapping the text semantic embedding vector, the speech semantic embedding vector and the image semantic embedding vector to a unified semantic space with a dimension of d , obtaining the unified semantic representation of each modality after alignment; the formula is as follows: ； ； ； in, , and These represent the aligned unified semantic representation of text, the aligned unified semantic representation of speech, and the aligned unified semantic representation of image, respectively. This indicates that the pre-trained Transformer encoder is followed by a projection head. ; This indicates that the pre-trained speech encoder is connected to the projection head. ; This indicates that the pre-trained visual encoder is followed by the projection head ViT (ViT). ); Constructing positive and negative sample pairs: Pairing aligned text unified semantic representations, aligned speech unified semantic representations, and aligned image unified semantic representations of the same user query across modalities to construct a set of positive sample pairs. The aligned unified semantic representations of text, speech, and images from different user queries are paired to construct a set of negative sample pairs. ; Calculate the cross-modal contrastive loss: Use the cross-modal NCE loss function for contrastive learning optimization, as shown in the following formula: ； in, , and Both represent local index variables, used to distinguish semantic representations of different modalities and different elements in sample pairs; Modal semantic representation and Cosine similarity between them; Indicates temperature hyperparameter; Represents the set of positive sample pairs; Represents the set of negative sample pairs; Generating multimodal fusion features: The aligned unified semantic representations of text, speech, and image are concatenated and then fused using attention weighting to generate multimodal fusion features, as shown in the following formula: ； wherein the attention weight ; denotes the learnable attention parameters of the modality ; T denotes the transpose; denotes the multi-modal fusion feature. 4.The method of claim 3, wherein, The lightweight intent recognition based on knowledge distillation is as follows: Generating soft labels for the teacher's large language model: Multimodal feature fusion using the teacher's large language model DeepSeek-V4-Pro Process to obtain intent category Soft label probability distribution ;in, and These represent the weight matrix and bias vector of the classification layer in the teacher's large language model, respectively. This represents the high-level semantic features extracted by the teacher's large language model; The hidden layer dimension represents the teacher's large language model; K The total number of intent categories, ranging from 10 to 100; the Softmax function is defined as follows: ; The original score vector representing the output of the classification layer is a K-dimensional real number vector; The original score vector representing the output of the classification layer. The element corresponding to the category index c is the original score of the classification layer predicting that the input sample belongs to the c-th intention category; This represents the k-th element in the original score vector x output by the classification layer, where k is the summation index variable. It iterates through all intent categories and is used for normalization calculation of the denominator. c Indicates a category index; Training and distilling the student intent recognition model: The student intent recognition model adopts a 6-layer Transformer encoder architecture, and multimodal features are fused through the student intent recognition model. Perform encoding processing to obtain the hard label prediction distribution. ;in, and These represent the weight matrix and bias vector of the classification layer in the student intent recognition model, respectively. This represents the high-level semantic features extracted by the student intent recognition model; The hidden layer dimension represents the dimension of the student intent recognition model, i.e., the dimension of the feature vector obtained after encoding each word vector in the Transformer encoder of the student intent recognition model; student represents the student intent recognition model; the total loss function of knowledge distillation is defined as: ;in, ; This represents the hard-label cross-entropy loss. Intent category in one-hot encoding of true intent label c The value corresponding to the dimension; The student intent recognition model predicts the first... c The probability of each category; ; Indicates distillation loss for soft labels; The teacher's large language model predicts the first c The probability of each category; This is the balance coefficient. 5.The method of claim 4, wherein, Domain-adaptive fine-tuning is as follows: Obtain domain-specific labeled samples: Obtain a set of labeled samples for the target application domain. ;in, Indicates the first Text queries for each sample; This represents the corresponding multimodal auxiliary data, namely, speech modal data or image modal data; ∈{1,2,…,K} represents the true intent label; Indicates the number of labeled samples; Initialize low-rank adaptation parameters: Freeze all original weight parameters of the student intent recognition model; for each weight matrix in the student intent recognition model that needs fine-tuning. Introducing trainable low-rank decomposition matrices and Where rank r ≪ min(d, k), and the low-rank decomposition matrix A is initialized using a random Gaussian distribution. ; Initialize to a zero matrix; Forward propagation and fine-tuning optimization: During fine-tuning, the original weight matrix W remains frozen, and only the low-rank matrices A and B are updated. The modified weight matrix is ​​as follows: Label the sample set in the target application domain. To minimize the cross-entropy loss, the formula is as follows: ； in, This indicates an indicator function that takes a value of 1 when the condition is true and 0 otherwise; and it uses the Adam optimizer with a learning rate of... Update the low-rank parameters A and B, set the batch size to 32, and the number of training epochs to 50. 6.The method of claim 5, wherein, The specific intent categories are as follows: Student intent recognition model intent inference: Input user query data into the penultimate layer of the fine-tuned student intent recognition model to obtain the high-level semantic feature vector extracted by the student intent recognition model. ;in, =384 is the hidden layer dimension of the student intent recognition model; then the high-level semantic feature vector Input the student intent recognition model's classification layer, and calculate the probability distribution of user queries belonging to each intent category using the Softmax function. The formula is as follows: ;in, and These represent the weight matrix and bias vector of the classification layer in the fine-tuned student intent recognition model, respectively; K represents the total number of intent categories; the Softmax function is defined as... ; c∈{1,2,…,K} is the category index, exp( ) represents an exponential function. Let be a probability distribution vector, satisfying ; Calculate prediction confidence: Calculate the confidence score Conf of the student intent recognition model for the predicted results of the user query, using the following formula: That is, the maximum value in the probability distribution, which takes the value range of [1 / K, 1]. Intent category decision: Decisions are made using a confidence threshold δ∈[0.5,0.9]. Intent category mapping: Mapping intent category labels Mapped to a set of preset intent categories The corresponding intent name It is output in a structured data format; wherein the structured data format contains at least an intent category identifier. Intent Category Name The confidence score (Conf) and the original content of the query text.

7. The method for multimodal dialogue intent recognition according to claim 6, characterized in that, The decision-making process using a confidence threshold δ∈[0.5,0.9] is as follows: If the confidence score Conf ≥ δ, output the intent category label and return the corresponding intent category name and confidence score. If the confidence score Conf < δ, it is determined to be an unidentifiable intent. The "Uncertain Intent" flag and the top T candidate intents with the highest probability distribution are returned for the user to confirm or ask clarifying follow-up questions. 8.A multi-modal dialogue intent recognition system based on, characterized in that, This system is used to implement the multimodal dialogue intent recognition method as described in any one of claims 1 to 7; the system comprises: The acquisition and preprocessing module is used to acquire query data input by the user, including text modal data, speech modal data and / or image modal data. It uses a pre-trained Transformer encoder, a pre-trained speech encoder and a pre-trained visual encoder to preprocess the text modal data, speech modal data and image modal data respectively, and extracts text semantic embedding vectors, speech semantic embedding vectors and image semantic embedding vectors. The cross-modal alignment module is used to map text semantic embedding vectors, speech semantic embedding vectors and image semantic embedding vectors to a unified semantic space through cross-modal contrastive learning, obtain a unified semantic representation after each modality alignment, and then concatenate the unified semantic representations after each modality alignment and then perform weighted fusion through an attention mechanism to generate multimodal fusion features. The lightweight intent recognition module is used to understand the intent of the fused features after multimodal semantic alignment by the teacher's big language model, generate soft label prediction distribution, and distill the intent understanding ability of the teacher's big language model into the student intent recognition model through the knowledge distillation loss function. The domain adaptive fine-tuning module is used to freeze all the original weights of the student intent recognition model, and use the LoRA method to perform domain adaptive fine-tuning of the student intent recognition model based on the labeled samples of the target application domain, update the parameters of the student intent recognition model, and obtain the fine-tuned student intent recognition model to adapt it to the intent recognition task of the target domain. The intent category acquisition module is used to input user query data into the fine-tuned student intent recognition model and obtain the intent category label corresponding to the user query.

9. An electronic device, comprising: include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the multimodal dialogue intent recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the multimodal dialogue intent recognition method as described in any one of claims 1 to 7.

Images