Feature decoupling-based multi-modal intention recognition method and related product

Abstract

The invention discloses a multi-modal intention recognition method based on feature decoupling and a related product, and belongs to the technical field of multi-modal representation learning. According to the method, domain confrontation and Wasserstein distance loss are designed to jointly align shared feature distribution of different modals, and a cross-modal Transform module is adopted to eliminate distribution differences among specific features of different modals, so that performance improvement of the multi-modal intention recognition method is realized; according to the method, more accurate sample multi-modal semantic representation can be obtained while consistency learning of shared features among modals is kept. On the premise of fully retaining feature information, dual alignment of shared feature distribution and a semantic level can be realized more effectively. Meanwhile, by means of modal specific features extracted by a cross-modal attention mechanism, learning and fusion of cross-modal semantics can be further promoted on the basis of keeping independence of each modal feature.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal representation learning, and in particular to a multi-modal intent recognition method based on feature decoupling and related products. BACKGROUND

[0002] Intent recognition is crucial in human communication and is an important research direction in current natural language understanding (NLU), aiming to determine the intent category using text information to achieve more optimal dialogue interaction. Although text-based intent recognition has achieved remarkable results, it mainly focuses on goal-oriented tasks within specific domains. The intent of these tasks usually comes from instructions or queries with clear semantic features, which is different from multi-modal language in the real world that contains rich emotional, attitudinal and behavioral information. Combining natural language with non-verbal signals such as expressions, body movements and tone of voice helps to analyze human intent from multiple perspectives and provides more friendly services.

[0003] In order to effectively perform fusion representation learning on data from various modalities, researchers have proposed many methods for multi-modal language understanding. Among them, the representative methods mainly include MAG-BERT which realizes text Embedding bias learning by early fusion of visual and audio features with text features; MULT which realizes multi-modal non-aligned correlation learning using cross-modal Transformer method; and MISA which greatly relieves the difficulty of feature fusion learning caused by the heterogeneity of multi-modal data by projecting each modality into two different feature subspaces (modality-invariant and modality-private space) through a feature decoupling strategy.

[0004] Subsequently, a series of improved methods based on the above three representative methods were proposed, further promoting the development of the field of multi-modal intent recognition. For example, SDIF-DA uses a data augmentation method based on ChatGPT to obtain more text samples similar to the target classification sample task, and based on these augmented text samples, the text feature learning module is pre-trained, then a shallow-to-deep interaction module is designed to gradually effectively align and fuse the features of text, visual and audio modalities; TCL-MAP develops a modality-aware prompt module (MAP) based on similarity-based modality alignment and cross-modal attention mechanism, and proposes a token-level contrastive learning method with modality-aware prompt to realize effective alignment and fusion of multi-modal features.

[0005] However, the field still faces two severe challenges. First, given that intent recognition is mainly a text-centered task, while limited transcribed text often has rich semantic information and may convey different intents in different actual dialogue scenarios. Non-verbal modal information and text information are usually complementary to each other, which can provide additional clues for semantic and emotional disambiguation, but the natural heterogeneity between different modalities increases the difficulty of robust multi-modal representation learning. Second, existing multi-modal intent recognition methods based on feature decoupling mostly use similarity measurement criteria to pull the features of modal invariant space closer, which forcibly guides the features to realize alignment in the hidden space while ignoring their inherent distribution differences, making the model less robust when the data distribution difference is large. SUMMARY

[0006] The purpose of the present application is to provide a multi-modal intent recognition method based on feature decoupling and related products to overcome the deficiencies in the prior art in learning the potential correlation between different modalities.

[0007] The present application solves the above technical problems through the following technical solutions: The present application provides a multi-modal intent recognition method based on feature decoupling, comprising the following steps: S1, based on the target intent recognition task dataset, a multi-modal sample pair is obtained; a pre-trained model is used to extract features from the multi-modal sample pair to obtain original modal features; S2, align the feature dimensions of the original modal features, and use a feature sharing encoder and a feature specific encoder to decouple the original modal features after aligning the feature dimensions to obtain modal-shared features and modal-specific features; wherein the feature sharing encoder consists of a time self-attention layer and a linear layer; the feature specific encoder consists of multiple cross-modal Transformer layers; S3, the modal-shared features and the modal-specific features are spliced to obtain fusion features; the fusion features are linearly transformed to obtain classification logic values; S4, calculate the total loss of the model, the total loss at least includes domain adversarial loss, Wasserstein distance loss and cross-entropy loss; wherein the domain adversarial loss and the Wasserstein distance loss are calculated based on the modal-shared features, and the cross-entropy loss is calculated based on the classification logic values and the true sample labels of the multi-modal sample pair; S5, based on the total loss, the reverse gradient propagation is performed, the gradient information of the model parameters is calculated and the model is optimized, and the optimized model is used to recognize the multi-modal sample pair.

[0008] The further improvement of the present application is that the feature dimensions of the original modal features are aligned, specifically: the feature dimensions of the original modal features are aligned through a one-dimensional convolutional neural network.

[0009] The further improvement of the present application is that the multi-modal sample pair is , wherein, is text; is visual; is audio; the multi-modal sample pair is subjected to feature extraction by using a pre-trained model to obtain original modal features, specifically:

[0010] , wherein, is a pre-trained text feature extraction model; is a pre-trained video feature extraction model; is a pre-trained audio feature extraction model; The feature dimensions of the original modal features are aligned through a one-dimensional convolutional neural network, specifically:

[0011] , wherein, is the original modal feature after feature dimension alignment, is the aligned feature dimension, is a transpose operator, is a one-dimensional convolutional neural network parameter.

[0012] The further improvement of the present application is that the original modal features are subjected to feature decoupling by using a feature sharing encoder to obtain modal-sharing features, specifically:

[0013] , wherein, is a modal-sharing feature; is a feature sharing encoder; is a feature sharing encoder parameter.

[0014] The further improvement of the present application is that the original modal features are subjected to feature decoupling by using a feature-specific encoder to obtain modal-specific features, specifically:

[0015]

[0016]

[0017] , wherein, is a video modal-specific feature; is a text modal-specific feature; For audio modal-specific features; For video-text joint modal representation; For joint video-audio modal representation; For text-video joint modal representation; For text-audio joint modal representation; For audio-text joint modal representation; For audio-video joint modal representation; For feature splicing module; It is a linear projection layer.

[0018] A further improvement of this invention is that the total loss further includes orthogonal loss and reconstruction loss, wherein the orthogonal loss and reconstruction loss are calculated based on modality-shared features; the total loss is specifically as follows:

[0019] in, Total loss; For classification loss; Losses due to domain confrontation; Wasserstein distance loss; Orthogonal loss; For reconstruction loss; For the control parameters of domain adversarial loss; These are the control parameters for Wasserstein distance loss; These are the control parameters for orthogonal loss; The control parameters for reconstructing the loss.

[0020] The present invention also provides a multimodal intent recognition method based on feature decoupling, comprising: The first module is used to obtain multimodal sample pairs based on the target intent recognition task dataset; and to extract features from the multimodal sample pairs using a pre-trained model to obtain the original modal features. The second module is used to align the feature dimensions of the original modal features. It uses a feature-sharing encoder and a feature-specific encoder to decouple the original modal features after feature dimension alignment, resulting in modality-shared features and modality-specific features. The feature-sharing encoder consists of a temporal self-attention layer and a linear layer; the feature-specific encoder consists of multiple cross-modal Transformer layers. The third module is used to concatenate modality-shared features and modality-specific features to obtain fused features; and to perform linear transformation on the fused features to obtain classification logical values. The fourth module is configured to calculate a total loss of the model, wherein the total loss at least comprises a domain adversarial loss, a Wasserstein distance loss and a cross-entropy loss; wherein the domain adversarial loss and the Wasserstein distance loss are calculated based on the modal-shared features, and the cross-entropy loss is calculated based on the classification logic value and the real sample label of the multi-modal sample pair; The fifth module is configured to perform reverse gradient propagation based on the total loss, calculate gradient information of the model parameters and optimize the model, and identify the multi-modal sample pair by using the optimized model.

[0021] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the multi-modal intent recognition method based on feature decoupling.

[0022] The application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the multi-modal intent recognition method based on feature decoupling.

[0023] Compared with the prior art, the application has the following advantages: The multi-modal intent recognition method based on feature decoupling provided by the application can jointly align the shared feature distribution of different modalities by designing the domain adversarial and Wasserstein distance loss, and can eliminate the distribution difference between the specific features of different modalities by using the cross-modal Transformer module, so that the performance of the multi-modal intent recognition method is improved, and more accurate sample multi-modal semantic representation can be obtained while maintaining the consistency learning of the shared features between modalities. Compared with the traditional method which only relies on similarity measurement, orthogonal constraint or reconstruction loss to constrain the feature distribution of each subspace, the present method can more effectively realize the double alignment of the shared feature distribution and the semantic level under the premise of fully preserving the feature information. At the same time, with the help of the modal-specific features extracted by the cross-modal attention mechanism, the learning and fusion of cross-modal semantics can be further promoted on the basis of maintaining the independence of each modal feature. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings are included to provide a further understanding of the application, and constitute a part of the application. The illustrative embodiments of the application and their description serve to explain the application without limiting the application.

[0025] Figure 1 is a method flowchart of a specific embodiment of the application.

[0026] Figure 2 is a schematic diagram of the overall framework structure in a specific embodiment of the application.

[0027] Figure 3is a structural schematic diagram of a modal fusion & detection module in a certain specific embodiment of the present application.

[0028] Figure 4 is a structural schematic diagram of a model training result in a certain specific embodiment of the present application.

[0029] Figure 5 is a structural schematic diagram of a system framework in a certain specific embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0031] In the description of the present application, it should be understood that the terms “comprise” and “include” indicate the presence of described features, integers, steps, operations, elements, and / or components, but do not exclude one or more other features, integers, steps, operations, elements, components, and / or sets thereof.

[0032] It should also be understood that the terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and the appended claims of the present application, unless otherwise clearly indicated by the context, the singular forms “a”, “an” and “the” are intended to include the plural forms as well.

[0033] It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present application to describe a predetermined range, etc., these predetermined ranges should not be limited to these terms. These terms are only used to distinguish the predetermined ranges from each other. For example, the first predetermined range can also be referred to as the second predetermined range, and similarly, the second predetermined range can also be referred to as the first predetermined range, without departing from the scope of the embodiments of the present application.

[0034] Depending on the context, the word “if’ as used herein can be interpreted to mean “when” or “while” or “in response to determining” or “in response to detecting”. Similarly, depending on the context, the phrase “if determined” or “if detecting (a stated condition or event)” can be interpreted to mean “when determined” or “in response to determining” or “when detecting (a stated condition or event)” or “in response to detecting (a stated condition or event)”.

[0035] The application will be further described in detail below in combination with the drawings and specific embodiments, which are an explanation rather than a limitation of the application.

[0036] A multi-modal intent recognition method based on feature decoupling, comprising the following steps: S1, based on the target intent recognition task data set, a multi-modal sample pair is obtained; a pre-trained model is used to extract features from the multi-modal sample pair to obtain original modal features; S2, align the feature dimensions of the original modal features, and use a feature sharing encoder and a feature specific encoder to decouple the original modal features after the feature dimension alignment to obtain modal-shared features and modal-specific features; wherein the feature sharing encoder is composed of a time sequence self-attention layer and a linear layer; the feature specific encoder is composed of multiple cross-modal Transformer layers; S3, the modal-shared features and the modal-specific features are spliced to obtain fusion features; the fusion features are linearly transformed to obtain classification logic values; S4, calculate the total loss of the model, the total loss at least includes domain adversarial loss, Wasserstein distance loss and cross-entropy loss; wherein the domain adversarial loss and the Wasserstein distance loss are calculated based on the modal-shared features, and the cross-entropy loss is calculated based on the classification logic values and the true sample labels of the multi-modal sample pair; S5, based on the total loss, the reverse gradient propagation is carried out, the gradient information of the model parameters is calculated and the model is optimized, and the optimized model is used to recognize the multi-modal sample pair.

[0037] The optimization method combining domain adversarial loss and Wasserstein distance loss effectively realizes the distribution alignment and syntax alignment of different modal features in the modal sharing space; at the same time, in order to solve the heterogeneity problem of each modal data in the modal specific space, the cross-modal attention mechanism is introduced, so that the real semantics of the sample can be captured more accurately, and the accuracy of intent recognition is improved.

[0038] In a specific embodiment of the application, a multi-modal intent recognition method based on feature decoupling, comprising the following steps: obtaining a {text-video-audio} sample pair for multi-modal intent recognition, symbolically represented as Text, visual, audio data respectively use BERT, Faster R-CNN, and WAV2VEC models which have been pre-trained on a specific large-scale data set to extract features to obtain original modal features , denoted as:

[0039] Wherein, for the modal original modality feature , is the sequence length of the original modality feature, is the feature dimension of the original modality feature.

[0040] In order to facilitate subsequent cross-modal feature decoupling calculation, a 1D CNN layer is used to perform preliminary feature dimension alignment operation on each modality feature, and the calculation process is represented as:

[0041] wherein, , represents the aligned feature dimension, represents the transpose operator, represents the parameters of the one-dimensional convolution layer used for this modality.

[0042] The natural heterogeneity between multi-modal data leads to a significant gap in feature distribution between each modality feature. In order to alleviate the data distribution gap to achieve better feature fusion, feature decoupling processing is performed on each modality feature. First, a feature shared encoder is designed to extract modality-shared features :

[0043] wherein, are the parameters of the feature shared encoder used for this modality. The shared encoder used here consists of a time self-attention Transformer layer and a linear layer, that is:

[0044] Then, a feature-specific encoder is designed to extract modality-specific features . Unlike other feature decoupling methods that use linear layers to form feature-specific encoders, cross-modal Transformer (CMT) is used to promote semantic alignment between modalities, and a reasonable multi-supervised loss is designed to realize joint optimization of model parameters.

[0045] Specifically, taking the video modality as an example to illustrate how to obtain the video modality-specific feature. First, for video-text attention learning, the video feature is used to generate the Query item, while the text feature is used to generate the Key and Value items. The CMT module learns the attention weight so that each sequence of the video segment can pay attention to all sequences of the text sentence to different degrees. After layer CMT module iterative processing, the visual-text joint specific modality representation is:

[0046] wherein, is the number of heads of the multi-head mechanism, , , , and is the learnable projection parameter; denotes the module stacking operation; denotes the exponential normalization function.

[0047] Similarly, for video-audio attention learning, the Query item is still generated using video features, but this time the Key and Value items are generated using audio features After passing through multiple layers of CMT modules, the visual-audio joint specific modality will be obtained.

[0048] The and are concatenated and input into a linear projection layer to obtain the final video modality-specific feature :

[0049] The same processing method is used for text and audio modalities to obtain text modality-specific features and audio modality-specific features :

[0050]

[0051] wherein, is the text-video joint specific modality representation; is the text-audio joint specific modality representation; is the audio-text joint specific modality representation; is the audio-video joint specific modality representation.

[0052] The modality-shared features , and modality-specific features , generated by the above steps are concatenated to obtain the fusion features , which are then processed by a multi-layer perceptron composed of two linear projection layers stacked and output the final classification :

[0053] in, Batch size; , This is the weight matrix; , It is the bias vector; Output the number of categories; The activation function (such as ReLU, GELU, etc.) introduces nonlinear transformation capabilities into the model; This indicates a random discard operation. This is the discard probability, used to prevent the model from overfitting during training.

[0054] To achieve the modality consistency and differential learning required by the model, multiple supervised losses were designed, which constitute the total learning loss:

[0055] in, These are the control parameters for each corresponding supervised loss term. Overall model loss. These control parameters are adjusted to balance the contribution of each regularization term.

[0056] The specific definitions and functions of each loss item are as follows: - Domain Adversarial Loss: This strategy aims to align the distribution of modality-shared features using a domain adversarial approach. First, gradient inversion is applied to the modality-shared features in the modality-invariant feature space.

[0057] in, It is a gradient inversion layer. The inversion coefficients are then used. A domain discriminant is then designed. To determine the modality category to which the modality-shared features belong, and to calculate the domain adversarial loss in batch processing:

[0058]

[0059] in, Modality-shared features , The true modality category one-hot encoded vector, It is the modality class probability distribution vector predicted by the domain discriminator. The parameters of the discriminator in this domain Represents the logarithmic function. Indicates taking the first Each sample. The above process achieves normal data transmission (i.e., identity transformation) during forward propagation, while during backward propagation, the gradients from subsequent layers (such as the domain discriminator) are multiplied by a negative inversion coefficient. The function of sending the data back later.

[0060] Wasserstein distance loss: The principle is to calculate the minimum "work" required to transform a source distribution into another target distribution. This "work" is directly defined by the Euclidean distance between the two distributions. By minimizing the total "work" during the transformation process, an optimal transmission scheme is obtained, which maps nearby points in the source distribution to nearby points in the target distribution, thereby naturally preserving the local topology in the source feature space and achieving semantic alignment. The distance is defined as follows:

[0061] in, Indicates the use of entropy regularization parameters The optimal transmission cost is solved using the Sinkhorn algorithm; Indicates Euclidean distance; This represents a combination of random modes.

[0062] - Orthogonal loss: aims to achieve non-redundant representation by establishing orthogonal constraints between modality-shared features and modality-specific features, and is defined as follows:

[0063]

[0064]

[0065] in, and These are matrices composed of modality-shared features and modality-specific features, respectively. This represents the square of the Frobenius norm.

[0066] - Reconstruction Loss: This loss aims to constrain the risk of redundancy in modal representation information during model learning. It is defined as follows:

[0067]

[0068] in, Indicates the refactoring module. This represents the parameters of the refactoring module. Represents reconstructed modal features. This represents the square of the L2-norm.

[0069] - Classification Loss: The final model will output the classification logits for each sample in the batch. The classification loss of the model is obtained by calculating the cross-entropy loss between the logits and the true sample labels.

[0070]

[0071] in, , , They represent the first The logits of each sample, the one-hot encoded vector of the true class, and the probability distribution vector of the class predicted by the model.

[0072] Through the above steps, a multimodal intent recognition model based on a cross-modal attention mechanism using feature decoupling is constructed. During training, the model is input with a batch of multimodal data samples (including visual, text, and audio). The data is first processed by the feature decoupling module, passing through a feature-sharing encoder and a feature-specific encoder respectively, to obtain the modality-shared features of each sample. and modality-specific features The two are then spliced ​​together to form a fusion feature. Finally, the data is input into a classification network consisting of fully connected layers, which outputs the predicted logits for each class. The total model loss is then calculated. During each training step, backpropagation is performed starting from the total loss. The Adam adaptive optimizer is used to optimize the model parameters based on the gradient information obtained from backpropagation, thereby guiding the model to achieve accurate intent recognition results based on multimodal inputs. Please refer to [link / reference]. Figure 1 The present invention provides a multimodal intent recognition method based on feature decoupling and cross-modal attention mechanism, which may specifically include the following steps: Step 1: Based on the target intent recognition task dataset, extract the modal features of each sample pair; Step 2: Input multimodal features, decouple the original modal features based on the multimodal intent recognition method, optimize feature learning using domain adversarial and Wasserstein distance loss, and finally fuse the features of each modality to obtain the final classification result.

[0073] The intent recognition method of this invention can be used for the classification of multimodal intent sample pairs, and the classification object is a labeled {text-video-audio} sample pair.

[0074] Referring to Figure 2 In the embodiment of the present application, the overall framework of the adopted intention recognition method includes: The feature extraction module is used for processing each modality data, wherein BERT, Faster R-CNN, and WAV2VEC are respectively used for extracting text, video, and audio features; The feature decoupling module is used for receiving the inputted original multi-modal features, and then respectively mapping each original modality feature to a shared and specific hidden space through a modality-shared and feature-specific encoder; The feature reconstruction module is used for ensuring that the loss of modality features after feature decoupling is avoided, by adding the shared and specific features of each modality obtained after feature decoupling, and then using a feature reconstruction encoder to obtain reconstructed features close to the original modality features; The feature fusion and classification prediction module is used for splicing each feature to obtain fusion features, and finally passing the output of the self-attention layer through a double-layer MLP to obtain the final classification logits.

[0075] Referring to Figure 3 The modality fusion and detection module in the embodiment of the present application includes two linear layers. After obtaining the shared and specific features of each modality, the module will perform deep feature fusion and generate the final classification result. First, six groups of feature vectors are subjected to a dimension splicing operation to form multi-modal fusion features The spliced feature tensor fuses shared semantic information and modality-specific information. Subsequently, the fusion features are subjected to nonlinear transformation and dimension reduction processing through a two-layer feedforward neural network. The first linear transformation layer performs dimension reduction mapping:

[0076] The feature dimension is reduced from to , realizing feature compression. In order to enhance the generalization ability of the model, random inactivation regularization is applied to the reduced features:

[0077] to prevent overfitting, and then a nonlinear transformation ability is introduced through an activation function:

[0078] wherein ReLU, GELU, or other activation functions can be used to enhance the representation ability of the model. Finally, the classification result is output through the second linear layer:

[0079] The output logits are subjected to The final category probability distribution can be obtained after normalization. The fusion architecture maintains the integrity of multi-modal information and avoids over-parameterization through a carefully designed dimension reduction strategy. Meanwhile, the introduction of Dropout regularization and nonlinear activation ensures the generalization performance and representation ability of the model.

[0080] Referring to Figure 4 In the embodiment of the present application, the multi-modal intention recognition data set MIntRec used contains 2224 sample pairs. The shorter feature sequence in each modality is padded with zeros, so that the text, video and audio modality feature sequence lengths are fixed at 30, 230 and 480 respectively. The target intention categories include 20 categories such as "complaint", "praise" and "apology". The average sample duration is 2.38s and the average sentence length is 7.04. The data set is randomly divided into a training set, a validation set and a test set, each containing 1334, 445 and 445 sample pairs respectively. Each time a batch of 16 sample pairs is input into the model. The training time of the embodiment of the present application on an 8-card server equipped with a Gold 6626R@2.90GHz CPU, 8G memory and a NVIDIA GeForce RTX3090 is about 1 hour.

[0081] In the embodiment of the present application, the specific implementation steps include: first, setting model-related hyperparameters and optimizer configurations. The present application uses Adam as the optimizer, with an initial learning rate of 2e-5, a batch size of 16 and a total number of rounds of 40 during training. The learning rate adjustment strategy is set to an exponential decay learning rate scheduler with a decay index of 0.9, and a linear learning rate decay scheduler with linear warm-up is used. The recognition network of the embodiment of the present application includes a feature decoupling module containing a modality-shared and modality-specific encoder, a feature reconstruction module to ensure the completion of modality information, and a feature fusion & classification module composed of two layers of Linear layers. The classification logits input into the final classification module will be used to calculate the classification loss of the model. In order to ensure the accuracy of the representation learning of each modality feature, multiple supervised losses are designed to optimize the network. In the model training, 16 samples are randomly selected from the training set each time, then the total task loss is obtained, and the selected Adam optimizer is used for parameter update. After each round of training, the model performance on the validation set is verified and recorded, and the early stopping strategy is adopted, with a threshold of 0.999, to ensure that the final model is the best on the training set. Figure 4 ​​​As shown, it can be seen that when the accuracy on the model validation set no longer rises for 8 consecutive training rounds, the model training is ended. Then the trained model is tested on the test set, and an accuracy of 72.81% is obtained, which confirms the effectiveness of the application in the multi-modal intent recognition task.

[0082] In summary, the embodiment of the application discloses a multi-modal intent recognition method based on cross-modal attention mechanism of feature decoupling. On the one hand, the method constrains the feature representation of the modal-shared space through domain adversarial technology and Wasserstein distance loss, wherein the domain adversarial technology applies a gradient reversal layer to the modal-shared feature, and simultaneously adopts a discriminator to distinguish the modal category of the feature, finally realizes the alignment on the shared feature distribution, while the Wasserstein distance loss can preserve the topological structure in the feature space, better ensure that the shared feature contains the real shared semantic information without excessive constraint, and realize the alignment on the feature semantics. On the other hand, in order to solve the heterogeneity problem in the modal-specific space of the traditional method, the cross-modal attention mechanism is adopted, and the orthogonality loss and reconstruction loss are designed to ensure the difference and integrity of the modal information. Finally, the embodiment of the application illustrates the effectiveness of the recognition method.

[0083] Please refer to Figure 5 The embodiment of the application discloses a multi-modal intent recognition system based on cross-modal attention mechanism of feature decoupling, comprising: A multi-modal sample acquisition module is configured to acquire a {text-video-audio} sample pair to be recognized. An original data feature extraction module is configured to extract features of original data of each modality in the acquired multi-modal sample. The original data feature extraction module comprises: A text feature extraction module: a BERT model pre-trained on a specific large-scale text dataset is used for feature extraction, and the obtained feature is .

[0084] A video feature extraction module: a Faster R-CNN model pre-trained on a specific large-scale video dataset is used for feature extraction, and the obtained feature is .

[0085] An audio feature extraction module: a WAV2VEC model pre-trained on a specific large-scale audio dataset is used for feature extraction, and the obtained feature is .

[0086] An identification module is configured to process the pre-trained identification model based on the {text-video-audio} sample pair to be recognized, and obtain an identification result. The identification model comprises: A feature extraction module for processing each modality data, wherein BERT, Faster R-CNN, and WAV2VEC are respectively used to extract text, video, and audio features; A feature decoupling module for receiving inputted original multi-modal features, and then respectively mapping each original modality feature to a shared and specific hidden space through a modality-shared and feature-specific encoder; A feature reconstruction module for ensuring that the loss of modality features after feature decoupling is avoided, by adding the shared and specific features of each modality obtained after feature decoupling, and then using a feature reconstruction encoder to obtain reconstructed features close to the original modality features; A feature fusion and classification prediction module for splicing each feature to obtain fused features, and finally passing the output of a self-attention layer through a double-layer MLP to obtain final classification logits.

[0087] The system effectively realizes the distribution alignment and modality alignment of different modality features in the modality shared space; and in order to solve the heterogeneity problem of each modality data in the modality specific space, a cross-modality attention mechanism is introduced, so that the real semantics of the sample can be more accurately captured, and the accuracy of intent recognition is improved.

[0088] Based on the same inventive concept, the embodiments of the present application provide a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the multi-modal intent recognition method based on feature decoupling when executing the computer program. The memory can include a memory, such as a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk memory. The processor, network interface, and memory are connected to each other through an internal bus, which can be an industry standard architecture bus, a peripheral component interconnect standard bus, an extended industry standard architecture bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory is used to store programs, specifically, the programs can include program codes, and the program codes include computer operation instructions. The memory can include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0089] Based on the same inventive concept, the embodiment of the present application provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps of the multi-modal intent recognition method based on feature decoupling. Specifically, the computer readable storage medium includes, but is not limited to, for example, a volatile memory and / or a non-volatile memory. The volatile memory can include a RAM (Random Access Memory), a cache, and the like. The non-volatile memory can include a ROM (Read-Only Memory), a hard disk, a flash memory, an optical disc, a magnetic disc, and the like.

[0090] Based on the same inventive concept, the embodiment of the present application provides a computer program product, which includes a computer program stored on a computer readable storage medium. The computer program includes program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the multi-modal intent recognition method based on feature decoupling.

[0091] Those skilled in the art should understand that the embodiments of the present application can be provided as a method or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM (Compact Disc Read-Only Memory), optical storage, and the like) containing computer-usable program code.

[0092] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0093] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0094] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that are executed on the computer or other programmable apparatus provide steps for implementing the functions specified in the flow or flows and / or blocks Figure 1 of the block or blocks. Figure 1

[0095] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to encompass within their scope all such variations and modifications as are included within the scope of the application.

[0096] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.​

Claims

1. A multi-modal intent recognition method based on feature decoupling, characterized in that, The method comprises the following steps: S1, based on a target intent recognition task dataset, a plurality of modal sample pairs are obtained; a pre-trained model is used to extract features of the plurality of modal sample pairs to obtain original modal features; S2, the feature dimensions of the original modal features are aligned, and the original modal features after the alignment of the feature dimensions are decoupled by using a feature sharing encoder and a feature specific encoder respectively to obtain modal-sharing features and modal-specific features; the feature sharing encoder comprises a time self-attention layer and a linear layer; the feature specific encoder comprises a plurality of cross-modal Transformer layers; S3, the modal-sharing features and the modal-specific features are spliced to obtain fusion features; the fusion features are linearly transformed to obtain classification logic values; S4, a total loss of the model is calculated, the total loss at least comprises a domain adversarial loss, a Wasserstein distance loss and a cross-entropy loss; the domain adversarial loss and the Wasserstein distance loss are calculated based on the modal-sharing features, and the cross-entropy loss is calculated based on the classification logic values and true sample labels of the plurality of modal sample pairs; S5, based on the total loss, the reverse gradient propagation is performed, the gradient information of the model parameters is calculated, and the model is optimized; the optimized model is used to recognize the plurality of modal sample pairs.

2. The method of claim 1, wherein, The alignment of the feature dimensions of the original modal features is specifically performed by using a one-dimensional convolutional neural network.

3. The method of claim 2, wherein, A multimodal sample pair is set wherein, is text; is visual; is audio; a pre-trained model is used to extract features from the multimodal sample pair to obtain original modality features, specifically: wherein, is a pre-trained text feature extraction model; is a pre-trained video feature extraction model; is a pre-trained audio feature extraction model; The alignment of the feature dimensions of the original modal features is specifically performed by using a one-dimensional convolutional neural network. wherein, is the original modality feature after feature dimension alignment, is the aligned feature dimension, is the transpose operator, is the one-dimensional convolutional neural network parameter.

4. The method of claim 3, wherein, The feature decoupling of the original modal features by using the feature sharing encoder to obtain the modal-sharing features is specifically performed by using a one-dimensional convolutional neural network. wherein, is a modal-sharing feature; is a feature sharing encoder; is a feature sharing encoder parameter.

5. The method of claim 4, wherein, The feature decoupling of the original modal features by using the feature specific encoder to obtain the modal-specific features is specifically performed by using a one-dimensional convolutional neural network. wherein, is a video modality-specific feature; is a text modality-specific feature; is an audio modality-specific feature; is a video-text joint-specific modality representation; is a video-audio joint-specific modality representation; is a text-video joint-specific modality representation; is a text-audio joint-specific modality representation; is an audio-text joint-specific modality representation; is an audio-video joint-specific modality representation; is a feature concatenation module; is a linear projection layer.

6. The method of claim 1, wherein, The total loss further comprises an orthogonal loss and a reconstruction loss, wherein the orthogonal loss and the reconstruction loss are calculated based on the modal-sharing features; and the total loss is specifically: wherein, is a total loss; is a classification loss; is a domain adversarial loss; is a Wasserstein distance loss; is an orthogonality loss; is a reconstruction loss; is a control parameter for the domain adversarial loss; is a control parameter for the Wasserstein distance loss; is a control parameter for the orthogonality loss; is a control parameter for the reconstruction loss.

7. A multi-modal intent recognition method based on feature decoupling, characterized in that, The method comprises the following steps: A first module is configured to obtain a plurality of modal sample pairs based on a target intent recognition task dataset, and extract features of the plurality of modal sample pairs by using a pre-trained model to obtain original modal features; A second module is configured to align the feature dimensions of the original modal features, and decouple the original modal features after the alignment of the feature dimensions by using a feature sharing encoder and a feature specific encoder respectively to obtain modal-sharing features and modal-specific features; the feature sharing encoder comprises a time self-attention layer and a linear layer; the feature specific encoder comprises a plurality of cross-modal Transformer layers; A third module is configured to splice the modal-sharing features and the modal-specific features to obtain fusion features, and linearly transform the fusion features to obtain classification logic values; A fourth module is configured to calculate a total loss of the model, wherein the total loss at least comprises a domain adversarial loss, a Wasserstein distance loss and a cross-entropy loss; the domain adversarial loss and the Wasserstein distance loss are calculated based on the modal-sharing features, and the cross-entropy loss is calculated based on the classification logic values and true sample labels of the plurality of modal sample pairs; A fourth module configured to calculate a total loss of the model, the total loss comprising at least a domain adversarial loss, a Wasserstein distance loss, and a cross-entropy loss; wherein the domain adversarial loss and the Wasserstein distance loss are calculated based on the modality-shared features, and the cross-entropy loss is calculated based on the classification logic values and the true sample labels of the multi-modal sample pairs; A fifth module configured to perform backward gradient propagation based on the total loss, calculate gradient information of the model parameters, and optimize the model, and use the optimized model to recognize the multi-modal sample pairs.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the multi-modal intent recognition method based on feature decoupling according to any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the multi-modal intent recognition method based on feature decoupling according to any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the multi-modal intent recognition method based on feature decoupling according to any one of claims 1-6.

Images