Semantic alignment system for multi-modal intent recognition

Abstract

The application discloses a semantic alignment system for multi-modal intention recognition, and relates to the technical field of intention recognition, comprising a modal data analysis unit, a quality score calculation unit, a first round determination unit, a second round determination unit, a third round determination unit and an intention recognition unit. The application can accurately distinguish between internal rejection caused by low data quality and external rejection caused by real semantic contradiction by calculating the rejection coefficient and the quality score, preferentially selects the mode with external rejection as the main mode for cyclic query, gives priority to the real contradiction point, dynamically adjusts the state and weight according to the rejection type, updates the state with new information when external conflict is detected, resets the weight vector, avoids being misled by early misunderstanding, and enables more accurate intention judgment in subsequent multi-view fusion decoding.

Description

Technical Field

[0001] This invention relates to the field of intent recognition technology, specifically a semantic alignment system for multimodal intent recognition. Background Technology

[0002] A semantic alignment system for multimodal intent recognition dynamically aligns multimodal semantics through exclusion-aware scheduling, hybrid granularity computation, and multi-view fusion to achieve accurate, efficient, and interpretable intent recognition. Patent application number 202511720428.8 discloses "an enhanced multimodal data semantic alignment method and system, comprising the steps of: preprocessing and extracting features from input image and text data to obtain image feature matrices and text feature matrices, and performing matrix inner product operations to obtain an initial cross-modal similarity matrix; treating the initial similarity matrix as an adjacency matrix of a graph structure, calculating the degree matrix, and constructing a Laplacian matrix; iteratively calculating a final similarity matrix that propagates and enhances deep associations based on the initial similarity matrix and the Laplacian matrix; comparing the final similarity matrix with known image-text matching relationships within a batch, calculating the contrastive learning loss value, and backpropagating to iteratively update the parameters of the feature extraction network to achieve deep semantic alignment of cross-modal features. This invention effectively alleviates the sparsity problem of the high-dimensional feature space of multimodal images and text, and significantly improves the alignment accuracy of the model."

[0003] The aforementioned existing technologies have solved the problem of difficulty in capturing deep, indirect semantic relationships between cross-modal samples. However, in practice, due to the lack of an effective mechanism for handling semantic exclusion between modalities, when the expressions of text, visual, and audio modalities are inconsistent, the system cannot determine whether this inconsistency is due to internal exclusion caused by low data quality in a certain modality or external exclusion caused by semantic contradictions in the user's actual expression. This confusion can lead the system to make incorrect intent judgments. Summary of the Invention

[0004] The purpose of this invention is to provide a semantic alignment system for multimodal intent recognition to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a semantic alignment system for multimodal intent recognition, comprising:

[0006] The modal data analysis unit is used to receive multimodal data, preprocess it, and extract multi-granularity features from each modal data to calculate the corresponding repulsion coefficient.

[0007] The quality score calculation unit is used to statistically analyze the modal data before preprocessing, determine the corresponding quality scores, and identify the rejection type of different modal data based on the rejection coefficient and quality score.

[0008] The first round determination unit is used to set the scheduler state parameters, obtain the dominant mode of the current round, determine the coarse-grained features of the dominant mode, and calculate the hidden state vector, confidence and weight vector of the current round.

[0009] The second round determination unit is used to obtain the weight vector of the previous round after entering the second round, determine the current quality score based on the weight vector, analyze the required granularity features of the dominant mode in the current round, and calculate the hidden state vector and confidence level of the current round using the required granularity features of the dominant mode.

[0010] The third round determination unit is used to receive the confidence levels from the first two rounds, determine the required granularity features of the current dominant mode according to the change in confidence level, and output the hidden state vectors of all rounds.

[0011] The intent recognition unit is used to fuse all hidden state vectors and input them into the classifier to obtain the intent type and confidence level.

[0012] Preferably, the modal data analysis unit includes a feature extraction module and a repulsion coefficient calculation module;

[0013] The feature extraction module receives multimodal data. The data is then standardized to obtain preprocessed modal data. ,in , , These represent the text modal data before and after preprocessing, respectively. These represent the video modal data before and after preprocessing, respectively. These represent the audio modal data before and after preprocessing, respectively. A multi-granularity feature extractor is used to... The corresponding multi-granularity features are obtained by processing them sequentially. ,in express Multi-granularity features, express Multi-granularity features, express Multi-granularity features;

[0014] The repulsion coefficient calculation module extracts coarse-grained features in Then, select Calculate and respectively similarity coefficient between ,in , ,use Analysis repulsion coefficient ,in Select in sequence , determine repulsion coefficient .

[0015] Preferably, the quality score calculation unit includes a text quality assessment module, a video quality assessment module, an audio quality assessment module, and an exclusion type determination module;

[0016] The text quality assessment module statistically analyzes the text modal data before preprocessing. Length value Number of special characters and fluency ,use Calculate Length rating ,in , This represents the optimal length threshold, through and Analyze the character ratio ,in ,according to Calculate mass fraction ,in , Indicates maximum fluency. express Weighting coefficients;

[0017] The video quality assessment module determines the video modal data before preprocessing. Analyze the Laplace variance of the grayscale image. Pixel brightness standard deviation Maximum brightness value and minimum brightness value Afterwards, according to Calculate the contrast score ,in ,pass Determine the luminance variance score ,in , This represents the optimal brightness variance. Representing control parameters, using and Calculate mass fraction ,in ,in This represents the maximum value of the Laplace variance. express Weighting coefficients;

[0018] The audio quality assessment module extracts audio modal data before preprocessing. signal-to-noise ratio and the proportion of audio frames ,according to and Calculate mass fraction ,in , express Weighting coefficients;

[0019] The rejection type determination module sets a rejection threshold. and quality threshold Select text modal data before and after preprocessing. The corresponding repulsion coefficients and quality fraction ,like If the current type is determined to be non-rejective, then... and If the current type is determined to be intrinsic rejection, then... and If the current type is determined to be external rejection, then select in sequence. and This will output the corresponding rejection type.

[0020] Preferably, the first round determination unit includes an initialization configuration module and an exclusion type acquisition module;

[0021] The initialization configuration module sets the scheduler status parameters, which include the current round, minimum round, maximum round, current round dominant mode, confidence history, and dominant mode history. Then, the current round and dominant mode in the status parameters are initialized respectively.

[0022] After the rejection type acquisition module obtains the dominant mode and the corresponding rejection type of the current round, if the rejection type is external rejection, it determines that the required granularity features of the dominant mode are coarse-grained and medium-grained features; if the type is not external rejection, it determines that the required granularity features are coarse-grained features.

[0023] Preferably, the first round determination unit further includes a result output module, a confidence level analysis module, and a confidence level comparison module;

[0024] The result output module extracts the granular features required for the dominant mode and the coarse-grained features of other modes, and then uses the coarse-grained features of the dominant mode as the query vector. , No. Coarse-grained features of other modalities serve as keys Sum ,according to and Calculate the weight vector ,in , Indicates the scaling factor. Representing the normalized exponential function, using and Analyze the query results ,in , The representation layer normalization function is repeated until all query results corresponding to all other modalities are parsed, and then fused to obtain the output result of the current round;

[0025] The confidence resolution module will analyze the hidden state vector. After initializing it as a zero vector, it is compared with the output of the current round. Perform joint analysis to obtain the current hidden state vector. ,in , Representing the forgetting factor, through Analyze the confidence level of the current round. ,in , Represents the weight matrix. This represents the bias vector. Number of categories representing intent;

[0026] After the confidence comparison module sets a safety threshold, if the current round is greater than or equal to the minimum round in the state parameters and the confidence is greater than the safety threshold, the predicted intent type is directly output; otherwise, the next round of the loop is entered.

[0027] Preferably, the second round determination unit includes a type adjustment module, a mode selection module, a result integration module, a gain coefficient calculation module, and a gain coefficient comparison module;

[0028] When the type adjustment module enters the second round, it obtains the weight vector and quality scores of each modality feature from the previous round, calculates the quality score after the first round, analyzes the similarity coefficient between the coarse-grained features of each modality and the current hidden state vector, calculates the repulsion coefficient of each modality according to the similarity coefficient, and uses the repulsion coefficient and quality score to determine the repulsion type of each modality data after the first round.

[0029] The modality selection module scans all modalities. If there is a modality of external rejection, the modality corresponding to the maximum rejection coefficient is selected as the dominant modality in the second round. Otherwise, it determines whether the confidence level in the first round is lower than the constraint threshold. If it is lower than the constraint threshold, the modality is directly designated as the dominant modality. If it is not lower than the constraint threshold, all modalities of internal rejection are excluded, and the dominant modality is selected sequentially from the remaining modalities.

[0030] After obtaining the second-round dominant mode and the corresponding rejection type, the result integration module analyzes the granularity features required by the dominant mode. Using the coarse-grained features of the dominant mode and other modes, it calculates the weight vector and the initial output result in the current round. If there are medium-grained features among the granular features required by the dominant mode, it identifies the key regions of other modes by combining the current weight vector, calculates the local output result through the medium-grained features of the dominant mode, and integrates the initial output result with the local output result to obtain the output result of the current round.

[0031] The gain coefficient calculation module utilizes the hidden state vector from the previous round. Update the current hidden state vector with the output of the current round. And the confidence level for the second round is calculated using this value. ,use and Analyze the gain coefficient of the current round. ,in , This represents the dimension-aligned weight matrix. The bias term indicating dimension alignment;

[0032] If the confidence level of the gain coefficient comparison module is greater than or equal to the standard threshold, it will terminate forcibly and output the prediction intent type. If the gain coefficient is less than the saturation threshold and the confidence level is less than the standard threshold but greater than the safety threshold, it will directly output the prediction intent type. Otherwise, it will enter the next round of loop.

[0033] Preferably, the third round determination unit includes a change value analysis module, a granularity feature analysis module, and a state vector update module;

[0034] When the change value analysis module enters the third round, it sets a standard interval threshold. and standard total threshold Receive confidence levels from the first and second rounds. and initial value ,use and Calculate the interval change value Total change ,in , ,like and If the confidence level is positive, it is determined that there is no increase in confidence; otherwise, it is determined that there is an increase in confidence.

[0035] The granular feature analysis module selects the remaining modes that have not yet become dominant in the previous two rounds as the dominant mode in this round. It scans the rejection type of the mode, counts the current remaining resources and task complexity. If the mode has external conflict, no confidence improvement and remaining resources greater than a preset value, and task complexity greater than 0.8, then the granular features required for the dominant mode are determined to be coarse-grained, medium-grained and fine-grained features. When the mode has external conflict, if at least one of the following is true: no confidence improvement, remaining resources not greater than a preset value, and task complexity not greater than 0.8, then the granular features required for the dominant mode are determined to be coarse-grained and medium-grained features. If the mode does not have external conflict, then the granular features required for the dominant mode are determined to be coarse-grained features.

[0036] The state vector update module calculates the output result of the current round using the granular features required by the dominant mode and the coarse-grained features of other modes, and then updates the output based on the hidden state vector of the second round. Determine the hidden state vector for the current round. After setting the confidence level and gain coefficient, since the third round has reached the maximum number of rounds, the loop will be forcibly terminated, and the hidden state vectors of all rounds will be output.

[0037] Preferably, the intent recognition unit includes a state vector fusion module and an explanatory text generation module;

[0038] The state vector fusion module receives the hidden state vectors from all rounds, maps all vectors to a unified dimension, and selects the mapped vectors from the first round. Hidden state vectors , determine Weighting coefficients ,in , All represent learnable parameters. After selecting all hidden state vectors in sequence, the corresponding weight coefficients are obtained. The hidden state vectors of all rounds are fused using the weight coefficients to obtain the final hidden state vector, which is then input into the classifier to output the intent type and confidence score.

[0039] The explanatory text generation module receives the weight vectors, dominant modes, exclusion types, and confidence levels for each round. It then determines the contribution ratio of each round based on the weight vectors, uses the exclusion types to locate key contradictions, traces key rounds through changes in confidence levels, and generates corresponding explanatory texts using natural language templates.

[0040] Compared with the prior art, the beneficial effects of the present invention are:

[0041] 1. This invention can accurately distinguish between internal rejection caused by low data quality and external rejection caused by real semantic contradictions by calculating the rejection coefficient and quality score. It prioritizes the selection of modalities with external rejection as the main focus for cyclical querying, so that the real contradictions are given priority attention. At the same time, it dynamically adjusts the state and weight according to the rejection type. When an external conflict is detected, it uses new information to guide the state update and reset the weight vector, avoiding being misled by early misunderstandings. This enables more accurate intent judgment to be generated in subsequent multi-view fusion decoding.

[0042] 2. After extracting multi-granularity features, this invention schedules them on demand. In the first round, only coarse-grained features are used for rapid preliminary understanding. In the second round, when external factors are detected as rejection, the extraction of granular features on demand is triggered. This mechanism avoids the redundant overhead of calculating all granularities for all modalities. At the same time, the granularity activation strategy is adjusted according to the system load to ensure that the response speed is maintained under high load. By utilizing the attention weights of each round, the changes in the dominant modality, and the confidence evolution trajectory, an interpretable report in natural language is generated. This transparent decision-making process enables managers to locate the cause of errors, and users can understand the basis of the system's judgment, providing a trust foundation for human-machine collaborative decision-making. Attached Figure Description

[0043] Figure 1 A schematic diagram of the overall system flow is provided for embodiments of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] Please see Figure 1 This invention provides a technical solution: a semantic alignment system for multimodal intent recognition, comprising:

[0046] The modal data analysis unit is used to receive multimodal data, preprocess it, and extract multi-granularity features from each modal data to calculate the corresponding repulsion coefficient.

[0047] The quality score calculation unit is used to statistically analyze the modal data before preprocessing, determine the corresponding quality scores, and identify the rejection type of different modal data based on the rejection coefficient and quality score.

[0048] The first round determination unit is used to set the scheduler state parameters, obtain the dominant mode of the current round, determine the coarse-grained features of the dominant mode, and calculate the hidden state vector, confidence and weight vector of the current round.

[0049] The second round determination unit is used to obtain the weight vector of the previous round after entering the second round, determine the current quality score based on the weight vector, analyze the required granularity features of the dominant mode in the current round, and calculate the hidden state vector and confidence level of the current round using the required granularity features of the dominant mode.

[0050] The third round determination unit is used to receive the confidence levels from the first two rounds, determine the required granularity features of the current dominant mode according to the change in confidence level, and output the hidden state vectors of all rounds.

[0051] The intent recognition unit is used to fuse all hidden state vectors and input them into the classifier to obtain the intent type and confidence level.

[0052] The modal data analysis unit includes a feature extraction module and a repulsion coefficient calculation module;

[0053] The feature extraction module receives multimodal data. The data is then standardized to obtain preprocessed modal data. ,in , , These represent the text modal data before and after preprocessing, respectively. These represent the video modal data before and after preprocessing, respectively. These represent the audio modal data before and after preprocessing, respectively. A multi-granularity feature extractor is used to... The corresponding multi-granularity features are obtained by processing them sequentially. ,in express Multi-granularity features, express Multi-granularity features, express Multi-granularity features;

[0054] Using a multi-granularity feature extractor to The corresponding multi-granularity features are obtained by processing them sequentially. The specific process is as follows:

[0055] S101. Transfer the preprocessed text modal data The input is fed into the BERT model, and after passing through multiple Transformer encoding layers, the final hidden state corresponding to the special label [CLS] is taken. This hidden state is then compressed to 64 dimensions through a fully connected mapping layer, thus obtaining... coarse-grained characteristics ;

[0056] S102. After obtaining all the hidden state sequences of tokens output by BERT, The data is divided into non-overlapping windows of length 2, meaning each pair of consecutive tokens forms a segment. Average pooling is performed on the two token vectors within each segment to obtain segment-level features. These features are then compressed to 128 dimensions using a fully connected layer. Medium-grained characteristics ;

[0057] S103. Take the hidden state sequence of all tokens output by BERT, and compress the dimension of each token to 256 dimensions through a fully connected layer to obtain... Fine-grained features ;

[0058] S104, will Combine and output Multi-granularity features ;

[0059] S105. Transfer the preprocessed video modal data The data is input frame by frame into the MobileNetV3 network. After feature extraction by the convolutional backbone network, a feature map is output for each frame. Global average pooling is performed on the feature map of each frame to compress the spatial dimension and obtain a frame-level feature sequence.

[0060] S106. Perform average pooling on the frame-level feature sequence, aggregate the vectors in the sequence to obtain a video-level vector, and compress it to 64 dimensions through a fully connected layer. coarse-grained characteristics ;

[0061] S107. Divide the frame-level feature sequence into non-overlapping windows of length 4, i.e., every 4 consecutive frames form a segment. Perform average pooling on the 4 frame-level feature vectors within each segment to obtain segment-level features. Then, compress the segment feature dimension to 128 dimensions through a fully connected layer. Medium-grained characteristics ;

[0062] S108, Directly use frame-level feature vector sequences as Fine-grained features ;

[0063] S109, will Combine and output Multi-granularity feature vector .

[0064] S110, process the preprocessed audio modal data The input is fed into the Wav2Vec2 model, where local acoustic features are extracted by a convolutional feature encoder, and long-term dependencies are captured by a Transformer encoder, outputting a time-frame feature sequence.

[0065] S111. Perform global average pooling on the time-frame feature sequence output by the Wav2Vec2 model, aggregate the vectors in the sequence into a global audio vector, and compress it to 64 dimensions through a fully connected layer to obtain... coarse-grained characteristics ;

[0066] S112. Divide the time frame feature sequence into non-overlapping windows of length 4. Each segment consists of 4 consecutive frames. Perform average pooling on the 4 frame-level vectors within each segment to obtain segment-level features. Compress the segment feature dimension to 128 dimensions using a fully connected layer. Medium-grained characteristics ;

[0067] S113, Directly take the time frame sequence output by the Wav2Vec2 model as... Fine-grained features ;

[0068] S114, will Combine and output Multi-granularity feature vector ;

[0069] Extraction of repulsion coefficient calculation module coarse-grained features in Then, select Calculate and respectively similarity coefficient between ,in , ,use Analysis repulsion coefficient ,in Select in sequence , determine repulsion coefficient ;

[0070] The quality score calculation unit includes a text quality assessment module, a video quality assessment module, an audio quality assessment module, and an exclusion type determination module;

[0071] The text quality assessment module statistically analyzes the text modal data before preprocessing. Length value Number of special characters and fluency ,use Calculate Length rating ,in , This represents the optimal length threshold, through and Analyze the character ratio ,in ,according to Calculate mass fraction ,in , Indicates maximum fluency. express Weighting coefficients;

[0072] The video quality assessment module determines the video modal data before preprocessing. Analyze the Laplace variance of the grayscale image. Pixel brightness standard deviation Maximum brightness value and minimum brightness value Afterwards, according to Calculate the contrast score ,in ,pass Determine the luminance variance score ,in , This represents the optimal brightness variance. Representing control parameters, using and Calculate mass fraction ,in ,in This represents the maximum value of the Laplace variance. express Weighting coefficients;

[0073] The audio quality assessment module extracts audio modal data before preprocessing. signal-to-noise ratio and the proportion of audio frames ,according to and Calculate mass fraction ,in , express Weighting coefficients;

[0074] The rejection type determination module sets the rejection threshold. and quality threshold Select text modal data before and after preprocessing. The corresponding repulsion coefficients and quality fraction ,like If the current type is determined to be non-rejective, then... and If the current type is determined to be intrinsic rejection, then... and If the current type is determined to be external rejection, then select in sequence. and This outputs the corresponding rejection type.

[0075] The first round of determination units includes an initialization configuration module and an exclusion type acquisition module;

[0076] The initialization configuration module sets the scheduler status parameters, which include the current round, minimum round, maximum round, current round dominant mode, confidence history, and dominant mode history. Then, the current round and dominant mode in the status parameters are initialized respectively.

[0077] After the rejection type acquisition module obtains the dominant mode and the corresponding rejection type of the current round, if the rejection type is external rejection, it determines that the required granularity features of the dominant mode are coarse-grained and medium-grained features; if the type is not external rejection, it determines that the required granularity features are coarse-grained features.

[0078] The first round of determination also includes a result output module, a confidence level analysis module, and a confidence level comparison module;

[0079] The output module extracts the granular features required for the dominant mode and the coarse-grained features of other modes, and then uses the coarse-grained features of the dominant mode as the query vector. , No. Coarse-grained features of other modalities serve as keys Sum ,according to and Calculate the weight vector ,in , Indicates the scaling factor. Representing the normalized exponential function, using and Analyze the query results ,in , The representation layer normalization function is repeated until all query results corresponding to all other modalities are parsed, and then fused to obtain the output result of the current round;

[0080] The confidence resolution module will use the hidden state vector After initializing it as a zero vector, it is compared with the output of the current round. Perform joint analysis to obtain the current hidden state vector. ,in , Representing the forgetting factor, through Analyze the confidence level of the current round. ,in , Represents the weight matrix. This represents the bias vector. Number of categories representing intent;

[0081] After setting a safety threshold, if the current round is greater than or equal to the minimum round in the state parameters and the confidence level is greater than the safety threshold, the predicted intent type will be output directly; otherwise, the next round of the loop will begin.

[0082] The second round of determination units includes a type adjustment module, a mode selection module, a result integration module, a gain coefficient calculation module, and a gain coefficient comparison module;

[0083] When the type adjustment module enters the second round, it obtains the weight vector and quality scores of each modality feature from the previous round, calculates the quality score after the first round, analyzes the similarity coefficient between the coarse-grained features of each modality and the current hidden state vector, calculates the repulsion coefficient of each modality according to the similarity coefficient, and uses the repulsion coefficient and quality score to determine the repulsion type of each modality data after the first round.

[0084] The modality selection module scans all modalities. If there is a modality of type external rejection, the modality corresponding to the maximum rejection coefficient is selected as the dominant modality in the second round. Otherwise, it determines whether the confidence level in the first round is lower than the constraint threshold. If it is lower than the constraint threshold, the modality is directly designated as the dominant modality. If it is not lower than the constraint threshold, all modalities of type internal rejection are excluded, and the dominant modality is selected in order from the remaining modalities.

[0085] After the result integration module obtains the dominant mode and the corresponding rejection type in the second round, it analyzes the granularity features required for the dominant mode. Using the coarse-grained features of the dominant mode and other modes, it calculates the weight vector and the initial output result in the current round. If there are medium-grained features among the granular features required for the dominant mode, it identifies the key regions of other modes by combining the current weight vector, calculates the local output result through the medium-grained features of the dominant mode, and integrates the initial output result with the local output result to obtain the output result of the current round.

[0086] The gain coefficient calculation module utilizes the hidden state vector from the previous round. Update the current hidden state vector with the output of the current round. And the confidence level for the second round is calculated using this value. ,use and Analyze the gain coefficient of the current round. ,in , This represents the dimension-aligned weight matrix. The bias term indicating dimension alignment;

[0087] If the confidence level of the gain coefficient comparison module is greater than or equal to the standard threshold, it will terminate forcibly and output the prediction intent type. If the gain coefficient is less than the saturation threshold and the confidence level is less than the standard threshold but greater than the safety threshold, it will directly output the prediction intent type. Otherwise, it will enter the next round of loop.

[0088] The third round of determination units includes a change value analysis module, a granularity feature analysis module, and a state vector update module;

[0089] When the change value analysis module enters the third round, it sets the standard interval threshold. and standard total threshold Receive confidence levels from the first and second rounds. and initial value ,use and Calculate the interval change value Total change ,in , ,like and If the confidence level is positive, it is determined that there is no increase in confidence; otherwise, it is determined that there is an increase in confidence.

[0090] The granular feature analysis module selects the remaining modes that have not yet become dominant in the previous two rounds as the dominant mode in this round. It scans the rejection type of the mode, counts the current remaining resources and task complexity. If the mode has external conflict, no confidence improvement and remaining resources greater than the preset value, and task complexity greater than 0.8, then the granular features required for the dominant mode are determined to be coarse-grained, medium-grained and fine-grained features. When the mode has external conflict, if at least one of the following is true: no confidence improvement, remaining resources not greater than the preset value, and task complexity not greater than 0.8, then the granular features required for the dominant mode are determined to be coarse-grained and medium-grained features. If the mode does not have external conflict, then the granular features required for the dominant mode are determined to be coarse-grained features.

[0091] The state vector update module calculates the output of the current round using the granular features required by the dominant mode and the coarse-grained features of other modes, based on the hidden state vector of the second round. Determine the hidden state vector for the current round. After setting the confidence level and gain coefficient, since the third round has reached the maximum number of rounds, the loop will be forcibly terminated, and the hidden state vectors of all rounds will be output.

[0092] The intent recognition unit includes a state vector fusion module and an interpretation text generation module;

[0093] The state vector fusion module receives the hidden state vectors from all rounds, maps all vectors to a unified dimension, and selects the mapped vectors from the first round. Hidden state vectors , determine Weighting coefficients ,in , All represent learnable parameters. After selecting all hidden state vectors in sequence, the corresponding weight coefficients are obtained. The hidden state vectors of all rounds are fused using the weight coefficients to obtain the final hidden state vector, which is then input into the classifier to output the intent type and confidence score.

[0094] After receiving the weight vectors, dominant modalities, exclusion types, and confidence levels for each round, the explanatory text generation module determines the contribution ratio of each round based on the weight vectors, locates key contradictions using the exclusion types, traces key rounds through changes in confidence levels, and generates corresponding explanatory texts using natural language templates.

[0095] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0096] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A semantic alignment system for multimodal intent recognition, characterized in that, include: The modal data analysis unit is used to receive multimodal data, preprocess it, and extract multi-granularity features from each modal data to calculate the corresponding repulsion coefficient. The quality score calculation unit is used to statistically analyze the modal data before preprocessing, determine the corresponding quality scores, and identify the rejection type of different modal data based on the rejection coefficient and quality score. The first round determination unit is used to set the scheduler state parameters, obtain the dominant mode of the current round, determine the coarse-grained features of the dominant mode, and calculate the hidden state vector, confidence and weight vector of the current round. The second round determination unit is used to obtain the weight vector of the previous round after entering the second round, determine the current quality score based on the weight vector, analyze the required granularity features of the dominant mode in the current round, and calculate the hidden state vector and confidence level of the current round using the required granularity features of the dominant mode. The third round determination unit is used to receive the confidence levels from the first two rounds, determine the required granularity features of the current dominant mode according to the change in confidence level, and output the hidden state vectors of all rounds. The intent recognition unit is used to fuse all hidden state vectors and input them into the classifier to obtain the intent type and confidence level.

2. The semantic alignment system for multimodal intent recognition according to claim 1, characterized in that, The modal data analysis unit includes a feature extraction module and a repulsion coefficient calculation module; The feature extraction module receives multimodal data. The data is then standardized to obtain preprocessed modal data. ,in , , These represent the text modal data before and after preprocessing, respectively. These represent the video modal data before and after preprocessing, respectively. These represent the audio modal data before and after preprocessing, respectively. A multi-granularity feature extractor is used to... The corresponding multi-granularity features are obtained by processing them sequentially. ; The repulsion coefficient calculation module extracts coarse-grained features in Then, select Calculate and respectively similarity coefficient between ,in , ,use Analysis repulsion coefficient ,in Select in sequence , determine repulsion coefficient .

3. The semantic alignment system for multimodal intent recognition according to claim 1, characterized in that, The quality score calculation unit includes a text quality assessment module, a video quality assessment module, an audio quality assessment module, and an exclusion type determination module. The text quality assessment module statistically analyzes the text modal data before preprocessing. Length value Number of special characters and fluency ,use Calculate Length rating ,in , This represents the optimal length threshold, through and Analyze the character ratio ,in ,according to Calculate mass fraction ,in , Indicates maximum fluency. express Weighting coefficients; The video quality assessment module determines the video modal data before preprocessing. Analyze the Laplace variance of the grayscale image. Pixel brightness standard deviation Maximum brightness value and minimum brightness value Afterwards, according to Calculate the contrast score ,in ,pass Determine the luminance variance score ,in , This represents the optimal brightness variance. Representing control parameters, using and Calculate mass fraction ,in ,in This represents the maximum value of the Laplace variance. express Weighting coefficients; The audio quality assessment module extracts audio modal data before preprocessing. signal-to-noise ratio and the proportion of audio frames ,according to and Calculate mass fraction ,in , express Weighting coefficients; The rejection type determination module sets a rejection threshold. and quality threshold Select text modal data before and after preprocessing. The corresponding repulsion coefficients and quality fraction ,like If the current type is determined to be non-rejective, then... and If the current type is determined to be intrinsic rejection, then... and If the current type is determined to be external rejection, then select in sequence. and This will output the corresponding rejection type.

4. The semantic alignment system for multimodal intent recognition according to claim 1, characterized in that, The first round determination unit includes an initialization configuration module and an exclusion type acquisition module; The initialization configuration module sets the scheduler status parameters, and initializes the current round and dominant mode in the status parameters respectively; After the rejection type acquisition module obtains the dominant mode and the corresponding rejection type of the current round, if the rejection type is external rejection, it determines that the required granularity features of the dominant mode are coarse-grained and medium-grained features; if the type is not external rejection, it determines that the required granularity features are coarse-grained features.

5. The semantic alignment system for multimodal intent recognition according to claim 4, characterized in that, The first round determination unit also includes a result output module, a confidence level parsing module, and a confidence level comparison module; The result output module extracts the granular features required for the dominant mode and the coarse-grained features of other modes, and then uses the coarse-grained features of the dominant mode as the query vector. , No. Coarse-grained features of other modalities serve as keys Sum ,according to and Calculate the weight vector ,in , Indicates the scaling factor. Representing the normalized exponential function, using and Analyze the query results ,in , The representation layer normalization function is repeated until all query results corresponding to all other modalities are parsed, and then fused to obtain the output result of the current round; The confidence resolution module will analyze the hidden state vector. After initializing it as a zero vector, it is compared with the output of the current round. Perform joint analysis to obtain the current hidden state vector. ,in , Representing the forgetting factor, through Analyze the confidence level of the current round. ,in , Represents the weight matrix. This represents the bias vector. Number of categories representing intent; After the confidence comparison module sets a safety threshold, if the current round is greater than or equal to the minimum round in the state parameters and the confidence is greater than the safety threshold, the predicted intent type is directly output; otherwise, the next round of the loop is entered.

6. The semantic alignment system for multimodal intent recognition according to claim 1, characterized in that, The second round determination unit includes a type adjustment module, a mode selection module, a result integration module, a gain coefficient calculation module, and a gain coefficient comparison module; When the type adjustment module enters the second round, it obtains the weight vector and quality scores of each modality feature from the previous round, calculates the quality score after the first round, analyzes the similarity coefficient between the coarse-grained features of each modality and the current hidden state vector, calculates the repulsion coefficient of each modality according to the similarity coefficient, and uses the repulsion coefficient and quality score to determine the repulsion type of each modality data after the first round. The modality selection module scans all modalities. If there is a modality of external rejection, the modality corresponding to the maximum rejection coefficient is selected as the dominant modality in the second round. Otherwise, it determines whether the confidence level in the first round is lower than the constraint threshold. If it is lower than the constraint threshold, the modality is directly designated as the dominant modality. If it is not lower than the constraint threshold, all modalities of internal rejection are excluded, and the dominant modality is selected sequentially from the remaining modalities. After obtaining the second-round dominant mode and the corresponding rejection type, the result integration module analyzes the granularity features required by the dominant mode. Using the coarse-grained features of the dominant mode and other modes, it calculates the weight vector and the initial output result in the current round. If there are medium-grained features among the granular features required by the dominant mode, it identifies the key regions of other modes by combining the current weight vector, calculates the local output result through the medium-grained features of the dominant mode, and integrates the initial output result with the local output result to obtain the output result of the current round. The gain coefficient calculation module utilizes the hidden state vector from the previous round. Update the current hidden state vector with the output of the current round. And the confidence level for the second round is calculated using this value. ,use and Analyze the gain coefficient of the current round. ,in , This represents the dimension-aligned weight matrix. The bias term indicating dimension alignment; If the confidence level of the gain coefficient comparison module is greater than or equal to the standard threshold, it will terminate forcibly and output the prediction intent type. If the gain coefficient is less than the saturation threshold and the confidence level is less than the standard threshold but greater than the safety threshold, it will directly output the prediction intent type. Otherwise, it will enter the next round of loop.

7. The semantic alignment system for multimodal intent recognition according to claim 1, characterized in that, The third round determination unit includes a change value analysis module, a granularity feature analysis module, and a state vector update module; When the change value analysis module enters the third round, it sets a standard interval threshold. and standard total threshold Receive confidence levels from the first and second rounds. and initial value ,use and Calculate the interval change value Total change ,in , ,like and If the confidence level is positive, it is determined that there is no increase in confidence; otherwise, it is determined that there is an increase in confidence. The granular feature analysis module selects the remaining modes that have not yet become dominant in the previous two rounds as the dominant mode in this round. It scans the rejection type of the mode, counts the current remaining resources and task complexity. If the mode has external conflict, no confidence improvement and remaining resources greater than a preset value, and task complexity greater than 0.8, then the granular features required for the dominant mode are determined to be coarse-grained, medium-grained and fine-grained features. When the mode has external conflict, if at least one of the following is true: no confidence improvement, remaining resources not greater than a preset value, and task complexity not greater than 0.8, then the granular features required for the dominant mode are determined to be coarse-grained and medium-grained features. If the mode does not have external conflict, then the granular features required for the dominant mode are determined to be coarse-grained features. The state vector update module calculates the output result of the current round using the granular features required by the dominant mode and the coarse-grained features of other modes, and then updates the output based on the hidden state vector of the second round. Determine the hidden state vector for the current round. After setting the confidence level and gain coefficient, since the third round has reached the maximum number of rounds, the loop will be forcibly terminated, and the hidden state vectors of all rounds will be output.

8. The semantic alignment system for multimodal intent recognition according to claim 1, characterized in that, The intent recognition unit includes a state vector fusion module and an explanation text generation module; The state vector fusion module receives the hidden state vectors from all rounds, maps all vectors to a unified dimension, and selects the mapped vectors from the first round. Hidden state vectors , determine Weighting coefficients ,in , All represent learnable parameters. After selecting all hidden state vectors in sequence, the corresponding weight coefficients are obtained. The hidden state vectors of all rounds are fused using the weight coefficients to obtain the final hidden state vector, which is then input into the classifier to output the intent type and confidence score. The explanatory text generation module receives the weight vectors, dominant modes, exclusion types, and confidence levels for each round. It then determines the contribution ratio of each round based on the weight vectors, uses the exclusion types to locate key contradictions, traces key rounds through changes in confidence levels, and generates corresponding explanatory texts using natural language templates.

Images