A multi-modal data-driven user consultation intent classification method and system

Abstract

The application relates to the technical field of artificial intelligence, and discloses a multi-modal data driven user consultation intention classification method and system, which aims to solve the problems of classification precision decline, calculation redundancy and high inference delay of edge devices caused by the forced fusion of time-asynchronous multi-modal data in the existing model. The method comprises the following steps: labeling high-precision timestamps for each modal data; extracting and enhancing the time sequence features; dynamically evaluating the timeliness of the features through a learnable modal-specific attenuation function, and gating and pruning the expired features; performing lightweight cross-modal attention fusion among the sparse effective features; and finally outputting the intention classification result and calibrating the confidence based on the timeliness score variance. The system comprises five modules of asynchronous receiving, feature extraction and time sequence encoding, effectiveness evaluation and gating, sparse fusion and classification calibration. Through the pre-screening and post-fusion mechanism, the application significantly reduces the calculation cost while ensuring high accuracy, and is suitable for edge deployment and multi-service scene adaptation.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically relating to a multimodal data-driven method and system for classifying user consultation intent. Background Technology

[0002] With the widespread application of artificial intelligence technology in systems such as intelligent customer service and virtual assistants, multimodal user intent classification has become a key step in improving the intelligence level of services. Existing methods are generally based on the Transformer architecture, assuming that the data of each modality are synchronized in time and that feature fusion is achieved through cross-modal attention. However, in real customer service environments, user input often exhibits asynchronous phenomena such as voice precedence and text lag, making it difficult for models that rely on forced synchronization to accurately resolve intent boundaries and leading to misjudgments.

[0003] Existing technologies exhibit multiple structural flaws when addressing asynchronous time issues: First, fixed-time-window strategies cannot adapt to the varying latency tolerances of different business scenarios. In e-commerce, reasonable delays caused by users verifying information are misjudged as invalid input, while in financial scenarios, excessively wide windows introduce security risks. Second, generative completion methods incorrectly equate "expired" with "missing," generating noise on top of existing expired features, which exacerbates semantic confusion. Third, even if some solutions attempt to dynamically adjust the window, their reliance on historical statistics makes them unsuitable for responding to sudden input interruptions or latency spikes. Finally, existing models suffer from severe computational redundancy when deployed on edge devices, with inference latency often exceeding industry real-time standards. Furthermore, the involvement of expired features in attention calculations leads to inflated confidence levels and inaccurate results.

[0004] Furthermore, in specific application scenarios such as customer service for payment institutions, classifying user inquiry intent faces another challenge: existing solutions largely rely on human customer service or basic AI customer service. Human customer service is inefficient, costly, and lacks standardized service, while basic AI customer service, due to its reliance on keyword matching and fixed templates, cannot handle complex expressions involving colloquialisms, typos, and contextual dependencies, resulting in low intent recognition accuracy, rigid responses, and outdated knowledge bases. For example, in payment inquiries, if a user enters "Why hasn't the 500 yuan I transferred yesterday arrived in the recipient's account yet?", basic AI customer service struggles to accurately identify the intent due to colloquial vocabulary and a lack of keywords, leading to a high misjudgment rate and severely impacting user experience and automated processing efficiency.

[0005] Therefore, there is an urgent need for a new paradigm for multimodal intent classification that can dynamically identify modal timeliness at the millisecond level, accurately eliminate expired noise, and take into account lightweight deployment at the edge. Summary of the Invention

[0006] The technical problem to be solved by this invention is that the existing multimodal fusion model for classifying user consultation intent relies heavily on the strict synchronization and alignment of each modality data in the time dimension. When faced with the asynchronous modal input time phenomenon that is common in industrial application scenarios, the forced fusion of expired or invalid modal features will lead to a significant decrease in intent classification accuracy, ineffective consumption of computing resources, and excessively high inference latency on edge devices.

[0007] To address the aforementioned technical issues, this invention provides a multimodal data-driven method and system for classifying user consultation intent. It introduces a learnable temporal validity metric coupled to the absolute timestamp of each input modal feature. Before entering the cross-modal fusion stage, the "timeliness" of each modal feature is dynamically quantified and screened, actively pruning expired features in the time dimension. Only features with high timeliness scores are allowed to participate in subsequent sparse cross-modal attention calculations, thereby eliminating semantic noise introduced by asynchronous inputs at its source and significantly reducing the computational complexity of the model. Ultimately, this achieves the goal of balancing high classification accuracy with real-time inference performance on edge devices.

[0008] According to one aspect of the present invention, a multimodal data-driven method for classifying user consultation intent is provided, comprising the following steps:

[0009] S1 acquires user input data streams containing text and voice in real time through a multimodal data asynchronous receiving interface, and accurately labels each frame of independent modal data with a high-precision system timestamp, forming a raw multimodal data sequence with timestamps;

[0010] S2, the original multimodal data sequence with timestamps is input into a parallel modality-independent feature extraction network. For each modality, the modality-independent feature extraction network uses a preset deep learning model to convert its original multimodal data sequence into an initial feature vector sequence, wherein each feature vector retains the timestamp information of its corresponding original multimodal data sequence.

[0011] S3. For each feature vector in the initial feature vector sequence, use a predefined temporal coding function to encode its timestamp information into a temporal position vector, and concatenate the temporal position vector with the corresponding initial feature vector to generate a temporally enhanced feature vector sequence.

[0012] S4. Input the temporal augmentation feature vector sequence into the temporal validity dynamic evaluation and gating module. This module calculates a temporal validity score for each temporal augmentation feature vector in the temporal augmentation feature vector sequence based on a learnable modality-specific decay function related to the current session context. The temporal validity score characterizes the semantic value of the temporal augmentation feature vector at the current decision time point.

[0013] S5. Based on a preset validity score threshold, a gating operation is performed on the time-enhanced feature vector sequence, setting feature vectors with scores below the threshold to zero, thereby generating a sparse feature vector sequence containing only time-sensitive features.

[0014] S6, feed the sparse feature vector sequence into a lightweight cross-modal sparse attention fusion module. This lightweight cross-modal sparse attention fusion module only performs cross-modal attention calculation between non-zero feature vectors in the sparse feature vector sequence to generate a fusion feature vector that can represent the current user's core intent.

[0015] S7 inputs the fused feature vector into the intent classifier, calculates the probability distribution of each predefined intent category through a fully connected network and output layer, and outputs the intent category with the highest probability value as the final classification result.

[0016] As one embodiment of the present invention, accurately labeling each frame of independent modal data with a high-precision system timestamp specifically includes:

[0017] Using the system clock synchronized with the network time protocol server, an absolute timestamp in milliseconds is generated for each received frame of text data, each slice of audio data, or each frame of image data.

[0018] As one embodiment of the present invention, the modality-independent feature extraction network specifically includes:

[0019] A text feature extraction sub-network for processing text data employs a distilled bidirectional encoder representation model with a four-layer encoder structure, a hidden layer dimension of 256, and four attention heads, which converts the input text sequence into a 768-dimensional word vector sequence as the initial text feature vector sequence.

[0020] A speech feature extraction subnetwork for processing speech data employs a time-delay neural network consisting of five stacked one-dimensional convolutional modules. The input speech signal is first converted into a Mel-frequency cepstral coefficient sequence, and then a 128-dimensional acoustic feature vector sequence is extracted by the network as the initial speech feature vector sequence.

[0021] In a preferred embodiment of the present invention, the text feature extraction subnetwork in the modality-independent feature extraction network further includes a preprocessing unit and a dedicated intent recognition unit:

[0022] The preprocessing unit is used to clean the text data, correct typos, and convert it into colloquial language, and to associate it with the context of multi-turn dialogue;

[0023] The dedicated intent recognition unit is based on a large model finely tuned from labeled data in the payment industry. Combined with natural language understanding technology, it outputs intent categories and key parameters, and supports context-dependent intent completion.

[0024] In addition, the intent classifier works in conjunction with the RAG enhanced knowledge base module to retrieve business knowledge based on intent categories and key parameters, and dynamically generate appropriate responses by combining user profiles, thereby further improving classification accuracy and service adaptability.

[0025] As one embodiment of the present invention, encoding timestamp information into a time-series position vector using a predefined time-series coding function specifically includes:

[0026] The millisecond-level timestamps are encoded using sine and cosine functions. The mathematical expression is: for a timestamp position pos and dimension index i, the component of the time-series position vector in the even-numbered dimension is the sine function value. The components in the odd-numbered dimensions are cosine function values. ,in is the dimension of the feature vector.

[0027] As one embodiment of the present invention, the learnable mode-specific decay function used in the timing effectiveness dynamic evaluation and gating module is specifically as follows:

[0028] An exponentially decaying function, in the form of: ,in For time series validity score, The current system time. The timestamp of the feature vector. The decay rate parameter for mode m; decay rate parameter It is not a fixed value, but is dynamically generated by a meta-learning network based on the business type identifier (finance, e-commerce, ticketing) of the current session, so as to achieve adaptive adjustment of time sensitivity under different business scenarios.

[0029] As one embodiment of the present invention, the lightweight cross-modal sparse attention fusion module is specifically implemented as follows:

[0030] The lightweight cross-modal sparse attention fusion module adopts a multi-head attention mechanism with two attention heads, and the dimensions of its query, key, and value matrices are all set to 128. When calculating the attention weight matrix, a pre-generated binary mask matrix is ​​used to force the row and column values ​​corresponding to the feature vectors in the sequence to be fused that are zeroed by the gating operation to be set to negative infinity. This makes the weights of these invalid features zero in the subsequent normalized exponential function calculation, so that information interaction and fusion only occur between valid temporal features.

[0031] In one embodiment of the present invention, the method further includes an intent classification confidence calibration step, which is performed after the intent classifier outputs the probability distribution, and specifically includes:

[0032] Calculate the variance of all temporal effectiveness scores in the sparse feature vector sequence before entering the lightweight cross-modal sparse attention fusion module;

[0033] The temperature scaling factor T before the output layer of the intent classifier is dynamically adjusted based on the variance value, and its calculation formula is as follows: Where Var is the variance operator. Let be the set of validity scores, and α be a preset hyperparameter;

[0034] The original logical value output by the intent classifier is divided by the temperature scaling factor T and then input into the normalized exponential function layer to smooth the probability distribution of the output, thereby reducing the risk of overconfidence and misjudgment when the system processes asynchronous inputs with huge time differences.

[0035] According to another aspect of the present invention, a multimodal data-driven user consultation intent classification system is provided, comprising:

[0036] The multimodal data asynchronous reception and timestamp annotation module is used to acquire user input data streams containing text, voice or images in real time, and accurately annotate each frame of independent modal data with a high-precision system timestamp to form a raw multimodal data sequence with timestamps;

[0037] The modality-independent feature extraction and temporal coding module is used to convert the original multimodal data sequence with timestamps into a temporally enhanced feature vector sequence, where each feature vector explicitly contains its timestamp information.

[0038] The time-series validity dynamic evaluation and gating module is used to calculate a time-series validity score for each feature vector in the time-enhanced feature vector sequence based on a learnable modality-specific decay function related to the current session context, and to perform gating operation on the sequence based on a preset threshold to generate a sparse feature vector sequence containing only time-series validity features.

[0039] The lightweight cross-modal sparse attention fusion module is used to perform cross-modal attention computation only among non-zero feature vectors in the sparse feature vector sequence, generating a fusion feature vector that can represent the core intent of the current user.

[0040] The intent classification and confidence calibration module is used to classify the fused feature vectors into predefined intent categories and dynamically calibrate the output confidence based on the temporal validity score distribution of the input features.

[0041] As one embodiment of the present invention, the modality-independent feature extraction and temporal coding module is specifically configured as follows:

[0042] A text processing pathway includes a lexicalization unit, a distilled bidirectional encoder representation model, and a text temporal coding unit for converting text data streams into temporally enhanced text feature vectors.

[0043] A speech processing pathway includes an acoustic feature extraction unit, a time-delayed neural network, and a speech temporal coding unit, used to convert speech data streams into time-enhanced acoustic feature vectors;

[0044] Both the text timing coding unit and the speech timing coding unit use sine and cosine functions to encode timestamps and concatenate them with the initial feature vectors of their respective modalities.

[0045] As one embodiment of the present invention, the core components of the time-series validity dynamic evaluation and gating module are a meta-learning network and a gating unit:

[0046] The meta-learning network has its input connected to a context interface that provides an identifier for the current session's service type. Its network structure is a feedforward neural network with two fully connected layers, and its output is a specific decay rate parameter for each modality in the current session.

[0047] The gating unit receives the attenuation rate parameter and the time-enhanced feature vector sequence, calculates the validity score of each feature vector through the built-in exponential decay function, and uses a hardware-implemented comparator circuit to set the values ​​of each dimension of the feature vector with a score lower than a preset hardware threshold to zero in parallel.

[0048] As one embodiment of the present invention, the hardware implementation of the lightweight cross-modal sparse attention fusion module is as follows:

[0049] A dedicated tensor processing unit integrates a computational core array for performing matrix multiplication. Before performing matrix multiplication operations to calculate attention weights, this unit directly shields memory access and computation operations on zero-value input vectors through a programmable mask logic unit, thereby saving computing resources at the hardware level and ensuring inference latency of less than 200 milliseconds on edge computing platforms.

[0050] In summary, this application includes at least one of the following beneficial technical effects:

[0051] I. By introducing a learnable temporal validity evaluation mechanism coupled with an absolute timestamp, this invention enables the model to quantify and perceive the freshness of each modal feature, fundamentally solving the technical problem that existing technologies cannot distinguish between valid inputs and expired noise, and significantly improving the accuracy of intent classification in time-asynchronous scenarios.

[0052] Second, the pre-screening and post-fusion architecture adopted in this invention actively prunes a large amount of invalid and outdated feature data before fusion through a time-series validity gating module. Combined with the subsequent lightweight sparse attention mechanism, this greatly reduces the computational complexity of cross-modal interactions. This not only avoids the waste of computational resources caused by waiting for delayed modalities, but also enables the entire model to meet the deployment requirements of resource-constrained edge devices in terms of parameter quantity and computational load, thus solving the imbalance between resource consumption and model accuracy.

[0053] Third, the modal decay rate parameter in this invention is dynamically generated by the meta-learning network according to the business scenario, which gives the system the ability to adaptively adjust according to the different time sensitivity requirements of different businesses (finance, e-commerce), and overcomes the shortcomings of the fixed time window or fixed threshold method in the existing technology in the case of poor applicability in the ever-changing industrial scenario.

[0054] Fourth, by introducing a confidence calibration mechanism based on the variance of time-series validity scores, this invention enables the system to proactively reduce the confidence of the output when facing extremely asynchronous inputs and high classification uncertainty, providing a more reliable basis for decision-making for downstream systems and effectively suppressing the phenomenon of high-confidence erroneous judgments in existing fusion models when dealing with semantic conflicts. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the overall technical solution architecture of the present invention;

[0056] Figure 2 This is a schematic diagram of the core principle framework of the dynamic evaluation and gating mechanism for time-series validity in this invention. Detailed Implementation

[0057] To further illustrate the technical means and effects adopted by the present invention in order to achieve the intended purpose, the following detailed description is provided in conjunction with the accompanying drawings and preferred embodiments, based on specific implementation methods of the present invention.

[0058] refer to Figure 1 and Figure 2 As shown, this invention provides a multimodal data-driven method and system for classifying user consultation intent, aiming to solve problems such as decreased intent classification accuracy, wasted computing resources, and excessively high inference latency of edge devices caused by asynchronous modal input time in industrial customer service scenarios.

[0059] A multimodal data-driven method for classifying user consultation intent includes the following steps:

[0060] S1 acquires user input data streams containing text and voice in real time through a multimodal data asynchronous receiving interface, and accurately labels each frame of independent modal data with a high-precision system timestamp, forming a raw multimodal data sequence with timestamps;

[0061] S2, the original multimodal data sequence with timestamps is input into a parallel modality-independent feature extraction network. For each modality, the modality-independent feature extraction network uses a preset deep learning model to convert the original multimodal data sequence into an initial feature vector sequence, wherein each feature vector retains the timestamp information of its corresponding original multimodal data sequence.

[0062] S3. For each feature vector in the initial feature vector sequence, use a predefined temporal coding function to encode its timestamp information into a temporal position vector, and concatenate the temporal position vector with the corresponding initial feature vector to generate a temporally enhanced feature vector sequence.

[0063] S4. Input the temporal augmentation feature vector sequence into the temporal validity dynamic evaluation and gating module. This module calculates a temporal validity score for each temporal augmentation feature vector in the temporal augmentation feature vector sequence based on a learnable modality-specific decay function related to the current session context. The temporal validity score characterizes the semantic value of the temporal augmentation feature vector at the current decision time point.

[0064] S5. Based on a preset validity score threshold, a gating operation is performed on the time-enhanced feature vector sequence, setting feature vectors with scores below the threshold to zero, thereby generating a sparse feature vector sequence containing only time-sensitive features.

[0065] S6, feed the sparse feature vector sequence into a lightweight cross-modal sparse attention fusion module. This lightweight cross-modal sparse attention fusion module only performs cross-modal attention calculation between non-zero feature vectors in the sparse feature vector sequence to generate a fusion feature vector that can represent the current user's core intent.

[0066] S7 inputs the fused feature vector into the intent classifier, calculates the probability distribution of each predefined intent category through a fully connected network and output layer, and outputs the intent category with the highest probability value as the final classification result.

[0067] In step S1, the system captures multi-source input from the user in real time through the multimodal data asynchronous receiving interface.

[0068] The multi-modal data asynchronous reception interface supports the access of data streams in three modalities: text messages, voice segments, and image frames. For each independent modality data frame, the system calls the system clock synchronized with the Network Time Protocol server to assign it an absolute timestamp in milliseconds. For example, when the user issues a voice "I want a refund" at the moment when t equals zero milliseconds, the system records the timestamp of this voice segment as zero; if the user sends the text "Order number 123456" at t equals 5200 ms, the timestamp of this text is marked as 5200.

[0069] All modality data and their corresponding timestamps together constitute an original multi-modal data sequence. This original multi-modal data sequence does not require the modalities to be aligned in time, but instead fully retains its original input time sequence structure.

[0070] In step S2, the original multi-modal data sequence is fed into the modality-independent feature extraction network.

[0071] The modality-independent feature extraction network consists of multiple parallel sub-networks, and each sub-network is dedicated to processing a specific modality.

[0072] For the text modality, that is, the text feature extraction sub-network, a distilled bidirectional encoder representation model with a four-layer encoder structure, a hidden layer dimension of 256, and four attention heads is adopted; this distilled bidirectional encoder representation model receives the text sequence after tokenization and outputs a 768-dimensional word vector sequence as the initial text feature vector sequence, and each word vector is bound to the timestamp of its original text segment.

[0073] For the voice modality, that is, the voice feature extraction sub-network, first, the original audio signal is converted into a Mel-frequency cepstral coefficient sequence with a sampling rate of 16 kHz, a frame length of 25 ms, and a frame shift of 10 ms. Subsequently, this Mel-frequency cepstral coefficient sequence is input into a time-delay neural network stacked by five one-dimensional convolutional modules, and this network outputs a 128-dimensional acoustic feature vector sequence as the initial voice feature vector sequence. Each acoustic feature vector also retains the timestamp of its corresponding voice frame.

[0074] In the processing of the text modality, the text feature extraction sub-network further performs the following steps:

[0075] Preprocessing: Clean the input text (remove special symbols), correct typos (based on the typo dictionary in the payment industry, such as correcting "zhang户" to "账户"), and convert colloquialisms (such as converting "昨儿" to "昨天"), and extract multi-round dialogue context information;

[0076] Dedicated Intent Recognition: Employs a finely tuned payment industry model (e.g., based on the DistilBERT architecture, trained using 500,000 sets of labeled data) to output intent categories (e.g., "transfer question") and key parameters (e.g., transfer time, amount), with a confidence threshold set at 90%.

[0077] RAG Knowledge Retrieval: Based on intent category and key parameters, retrieve matching business knowledge (such as transfer arrival time) from the vector database, and rewrite the knowledge content in combination with the user's VIP level;

[0078] Emotional understanding and script generation: Identify user emotional tendencies and dynamically generate reassuring or empowering scripts based on user profiles to improve response suitability.

[0079] For example, in a payment consultation scenario, if a user enters "Why hasn't the 500 yuan I transferred yesterday arrived in my friend's card yet?", the system will preprocess it to convert it into "500 yuan transferred to my friend's bank card yesterday has not arrived," and the intent recognition outputs "Transfer problem (confidence level 97%)." After retrieving knowledge, the system will generate a reply: "Since you are a VIP user, transfers usually arrive in real time. If the funds have not arrived, it may be due to a delay in bank clearing. We suggest you check your transfer records first."

[0080] In step S3, the system performs temporal position encoding on each initial feature vector. This temporal position encoding uses a combination of sine and cosine functions to map the millisecond-level timestamp into a temporal position vector that matches the feature dimension.

[0081] Specifically, for timestamp position pos and dimension index i, the components of the time-series position vector in the even-numbered dimensions are: The components in the odd-numbered dimensions are ,in is the dimension of the feature vector. For text features, Take 768; for speech features, Take 128.

[0082] After encoding, the temporal position vector is concatenated with the corresponding initial feature vector along the channel dimension to generate a temporally enhanced feature vector. For example, concatenating a 768-dimensional text feature vector with a 768-dimensional temporal position vector results in a 1536-dimensional temporally enhanced text feature vector. Similarly, the speech feature vector is expanded to 256 dimensions.

[0083] In step S4, the time-enhanced feature vector sequence is input to the time-series validity dynamic evaluation and gating module. The core of this module is a learnable mode-specific decay function, whose mathematical expression is:

[0084]

[0085] in, For time series validity score, This is the current system time (i.e., the moment the evaluation is performed). The timestamp of the feature vector. Let be the decay rate parameter of mode m.

[0086] The attenuation rate parameter is not globally fixed, but is dynamically generated by a meta-learning network based on the service type identifier of the current session.

[0087] The meta-learning network is a feedforward neural network containing two fully connected layers. Its input is a business type identifier (e.g., finance, e-commerce, ticketing), and its output is the identifier for both the text and speech modalities. and For example, in financial business scenarios, the system has extremely high time sensitivity requirements, and the meta-learning network outputs a large λ value (such as...). =0.002, =0.003), causing features exceeding two seconds to rapidly decay to near zero; however, in e-commerce scenarios, users often need to verify order information, allowing for longer delays, and the meta-learning network outputs a smaller λ value (such as 0.003), which causes features exceeding two seconds to rapidly decay to near zero; while in e-commerce scenarios, users often need to verify order information, allowing for longer delays, and the meta-learning network outputs a smaller λ value (such as 0.003). =0.0008, =0.0012), ensuring that features within four seconds remain highly effective. This mechanism enables adaptive modeling of time tolerance for different business scenarios.

[0088] In step S5, the system sets a global validity score threshold, for example, 0.1.

[0089] For each vector in the time-enhanced feature vector sequence, if its If the value is below this threshold, all its dimensions are set to zero; otherwise, the original value is retained. This operation can be implemented at the hardware level using a parallel comparator circuit, ensuring low-latency processing.

[0090] After this gating operation, the original dense sequence is transformed into a sparse feature vector sequence, which contains only features that still have semantic value at the current decision time. For example, if the current time is 6000ms and the timestamp of a certain speech feature is 1000ms, in a financial scenario its... It might be only 0.05, below the threshold, and therefore set to zero; however, in e-commerce scenarios, its... It may be 0.15, which is higher than the threshold, and thus it is retained.

[0091] In step S6, the sparse feature vector sequence is fed into a lightweight cross-modal sparse attention fusion module. This lightweight cross-modal sparse attention fusion module employs a multi-head attention mechanism with two attention heads, and the dimensions of the query, key, and value matrices are all set to 128.

[0092] When calculating the attention weight matrix, the system pre-generates a binary mask matrix M with the same dimension as the feature sequence length. For any position i in the sequence, if its eigenvector has been set to zero, all elements in the i-th row and i-th column of the mask matrix M are set to negative infinity. In the subsequent calculation of the normalized exponential function, the attention weights at these positions are automatically set to zero, thus ensuring that cross-modal interactions only occur between valid non-zero features.

[0093] This design avoids invalid computations on expired features, significantly reducing computational complexity. For example, if the original sequence contains ten feature vectors, six of which are gated to zero, then attention computation is performed only on the remaining four valid features.

[0094] In step S7, the fused feature vector is input to the intent classifier. This intent classifier consists of two fully connected layers: the first layer has an output dimension of 512, and the second layer has an output dimension equal to the predefined total number of intent categories (e.g., twenty categories). The output layer uses a normalized exponential function to generate a probability distribution.

[0095] Furthermore, the present invention includes an intent classification confidence calibration step. This intent classification confidence calibration step calculates the variance of all time-series validity scores in the sparse feature vector sequence. And adjust the temperature scaling factor T dynamically accordingly:

[0096]

[0097] Where α is a preset hyperparameter with a value of 0.5. If the timestamps of the input features are highly dispersed (e.g., simultaneously containing features at t=0ms and t=5000ms), then A larger T results in a smoother probability distribution in the output of the normalized exponential function, reducing the maximum probability value and thus suppressing overconfident misjudgments in semantic conflict scenarios. Conversely, if all features are concentrated in a short period of time, then... When T is small and close to 1, the system maintains a high confidence output.

[0098] The implementation of the above method relies on a complete system architecture. A multimodal data-driven user consultation intent classification system includes:

[0099] The multimodal data asynchronous reception and timestamp annotation module is used to acquire user input in real time and annotate it with timestamps;

[0100] The modality-independent feature extraction and temporal coding module includes text processing and speech processing pathways, which respectively complete feature extraction and temporal enhancement.

[0101] The temporal effectiveness dynamic evaluation and gating module integrates meta-learning networks and gating units to achieve dynamic decay and feature pruning;

[0102] The lightweight cross-modal sparse attention fusion module employs a dedicated tensor processing unit to shield the computation of zero-value vectors at the hardware level.

[0103] The intent classification and confidence calibration module performs the final classification and dynamically adjusts the output confidence level.

[0104] In the modality-independent feature extraction and temporal coding module, the text processing pathway consists of a lexicalization unit, a distilled bidirectional encoder representation model, and a temporal coding unit; the speech processing pathway consists of an acoustic feature extraction unit, a time-delay neural network, and a temporal coding unit. Both pathways' temporal coding units employ the same sine-cosine coding function to ensure consistency in the representation of temporal information.

[0105] The meta-learning network of the time-series validity dynamic evaluation and gating module receives the service type identifier from the context interface and outputs mode-specific attenuation rate parameters. The gating unit receives these parameters and the time-enhanced feature sequence, and calculates vector values ​​one by one. The zeroing operation is performed via a hardware comparator. This design allows the gating process to be completed within microseconds, meeting real-time requirements.

[0106] When deployed on edge devices, the lightweight cross-modal sparse attention fusion module is implemented using a dedicated tensor processing unit. This unit incorporates programmable masking logic, which skips memory accesses and arithmetic operations on zero-value vectors before performing matrix multiplication, thereby keeping inference latency below 200ms.

[0107] In summary, this invention fundamentally solves the semantic noise problem caused by temporal asynchrony by introducing a dynamic evaluation mechanism for temporal validity and proactively removing expired modal features before fusion. Through sparse attention and hardware optimization, it achieves a balance between high accuracy and low latency. Meta-learning-driven adaptive decay rate enhances the system's generalization ability across multiple business scenarios. Confidence calibration improves decision reliability. These combined technical approaches constitute an efficient, robust, and deployable multimodal intent classification solution suitable for industrial-grade customer service systems.

[0108] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0109] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment includes only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A multimodal data-driven method for classifying user consultation intent, characterized in that, include: The system acquires user input data streams containing text and voice in real time through a multimodal data asynchronous receiving interface, and marks each frame of modal data with a high-precision system timestamp to form a raw multimodal data sequence with timestamps. The original multimodal data sequence with timestamps is input into a parallel modality-independent feature extraction network and converted into an initial feature vector sequence, where each feature vector retains the timestamp information of its corresponding original multimodal data. For each initial feature vector, its timestamp information is encoded into a temporal position vector using a predefined temporal coding function, and then concatenated with the corresponding initial feature vector to generate a time-enhanced feature vector sequence; The time-enhanced feature vector sequence is input into the time-series effectiveness dynamic evaluation and gating module, and a time-series effectiveness score is calculated for each time-enhanced feature vector based on a learnable modality-specific decay function. Based on a preset validity score threshold, a gating operation is performed on the time-enhanced feature vector sequence, setting feature vectors with scores below the threshold to zero, thereby generating a sparse feature vector sequence containing only time-sensitive features. The sparse feature vector sequence is fed into the lightweight cross-modal sparse attention fusion module, which performs cross-modal attention computation only between non-zero feature vectors to generate a fused feature vector. The fused feature vector is input into the intent classifier, and the probability distribution of each predefined intent category is calculated through a fully connected network and an output layer, outputting the intent classification result.

2. The multimodal data-driven user consultation intent classification method according to claim 1, characterized in that, Each frame of modal data is labeled with a high-precision system timestamp, including: Using the system clock synchronized with the network time protocol server, an absolute timestamp in milliseconds is generated for each received frame of text data, each slice of audio data, or each frame of image data.

3. The multimodal data-driven user consultation intent classification method according to claim 1, characterized in that, Modality-independent feature extraction networks include: The text feature extraction sub-network for processing text data adopts a distilled bidirectional encoder representation model with a four-layer encoder structure, a hidden layer dimension of 256, and a number of attention heads of four, which converts the input text sequence into a 768-dimensional word vector sequence as the initial text feature vector sequence. The speech feature extraction subnetwork used to process speech data employs a time-delay neural network consisting of five stacked one-dimensional convolutional modules. It first converts the input speech signal into a Mel-frequency cepstral coefficient sequence, and then extracts a 128-dimensional acoustic feature vector sequence as the initial speech feature vector sequence through the network.

4. The multimodal data-driven user consultation intent classification method according to claim 1, characterized in that, Timing coding uses sine and cosine functions to encode millisecond-level timestamps.

5. The multimodal data-driven user consultation intent classification method according to claim 4, characterized in that, The learnable modality-specific decay function used in the timing validity dynamic evaluation and gating module is an exponential decay function, and its decay rate is dynamically generated by a meta-learning network based on the service type identifier of the current session.

6. The multimodal data-driven user consultation intent classification method according to claim 1, characterized in that, The lightweight cross-modal sparse attention fusion module is specifically implemented as follows: The lightweight cross-modal sparse attention fusion module adopts a multi-head attention mechanism with two attention heads, and the dimensions of its query, key, and value matrices are all set to 128. When calculating the attention weight matrix, a pre-generated binary mask matrix is ​​used to force the row and column values ​​corresponding to the feature vectors in the sequence to be fused that are zeroed by the gating operation to be set to negative infinity, so that these invalid features have zero weight in the subsequent normalized exponential function calculation.

7. The multimodal data-driven user consultation intent classification method according to claim 6, characterized in that, It also includes an intent classification confidence calibration step, which is performed after the intent classifier outputs the probability distribution and includes: Calculate the variance of all time-series validity scores in the sparse feature vector sequence before entering the lightweight cross-modal sparse attention fusion module; The temperature scaling factor before the output layer of the intent classifier is dynamically adjusted based on the variance value. If the timestamps of the input features are highly dispersed, the intent classification probability distribution is actively smoothed to reduce the confidence of system misjudgment. The original logical value output by the intent classifier is divided by the temperature scaling factor and then fed into the normalized exponential function layer.

8. The multimodal data-driven user consultation intent classification method according to claim 7, characterized in that, The meta-learning network is a feedforward neural network containing two fully connected layers. Its input is the service type identifier of the current session, and its output is the decay rate parameters for the text modality and the speech modality, respectively.

9. The multimodal data-driven user consultation intent classification method according to claim 3, characterized in that, The text feature extraction subnetwork in the modality-independent feature extraction network further includes: The preprocessing unit is used to clean the text data, correct typos, and convert it into colloquial language, and to associate it with the context of multi-turn dialogues; A dedicated intent recognition unit, based on a large model finely tuned from labeled data in the payment industry, combined with natural language understanding technology, outputs intent categories and key parameters, and supports context-dependent intent completion.

10. A multimodal data-driven user consultation intent classification system, characterized in that, include: The multimodal data asynchronous reception and timestamp annotation module is used to acquire user input data streams containing text and voice in real time, and to accurately annotate each frame of independent modal data with a high-precision system timestamp, forming a raw multimodal data sequence with timestamps; The modality-independent feature extraction and temporal coding module is used to convert the original multimodal data sequence with timestamps into a temporally enhanced feature vector sequence, where each feature vector explicitly contains its timestamp information. The time-series validity dynamic evaluation and gating module is used to calculate a time-series validity score for each feature vector in the time-enhanced feature vector sequence based on a learnable modality-specific decay function related to the current session context, and to perform gating operation on the sequence based on a preset threshold to generate a sparse feature vector sequence containing only time-series validity features. The lightweight cross-modal sparse attention fusion module is used to perform cross-modal attention computation only among non-zero feature vectors in the sparse feature vector sequence, generating a fusion feature vector that can represent the core intent of the current user. The intent classification and confidence calibration module is used to classify the fused feature vectors into predefined intent categories and dynamically calibrate the output confidence based on the temporal validity score distribution of the input features.

Images