Multi-modal learning-based intention recognition method and device, equipment and medium

Abstract

The invention relates to the technical field of data analysis, and discloses an intention recognition method and device based on multi-modal learning, equipment and a medium, diversified information requested by a client can be captured by collecting multi-modal business data, and information missing or one-sidedness possibly brought by single-modal data is avoided. The multi-modal feature extraction model is utilized, key feature information can be mined from data of different modals, and the business requirements of clients can be described more accurately. Based on the preset feature fusion strategy, the multi-modal data features are fused, the defect of a single mode is overcome, the multi-modal fusion features with higher expressive power are formed, and therefore the intention of a customer is reflected more accurately. And the multi-modal fusion features are input into the intention recognition model matched with the fusion strategy, intention feature recognition analysis is performed, a more accurate intention recognition result is generated, and the accuracy of insurance service demand recognition of the target customer in the fields of finance, medical treatment and the like is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, and in particular to an intent recognition method, apparatus, device, and medium based on multimodal learning. Background Technology

[0002] In today's insurance industry, intent recognition technology is increasingly widely used. It helps insurance companies better understand customer needs and provide personalized services, thereby improving customer satisfaction and business efficiency. However, current insurance intent recognition technology faces many challenges in practical applications, mainly in terms of training data and semantic understanding.

[0003] The performance of intent recognition models is highly dependent on the quality and quantity of training data. In real-world business scenarios, the amount of multimodal data collected (such as text, speech, and images) is often limited, especially high-quality labeled data. Insufficient data can prevent the model from fully learning features across various scenarios, resulting in poor performance when handling unseen data. Furthermore, the collected data may only cover common scenarios, while data on rare or special scenarios may be missing. This uneven distribution of data can lead to insufficient generalization ability of the model when faced with unseen scenarios, making it unable to accurately identify customer intent.

[0004] Meanwhile, textual data in insurance business (such as customer inquiries and accident descriptions) often contains complex sentence structures and technical terms, such as financial insurance, medical insurance, and auto insurance. Although natural language processing (NLP) technology has made great strides, intent recognition models may still face difficulties in understanding complex, ambiguous, or vague sentences.

[0005] Therefore, improving the accuracy of customer intent recognition has become an urgent technical problem to be solved. Summary of the Invention

[0006] This invention provides an intent recognition method, apparatus, computer device, and medium based on multimodal learning to solve the technical problem of low accuracy in customer intent recognition.

[0007] Firstly, an intent recognition method based on multimodal learning is provided, including:

[0008] Collect multimodal business data related to auto insurance business requested by target customers, wherein the multimodal business data includes business data of at least two modal types;

[0009] Based on the multimodal feature extraction model, feature extraction is performed on the multimodal business data to obtain multimodal data features;

[0010] Based on a preset feature fusion strategy, multimodal data features are fused to obtain multimodal fused features;

[0011] The multimodal fusion features are input into an intent recognition model that matches the feature fusion strategy. The multimodal fusion features are then analyzed for intent feature recognition to generate intent recognition results for the target customer's needs for auto insurance business.

[0012] Secondly, an intent recognition device based on multimodal learning is provided, comprising:

[0013] The data acquisition module is used to collect multimodal business data related to the auto insurance business requested by the target customer, wherein the multimodal business data includes business data of at least two modal types;

[0014] The feature extraction module is used to extract features from the multimodal business data based on a multimodal feature extraction model to obtain multimodal data features;

[0015] The feature fusion module is used to perform multimodal feature fusion on multimodal data features based on a preset feature fusion strategy to obtain multimodal fused features;

[0016] The intent recognition module is used to input the multimodal fusion features into an intent recognition model that matches the feature fusion strategy, perform intent feature recognition analysis on the multimodal fusion features, and generate intent recognition results for the target customer's needs for auto insurance business.

[0017] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described intention recognition method based on multimodal learning.

[0018] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described intention recognition method based on multimodal learning.

[0019] The aforementioned scheme, implemented using multimodal learning-based intent recognition methods, devices, computer equipment, and storage media, captures diverse information about customer requests by collecting multimodal business data, avoiding information gaps or biases that may arise from single-modal data. Utilizing multimodal feature extraction models, key feature information can be mined from data of different modalities, extracting rich semantic and contextual content, which helps to more accurately describe and characterize customer business needs. Based on a preset feature fusion strategy, multimodal data features are fused, integrating the advantages of each modality, compensating for the shortcomings of a single modality, and forming more complete and expressive multimodal fusion features, thereby more accurately reflecting customer intent. Inputting the multimodal fusion features into an intent recognition model that matches the fusion strategy fully leverages the model's processing capabilities for fusion features, performing intent feature recognition analysis, generating more reliable and accurate intent recognition results, and effectively improving the accuracy of identifying the target customer's auto insurance business needs. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of an application environment for an intent recognition method based on multimodal learning in one embodiment of the present invention;

[0022] Figure 2 A flowchart illustrating the first embodiment of the intent recognition method based on multimodal learning provided in this invention.

[0023] Figure 3 This is a schematic diagram of the intent recognition application process provided in an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of an intent recognition device based on multimodal learning in one embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0026] Figure 6 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] The intent recognition method based on multimodal learning provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. After receiving a car insurance business request from the client, the server can collect multimodal business data related to the car insurance business requested by the target customer. The multimodal business data includes business data of at least two modalities. Based on a multimodal feature extraction model, features are extracted from the multimodal business data to obtain multimodal data features. Based on a preset feature fusion strategy, multimodal feature fusion is performed on the multimodal data features to obtain multimodal fused features. The multimodal fused features are then input into an intent recognition model that matches the feature fusion strategy to perform intent feature recognition analysis on the multimodal fused features, generating an intent recognition result for the target customer's car insurance business needs.

[0029] This invention addresses the technical problem of low accuracy in recognizing customer intent in insurance businesses across finance, healthcare, and other fields. By collecting multimodal business data, diverse information about customer requests can be captured, avoiding the information gaps or biases that may result from single-modal data. Utilizing a multimodal feature extraction model, key feature information can be mined from data across different modalities, extracting rich semantic and contextual content, which helps to more accurately describe and characterize customer business needs. Based on a preset feature fusion strategy, multimodal data features are fused, integrating the advantages of each modality and compensating for the shortcomings of a single modality to form a more complete and expressive multimodal fusion feature, thereby more accurately reflecting customer intent. Inputting the multimodal fusion feature into an intent recognition model that matches the fusion strategy fully leverages the model's processing capabilities for fusion features, performing intent feature recognition analysis to generate more reliable and accurate intent recognition results, effectively improving the accuracy of recognizing target customer auto insurance business needs.

[0030] The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0031] Please see Figure 2 As shown, Figure 2Schematic flowchart of the first embodiment of the intention recognition method based on multimodal learning provided by an embodiment of the present invention, including the following steps:

[0032] S101: Collect multimodal business data related to the auto insurance business requested by the target customer, where the multimodal business data includes business data of at least two modal types;

[0033] In one embodiment, the multimodal business data includes text data, voice data, image data, and business data of other modal types. The modal type refers to modal data types such as text, voice, and image.

[0034] Specifically, when the target customer initiates a business request related to the auto insurance business, collect the text information submitted by the customer, such as accident descriptions, problem consultations, etc., as text data; collect the customer's voice call records and convert the voice data into text, as voice data; collect the accident scene photos or other relevant pictures uploaded by the customer, as image data; it may also include other modal type business data such as the customer's personal information and historical claim records.

[0035] After collecting the multimodal business data, data preprocessing can be performed on the multimodal business data first. Different data preprocessing methods can be used for business data of different modal types in the multimodal business data.

[0036] For example, for text data, data cleaning can be performed on the text data to remove irrelevant characters, such as punctuation marks, special symbols, etc. Then perform word segmentation, stemming, etc., break the sentence into words or phrases, and restore the words to their root forms. Use regular expressions to remove irrelevant characters and perform semantic verification to check if it conforms to common sentence patterns in the auto insurance scenario.

[0037] Among them, word segmentation refers to breaking a sentence into words or phrases:

[0038] sentence = tokenize(raw text )

[0039] Among them, raw text represents the original input text string, such as the accident description or consultation content submitted by the user. tokenize(·) is a word segmentation function that splits the sentence into words or phrases according to spaces, punctuation marks, or a dictionary. sentence is the result of word segmentation, usually a list of words (such as ["I", "car", "was hit", "by"]).

[0040] Stemming refers to restoring a word to its root form.

[0041] stemmed_words = stem(words)

[0042] Here, `words` is the list of words obtained from word segmentation. `stem(·)` is the stemming function, which restores words to their root forms (e.g., "running" → "run"). `stemed_words` is the stemmed word list, used to unify word forms and reduce lexical redundancy.

[0043] For voice data, noise reduction techniques are used to remove background noise and improve speech clarity. Speech recognition technologies such as Automatic Speech Recognition (ASR) can be used to convert speech into text.

[0044] transcribed = ASR(audio)

[0045] Here, represents the original speech signal, usually a recording file. ASR(·) is the Automatic Speech Recognition function, which converts speech into text. transcribed is the recognized text string, for example, the user's speech "My car was hit" transcribed into text.

[0046] After converting speech to text, the speech-text can be processed by data cleaning, word segmentation, and other methods.

[0047] For image data, image standardization processing can be performed, such as adjusting resolution and contrast, cropping, scaling, and enhancement, to improve image usability and ensure that image quality meets the input requirements of feature recognition models or networks.

[0048] Image cropping refers to cropping an image to a fixed size.

[0049] cropped_image=crop(image,size)

[0050] Here, `image` represents the original input image (such as a user-uploaded photo of an accident scene). `size` represents the specified cropping size, such as (512, 512). `crop(·)` represents the image cropping function, which extracts a fixed region from the original image. `cropped_image` represents the cropped image, used to remove irrelevant background.

[0051] Image scaling refers to scaling an image to the size required for the model input.

[0052] resized_image=resize(image,target_size)

[0053] Here, `image` represents the input image. `target_size` represents the required resolution of the model input, such as (224, 224). `resize(·)` is the image scaling function that adjusts the image to the size required by the model. `resized_image` represents the scaled image.

[0054] S102: Based on the multimodal feature extraction model, perform feature extraction on the multimodal business data to obtain multimodal data features;

[0055] In one embodiment, the multimodal feature extraction model may include multiple feature extraction sub-models corresponding to different modal types, used for feature recognition and extraction of business data of different modal types.

[0056] Multimodal business data may include one or more of the following: text modality, voice modality, image modality, and other modality types of business data.

[0057] Specifically, the multimodal feature extraction model may include a text feature extraction sub-model, used to extract features from business data of text modality type, in order to extract text data features; the multimodal feature extraction model may also include a speech feature extraction sub-model, used to extract features from business data of speech modality type, in order to extract speech data features; the multimodal feature extraction model may also include an image feature extraction sub-model, used to extract features from business data of image modality type, in order to extract image data features.

[0058] For example, for business data of text modality type, word embedding techniques (such as...) can be used.

[0059] Word2Vec, GloVe, or context-based pre-trained models (such as BERT) can be used as text feature extraction sub-models to extract text features. For example, pre-trained word embedding models (such as Word2Vec, GloVe) can be used to convert text into vectors.

[0060] X text = word_embedding(tokenized)

[0061] Here, `tokenized` represents the text data after tokenization, typically a list of words or phrases. `word_embedding(·)` represents a pre-trained word embedding model (such as Word2Vec or GloVe) used to map words to a fixed-dimensional vector space. text This represents the extracted text feature vector, where each word is represented as a vector that reflects its semantic information.

[0062] Extracting text features using the BERT model:

[0063] X BERT =BERT(tokenized)

[0064] Here, `tokenized` represents the text data after tokenization. `BERT(·)` indicates that it is based on...

[0065] Transformer's pre-trained language models are able to capture the contextual semantic information of text. BERT This means that the text feature vectors extracted by BERT can better understand the semantics and contextual relationships in the text.

[0066] For business data of the speech modality type, speech features can be obtained by extracting MFCC (Mel frequency cepstral coefficients) and acoustic features. For example, the extraction of Mel frequency cepstral coefficients can be expressed as:

[0067] X MFCC =MFCC(audio)

[0068] Where audio represents the original speech signal data. MFCC(·) represents the Mel-frequency cepstral coefficient extraction method, used to extract acoustic features from the speech signal. X MFCC This represents the extracted speech feature vector, which reflects the acoustic characteristics of the speech.

[0069] For image-based business data, convolutional neural networks (such as ResNet and Inception) can be used to extract image features. For example, pre-trained CNN models (such as...) can be used.

[0070] ResNet and Inception extract image features:

[0071] X image =CNN(preprocessed) image )

[0072] Among them, preprocessed image This represents image data after preprocessing (such as cropping, scaling, enhancement, etc.). CNN(·) represents a convolutional neural network model (such as ResNet, Inception), used to extract spatial features from images. X image This represents the extracted image feature vector, which reflects the visual information in the image.

[0073] The multimodal feature extraction model can extract effective features from business data of different modalities.

[0074] S103: Based on a preset feature fusion strategy, perform multimodal feature fusion on multimodal data features to obtain multimodal fused features;

[0075] In one embodiment, the feature fusion strategy can be selected based on the characteristics of the multimodal data features and application requirements, and appropriate feature fusion strategies can be adopted for multimodal data features with different characteristics.

[0076] For example, feature fusion strategies may include a first fusion strategy, a second fusion strategy, and a third fusion strategy.

[0077] In one embodiment, the similarity and differences between modal features can be understood by calculating the feature correlation between multimodal data features, and an appropriate feature fusion strategy can be selected. Specifically, the feature correlation between multimodal data features is calculated; based on the feature correlation between the multimodal data features, a feature fusion strategy is determined for the multimodal data features corresponding to business data of different modal types.

[0078] In one embodiment, the similarity between features of multimodal data can be used to characterize their feature correlation. Methods for calculating feature similarity may include Pearson correlation coefficient, mutual information, and cosine similarity.

[0079] For example, cosine similarity measures the cosine angle between two vectors, thus reflecting their directional similarity. The calculation formula is:

[0080]

[0081] Among them, X j and X k Let represent the feature vectors of the j-th and k-th modes, respectively, and let ||·| represent the L2 norm of the vectors. The cosine similarity value ranges from [-1, 1], where 1 indicates similarity in direction, -1 indicates opposite direction, and 0 indicates no correlation.

[0082] In one embodiment, when the feature relevance is greater than or equal to a relevance threshold, the feature fusion strategy is determined to be at least one of a first fusion strategy, a second fusion strategy, and a third fusion strategy; when the feature relevance is less than the relevance threshold, the feature fusion strategy is determined to be at least one of a second fusion strategy and the third fusion strategy.

[0083] Specifically, a relevance threshold can be set based on actual application scenarios and experience to distinguish between high and low feature relevance. For example, when the feature relevance is greater than or equal to the relevance threshold, the features are considered to have a strong correlation; when the feature relevance is less than the relevance threshold, the features are considered to have a weak correlation.

[0084] When the correlation between features in multimodal data is greater than or equal to the correlation threshold, it indicates a strong correlation and complementarity between features of different modalities. At least one of the following fusion strategies can be selected: the first fusion strategy (early fusion), the second fusion strategy (mid-term fusion), and the third fusion strategy (late fusion). The early fusion strategy fully utilizes the strong correlation between modalities, fusing features from different modalities at a lower level to better uncover their interrelationships. The mid-term fusion strategy fuses features from different modalities at an intermediate level after feature extraction, also effectively utilizing the correlation between modalities while preserving a certain degree of modal independence.

[0085] When the correlation between features in multimodal data is less than a correlation threshold, it indicates that the features of different modalities are weakly correlated and highly independent. In this case, forcibly adopting early or mid-term fusion strategies may lead to information confusion or loss of independent modal information. Therefore, at least one of the second (mid-term) and third (late) fusion strategies is more suitable. The late fusion strategy trains a model independently for each modality's features and then fuses the outputs of each model. This better preserves the independent information of each modality and, by using appropriate fusion methods (such as weighted averaging or voting) to synthesize the prediction results of each modality, improves the accuracy and stability of intent recognition.

[0086] In one embodiment, when the feature fusion strategy is the first fusion strategy, the data features corresponding to each modality type in the multimodal data features are directly concatenated to obtain the first concatenated feature; the first concatenated feature is normalized and dimension aligned to obtain the multimodal fused feature.

[0087] For example, the first fusion strategy can be an early fusion strategy, which involves concatenating features from different modalities during the feature extraction stage to form a high-dimensional multimodal fusion feature vector, and then inputting it into the downstream model.

[0088] In one embodiment, the first fusion strategy is to directly concatenate the multimodal data features (i.e., text features, speech features, and image features, etc.) extracted from multimodal business data to form a high-order first concatenated feature.

[0089] Specifically, the extracted modal data features are preprocessed, including normalization and standardization, to eliminate scale differences between different modal data and make them comparable. The preprocessed data features corresponding to each modality type are then directly concatenated in a specific order to form a long feature vector, i.e., the first concatenated feature. For example, text feature vectors, speech feature vectors, and image feature vectors are sequentially concatenated to form a feature vector that integrates information from multiple modalities.

[0090] Features from different modalities are concatenated together to form a high-dimensional feature vector.

[0091] X fused =[X text ,X voice ,X image ]

[0092] Among them, X text X represents text features. voice Representing speech features, X image Represents image features.

[0093] Business data features of different modalities need to be aligned in time or space to ensure that they correspond to the same scenario or event. The data feature dimensions of each modal type need to be the same, or the data features of different modal types can be converted to the same dimension.

[0094] The first concatenated feature is normalized to ensure that the numerical range of the entire feature vector falls within a specific interval, such as [0,1] or [-1,1]. Dimension alignment is then performed on the normalized first concatenated feature to ensure that the dimension of the multimodal fusion feature matches the input requirements of the downstream model. If the dimension of the concatenated feature is too high or too low, it can be adjusted using methods such as dimensionality reduction (e.g., PCA, t-SNE) or dimensionality increase (e.g., linear interpolation, zero-padding) to meet the dimensionality requirements of the model input.

[0095] In one embodiment, when the feature fusion strategy is the second fusion strategy, feature preprocessing is performed on the data features of each modality type in the multimodal data features to obtain preprocessed features of different modality types; the preprocessed features of different modality types are spliced ​​and fused to obtain the multimodal fused features.

[0096] For example, the second fusion strategy can be an intermediate fusion strategy, which means that after feature extraction, each different modality type of data feature in the multimodal data features is subjected to preliminary processing to adapt to the modality type, and then the features after preliminary processing are fused.

[0097] Specifically, feature preprocessing of data features of different modal types (text features, speech features, and image features, etc.) in multimodal data features can be performed by using a multilayer perceptron (MLP) to map the data features of different modal types to the same semantic space, thereby enhancing the representational ability and comparability of the features.

[0098] Preprocessing features can include text mapping features, speech mapping features, and image mapping features.

[0099] For example, text modality features (i.e., text features) from multimodal data features are input into the MLP network corresponding to the text modality type. After nonlinear transformation, the mapped text features are obtained. Similarly, speech modality features (i.e., speech features) from multimodal data features are input into the MLP network corresponding to the speech modality type. After nonlinear transformation, the mapped speech features are obtained. Likewise, image modality features (i.e., image features) from multimodal data features are input into the MLP network corresponding to the image modality type. After nonlinear transformation, the mapped image features are obtained.

[0100] The feature vectors of different modalities after mapping are concatenated and fused. This involves linking preprocessed features such as text mapping features, speech mapping features, and image mapping features together in a specific order to form a multimodal fusion feature. The concatenation method can be a simple concatenation operation, linking the feature vectors of each modality end-to-end to form a long fusion feature vector.

[0101] For example, the feature fusion process of the second fusion strategy can be represented as:

[0102]

[0103] Among them, X text H represents text features. text Represents text mapping features; X voice H represents speech features. voice Represents speech mapping features; X image H represents image features. image This represents the image mapping features. MLP(·) represents a multilayer perceptron. X fused This represents the multimodal fusion feature obtained by fusing text mapping features, speech mapping features, and image mapping features.

[0104] In one embodiment, when the feature fusion strategy is the third fusion strategy, the data features of each modality in the multimodal data features are respectively input into the corresponding single-modal intent recognition sub-model, and the single-modal features corresponding to each modality type are output; based on the feature fusion weights corresponding to the business data of each modality type, the single-modal features corresponding to each modality type are weighted and fused to obtain the multimodal fused features.

[0105] For example, the third fusion strategy can be a late fusion strategy, which involves training independent models for each modality type of data feature in the multimodal data features, and finally fusing the outputs of each model using methods such as weighted averaging or voting. This can be expressed by the following formula:

[0106]

[0107] Among them, X text X represents text features. voice Representing speech features, X image Represents image features. (model) text (·), model voice (·), model image (·) represent the single-modal intent recognition sub-models corresponding to text, speech, and image, respectively. w1, w2, and w3 are the weights of the text, speech, and image modalities, respectively, satisfying w1 + w2 + w3 = 1 and w1, w2, w3 ≥ 0. fused This represents the multimodal fusion feature obtained by weighted fusion of the single-modal features corresponding to each modality type output by the single-modal intent recognition sub-model.

[0108] Specifically, the extracted text feature vectors are input into the text intent recognition sub-model for training. The text intent recognition sub-model can employ a multilayer perceptron (MLP) or other suitable model architecture. Through training, the model learns the intent recognition features in the text modality and outputs the single-modal feature y corresponding to the text modality. text .

[0109] The extracted speech feature vectors are input into the speech intent recognition sub-model for training. The speech intent recognition sub-model can employ a convolutional neural network (CNN) or other model structures suitable for speech data. Through training, the model learns the intent recognition features in the speech modality and outputs the single-modal feature y corresponding to the speech modality. voice .

[0110] The extracted image feature vectors are input into the image intent recognition sub-model for training. The image intent recognition sub-model can employ a multilayer perceptron (MLP) or other suitable model architecture. Through training, the model learns the intent recognition features of the image modality and outputs the single-modal feature y corresponding to the image modality.image .

[0111] The feature fusion weights corresponding to business data of each modality can be determined through methods such as grid search or Bayesian optimization on the validation set of historical data. The single-modal features of each modality are then weighted and fused according to the determined weights to obtain the multimodal fused feature y. fused .

[0112] In one embodiment, such as Figure 3 As shown, the multimodal fusion features obtained by the feature fusion strategy are incorporated into the pre-trained intent recognition model, and the intent recognition model is used to identify and analyze the intent features of the multimodal fusion features.

[0113] In one embodiment, multimodal training data is collected and preprocessed, the multimodal training data including at least text type data, speech type data, and image type data; feature extraction is performed on the multimodal training data to obtain multimodal training data features; each of the aforementioned feature fusion strategies is executed on the multimodal training data features to generate training data fusion features corresponding to each of the aforementioned feature fusion strategies; based on the training data fusion features corresponding to each of the aforementioned feature fusion strategies, the pre-trained model is iteratively trained to generate an intent recognition model adapted to each of the aforementioned feature fusion strategies.

[0114] Specifically, multimodal training data includes historical data of various modalities such as text, voice, and images. It collects text information submitted by customers, such as accident descriptions and inquiries; data sources may include online forms filled out by customers and customer service chat logs. It also collects customer voice call records; data sources may include customer service hotline calls and voicemail messages. Finally, it collects accident scene photos or other relevant images uploaded by customers; data sources may include images uploaded by customers through mobile applications and on-site investigation photos.

[0115] The collected multimodal training data is preprocessed. Specifically, for text data, preprocessing operations such as text cleaning, word segmentation, and stemming can be performed; for speech data, preprocessing operations such as noise reduction and speech-to-text conversion can be performed; and for image data, preprocessing operations such as image cropping, scaling, and enhancement can be performed.

[0116] The preprocessed multimodal training data and corresponding labels (such as intent categories) are combined to form the training data. The training data is then divided into a training set, a validation set, and a test set, typically in a 7:2:1 ratio. The training set is used for model training, the validation set is used for hyperparameter tuning and model selection, and the test set is used for final model performance evaluation.

[0117] For the training data, feature extraction is performed on the multimodal training data and its corresponding labels to obtain multimodal training data features, including text training data features, speech training data features and image training data features.

[0118] Based on pre-defined feature fusion strategies—namely, a first fusion strategy (early fusion), a second fusion strategy (mid-term fusion), and a third fusion strategy (late fusion)—corresponding data fusion operations are performed on the training data features of each modality type within the multimodal training data features. For example, the first fusion strategy concatenates the multimodal training data features; the second fusion strategy preprocesses the training data features of different modality types before concatenation; or the third fusion strategy trains independent models for the training data features of different modality types, and then weights and fuses the outputs to obtain the fused training data features corresponding to different feature fusion strategies.

[0119] For intent recognition models corresponding to different feature fusion strategies, different model architectures can be selected as pre-trained models. For example, for the first fusion strategy (early fusion), models capable of handling high-dimensional features and possessing strong feature learning capabilities can be selected, such as Deep Multilayer Perceptron (Deep MLP) and Transformer; for the second fusion strategy (mid-term fusion), models suitable for processing semantic information can be selected, such as RNNs, CNNs with attention mechanisms, or lightweight Transformers; for the third fusion strategy (late fusion), the independent models of each modality can be fully trained first, and then the optimal fusion weights of the outputs of each modality model can be determined by grid search or Bayesian optimization methods using a validation set.

[0120] During model training, the parameters of the pre-trained model are initialized, which can be done using random initialization or pre-trained model parameter initialization. A suitable loss function is chosen, such as cross-entropy loss, to measure the difference between the model's predictions and the true labels. A suitable optimization algorithm is selected, such as stochastic gradient descent (SGD) or Adam, to update the model parameters and reduce the value of the loss function.

[0121] training set Inputting data into the model in batches and performing forward propagation calculations yields the model's predictions. The loss function between the predicted results and the true labels is then calculated. The loss function can be expressed as:

[0122]

[0123] in, θ is the loss function, and θ is the model parameter. y is the multimodal fusion feature of the i-th training sample.(i) is the true label of the i-th training sample, and N is the number of samples in the training set.

[0124] The gradient is calculated using the backpropagation algorithm, and the model parameters are updated using the optimizer. These steps are repeated until the model's loss function value on the training set converges, or the preset number of training epochs is reached.

[0125] In one embodiment, during training, model performance can be evaluated using metrics such as accuracy, precision, recall, and F1 score. Cross-validation is used to assess the model's stability and generalization ability. Furthermore, the model's hyperparameters are optimized using methods such as grid search and random search.

[0126] During application, the model's performance in the production environment can be monitored, allowing for timely identification and resolution of issues. Training data can also be updated regularly, new cases and scenarios added, and the model retrained periodically. Furthermore, the model can be continuously iterated and optimized based on user feedback and business needs.

[0127] S104: Input the multimodal fusion features into an intent recognition model that matches the feature fusion strategy, perform intent feature recognition analysis on the multimodal fusion features, and generate intent recognition results for the target customer's needs for auto insurance business.

[0128] The multimodal fusion features obtained after fusion processing are used as input to the intent recognition model. These fusion features integrate data information from different modalities, which can more comprehensively reflect the customer's intent regarding their needs for auto insurance business.

[0129] In one embodiment, the intent recognition model can be a classification model such as logistic regression, support vector machine (SVM), or multilayer perceptron (MLP). The choice of model should be based on a comprehensive consideration of specific task requirements and data characteristics.

[0130] In one embodiment, a suitable model architecture is selected based on a feature fusion strategy.

[0131] For example, since the feature vectors obtained by the second fusion (early fusion) strategy have a high dimensionality and incorporate original information from different modalities, it is suitable to select models with strong nonlinear modeling and feature learning capabilities, such as Multilayer Perceptrons (MLPs) and Transformers, to fully explore the complex relationships and potential patterns in the fused features. During training, the model simultaneously learns feature representations from different modal data and the correlations between them. By training with a large amount of fused feature data labeled with intent, the model gradually adjusts its parameters to minimize the difference between the predicted intent and the true intent.

[0132] For new customer business request data, the corresponding multimodal data undergoes early fusion (first fusion strategy). The resulting multimodal fusion feature, obtained by concatenating and fusing the data features corresponding to each modality type, is then input into the trained intent recognition model. Based on the learned feature mapping relationship, the intent recognition model outputs the probability distribution of the customer's intent regarding auto insurance business needs, and the final intent recognition result is determined based on the probability magnitude.

[0133] For example, the multimodal fusion features after the second fusion (intermediate fusion) strategy have already extracted and integrated the semantic features of each modality to a certain extent. Therefore, some model structures that are sensitive to semantic information and have good generalization ability can be selected, such as recurrent neural networks (RNNs) and their variants (such as LSTM and GRU), convolutional neural networks (CNNs), etc. During the training phase, the intent recognition model focuses on learning the semantic information contained in the intermediate fusion features and the correlation between the semantics of different modalities. Through learning from a large amount of labeled data, the intent recognition model can understand the mapping relationship between different semantic features and the intent of auto insurance business. For example, by using intermediate fusion feature data with intent labels, methods such as cross-validation can be used to optimize model parameters and improve the accuracy and generalization ability of the model.

[0134] When performing intent recognition on new customer data, a second fusion (intermediate fusion) strategy is first applied to the multimodal data to obtain multimodal fusion features, which are then input into the trained model. Based on the learned semantic mapping relationships, the model calculates the probability of different auto insurance business intents and selects the intent with the highest probability as the final recognition result.

[0135] For example, late-stage fusion combines the outputs of independent models from each modality. Therefore, for each independent model, a suitable model can be selected based on the characteristics of the modality data and the task requirements. For instance, a Transformer-based model can be chosen for text modality, a Convolutional Neural Network (CNN) or Recurrent Neural Network (RNN) can be chosen for speech modality, and a CNN can be chosen for image modality. The final fusion model can be a simple linear model (such as weighted average) or a model that can comprehensively consider multiple inputs (such as stacked generalization, voting mechanisms, etc.).

[0136] During training, the single-modal feature sub-models for each modality are first trained separately. When training each single-modal feature sub-model, data and intent labels of the corresponding modality type are used, and the model parameters are adjusted through optimization algorithms to accurately recognize intents based on the modality data. Then, after obtaining the outputs of each modality model, they are fused into the intent recognition model using fusion methods (such as weighted averaging and voting). The parameters of the intent recognition model are adjusted based on the difference between the fused result and the true intent to determine the optimal fusion method.

[0137] For each new customer request, its corresponding multimodal data features are input into a separate, pre-trained unimodal feature sub-model to obtain the intent recognition result for each unimodal feature sub-model. Then, the intent recognition results of these unimodal feature sub-models are fused according to the fusion method set in the intent recognition model. For example, the final intent probability distribution is calculated using a weighted average formula, or a voting mechanism is used to select the intent that appears most frequently as the final result, thereby generating the intent recognition result for the customer's need for auto insurance services.

[0138] In one embodiment, the intent recognition model outputs one or more probability values ​​representing customer intent categories. These probability values ​​reflect the likelihood of a customer having different needs for auto insurance services. Based on the probability distribution output by the model, the intent category with the highest probability is selected as the final intent recognition result. Simultaneously, a certain probability threshold can be set according to business needs. Intent categories below the threshold can be further analyzed or labeled as unknown intents for subsequent manual intervention or model optimization.

[0139] Through the above process, AI-based multimodal learning can achieve more accurate and efficient intent recognition in auto insurance customer intent identification, precisely identifying customers' relevant insurance needs. This improves the customer experience and related service quality, better serving customers and meeting their insurance needs, and greatly avoiding customer complaints due to inaccurate identification. Secondly, the improved accuracy of intent recognition facilitates subsequent data mining, ensuring the accuracy and efficiency of subsequent analysis. Continuous optimization of anomaly data allows for the improvement of the completeness of anomaly data, enabling full utilization of this data to provide customers with a better insurance experience and ensure the company's continued stable operation.

[0140] As can be seen, the above solution captures diverse information about customer requests by collecting multimodal business data, avoiding the information gaps or biases that may result from single-modal data. Utilizing a multimodal feature extraction model, key feature information can be mined from data of different modalities, extracting rich semantic and contextual content, which helps to more accurately describe and characterize customer business needs. Based on a preset feature fusion strategy, the multimodal data features are fused, integrating the advantages of each modality, compensating for the shortcomings of a single modality, and forming a more complete and expressive multimodal fusion feature, thus more accurately reflecting customer intent. Inputting the multimodal fusion features into an intent recognition model that matches the fusion strategy can fully leverage the model's processing capabilities for fusion features, performing intent feature recognition analysis, generating more reliable and accurate intent recognition results, and effectively improving the accuracy of identifying the target customer's auto insurance business needs.

[0141] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0142] In one embodiment, a multimodal learning-based intent recognition device is provided, which corresponds one-to-one with the multimodal learning-based intent recognition method described in the above embodiments. For example... Figure 4 As shown, the intent recognition device based on multimodal learning includes: a data acquisition module 201, a feature extraction module 202, a feature fusion module 203, and an intent recognition module 204. Detailed descriptions of each functional module are as follows:

[0143] The data acquisition module 201 is used to collect multimodal business data related to the auto insurance business requested by the target customer, wherein the multimodal business data includes business data of at least two modal types;

[0144] The feature extraction module 202 is used to extract features from the multimodal business data based on a multimodal feature extraction model to obtain multimodal data features;

[0145] The feature fusion module 203 is used to perform multimodal feature fusion on multimodal data features based on a preset feature fusion strategy to obtain multimodal fused features;

[0146] The intent recognition module 204 is used to input the multimodal fusion features into an intent recognition model that matches the feature fusion strategy, perform intent feature recognition analysis on the multimodal fusion features, and generate intent recognition results for the target customer's needs for auto insurance business.

[0147] In one embodiment, the feature fusion module 203 includes:

[0148] The feature splicing unit is used to directly splice the data features corresponding to each modality type in the multimodal data features to obtain the first spliced ​​feature when the feature fusion strategy is the first fusion strategy.

[0149] The splicing feature normalization unit is used to normalize and align the dimensions of the first splicing feature to obtain the multimodal fusion feature.

[0150] In one embodiment, the feature fusion module 203 further includes:

[0151] The feature preprocessing unit is used to perform feature preprocessing on the data features of each modality type in the multimodal data features when the feature fusion strategy is the second fusion strategy, so as to obtain preprocessed features of different modality types.

[0152] The preprocessing feature fusion unit is used to splice and fuse preprocessed features of different modal types to obtain the multimodal fused features.

[0153] In one embodiment, the feature fusion module 203 further includes:

[0154] A single-modal feature extraction unit is used to input the data features of each modality in the multimodal data features into the corresponding single-modal intent recognition sub-model when the feature fusion strategy is the third fusion strategy, and output the single-modal features corresponding to each modality type.

[0155] The single-modal feature weighted fusion unit is used to perform weighted fusion of the single-modal features corresponding to each modality type based on the feature fusion weights corresponding to the business data of each modality type, so as to obtain the multimodal fusion features.

[0156] In one embodiment, the intent recognition device based on multimodal learning further includes a feature fusion strategy determination module, comprising:

[0157] The feature correlation calculation unit is used to calculate the feature correlation between features in multimodal data.

[0158] The feature fusion strategy determination unit is used to determine the feature fusion strategy between the multimodal data features corresponding to business data of different modal types based on the feature correlation between the multimodal data features.

[0159] In one embodiment, the feature fusion strategy determination unit includes:

[0160] The fusion strategy determines a first subunit, which is used to determine that the feature fusion strategy is at least one of a first fusion strategy, a second fusion strategy, and a third fusion strategy when the feature relevance is greater than or equal to the relevance threshold;

[0161] The fusion strategy determines a second sub-unit, which is used to determine the feature fusion strategy as at least one of the second fusion strategy and the third fusion strategy when the feature relevance is less than the relevance threshold.

[0162] In one embodiment, the intent recognition device based on multimodal learning further includes a model training module, comprising:

[0163] The training data collection unit is used to collect and preprocess multimodal training data, which includes at least text data, speech data and image data.

[0164] The training data feature extraction unit is used to extract features from the multimodal training data to obtain multimodal training data features;

[0165] The training data feature fusion unit is used to execute each of the feature fusion strategies on the multimodal training data features to generate training data fusion features corresponding to each feature fusion strategy.

[0166] The model training unit is used to iteratively train the pre-trained model based on the fusion features of the training data corresponding to each of the feature fusion strategies, and generate an intent recognition model that is adapted to each of the feature fusion strategies.

[0167] This invention provides an intent recognition device based on multimodal learning. By collecting multimodal business data, it can capture diverse information from customer requests, avoiding the information gaps or biases that may result from single-modal data. Utilizing a multimodal feature extraction model, key feature information can be mined from data of different modalities, extracting rich semantic and contextual content, which helps to more accurately describe and characterize customer business needs. Based on a preset feature fusion strategy, multimodal data features are fused, integrating the advantages of each modality and compensating for the shortcomings of a single modality, forming a more complete and expressive multimodal fusion feature, thus more accurately reflecting customer intent. Inputting the multimodal fusion feature into an intent recognition model that matches the fusion strategy can fully leverage the model's processing capabilities for fusion features, performing intent feature recognition analysis, generating more reliable and accurate intent recognition results, and effectively improving the accuracy of identifying the target customer's auto insurance business needs.

[0168] Specific limitations regarding the multimodal learning-based intent recognition device can be found in the limitations of the multimodal learning-based intent recognition method described above, and will not be repeated here. Each module in the aforementioned multimodal learning-based intent recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0169] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side intent recognition method based on multimodal learning.

[0170] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of an intent recognition method based on multimodal learning.

[0171] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0172] Collect multimodal business data related to auto insurance business requested by target customers, wherein the multimodal business data includes business data of at least two modal types;

[0173] Based on the multimodal feature extraction model, feature extraction is performed on the multimodal business data to obtain multimodal data features;

[0174] Based on a preset feature fusion strategy, multimodal data features are fused to obtain multimodal fused features;

[0175] The multimodal fusion features are input into an intent recognition model that matches the feature fusion strategy. The multimodal fusion features are then analyzed for intent feature recognition to generate intent recognition results for the target customer's needs for auto insurance business.

[0176] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0177] Collect multimodal business data related to auto insurance business requested by target customers, wherein the multimodal business data includes business data of at least two modal types;

[0178] Based on the multimodal feature extraction model, feature extraction is performed on the multimodal business data to obtain multimodal data features;

[0179] Based on a preset feature fusion strategy, multimodal data features are fused to obtain multimodal fused features;

[0180] The multimodal fusion features are input into an intent recognition model that matches the feature fusion strategy. The multimodal fusion features are then analyzed for intent feature recognition to generate intent recognition results for the target customer's needs for auto insurance business.

[0181] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0182] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0183] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0184] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for intent recognition based on multi-modal learning, characterized in that, The method comprises: Collecting multi-modal business data related to the target customer's request for vehicle insurance business, wherein the multi-modal business data comprises business data of at least two modal types; Performing feature extraction on the multi-modal business data based on a multi-modal feature extraction model to obtain multi-modal data features; Performing multi-modal feature fusion on the multi-modal data features based on a preset feature fusion strategy to obtain multi-modal fusion features; Inputting the multi-modal fusion features into an intent recognition model matched with the feature fusion strategy to perform intent feature recognition analysis on the multi-modal fusion features, and generating an intent recognition result of the target customer's demand for vehicle insurance business. 2.The intent recognition method based on multi-modal learning according to claim 1, characterized in that, The multi-modal feature fusion based on the preset feature fusion strategy comprises: When the feature fusion strategy is a first fusion strategy, directly concatenating data features corresponding to each modal type in the multi-modal data features to obtain first concatenated features; Performing normalization and dimension alignment on the first concatenated features to obtain the multi-modal fusion features. 3.The multi-modal learning based intent recognition method of claim 1, wherein, The multi-modal feature fusion based on the preset feature fusion strategy comprises: When the feature fusion strategy is a second fusion strategy, performing feature preprocessing on data features of each modal type in the multi-modal data features to obtain preprocessed features of different modal types; Concatenating and fusing the preprocessed features of different modal types to obtain the multi-modal fusion features. 4.The method of claim 1, wherein, The multi-modal feature fusion based on the preset feature fusion strategy comprises: When the feature fusion strategy is a third fusion strategy, inputting data features of each modal in the multi-modal data features into corresponding single-modal intent recognition sub-models to output single-modal features corresponding to each modal type; Based on the feature fusion weights corresponding to the business data of each modal type, weighting and fusing the single-modal features corresponding to each modal type to obtain the multi-modal fusion features. 5.The multi-modal learning based intent recognition method according to claim 1, wherein, Before the multi-modal feature fusion based on the preset feature fusion strategy, the method further comprises: Calculating feature correlation between the multi-modal data features; Based on the feature correlation between the multi-modal data features, determining the feature fusion strategy between the multi-modal data features corresponding to the business data of different modal types. 6.The method of claim 5, wherein, The determination of the feature fusion strategy between the multi-modal data features corresponding to the business data of different modal types based on the feature correlation between the multi-modal data features comprises: When the feature correlation is greater than or equal to a correlation threshold, determining the feature fusion strategy as at least one of the first fusion strategy, the second fusion strategy, and the third fusion strategy; When the feature correlation is less than the correlation threshold, determining the feature fusion strategy as at least one of the second fusion strategy and the third fusion strategy. 7.The multi-modal learning based intent recognition method according to claim 1, wherein, Before the inputting the multi-modal fusion feature into the intent recognition model matched with the feature fusion strategy, performing intent feature recognition analysis on the multi-modal fusion feature, and generating the intent recognition result of the target customer for the vehicle insurance business demand, the method further includes: collecting and preprocessing multi-modal training data, the multi-modal training data at least including text type data, speech type data and image type data; performing feature extraction on the multi-modal training data to obtain multi-modal training data features; performing each feature fusion strategy on the multi-modal training data features to generate training data fusion features corresponding to each feature fusion strategy; based on the training data fusion features corresponding to each feature fusion strategy, iteratively training the pre-training model to generate an intent recognition model adapted to each feature fusion strategy. 8.An intent recognition apparatus based on multi-modal learning, characterized by comprising: The intent recognition device based on multi-modal learning includes: a data acquisition module configured to acquire multi-modal business data related to a vehicle insurance business requested by a target customer, the multi-modal business data including business data of at least two modal types; a feature extraction module configured to perform feature extraction on the multi-modal business data based on a multi-modal feature extraction model to obtain multi-modal data features; a feature fusion module configured to perform multi-modal feature fusion on the multi-modal data features based on a preset feature fusion strategy to obtain multi-modal fusion features; an intent recognition module configured to input the multi-modal fusion features into an intent recognition model matched with the feature fusion strategy, perform intent feature recognition analysis on the multi-modal fusion features, and generate an intent recognition result of the target customer for the vehicle insurance business demand.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the intent recognition method based on multi-modal learning as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the steps of the intent recognition method based on multi-modal learning as claimed in any one of claims 1 to 7.

Images