Artificial intelligence interaction and decision analysis system based on multi-modal technology

Abstract

The invention discloses an artificial intelligence interaction and decision analysis system based on a multi-modal technology. The invention discloses the artificial intelligence interaction and decision analysis system based on the multi-modal technology. The data acquisition module is used for acquiring a plurality of multi-modal feedback suggestions and suggestion parameters published by a plurality of participants for different subject discussions; the analysis module traverses each subject discussion, determines the emotional polarity and emotional intensity of the multi-modal feedback opinions by using an emotional analysis model, determines the evaluation quality scores of the multi-modal feedback opinions according to opinion parameters, and determines the popularity of the subject discussion in combination with the analysis result; the decision-making module clusters a plurality of multi-modal feedback suggestions under a plurality of topic discussions, and determines the processing emergency degree of each cluster according to the popularity of the topic discussions to which each multi-modal feedback suggestion in each cluster belongs; and supervising the multi-modal feedback suggestions in the plurality of clusters according to the processing emergency degree. The technical problems that the processing efficiency is low and the user experience feeling is poor when supervision is carried out in sequence according to the publishing sequence of the user feedback opinions under the subject discussion in the related technology are solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an artificial intelligence interaction and decision analysis system based on multimodal technology. Background Art

[0002] In traditional community management and feedback processing, user feedback is typically collected and processed in the order in which it was posted in the discussion forum. While this approach is intuitive and simple, it actually exposes a series of efficiency deficiencies. Especially on platforms with a large user base and high-frequency interactions, chronologically processing user feedback (whether text comments, voice messages, or illustrated suggestions) leads to several key issues:

[0003] (1) Important feedback information is drowned out: Due to the lack of prior assessment of the importance or urgency of feedback, some issues containing key information or that need to be resolved urgently may be postponed due to late release time, missing the best time to respond.

[0004] (2) Uneven resource allocation: Processing in chronological order means that processing resources are evenly distributed regardless of the complexity of the feedback. This one-size-fits-all approach ignores the diversity of problems and the differences in the difficulty of solving them, resulting in wasted resources on low-value or low-priority tasks.

[0005] (3) Low supervision efficiency: In the absence of intelligent analysis, the manual review team needs to review the feedback one by one, and then classify and supervise based on experience and judgment. This is not only time-consuming, but also prone to misjudgment or omission of key issues due to subjective judgment, affecting the efficiency and quality of problem solving.

[0006] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0007] An embodiment of the present application provides an artificial intelligence interaction and decision analysis system based on multimodal technology to at least solve the technical problems of low processing efficiency and poor user experience in related technologies that supervise user feedback in the order of publication in special discussions.

[0008] According to one aspect of the embodiments of the present application, a method for an artificial intelligence interaction and decision analysis system based on multimodal technology is provided. The system at least includes: a data acquisition module, an analysis module, and a decision module. Among them, the data acquisition module is used to obtain multiple multimodal feedback opinions published by multiple participants for different topic discussions and the opinion parameters of each multimodal feedback opinion; the analysis module is used to traverse each topic discussion, use an emotion analysis model to determine the sentiment polarity and sentiment intensity of each multimodal feedback opinion under the topic discussion, determine the evaluation quality score of each multimodal feedback opinion under the topic discussion according to the opinion parameters, and determine the popularity of the topic discussion according to the evaluation quality score, sentiment polarity and sentiment intensity of each multimodal feedback opinion under the topic discussion; the decision module is used to cluster multiple multimodal feedback opinions published by multiple participants for different topic discussions to obtain multiple clustering clusters, and determine the processing urgency of the clustering clusters according to the popularity of the topic discussions to which the multimodal feedback opinions in each clustering cluster belong, where the similarity of the multimodal feedback opinions in the clustering cluster is higher than a preset similarity threshold and the sentiment polarity is consistent; and supervise the multimodal feedback opinions in multiple clustering clusters in sequence according to the processing urgency.

[0009] Optionally, the system further includes: a model training module, and the model training module is used to train the emotion analysis model according to the following steps: construct an initial learning model, where the initial learning model at least includes: a multimodal sentiment classification network and a sentiment intensity evaluation network; obtain multiple groups of training sample data, where each group of training sample data includes: a multimodal feedback opinion sample and the sentiment polarity label and sentiment intensity label corresponding to the multimodal feedback opinion sample, and the sentiment polarity label includes: positive emotion, negative emotion; for each training batch in the iterative training process, input the multimodal feedback opinion samples of the training batch into the multimodal sentiment classification network and the sentiment intensity evaluation network to obtain the predicted sentiment polarity corresponding to each multimodal feedback opinion sample output by the multimodal sentiment classification network, and obtain the predicted sentiment intensity corresponding to each multimodal feedback opinion sample output by the sentiment intensity evaluation network; use the sentiment polarity label and sentiment intensity label, predicted sentiment polarity and predicted sentiment intensity corresponding to each multimodal feedback opinion sample to construct an objective loss function, where the objective loss function includes: a regression loss function, a classification loss function; adjust the model parameters of the initial learning model according to the objective loss function until the model parameters converge to obtain the trained emotion analysis model.

[0010] Optionally, the analysis module at least includes: a feature extraction unit and a first calculation unit, where the feature extraction unit is configured to extract features from each multimodal feedback opinion under the special topic discussion to obtain a fused feature vector corresponding to each multimodal feedback opinion; the first calculation unit is configured to calculate the evaluation quality of each multimodal feedback opinion based on the opinion parameter of the multimodal feedback opinion and the corresponding fused feature vector.

[0011] Optionally, the multimodal feedback opinion at least includes: a first feedback opinion in the form of an image, a second feedback opinion in the form of speech, and a third feedback opinion in the form of text. Wherein, the feature extraction unit is further configured to extract features from the multimodal feedback opinion according to the following steps to obtain a corresponding fused feature vector: when the multimodal feedback opinion includes the first feedback opinion, perform a first preprocessing operation on the first feedback opinion, where the first preprocessing operation includes at least one of the following: grayscale processing, text region detection and segmentation, text direction correction; use an optical character recognition model to extract the text data corresponding to the preprocessed first feedback opinion; when the multimodal feedback opinion includes the second feedback opinion, perform a second preprocessing operation on the second feedback opinion, where the second preprocessing operation includes at least one of the following: denoising, loudness adjustment, pre-emphasis; use a dialect recognition model to identify the preprocessed second feedback opinion to determine the corresponding dialect type, and correct the preprocessed second feedback opinion according to a preset dialect correction rule, where the dialect recognition model is obtained by training a neural network model with multiple low-rank bypasses using a training data set fine-tuned with multiple dialect data, and each low-rank bypass is used to learn a kind of dialect data; use automatic speech recognition technology to identify the text data corresponding to the corrected second feedback opinion; when the multimodal feedback opinion includes the third feedback opinion, perform a third preprocessing operation on the third feedback opinion, where the third preprocessing operation includes at least one of the following: text cleaning, lemmatization, uniform case; map the text data corresponding to the preprocessed first feedback opinion, the text data corresponding to the corrected second feedback opinion, and the preprocessed third feedback opinion into the same vector space to obtain three text feature vectors, and use an attention mechanism to fuse the three text feature vectors to obtain a fused feature vector.

[0012] Optionally, the opinion parameter at least includes the interaction frequency. Among them, the first calculation unit is further configured to calculate the evaluation quality of each multimodal feedback opinion according to the opinion parameter of the multimodal feedback opinion and the corresponding fusion feature vector, and in accordance with the following steps: analyze the fusion feature vector corresponding to the multimodal feedback opinion by using the N-Gram algorithm to obtain the novelty of the multimodal feedback opinion; determine the interaction degree of the multimodal feedback opinion according to the matching result between the preset interaction rate quantization rule and the interaction frequency of the multimodal feedback opinion, where the corresponding relationship between different interaction frequencies and interaction degrees is defined in the interaction rate quantization rule; calculate the similarity between the fusion feature vector corresponding to the multimodal feedback opinion and multiple target knowledge texts corresponding to the topic discussion, and take the average value of the multiple similarities as the professionalism of the multimodal feedback opinion, where the target knowledge text is the knowledge text with the highest number of searches in the knowledge text set corresponding to the topic discussion; comprehensively determine the comment quality score of the multimodal feedback opinion based on the novelty, interaction degree, and professionalism.

[0013] Optionally, the first calculation unit is further configured to analyze the fusion feature vector corresponding to the multimodal feedback opinion by using the N-Gram algorithm according to the following steps to obtain the novelty of the multimodal feedback opinion: divide the fusion feature vector corresponding to the multimodal feedback opinion according to the preset length to obtain a first number of N-Gram vectors, and calculate the similarity between every two N-Gram vectors; take the N-Gram vector pairs with similarity higher than the preset second similarity threshold as duplicate N-Gram vectors, and count the second number of duplicate N-Gram vectors in the fusion feature vector corresponding to the multimodal feedback opinion; take 1 minus the quotient of the second number and the first number as the novelty of the multimodal feedback opinion.

[0014] Optionally, the analysis module further includes: a second calculation unit, and the second calculation unit is configured to determine the popularity of the topic discussion according to the evaluation quality score, emotional polarity, and emotional intensity of each multimodal feedback opinion under the topic discussion, and in accordance with the following steps: determine the total number of multimodal feedback opinions published under the topic discussion; take the average value of the evaluation quality scores of the multiple multimodal feedback opinions under the topic discussion as the overall comment quality score of the topic discussion; determine the proportion of the topic discussion under different emotional polarities and different emotional intensities according to the emotional polarity and emotional intensity of each multimodal feedback opinion under the topic discussion, and perform weighted summation on the proportions of the topic discussion under different emotional polarities and different emotional intensities to obtain the overall emotional score of the topic discussion; perform weighted summation on the total number, the overall comment quality score, and the overall emotional score to obtain the popularity of the topic discussion.

[0015] Optionally, the decision-making module is further configured to supervise the multi-modal feedback opinions in multiple clustering clusters in the order of processing urgency according to the following steps: For each clustering cluster, obtain the set of historical multi-modal feedback opinion response results of the participants who issued the multi-modal feedback opinions within the clustering cluster, where the set of historical multi-modal feedback opinion response results includes: multiple historical multi-modal feedback opinions of the participants within the historical time period, and the supervision result and processing time consumption of each historical multi-modal feedback opinion; construct a user portrait of the corresponding participant based on the set of historical multi-modal feedback opinion response results; determine at least one high-quality multi-modal feedback opinion from within the clustering cluster based on the user portrait of the participant who issued the multi-modal feedback opinion and the evaluation quality score of the multi-modal feedback opinion, where the high-quality multi-modal feedback opinion is a multi-modal feedback opinion with a high supervision success rate and short processing time consumption within the clustering cluster, and the evaluation quality score published by the participant is higher than a preset evaluation threshold; supervise each high-quality multi-modal feedback opinion in multiple clustering clusters in the order of processing urgency.

[0016] In the embodiment of the present application, the data acquisition module may first acquire multiple multi-modal feedback opinions issued by multiple participants for different topic discussions and the opinion parameters of each multi-modal feedback opinion; then, the analysis module determines the popularity of the topic discussion based on the multiple multi-modal feedback opinions issued by multiple participants under each topic discussion and the opinion parameters of each multi-modal feedback opinion; then, the decision-making module clusters the multiple multi-modal feedback opinions issued by multiple participants for different topic discussions to obtain multiple clustering clusters, and determines the processing urgency of the clustering clusters based on the popularity of the topic discussions to which the multi-modal feedback opinions within each clustering cluster belong, and supervises the multi-modal feedback opinions in multiple clustering clusters in the order of processing urgency, achieving efficient classification and priority sorting of user opinions, achieving the purpose of quickly responding to user demands and improving the problem-solving efficiency, and thus solving the technical problems of low processing efficiency and poor user experience in the related art of supervising in the order of the publication sequence of user feedback opinions under topic discussions. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0018] Figure 1 is a schematic structural diagram of an optional artificial intelligence interaction and decision analysis system based on multi-modal technology according to an embodiment of the present application;

[0019] Figure 2 is a detailed schematic structural diagram of another optional artificial intelligence interaction and decision analysis system based on multi-modal technology according to an embodiment of the present application. Specific implementation manner

[0020] In order to enable those skilled in the art of the present technology to better understand the solution of this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of this application.

[0021] It should be noted that the terms "first", "second", etc. in the description, claims and drawings of this application are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0022] Figure 1 is a schematic structural diagram of an artificial intelligence interaction and decision analysis system based on multi-modal technology according to an embodiment of this application, as Figure 1 shown. The system at least includes: a data acquisition module 11, an analysis module 12, and a decision module 13. Among them,

[0023] First, the data acquisition module 11 acquires a plurality of multi-modal feedback opinions published by multiple participants for discussions on different topics and the opinion parameters of each multi-modal feedback opinion.

[0024] Among them, the multi-modal feedback opinion is the feedback opinion published by the participants in the topic discussion through various media, and its forms include but are not limited to: comments input in text, suggestions recorded in voice, attachments of handwritten or image content, and viewpoints expressed in video form. And these feedback opinions can be suggestions on product functions, improvement opinions on service processes, reports on system defects, or feedback on specific policies, which are used to help financial institutions continuously understand and respond to the personalized needs of users. The opinion parameter of the multi-modal feedback opinion is metadata used to describe and quantify the characteristics of each multi-modal feedback opinion, including but not limited to the timestamp when the feedback opinion is published, the basic information of the feedbacker (such as department affiliation, job level, etc.), the type of the feedback content (such as text, voice, image), the length of the feedback content, and the interaction situation.

[0025] Then, the analysis module 12 can traverse each forum discussion, use the sentiment analysis model to determine the sentiment polarity and sentiment intensity of each multimodal feedback opinion under the forum discussion, determine the evaluation quality score of each multimodal feedback opinion under the forum discussion according to the opinion parameters, and determine the popularity of the forum discussion according to the evaluation quality score, sentiment polarity and sentiment intensity of each multimodal feedback opinion under the forum discussion.

[0026] For each forum discussion, the analysis module 12 can use natural language processing techniques combined with deep learning algorithms to analyze and quantify the sentiment color and intensity in the user feedback. At the same time, score the multimodal feedback opinions according to the characteristics of the feedback opinions to obtain the evaluation quality score of each multimodal feedback opinion; determine the popularity of the forum discussion according to the evaluation quality score, sentiment polarity and sentiment intensity of each multimodal feedback opinion under the forum discussion, which is the key basis for reflecting the importance and urgency of the forum discussion.

[0027] Finally, the decision-making module 13 clusters the multiple multimodal feedback opinions published by multiple participants for different forum discussions to obtain multiple clusters, and determines the processing urgency of the clusters according to the popularity of the forum discussions to which each multimodal feedback opinion in each cluster belongs, and supervises the multimodal feedback opinions in multiple clusters in turn according to the processing urgency.

[0028] Specifically, the decision-making module 13 can cluster the multiple multimodal feedback opinions published by multiple participants for different forum discussions to form multiple clusters with clear themes and consistent sentiment tendencies. Therefore, the similarity of each multimodal feedback opinion in the cluster is higher than the preset similarity threshold and the sentiment polarities are the same, that is, each cluster represents a group of user feedback opinions around a specific issue, so that subsequent management personnel can identify and solve the problems generally concerned by users in a targeted manner. Immediately afterwards, determine the processing urgency of the clusters according to the popularity of the forum discussions to which each multimodal feedback opinion in each cluster belongs, so as to judge which clusters reflect the issues that users are most concerned about and have strong emotions, and then subsequent processing and response can be given priority to ensure that highly concerned issues with strong emotions, high quality and wide influence are solved first. Furthermore, sort the processing urgencies of multiple clusters from high to low, and convert any multimodal feedback opinion in the cluster into a specific supervision task in turn according to the sorting result, and automatically distribute it to the corresponding responsible department or individual to ensure that the problem is responded to and solved in a timely manner.

[0029] The following will combine Figure 2 the structure diagram shown below to explain the functions of each module of the artificial intelligence interaction and decision analysis system based on multimodal technology.

[0030] According to some alternative embodiments of the present application, the data acquisition module 11 can use web crawling technology and API interfaces to capture in real time multiple multimodal feedback opinions and the opinion parameters of each multimodal feedback opinion published by multiple participants for different topic discussions, and store these opinions and parameters in a preset database for subsequent retrieval and analysis. For example, taking the topic discussion of "Cultural Bank" initiated by Bank A as an example, participants can publish suggestions, ideas, or problem descriptions regarding the implementation of the "Cultural Bank" strategy, such as suggesting to hold customer activities with cultural themes (such as Chinese painting workshops, etc.), suggesting to increase training related to cultural banks (such as historical financial knowledge, cultural sensitivity training, etc.), suggesting to integrate cultural elements into financial products and services (such as developing wealth management products related to cultural heritage, providing customized cultural bank services), etc.

[0031] The above-described forms include but are not limited to: participants inputting text descriptions in the discussion area, participants providing opinions through the voice message function, participants uploading image files containing handwritten suggestions or charts, and participants uploading self-recorded video feedback. For example, an employee in Bank A published a composite feedback opinion on "pain points in promoting digital transformation" under the topic discussion of "employees submit suggestions for digital transformation", including: a text comment describing the problem of "low cross-departmental collaboration efficiency in agile organization building", a voice message supplementing the description that "the existing system cannot support the rapid iteration requirements of the No. 1 combined financial product", and a screenshot attachment of a handwritten flowchart annotating the process bottlenecks (such as "the approval process requires the serial stamping of 3 departments").

[0032] According to some other alternative embodiments of the present application, the artificial intelligence interaction and decision analysis system based on multimodal technology further includes a model training module 14, and the model training module 14 can train an emotion analysis model according to the following steps, including:

[0033] The first step: Construct an initial learning model. Among them, the initial learning model at least includes: a multimodal sentiment classification network and a sentiment intensity evaluation network, and the sentiment classification network is used to predict the sentiment polarity of the feedback opinion, while the sentiment intensity evaluation network is used to quantify the sentiment intensity.

[0034] For example, the grid architecture of the initial learning model can be an improved BERT-CRF+BiLSTM hybrid model. Among them, BERT (Bidirectional Encoder Representations from Transformers), as a pre-trained model based on the Transformer architecture, is used to capture complex semantic relationships and context information in the text, generate deep semantic vector representations for each word, and help understand the emotional color. And CRF (Conditional Random Field), as a sequence labeling model, can classify the emotional polarity. BiLSTM (Bidirectional Long Short-Term Memory), as a bidirectional recurrent neural network, can transmit information in both the forward and backward directions simultaneously, capture the context before and after the emotional words in the text, accurately identify the position information of the emotional words and their influence on the surrounding words, so as to refine the evaluation of the emotional intensity. The design of this hybrid model allows the model to not only identify the emotional polarity (such as positive or negative), but also carefully evaluate the emotional intensity (such as very negative, slightly negative, neutral, slightly positive, very positive, etc.), and has better generalization ability and robustness when dealing with long texts or multimodal data.

[0035] Step 2: Obtain multiple sets of training sample data.

[0036] Specifically, the multiple sets of training sample data can be obtained by manually annotating the historical feedback opinions in financial institutions. Among them, each set of training sample data includes: multimodal feedback opinion samples, and the emotional polarity labels and emotional intensity labels corresponding to the multimodal feedback opinion samples. The emotional polarity labels include: positive emotion, negative emotion.

[0037] Step 3: For each training batch in the iterative training process, input the multimodal feedback opinion samples of the training batch into the multimodal emotion classification network and the emotional intensity evaluation network, obtain the predicted emotional polarity (such as positive or negative) corresponding to each multimodal feedback opinion sample output by the multimodal emotion classification network, and obtain the predicted emotional intensity (such as level 0 to 5) corresponding to each multimodal feedback opinion sample output by the emotional intensity evaluation network.

[0038] Among them, the initial learning model also includes a feature extraction module, and the feature extraction module can refer to the text vectorization process in step S1042 below to extract features from the multimodal feedback opinion samples.

[0039] Step 4: Construct an objective loss function using the emotional polarity labels and emotional intensity labels, predicted emotional polarity, and predicted emotional intensity corresponding to each multimodal feedback opinion sample.

[0040] Specifically, the above-mentioned target loss function includes: a regression loss function and a classification loss function. Among them, the classification loss function is used to measure the gap between the predicted sentiment polarity output by the sentiment classification network and the true sentiment polarity label. Therefore, the cross-entropy loss function can be used as the classification loss function; while the regression loss function is used to evaluate the error between the predicted sentiment intensity output by the sentiment intensity evaluation network and the true sentiment intensity label. Therefore, the Root Mean Squared Error (RMSE) function or the Mean Squared Error (MSE) function can be used as the regression loss function.

[0041] Step 5: Adjust the model parameters of the initial learning model according to the target loss function until the model parameters converge, and obtain the trained sentiment analysis model.

[0042] Specifically, the model training module 14 can adjust the model parameters according to the value of the target loss function through the backpropagation algorithm. Among them, the gradient of the target loss function guides the parameter update to minimize the gap between the prediction result and the true label. This process can be continuously carried out in multiple training batches using stochastic gradient descent, Adam or other optimization algorithms until the model parameters converge, indicating that the model has learned an effective pattern for predicting sentiment polarity and sentiment intensity from multi-modal feedback opinions.

[0043] It should be noted that in order to make the sentiment analysis model better adapt to the financial scenario, a domain adaptation stage can be added in the later stage of model training. Through fine-tuning of the training sample data or transfer learning, the sentiment analysis model can be made more familiar with financial terms, expressions related to financial service processes, and industry-specific sentiment expression patterns, so as to improve the generalization ability of the model in the financial field. Among them, fine-tuning of the training sample data is to use the multi-modal feedback opinions collected in the financial field to train at least one newly added fully connected layer in the output layer of the sentiment analysis model, which is used to capture specific expressions and sentiment patterns in the financial field. And transfer learning is to train the sentiment analysis model on the task of analyzing multi-modal feedback opinions in the financial scenario, so that the model can learn how to identify the sentiment polarity and sentiment intensity of multi-modal feedback opinions in the financial scenario.

[0044] In addition, during the model training process, a part of the data can be reserved in advance as a test set to monitor the model performance of the sentiment analysis model and prevent overfitting. Once the training is completed, an independent test set is used to comprehensively test the sentiment analysis model to ensure that its performance on unseen data meets the expectations, can accurately distinguish positive and negative emotions, and can accurately evaluate the sentiment intensity.

[0045] Therefore, the analysis module 12 can traverse each topic discussion and call the sentiment analysis model pre-trained by the model training module 14 to determine the sentiment polarity and sentiment intensity of each multimodal feedback opinion under the topic discussion. Among them, the above sentiment polarity is the overall emotional tendency presented by the multimodal feedback opinion, such as positive emotion and negative emotion, while the sentiment intensity quantifies the intensity of the emotional tendency, usually represented by a numerical value, ranging from 0 to 1, and the larger the value, the stronger the emotional tendency.

[0046] Furthermore, the analysis module 12 can also determine the evaluation quality score of each multimodal feedback opinion under the topic discussion according to the opinion parameters. Specifically, the analysis module 12 may include: a feature extraction unit 121 and a first calculation unit 122, where:

[0047] The feature extraction unit 121 can extract features from each multimodal feedback opinion traversed under the topic discussion to obtain a corresponding fused feature vector. Among them, the multimodal feedback opinion at least includes: a first feedback opinion in the form of an image, a second feedback opinion in the form of speech, and a third feedback opinion in the form of text.

[0048] The first calculation unit 122 can calculate the evaluation quality of each multimodal feedback opinion according to the opinion parameters of the multimodal feedback opinion and the corresponding fused feature vector.

[0049] Optionally, the feature extraction unit 121 can extract features from the multimodal feedback opinion according to the following steps to obtain a corresponding fused feature vector:

[0050] In the case where the multimodal feedback opinion includes the first feedback opinion, the feature extraction unit 121 can process it according to the following steps:

[0051] First, perform a first preprocessing operation on the first feedback opinion. Among them, the first preprocessing operation includes but is not limited to: grayscale processing (used to remove the interference of the image background color, reduce the computational complexity, and ensure the clarity and integrity of the key text information are not affected), using computer vision techniques (such as convolutional neural networks or deep learning models) to detect and segment the text area of the image (used to separate the text part in the image from the non-text background), and using image transformation techniques to correct the text direction, etc.

[0052] Then, use the optical character recognition model to extract the text data corresponding to the preprocessed first feedback opinion. Among them, the model architecture of the optical character recognition model can be a distortion resistant convolutional network (DRCN) based on the attention mechanism, and it is trained based on a text image dataset including printed or handwritten texts, so as to ensure that the optical character recognition model not only has the ability to process deformed texts with different writing styles in complex images, but also can intelligently focus on identifying key text information and ignore irrelevant background interference.

[0053] In addition, considering that the OCR recognition result may contain a small number of errors, therefore, the feature extraction unit 121 can also check and correct the text data corresponding to the preprocessed first feedback opinion, such as spelling check, error correction based on the language model, and homophone discrimination.

[0054] In the case where the multi-modal feedback opinion includes the second feedback opinion, the feature extraction unit 121 can process it according to the following steps:

[0055] First, perform a second preprocessing operation on the second feedback opinion. Among them, the second preprocessing operation includes but is not limited to: applying frequency domain filtering techniques (such as band-pass filtering, adaptive noise suppression, and spectral subtraction, etc.) for denoising, using loudness normalization techniques for loudness adjustment (used to adjust the loudness of the speech signal to make it reach the best recognition level), pre-emphasis (used to compensate for the frequency characteristic attenuation during speech transmission to enhance the high-frequency part, especially to compensate for the common low-frequency roll-off phenomenon in mobile phone or telephone calls), etc.

[0056] Then, use the dialect recognition model to recognize the preprocessed second feedback opinion to determine the corresponding dialect type. Among them, the dialect recognition model is obtained by training a neural network model with multiple low-rank bypasses (LRB) on a training dataset fine-tuned with various dialect data, so that the model can learn to stably recognize speech in different dialect environments and noise conditions. Among them, each low-rank bypass is used to learn the characteristics of a kind of dialect data, such as syllables, intonation patterns, pronunciation habits, etc. Therefore, during the model training process, the parameters of each low-rank bypass can be independently updated, while the original parameters of the other main networks in the neural network except for each low-rank bypass remain unchanged, so as to reduce the amount of parameter adjustment during the training process of the dialect recognition model for different dialect recognition tasks in the financial scenario.

[0057] Then, the preprocessed second feedback opinion is corrected according to preset dialect correction rules, where these rules include but are not limited to lexical replacement, grammar structure adjustment, and pronunciation correction, with the aim of converting the dialect speech into the speech characteristics of standard Mandarin or English.

[0058] Finally, the automatic speech recognition technology is used to recognize the text data corresponding to the corrected second feedback opinion.

[0059] In addition, considering that the ASR recognition result may contain a small number of errors, the feature extraction unit 121 can also verify and correct the text data corresponding to the corrected second feedback opinion, such as spelling correction, grammar repair, etc., so as to improve the readability and accuracy of the text.

[0060] In the case where the multi-modal feedback opinion includes the third feedback opinion, the feature extraction unit 121 can directly perform a third preprocessing operation on the third feedback opinion, where the third preprocessing operation includes but is not limited to: text cleaning based on regular expressions or natural language processing techniques (to remove irrelevant or interfering content in the text, including but not limited to hypertext markup language HTML tags, special characters, emojis, uniform resource locator URL links, etc.), lemmatization using natural language processing libraries or custom dictionaries (used to convert the vocabulary in the text into its basic form, such as converting the past participle form of a verb into the base verb form, or removing the comparative and superlative degrees of adjectives and adverbs), and unifying the case (converting the text into a unified case format, usually converting to lowercase).

[0061] Finally, a unified word vector model is adopted or domain adaptation fine-tuning of the model is performed to map the text data corresponding to the preprocessed first feedback opinion, the text data corresponding to the corrected second feedback opinion, and the preprocessed third feedback opinion into the same vector space to obtain three text feature vectors, and the attention mechanism is used to fuse the three text feature vectors to obtain a fused feature vector. Among them, the attention mechanism can automatically assign higher weights to those feature vectors that contribute more to the fusion result according to the correlation between the three text feature vectors, and then perform weighted summation on the three text feature vectors based on the weights calculated by the attention mechanism to generate the final fused feature vector. This fusion method can highlight the information that is more important for subsequent analysis while suppressing the influence of irrelevant or redundant information.

[0062] Optionally, the first calculation unit 122 can calculate the evaluation quality of each multi-modal feedback opinion according to the opinion parameters of the multi-modal feedback opinion and the corresponding fused feature vector, and follow the steps below:

[0063] Step 1: Analyze the fused feature vectors corresponding to the multi-modal feedback using the N-Gram algorithm to obtain the novelty of the multi-modal feedback.

[0064] Specifically, the first calculation unit 122 can divide the fused feature vectors corresponding to the multi-modal feedback according to a preset length (i.e., a fixed N words), obtain the first number of N-Gram vectors, and calculate the similarity between every two N-Gram vectors by methods such as cosine similarity, Euclidean distance, or Jaccard similarity; regard the N-Gram vector pairs with similarity higher than the preset second similarity threshold as duplicate N-Gram vectors, and count the second number of duplicate N-Gram vectors in the fused feature vectors corresponding to the multi-modal feedback; take 1 minus the quotient of the second number and the first number (i.e., the duplication degree) as the novelty of the multi-modal feedback.

[0065] Among them, if the novelty value is closer to 1, it means that the content in the fused feature vectors is more unique and the repeatability is lower, which means that the insights and requirements contained in the multi-modal feedback may be more novel and of greater value to strategy formulation and product optimization; on the contrary, if the novelty value is closer to 0, it means that there is more repetitive content in the feedback, so the reasons behind the repetition can be further analyzed to determine whether it is the common needs or problems of multiple users.

[0066] Step 2: Determine the interaction degree of the multi-modal feedback according to the matching result between the preset interaction rate quantization rule and the interaction frequency of the multi-modal feedback. Among them, the interaction rate quantization rule defines the corresponding relationship between different interaction frequencies and interaction degrees.

[0067] Among them, the interaction frequency reflects the number of times the multi-modal feedback is viewed, liked, replied to, or shared by users within a certain time range (such as one month, one week, one day) after the participant posts the multi-modal feedback, and the interaction degree is an abstract quantification of this interaction frequency, reflecting the focus and participation of users in the multi-modal feedback. Therefore, the interaction rate quantization rule pre-defines multiple interaction degree levels, and each interaction degree level corresponds to an interaction frequency interval. For example: if the interaction frequency is less than or equal to 10 times per day, it means that the interaction degree level is low; if the interaction frequency is between 11 times per day and 100 times per day, it means that the interaction degree level is medium; if the interaction frequency is greater than 100 times per day, it means that the interaction degree level is high. Furthermore, the system can match the interaction frequency of the multi-modal feedback collected regularly or in real time with the interaction rate quantization rule to obtain the interaction degree of the multi-modal feedback. For example, if a feedback containing text and voice is viewed 50 times, liked 50 times, and replied to 40 times within a week, then its total interaction frequency may be the sum of these three items divided by 7 (days), that is, the average number of interactions per day is 20 times per day. By matching, the interaction degree of this multi-modal feedback can be obtained as being at the medium level.

[0068] Step 3: Determine the similarity between the fusion feature vector corresponding to the multimodal feedback opinion and multiple target knowledge texts corresponding to the special topic discussion, and use the average value of the multiple similarities as the professionalism of the multimodal feedback opinion.

[0069] Specifically, the target knowledge text is the knowledge text with the highest number of searches in the knowledge text set corresponding to the special topic discussion (i.e., internal documents, industry reports, professional articles, etc. around the theme of the special topic discussion). Therefore, the target indication text can represent the most core and well-known professional information in this special topic discussion. Therefore, the first calculation unit 122 can use the same word embedding technology as the multimodal feedback opinion or a more advanced semantic understanding model to generate the feature vector of the target knowledge text, and use cosine similarity, Euclidean distance or other similarity measurement methods to calculate the similarity between the fusion feature vector of the multimodal feedback opinion and the feature vectors of multiple target knowledge texts, and use the average value of these similarities as the professionalism of the multimodal feedback opinion. This can reduce the impact of a single target knowledge text on the professionalism evaluation, and at the same time comprehensively consider the semantic relevance between all high-search-frequency knowledge texts and the multimodal feedback opinion. Therefore, the higher the professionalism value, the closer the feedback opinion is semantically to the target knowledge text, and the richer the professional information it contains.

[0070] Step 4: Comprehensively determine the comment quality score of the multimodal feedback opinion based on novelty, interactivity and professionalism.

[0071] Specifically, the first calculation unit 122 can perform weighted summation on novelty, interactivity and professionalism, and combine a non-linear adjustment term to obtain the comment quality score of the multimodal feedback opinion. Among them, the non-linear adjustment term is mainly used to adjust the product effect between novelty and professionalism. For example, comments with high novelty often also have high professionalism, which has an additional positive impact on the evaluation of comment quality, and vice versa. In addition, regarding the weighting coefficient, it can be determined based on the analysis of historical data, such as the actual contribution ratio of different types of comments to business improvement, or through expert evaluation, comprehensively considering the relative importance of novelty (innovation value), interactivity (user participation) and professionalism (content depth) to the comment quality score.

[0072] According to some other optional embodiments of the present application, the analysis module 12 further includes: a second calculation unit 123, and the second calculation unit 123 determines the popularity of the special topic discussion according to the evaluation quality score, sentiment polarity and sentiment intensity of each multimodal feedback opinion under the special topic discussion, and follows the following steps:

[0073] Step 1: Determine the total number of multimodal feedback opinions published under the special topic discussion.

[0074] Step 2: Take the average of the evaluation quality scores of multiple multimodal feedback opinions under the special topic discussion as the overall comment quality score of the special topic discussion.

[0075] Step 3: Determine the proportions of the special topic discussion under different emotional polarities and emotional intensities based on the emotional polarity and emotional intensity of each multimodal feedback opinion under the special topic discussion, and perform a weighted sum of the proportions of the special topic discussion under different emotional polarities and emotional intensities to obtain the overall emotional score of the special topic discussion.

[0076] For example, if the emotional polarity is divided into positive and negative, and the emotional intensity is divided into strong, medium, and mild, then the multimodal feedback opinions can be grouped according to the emotional polarity and emotional intensity. For example, all multimodal feedback opinions with an emotional polarity of "positive" and an emotional intensity of "strong" are grouped into one group, all multimodal feedback opinions with an emotional polarity of "positive" and an emotional intensity of "mild" are grouped into one group, all multimodal feedback opinions with an emotional polarity of "positive" and an emotional intensity of "medium" are grouped into one group, all multimodal feedback opinions with an emotional polarity of "negative" and an emotional intensity of "strong" are grouped into one group, all multimodal feedback opinions with an emotional polarity of "negative" and an emotional intensity of "mild" are grouped into one group, and all multimodal feedback opinions with an emotional polarity of "negative" and an emotional intensity of "medium" are grouped into one group. Calculate the ratio of the number of multimodal feedback opinions included in different groups to the total number of all multimodal feedback opinions under this special topic discussion, and perform a weighted sum according to the preset weight coefficients to obtain the overall emotional score of the special topic discussion. Among them, the weighting coefficient can be adjusted according to business requirements or the importance of emotional types. For example, in some cases, negative but strong emotional feedback may be more important than positive emotional feedback because they can reveal potential problems or dissatisfaction, and these problems may require managers to respond promptly. Therefore, the above multi-dimensional emotional analysis method can not only more comprehensively capture the delicate emotions in user feedback, but also convert this emotional information into actionable data to guide decision-making and optimize services.

[0077] Step 4: Perform a weighted sum of the total quantity, the overall comment quality score, and the overall emotional score to obtain the popularity of the special topic discussion.

[0078] Through the above embodiments, the second calculation unit 123 comprehensively determines the popularity of the special topic discussion by quantifying the quantity and quality of the multi-dimensional feedback opinions provided by users and combining emotional analysis. This method of determining popularity not only reflects the activity of the special topic discussion, but also takes into account the quality and emotional tendency of the feedback opinions.

[0079] Furthermore, the system takes into account that participants, out of enthusiasm for a specific topic or to attract more attention, may post the same or similar multimodal feedback under multiple topic discussions. Such behavior may lead to distortion of the popularity index and affect the accuracy of decision-making.

[0080] Therefore, the decision-making module 13 can cluster multiple multimodal feedbacks issued by multiple participants for different topic discussions, so as to classify the multimodal feedbacks with similarity higher than the preset similarity threshold and consistent sentiment polarity into one category, obtaining multiple clustering clusters, making the clustered feedbacks better reflect the essence of the problem, facilitating accurate positioning and efficient solution. Then, sum up the popularity of the topic discussions to which each multimodal feedback in each clustering cluster belongs, sort them in descending order, and determine the processing urgency of the clustering clusters according to the sorting results. Among them, the clustering clusters with higher popularity will be given higher weights, which means that the clustering clusters will be preferentially processed to ensure that important problems can be promptly concerned and solved. Finally, use a sorting algorithm (such as quicksort, mergesort, etc.) to arrange the clustering clusters in descending order according to their processing urgency, ensuring that the urgent and highly concerned clustering clusters are processed first. For each sorted clustering cluster, the system automatically starts the supervision process, and distributes the multimodal feedbacks in the clustering cluster to the corresponding responsible department or individual for processing, ensuring that the feedbacks can be quickly and effectively conveyed to the processing end.

[0081] Optionally, the decision-making module 13 can implement the supervision of the multimodal feedbacks in multiple clustering clusters in turn according to the following method:

[0082] For each clustering cluster, obtain the set of historical multimodal feedback response results of the participants who issued each multimodal feedback in the clustering cluster. Among them, the set of historical multimodal feedback response results includes metadata such as multiple historical multimodal feedbacks of the participants in the historical time period and the supervision results (whether the supervision is successful) and processing time (the time from the opinion being proposed to the end of the supervision) of each historical multimodal feedback; construct a user profile for the corresponding participants based on the set of historical multimodal feedback response results. The user profile includes, but is not limited to, feedback tendencies (tend to propose system optimization suggestions, service process improvement opinions, etc.), supervision success rate, processing time, feedback frequency and other indicators, which are used to identify which users are more likely to provide valuable opinions; determine at least one high-quality multimodal feedback from the clustering cluster based on the user profile of the participants who issued the multimodal feedback and the evaluation quality score of the multimodal feedback. The high-quality multimodal feedback is a multimodal feedback with a high supervision success rate and a short processing time issued by a participant with an evaluation quality score higher than the preset evaluation threshold in the clustering cluster.

[0083] Then, follow the order of handling urgency to supervise the implementation of each high-quality multimodal feedback in multiple clusters one by one. The purpose of doing this is that since high-quality multimodal feedback is usually provided by participants with high supervision success rate and low processing time consumption, such participants can usually put forward practical and easily implementable suggestions. Therefore, giving priority to supervising the high-quality multimodal feedback with high evaluation quality scores they put forward can not only quickly solve problems, ensure that resources are efficiently utilized on the most potential improvement points, avoid waste of resources, but also send a positive signal to all participants, that is, the system attaches importance to and actively responds to valuable feedback, and provides useful experience for other participants through their successful cases, promoting the improvement of the overall feedback quality.

[0084] In addition, the artificial intelligence interaction and decision analysis system based on multimodal technology may further include a task execution module 15, and the task execution module 15 may create intelligent work orders according to each high-quality multimodal feedback, and the intelligent work orders contain information such as problem description, expected resolution time, responsible department, etc. Furthermore, the task execution module 1 then uses a rule engine and a priority algorithm to automatically distribute the intelligent work orders to the responsible departments, taking into account factors such as the department's processing capacity, historical response speed, and resolution rate, so as to achieve optimal resource allocation and problem-solving efficiency.

[0085] In addition, the system can also implement a feedback closed-loop through an intelligent supervision mechanism, such as functions of tracking the progress of work order processing, automatically reminding the processing deadline, collecting processing results and feeding them back to the cluster to which the high-quality multimodal feedback belongs, so that the system only needs to process the high-quality multimodal feedback within the cluster to ensure that all multimodal feedback within the cluster can be processed in a timely and effective manner, thus greatly improving the processing efficiency of user feedback.

[0086] In the above embodiments of the present application, the descriptions of each embodiment have their own emphases. For parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0087] In several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of units can be a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces, and the indirect coupling or communication connection of units or modules can be in an electrical or other form.

[0088] The unit described as a separation component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place or distributed across multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0089] In addition, each functional unit in various embodiments of the present application may be integrated in a processing unit, may exist separately as individual physical units, or two or more units may be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0090] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of the present application. The aforementioned storage medium includes: USB flash drives, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), mobile hard disks, magnetic disks, or optical discs and other various media that can store program codes.

[0091] The above are only the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. An artificial intelligence interaction and decision analysis system based on multi-modal technology, characterized in that, The system at least includes: a data acquisition module, an analysis module, and a decision-making module. Among them, the data acquisition module is used to acquire multiple multimodal feedback opinions expressed by multiple participants for different topic discussions and the opinion parameters of each of the multimodal feedback opinions; the analysis module is used to traverse each of the topic discussions, use an emotion analysis model to determine the sentiment polarity and sentiment intensity of each multimodal feedback opinion under the topic discussion, determine the evaluation quality score of each multimodal feedback opinion under the topic discussion according to the opinion parameters, and determine the popularity of the topic discussion according to the evaluation quality score, sentiment polarity and sentiment intensity of each multimodal feedback opinion under the topic discussion; the decision-making module is used to cluster the multiple multimodal feedback opinions expressed by the multiple participants for different topic discussions to obtain multiple clustering clusters, and determine the processing urgency of the clustering clusters according to the popularity of the topic discussions to which the multimodal feedback opinions in each clustering cluster belong, where the similarity of the multimodal feedback opinions in each clustering cluster is higher than a preset similarity threshold and the sentiment polarities are consistent; supervise the multimodal feedback opinions in multiple clustering clusters in sequence according to the processing urgency.

2. The system according to claim 1, wherein The system also includes: a model training module, and the model training module is used to train the emotion analysis model according to the following steps: Construct an initial learning model, where the initial learning model at least includes: a multimodal sentiment classification network and a sentiment intensity evaluation network; Obtain multiple groups of training sample data, where each group of the training sample data includes: a multimodal feedback opinion sample and the sentiment polarity label and sentiment intensity label corresponding to the multimodal feedback opinion sample, and the sentiment polarity label includes: positive emotion, negative emotion; For each training batch in the iterative training process, input the multimodal feedback opinion samples of the training batch into the multimodal sentiment classification network and the sentiment intensity evaluation network to obtain the predicted sentiment polarity corresponding to each multimodal feedback opinion sample output by the multimodal sentiment classification network, and obtain the predicted sentiment intensity corresponding to each multimodal feedback opinion sample output by the sentiment intensity evaluation network; Construct an objective loss function by using the sentiment polarity label and sentiment intensity label, predicted sentiment polarity and predicted sentiment intensity corresponding to each multimodal feedback opinion sample, where the objective loss function includes: a regression loss function, a classification loss function; Adjust the model parameters of the initial learning model according to the objective loss function until the model parameters converge to obtain the trained emotion analysis model.

3. The system according to claim 1, wherein The analysis module at least includes: a feature extraction unit and a first calculation unit. Among them, the feature extraction unit is used to traverse each multimodal feedback opinion under the topic discussion, extract features from the multimodal feedback opinion to obtain a corresponding fused feature vector; the first calculation unit is used to calculate the evaluation quality of each multimodal feedback opinion according to the opinion parameters of the multimodal feedback opinion and the corresponding fused feature vector.

4. The system according to claim 3, characterized in that, The multi-modal feedback opinions at least include: a first feedback opinion in the form of an image, a second feedback opinion in the form of speech, and a third feedback opinion in the form of text. Among them, the feature extraction unit is further configured to extract features from the multi-modal feedback opinions according to the following steps to obtain corresponding fused feature vectors: When the multi-modal feedback opinions include the first feedback opinion, perform a first preprocessing operation on the first feedback opinion, where the first preprocessing operation includes at least one of the following: grayscale processing, text region detection and segmentation, text direction correction; use an optical character recognition model to extract the text data corresponding to the preprocessed first feedback opinion; When the multi-modal feedback opinions include the second feedback opinion, perform a second preprocessing operation on the second feedback opinion, where the second preprocessing operation includes at least one of the following: denoising, loudness adjustment, pre-emphasis; use a dialect recognition model to recognize the preprocessed second feedback opinion to determine the corresponding dialect type, and correct the preprocessed second feedback opinion according to a preset dialect correction rule, where the dialect recognition model is obtained by training a neural network model with multiple low-rank bypasses using a training data set fine-tuned with multiple dialect data, and each low-rank bypass is used to learn one dialect data; use automatic speech recognition technology to recognize the text data corresponding to the corrected second feedback opinion; When the multi-modal feedback opinions include the third feedback opinion, perform a third preprocessing operation on the third feedback opinion, where the third preprocessing operation includes at least one of the following: text cleaning, lemmatization, case unification; Map the text data corresponding to the preprocessed first feedback opinion, the text data corresponding to the corrected second feedback opinion, and the preprocessed third feedback opinion into the same vector space to obtain three text feature vectors, and use an attention mechanism to fuse the three text feature vectors to obtain the fused feature vector.

5. The system according to claim 3, characterized in that, The opinion parameters at least include the interaction frequency. Among them, the first calculation unit is further configured to calculate the evaluation quality of each multi-modal feedback opinion according to the opinion parameters of the multi-modal feedback opinion and the corresponding fused feature vector according to the following steps: Use the N-Gram algorithm to analyze the fused feature vector corresponding to the multi-modal feedback opinion to obtain the novelty of the multi-modal feedback opinion; Determine the interaction degree of the multi-modal feedback opinion according to the matching result between the preset interaction rate quantization rule and the interaction frequency of the multi-modal feedback opinion, where the corresponding relationship between different interaction frequencies and interaction degrees is defined in the interaction rate quantization rule; Calculate the similarity between the fused feature vector corresponding to the multi-modal feedback opinion and multiple target knowledge texts corresponding to the special topic discussion, and use the average value of the multiple similarities as the professionalism of the multi-modal feedback opinion, where the target knowledge text is the knowledge text with the highest number of searches in the knowledge text set corresponding to the special topic discussion; Comprehensively determine the comment quality score of the multimodal feedback opinion based on the novelty, the interactivity, and the professionalism.

6. The system according to claim 5, characterized in that, The first calculation unit is further configured to analyze the fusion feature vector corresponding to the multimodal feedback opinion by using the N-Gram algorithm according to the following steps to obtain the novelty of the multimodal feedback opinion: Divide the fusion feature vector corresponding to the multimodal feedback opinion according to a preset length to obtain a first number of N-Gram vectors, and calculate the similarity between every two N-Gram vectors; Take the N-Gram vector pairs with the similarity higher than a preset second similarity threshold as duplicate N-Gram vectors, and count the second number of duplicate N-Gram vectors in the fusion feature vector corresponding to the multimodal feedback opinion; Take 1 minus the quotient of the second number and the first number as the novelty of the multimodal feedback opinion.

7. The system according to claim 1, wherein The analysis module further includes: a second calculation unit, and the second calculation unit is configured to determine the heat of the special topic discussion according to the evaluation quality score, the sentiment polarity, and the sentiment intensity of each multimodal feedback opinion under the special topic discussion, and according to the following steps: Determine the total number of multimodal feedback opinions published under the special topic discussion; Take the average value of the evaluation quality scores of the multiple multimodal feedback opinions under the special topic discussion as the overall comment quality score of the special topic discussion; Determine the proportions of the special topic discussion under different sentiment polarities and different sentiment intensities according to the sentiment polarity and the sentiment intensity of each multimodal feedback opinion under the special topic discussion, and perform weighted summation on the proportions of the special topic discussion under different sentiment polarities and different sentiment intensities to obtain the overall sentiment score of the special topic discussion; Perform weighted summation on the total number, the overall comment quality score, and the overall sentiment score to obtain the heat of the special topic discussion.

8. The system according to claim 1, wherein The decision module is further configured to supervise the multimodal feedback opinions in multiple clustering clusters in sequence according to the following steps based on the processing urgency: For each clustering cluster, obtain the set of historical multimodal feedback opinion response results of the participants who published the multimodal feedback opinions in the clustering cluster, where the set of historical multimodal feedback opinion response results includes: multiple historical multimodal feedback opinions of the participants in the historical time period and the supervision result and processing time consumption of each historical multimodal feedback opinion; construct a user portrait of the corresponding participant according to the set of historical multimodal feedback opinion response results; determine at least one high-quality multimodal feedback opinion from the clustering cluster according to the user portrait of the participant who published the multimodal feedback opinion and the evaluation quality score of the multimodal feedback opinion, where the high-quality multimodal feedback opinion is a multimodal feedback opinion with a high supervision success rate and short processing time consumption published by a participant in the clustering cluster and with an evaluation quality score higher than a preset evaluation threshold; Supervise each high-quality multimodal feedback opinion in multiple clustering clusters in sequence according to the processing urgency.

Images