Multi-modal fusion question and answer method and system and electronic equipment

Abstract

The invention discloses a multi-modal fusion question and answer method and system and electronic equipment, and relates to the technical field of data processing, and the method comprises the steps: firstly, obtaining text data, video data and voice data inputted by a user at the current time, obtaining a first emotion feature corresponding to the text data, a second emotion feature corresponding to the video data and a third emotion feature corresponding to the voice data; then, carrying out feature fusion on the first emotion feature, the second emotion feature and the third emotion feature by adopting an attention mechanism, and outputting a fused emotion label; and finally, generating a question-answer result based on the fused emotion label and the historical user portrait, verifying the question-answer result step by step in sequence according to multiple levels of preset rules, and outputting the question-answer result when the verification of any level of preset rule is passed. By means of the method, the question and answer result generation accuracy can be improved, non-compliant content does not exist, user requirements can be better met, and therefore the user experience feeling is improved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a multi-modal fusion question and answer method, system, and electronic device. Background Art

[0002] With the development of artificial intelligence technology, intelligent dialogue systems have been widely used in various fields. For example, in customer service conversations, information consultations, etc., intelligent dialogue systems have gradually replaced traditional manual services, greatly reducing the workload of staff.

[0003] In the related art, existing intelligent dialogue systems mainly rely on single text input for semantic understanding and generating dialogue content. However, relying solely on single text input for generating dialogue content lacks the recognition of the user's emotional state and cannot comprehensively capture the user's emotions, resulting in the inability to well understand the user's intention in scenarios where multi-modal information is required to understand the user's intention, and there is a large deviation between the generated content and the user's true needs, leading to low user satisfaction. Moreover, in specific fields (such as the financial field), the content generated by intelligent dialogue systems may contain unvalidated financial terms or illegal commitments, resulting in the generated content violating real-time regulatory terms and not meeting financial regulatory requirements.

[0004] In view of this, how to provide a question and answer method that can accurately understand the user's intention and generate legal and compliant content is a technical problem that urgently needs to be solved at present. Summary of the Invention

[0005] Embodiments of this application provide a multi-modal fusion question and answer method, system, and electronic device to solve the problem that current intelligent dialogue systems cannot accurately identify the user's intention and the generated content may not meet financial regulatory requirements. It can identify the user's emotions from multi-dimensional features and improve the accuracy of identifying the user's true needs, thereby accurately generating answer content and avoiding the appearance of illegal and non-compliant content.

[0006] In a first aspect, this application provides a multi-modal fusion question and answer method, and the method includes:

[0007] Obtain the text data, video data, and voice data input by the user at the current time, and respectively obtain the first emotional feature corresponding to the text data, the second emotional feature corresponding to the video data, and the third emotional feature corresponding to the voice data;

[0008] Use the attention mechanism to perform feature fusion on the first emotional feature, the second emotional feature, and the third emotional feature, and output a fused emotional label;

[0009] Generate question-and-answer results based on the fused emotion tags and historical user profiles, and verify the question-and-answer results step by step according to multiple levels of preset rules. When any level of preset rules is verified successfully, output the question-and-answer results.

[0010] Through the above method, it is possible to accurately determine the user's current emotional state in a multi-modal fusion manner, dynamically adjust the question-and-answer result generation strategy based on the user's emotional state, and verify the question-and-answer results after generating them to ensure that the generated content is legal and compliant and can meet the user's current needs, thereby improving the user experience and satisfaction.

[0011] In an alternative embodiment, the attention mechanism is used to perform feature fusion on the first emotional feature, the second emotional feature, and the third emotional feature, and the output fused emotion tag includes:

[0012] Respectively determine the weight parameters corresponding to the first emotional feature, the second emotional feature, and the third emotional feature;

[0013] Based on the weight parameters corresponding to the first emotional feature, the second emotional feature, and the third emotional feature, perform feature fusion on the first emotional feature, the second emotional feature, and the third emotional feature to obtain a fused emotional feature;

[0014] Input the fused emotional feature into a preset classifier to obtain a fused emotion tag.

[0015] Through the above method, the attention mechanism is used to fuse the features of multiple modalities, and the user's emotional state can be determined in real time, thereby generating the user's emotion tag.

[0016] In an alternative embodiment, verifying the question-and-answer results step by step according to multiple levels of preset rules includes:

[0017] In response to executing the first preset rule to verify the question-and-answer results, extract keywords from the question-and-answer results and match the keywords with the preset keywords in the preset keyword library. If the matching confidence is less than the preset threshold, the verification passes and the question-and-answer results are output. If the verification fails, execute the second preset rule;

[0018] In response to executing the second preset rule to verify the question-and-answer results, use dependency syntactic analysis to extract the main sentence of the question-and-answer results and match it with the preset violation sentences. If the matching confidence is less than the preset threshold, the verification passes and the question-and-answer results are output. If the verification fails, execute the third preset rule;

[0019] In response to executing the third preset rule to verify the question-and-answer results, determine the user's intention in combination with the context content. If the user's intention matches the question-and-answer results and there are no violation words, the verification passes and the question-and-answer results are output.

[0020] Through the above method, by adopting a multi-level verification method, it can ensure that the generated Q&A content does not contain non-compliant content and can meet regulatory requirements.

[0021] In an optional implementation manner, if the Q&A result verification fails, it further includes:

[0022] If there are violation words in the Q&A result, match the compliant words corresponding to the violation words in the preset replacement word library; wherein, the preset replacement word library stores compliant words corresponding to each violation word one by one;

[0023] Based on the compliant words, correct the Q&A result to generate a compliant Q&A result.

[0024] Through the above method, when the Q&A result verification fails, the Q&A result can be dynamically corrected to ensure that there are no violation words in the Q&A result.

[0025] In an optional implementation manner, generating a Q&A result based on the fused sentiment tags and the historical user portrait includes:

[0026] Based on the fused sentiment tags and the historical user portrait, determine the risk level corresponding to the user;

[0027] Based on the risk level and the context content of the current conversation, determine the conversation strategy, and generate a Q&A result according to the conversation strategy.

[0028] In an optional implementation manner, after correcting the Q&A result based on the compliant words to generate a compliant Q&A result, it further includes:

[0029] Generate an error report according to the Q&A result that fails verification and the adjusted compliant Q&A result; wherein, the error report at least includes the Q&A result that fails verification and the corresponding correction method, and the error report is used to update the multi-level preset rules.

[0030] Through the above method, the existing preset rules and generation methods can be adjusted to ensure that the subsequent generated content can better meet the user's needs and comply with the compliance requirements.

[0031] In a second aspect, the present application provides a multi-modal fusion Q&A system, and the system includes:

[0032] A processing module, configured to obtain the text data, video data, and voice data input by the user at the current time, and respectively obtain the first sentiment feature corresponding to the text data, the second sentiment feature corresponding to the video data, and the third sentiment feature corresponding to the voice data;

[0033] A fusion module, which is used to fuse the first emotional feature, the second emotional feature, and the third emotional feature by using an attention mechanism and output a fused emotional label;

[0034] A generation module, which is used to generate a Q&A result based on the fused emotional label and the historical user profile, and verify the Q&A result step by step according to multiple preset rules. When any level of preset rules is verified successfully, the Q&A result is output.

[0035] In an alternative embodiment, when fusing the first emotional feature, the second emotional feature, and the third emotional feature by using an attention mechanism and outputting a fused emotional label, the fusion module is specifically configured to:

[0036] Determine the respective weight parameters corresponding to the first emotional feature, the second emotional feature, and the third emotional feature;

[0037] Based on the respective weight parameters corresponding to the first emotional feature, the second emotional feature, and the third emotional feature, perform feature fusion on the first emotional feature, the second emotional feature, and the third emotional feature to obtain a fused emotional feature;

[0038] Input the fused emotional feature into a preset classifier to obtain a fused emotional label.

[0039] In an alternative embodiment, when verifying the Q&A result step by step according to multiple preset rules, the generation module is specifically configured to:

[0040] In response to verifying the Q&A result by executing the first preset rule, extract keywords from the Q&A result and match the keywords with the preset keywords in the preset keyword library. If the matching confidence is less than the preset threshold, the verification is passed and the Q&A result is output. If the verification fails, the second preset rule is executed;

[0041] In response to verifying the Q&A result by executing the second preset rule, use dependency syntactic analysis to extract the main sentence of the Q&A result and match it with the preset violation sentences. If the matching confidence is less than the preset threshold, the verification is passed and the Q&A result is output. If the verification fails, the third preset rule is executed;

[0042] In response to verifying the Q&A result by executing the third preset rule, determine the user intention in combination with the context content. If the user intention matches the Q&A result and there are no violation words, the verification is passed and the Q&A result is output.

[0043] In an alternative embodiment, when the verification of the Q&A result fails, the generation module is further configured to:

[0044] If there are violation words in the Q&A result, match the compliant words corresponding to the violation words in the preset replacement word library; wherein, the preset replacement word library stores the compliant words corresponding to each violation word one by one;

[0045] Correct the Q&A result based on the compliant words to generate a compliant Q&A result.

[0046] In an alternative embodiment, when generating the Q&A result based on the fused sentiment label and the historical user profile, the generating module is specifically configured to:

[0047] Determine the risk level corresponding to the user based on the fused sentiment label and the historical user profile;

[0048] Determine the conversation strategy based on the risk level and the context content of the current conversation, and generate the Q&A result according to the conversation strategy.

[0049] In an alternative embodiment, after correcting the Q&A result based on the compliant words to generate a compliant Q&A result, the generating module is further configured to:

[0050] Generate an error report based on the Q&A result that fails verification and the adjusted compliant Q&A result; wherein, the error report at least includes the Q&A result that fails verification and the corresponding correction method, and the error report is used to update the multi-level preset rules.

[0051] In a third aspect, the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor is caused to execute the steps of the multi-modal fusion Q&A method described in the first aspect above.

[0052] In a fourth aspect, the present application provides a computer-readable storage medium, which includes program code, and when the program code runs on an electronic device, the program code is used to cause the electronic device to execute the steps of the multi-modal fusion Q&A method described in the first aspect above.

[0053] In a fifth aspect, the present application provides a computer program product, and when the computer program product is called by a computer, the computer is caused to execute the steps of the multi-modal fusion Q&A method as described in the first aspect.

[0054] In addition, other features and advantages of the present application will be described in the subsequent specification, and part of them will become obvious from the specification, or will be understood by implementing the present application. The objectives and other advantages of the present application can be achieved and obtained through the structures specifically pointed out in the written specification, claims, and drawings. Description of the Drawings

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings. In the drawings:

[0056] Figure 1 It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0057] Figure 2 It is a schematic diagram of the implementation process of a multi-modal fusion question-answering method provided by an embodiment of the present application;

[0058] Figure 3 It is a schematic diagram of the structure of a multi-modal fusion question-answering system provided by an embodiment of the present application;

[0059] Figure 4 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions of the present application in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some, rather than all, of the embodiments of the technical solutions of the present application. Based on the embodiments recorded in this application document, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope protected by the technical solutions of the present application.

[0061] It should be noted that in the description of the present application, "a plurality of" is understood as "at least two". "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The connection between A and B can represent: A is directly connected to B and A is connected to B through C. In addition, in the description of the present application, terms such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0062] In addition, in the technical solutions of the present application, the collection, dissemination, use, etc. of data all comply with the requirements of relevant national laws and regulations.

[0063] The following briefly introduces the design concept of the embodiments of the present application:

[0064] With the development of artificial intelligence technology, intelligent dialogue systems have been widely used in various fields. For example, in customer service conversations, information consultations, etc., intelligent dialogue systems have gradually replaced traditional manual services, greatly reducing the workload of staff.

[0065] In related technologies, existing intelligent dialogue systems mainly rely on single text input for semantic understanding and generating dialogue content. However, relying solely on single text input to generate dialogue content lacks the recognition of the user's emotional state and cannot comprehensively capture the user's emotions. As a result, in scenarios where it is necessary to understand the user's intention based on multimodal information, it is unable to well understand the user's intention, and there is a large deviation between the generated content and the user's real needs, resulting in low user satisfaction. Moreover, in specific fields (such as the financial field), the content generated by the intelligent dialogue system may contain unverified financial terms or illegal commitments, resulting in the generated content violating real-time regulatory terms and not meeting financial regulatory requirements.

[0066] In view of this, the present application provides a multimodal fusion question-and-answer method, which includes: First, obtain the text data, video data, and voice data input by the user at the current time, and respectively obtain the first emotional feature corresponding to the text data, the second emotional feature corresponding to the video data, and the third emotional feature corresponding to the voice data; Then, use the attention mechanism to perform feature fusion on the first emotional feature, the second emotional feature, and the third emotional feature, and output a fused emotional label; Finally, generate a question-and-answer result based on the fused emotional label and the historical user profile, and verify the question-and-answer result step by step according to multiple levels of preset rules. When any level of preset rules is verified to pass, output the question-and-answer result. Through the above method, it is possible to combine text data, video data, and voice data to comprehensively obtain the user's current emotional state from multiple modal data, so as to be able to adaptively adjust the current dialogue strategy based on the user's emotional state, generate dialogue content that meets the user's needs, and can also verify the generated content to avoid the appearance of non-compliant content.

[0067] The following describes the multimodal fusion question-and-answer method provided by the exemplary embodiment of the present application with reference to the accompanying drawings.

[0068] Refer to Figure 1 As shown, it is a schematic diagram of a possible application scenario in an embodiment of the present application. In this application scenario diagram, it includes a server 11 and terminal devices 12 (including terminal devices 121, 122... 12n).

[0069] The server 11 can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms. The terminal device 12 and the server 11 can be directly or indirectly connected through wired or wireless communication methods, and this application does not limit this here.

[0070] The terminal device 12 includes, but is not limited to, devices such as mobile phones, tablet computers, laptop computers, desktop computers, e-readers, intelligent voice interaction devices, smart home appliances, in-vehicle terminals, etc.; various software can be installed on the terminal device, such as application programs and applets, etc.

[0071] It should be noted that Figure 1 The above is only an example. In fact, the number of the terminal device 12 and the server 11 is not limited, and no specific limitation is made in the embodiments of this application.

[0072] Exemplarily, the terminal device 12 can access the network through cellular mobile communication technology, so as to communicate with the server 11, and then send the text data to be recognized to the server. Among them, the flavor mobile communication technology, for example, includes the fifth-generation mobile communication (5th Generation Mobile Networks, 5G) technology.

[0073] Optionally, the terminal 12 can access the network through short-range wireless communication means, so as to communicate with the server 11, and then send the text data, video data, and voice data input by the user to the server. After receiving the text data, video data, and voice data input by the user at the current time, the server can process them, generate the final question-and-answer result, and feedback it to the terminal device. Among them, the short-range wireless communication means, for example, includes Wireless Fidelity (Wi-Fi) technology.

[0074] Refer to Figure 2 As shown, it is a schematic diagram of the implementation process of a multi-modal fusion question-and-answer method provided by the embodiments of this application. The specific implementation process of this method is as follows:

[0075] S1: Obtain the text data, video data, and voice data input by the user at the current time, and respectively obtain the first emotional feature corresponding to the text data, the second emotional feature corresponding to the video data, and the third emotional feature corresponding to the voice data.

[0076] For an intelligent question-and-answer system, when a user needs to consult or ask questions, they can input text data through the intelligent chat interface to achieve a conversation with the intelligent question-and-answer system. However, the user's current emotional state cannot be well recognized only through text data.

[0077] Therefore, in the embodiments of the present application, first, a multi-modal fusion method is adopted to recognize the user's emotional state, and then corresponding question-and-answer results are generated according to the user's needs, so as to accurately match the user's needs according to the user's emotional state and improve the user's satisfaction.

[0078] Specifically, the intelligent question-and-answer system can be provided with an intelligent chat interface. When the user needs to have a conversation with the intelligent question-and-answer system at the current time, they can input text data through the dialog box in the intelligent chat interface. Then, the embodiments of the present application can also collect the video data of the user when using the intelligent dialogue system through the camera set on the terminal device, and collect the voice data of the user by using the microphone.

[0079] For example, the user inputs text data "Please recommend a financial product for me" through the dialog box. At the same time, the camera collects the video data corresponding to the user's current facial expression, and the microphone collects the voice data "Please recommend a financial product for me" input by the user.

[0080] However, the user's current emotional state cannot be well recognized only from the text data. However, the video data corresponding to the user's facial expression collected by the camera and the voice data collected by the microphone can both reflect the user's current emotional state to a certain extent.

[0081] Furthermore, the obtained text data, video data, and voice data can be preprocessed respectively. Specifically, the obtained text data can be subjected to word segmentation and semantic parsing processing, and duplicate words and / or meaningless words can be removed. For the obtained video data, key frames that can represent the user's current facial expression can be extracted, and preprocessing such as image sharpening, contrast enhancement, grayscale conversion, and filtering can be performed on the key frames. The key frames can also be converted into images with a resolution of 224×224. For the obtained voice data, preprocessing such as noise reduction and segmentation can be performed. Preprocessing the obtained text data, video data, and voice data is beneficial for subsequent feature extraction.

[0082] Furthermore, feature extraction can be performed separately on the preprocessed text data, video data, and speech data. In the embodiments of the present application, a Bidirectional Encoder Representations from Transformers (BERT) model can be used to extract the first sentiment feature corresponding to the text data; for the video data, the key frames extracted from the video data can be input into a Residual Network 50 layers (ResNet50) network for feature extraction to obtain the second sentiment feature corresponding to the video data; then, the Mel-scale Frequency Cepstral Coefficients (MFCC) corresponding to the speech data can be extracted. MFCC can characterize the spectral shape of the speech data and assist in capturing the timbre changes. Then, the MFCC is input into a Bidirectional Long Short Term Memory (Bi-LSTM) network to extract the third sentiment feature corresponding to the speech data. The forward hidden state corresponding to the MFCC can be obtained through the forward long short term memory network of the Bi-LSTM, and the backward hidden state corresponding to the MFCC can be obtained through the candidate long short term memory network of the Bi-LSTM. Then, the obtained forward hidden state and backward hidden state are concatenated to obtain the third sentiment feature in the embodiments of the present application. The current emotional changes of the user can be judged through the intonation fluctuations.

[0083] S2: Use the attention mechanism to perform feature fusion on the first sentiment feature, the second sentiment feature, and the third sentiment feature, and output the fused sentiment label.

[0084] In the embodiments of the present application, the method of multi-modal feature fusion can be used to more accurately determine the current emotional state of the user.

[0085] Specifically, when performing feature fusion processing on the first sentiment feature, the second sentiment feature, and the third sentiment feature, it is also necessary to map the first sentiment feature, the second sentiment feature, and the third sentiment feature to the same feature dimension through a fully connected layer to eliminate the dimensional difference between different features.

[0086] In an alternative embodiment, it is also necessary to determine the weight parameters corresponding to the first sentiment feature, the second sentiment feature, and the third sentiment feature respectively. Specifically, the weight parameters of the first sentiment feature, the second sentiment feature, and the third sentiment feature are calculated through learnable parameters. When calculating the weight parameters, the following formula can be used for calculation:

[0087]

[0088] Among them, W a represents learnable parameters, represents a concatenation operation, V speech represents the weight parameter corresponding to the first emotional feature, V uisual represents the weight parameter corresponding to the second emotional feature, V text represents the weight parameter corresponding to the third emotional feature.

[0089] After obtaining the weight parameters corresponding to each emotional feature respectively, feature fusion can be performed. In the embodiments of the present application, weighted fusion can be used to perform feature fusion on each emotional feature, so as to obtain a fused emotional feature. Specifically, the following formula can be used for weighted fusion:

[0090] V joint = a text V text + a visual V uisual + a speech V speech ;

[0091] Among them, V joint represents the fused emotional feature, a text represents the first emotional feature, V text represents the weight parameter corresponding to the first emotional feature, a visual represents the second emotional feature, V uisual represents the weight parameter corresponding to the second emotional feature, a speech represents the third emotional feature, V speech represents the weight parameter corresponding to the third emotional feature.

[0092] Through the above method, the attention mechanism can be used to perform feature fusion on each emotional feature, so as to obtain a fused emotional feature. Using the fused emotional feature can comprehensively consider the user's current emotional state from multiple dimensions.

[0093] Then, input the fused emotional feature into a preset classifier, and the user's current fused emotional label and the corresponding confidence level can be obtained. For example, if anxiety corresponds to a confidence level of 0.8, optimism corresponds to a confidence level of 0.4, and sadness corresponds to a confidence level of 0.6, then the confidence level corresponding to anxiety is relatively high, indicating that the user is currently in an anxious state.

[0094] S3: Generate a question-and-answer result based on the fused emotional label and the historical user portrait, and verify the question-and-answer result step by step according to multiple levels of preset rules. When any level of preset rules is verified to pass, output the question-and-answer result.

[0095] In an embodiment of the present application, after obtaining the fused emotion label, the Q&A result can be generated in combination with the historical user profile. The historical user profile can be obtained based on the user's historical data, such as the user's historical transaction data, asset scale, real-time behavior data, etc. Then, it is encoded using the one-hot encoding method to generate a feature vector, and then the fused emotion label and the historical user profile can be input into the Q&A generation module. Based on the fused emotion label and the historical user profile, the risk level corresponding to the user is determined. For example, if the fused emotion label of user A is anxiety and the historical user profile indicates that the user has a poor risk tolerance, then it is determined that the user's risk level is high. Then, based on the risk level and the context content of the current conversation, the conversation strategy is determined, and the Q&A result is generated according to the conversation strategy. For example, based on the context content, it can be known that the user's demand is "Please recommend a financial product for me", but now it is determined that the user has a high risk level and is anxious, then it can be determined that the conversation strategy is to optimize the affinity of the words and recommend financial products with lower risks. Based on the above conversation strategy, the corresponding Q&A result can be generated according to the user's demand.

[0096] In an embodiment of the present application, the generated Q&A result may contain content that does not meet the regulatory conditions, so it is necessary to verify the generated Q&A result, and the Q&A result is output when any level of preset rules is verified.

[0097] In an embodiment of the present application, multi-level preset rules are used to verify the Q&A result step by step in sequence. For example, the first preset rule, the second preset rule, and the third preset rule are set, and the Q&A result is verified by the first preset rule, the second preset rule, and the third preset rule in sequence. During the execution process, the Q&A result can be output when any level of preset rules is verified, and if it is not verified, the next level of preset rules will continue to verify. Specifically, first, in response to verifying the Q&A result by executing the first preset rule, the first preset rule can be verified based on keywords. Then, the keywords can be extracted from the Q&A result and matched with the preset keywords in the preset keyword library. The preset keyword library stores non-compliant keywords collected based on historical data. For example, the generated Q&A result contains "risk-free", and the preset keyword matched based on the preset keyword library is "no loss", and the confidence level between the two is 90%, which is greater than the preset threshold of 90%. Then, the word "risk-free" obviously does not meet the requirements. Another example is that the generated Q&A result contains "low risk", and the preset keyword matched based on the preset keyword library is "risk-free", and the confidence level between the two is 80%, which is less than the preset threshold of 90%. Then, "low risk" belongs to the acceptable content, and the verification is passed and the Q&A result is output.

[0098] When the first preset rule fails to pass the verification, the second preset rule is executed to verify the Q&A result. Dependency syntactic analysis is used to extract the main sentence of the Q&A result, which can clarify the core meaning of the Q&A result. Then, the extracted main sentence is matched with the preset violation sentences. If the matching confidence is less than the preset threshold, the verification passes and the Q&A result is output. If the verification fails, the third preset rule is continued to be executed.

[0099] When the third preset rule is executed, the user's intention can be determined in combination with the context of the conversation. For example, if the user asks about the interest rate multiple times, it means that the user is only interested in the specific value of the interest rate. And if the current Q&A result also gives the current interest rate situation and there are no violation words, then the third preset rule passes the verification and the Q&A result can be output. If the user mentions words like "high yield" multiple times, it means that the user has a high-risk demand currently. If the Q&A result mentions that Product A can obtain high yields, but "high yield" is a violation word, then the third preset rule cannot pass the verification of this Q&A result.

[0100] In an alternative implementation, assuming that the verification of the Q&A result using multiple-level preset rules all fails, the Q&A result can also be corrected to ensure that the generated Q&A result is compliant.

[0101] Specifically, if any level of the preset rule fails the verification, the violation words existing in the generated Q&A result are determined, and then the compliant words corresponding to the violation words are matched in the preset replacement word library. It should be noted that the preset replacement word library can save the compliant words corresponding to each violation word according to historical data. For example, "principal protection" is a violation word, and its corresponding compliant word is "net value type". Then, after obtaining the compliant words, the Q&A result that fails the verification can be corrected to generate a compliant Q&A result. For example, if the user inputs a description related to "principal protection wealth management", the Q&A result can be corrected to "net value type products".

[0102] In an alternative implementation, after outputting the final Q&A result to the user, an error report can also be generated based on the Q&A result that fails the verification and the adjusted compliant Q&A result.

[0103] The error report at least includes the Q&A result that fails the verification and the corresponding correction method. Based on the error report, the multiple-level preset rules can be updated so that the subsequent generated Q&A results can meet the requirements.

[0104] Furthermore, based on the same technical concept, the embodiments of the present application provide a multi-modal fusion Q&A system, and the multi-modal fusion Q&A system is used to implement the above method flow of the embodiments of the present application. Refer to Figure 3As shown in the figure, the system includes: a processing module 301, a fusion module 302, and a generation module 303, where,

[0105] The processing module 301 is configured to obtain text data, video data, and voice data input by the user at the current time, and respectively obtain a first emotional feature corresponding to the text data, a second emotional feature corresponding to the video data, and a third emotional feature corresponding to the voice data;

[0106] The fusion module 302 is configured to fuse the first emotional feature, the second emotional feature, and the third emotional feature by using an attention mechanism, and output a fused emotional label;

[0107] The generation module 303 is configured to generate a question-and-answer result based on the fused emotional label and the historical user profile, and sequentially verify the question-and-answer result according to multiple levels of preset rules, and output the question-and-answer result when any level of preset rules is verified successfully.

[0108] In an optional implementation manner, when fusing the first emotional feature, the second emotional feature, and the third emotional feature by using an attention mechanism and outputting a fused emotional label, the fusion module 302 is specifically configured to:

[0109] respectively determine weight parameters corresponding to the first emotional feature, the second emotional feature, and the third emotional feature;

[0110] Based on the weight parameters corresponding to the first emotional feature, the second emotional feature, and the third emotional feature, perform feature fusion on the first emotional feature, the second emotional feature, and the third emotional feature to obtain a fused emotional feature;

[0111] Input the fused emotional feature into a preset classifier to obtain a fused emotional label.

[0112] In an optional implementation manner, when sequentially verifying the question-and-answer result according to multiple levels of preset rules, the generation module 303 is specifically configured to:

[0113] In response to verifying the question-and-answer result by executing the first preset rule, extract keywords from the question-and-answer result, and match the keywords with preset keywords in a preset keyword library. If the matching confidence is less than a preset threshold, the verification is passed and the question-and-answer result is output. If the verification fails, the second preset rule is executed;

[0114] In response to verifying the question-and-answer result by executing the second preset rule, extract the sentence backbone from the question-and-answer result by using dependency syntax analysis, and match it with a preset violation sentence. If the matching confidence is less than a preset threshold, the verification is passed and the question-and-answer result is output. If the verification fails, the third preset rule is executed;

[0115] Verify the Q&A result in response to the execution of the third preset rule, then determine the user's intention in combination with the context content. If the user's intention matches the Q&A result and there are no violation words, the verification passes and the Q&A result is output.

[0116] In an alternative embodiment, when the Q&A result verification fails, the generation module 303 is further configured to:

[0117] If there are violation words in the Q&A result, match the compliance word corresponding to the violation word in the preset replacement word library; wherein, the preset replacement word library stores the compliance words corresponding to each violation word one by one;

[0118] Correct the Q&A result based on the compliance word to generate a compliant Q&A result.

[0119] In an alternative embodiment, when generating the Q&A result based on the fused sentiment label and the historical user profile, the generation module 303 is specifically configured to:

[0120] Determine the risk level corresponding to the user based on the fused sentiment label and the historical user profile;

[0121] Determine the conversation strategy based on the risk level and the context content of the current conversation, and generate the Q&A result according to the conversation strategy.

[0122] In an alternative embodiment, after correcting the Q&A result based on the compliance word to generate a compliant Q&A result, the generation module 303 is further configured to:

[0123] Generate an error report based on the Q&A result that fails the verification and the adjusted compliant Q&A result; wherein, the error report includes at least the Q&A result that fails the verification and the corresponding correction method, and the error report is used to update the multi-level preset rules.

[0124] Based on the same technical concept, an embodiment of the present application further provides an electronic device, which can implement the multi-modal fusion Q&A method flow provided in the above embodiments of the present application. In one embodiment, the electronic device can be a server, or a terminal device or other electronic devices. Refer to Figure 4 As shown, the electronic device may include:

[0125] At least one processor 401, and a memory 402 connected to at least one processor 401. In the embodiments of the present application, the specific connection medium between the processor 401 and the memory 402 is not limited. Figure 4 It is taken as an example that the processor 401 and the memory 402 are connected through a bus 400. The bus 400 is in Figure 4The connection between the center is represented by a thick line, and the connection methods between other components are only for illustrative purposes and are not limited thereto. The bus 400 can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 4 it is only represented by a thick line in the figure, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 401 can also be called a controller, and there is no restriction on the name.

[0126] In the embodiment of the present application, the memory 402 stores instructions that can be executed by at least one processor 401. By executing the instructions stored in the memory 402, at least one processor 401 can execute a multi-modal fusion question-and-answer method described above. The processor 401 can implement Figure 3 the functions of each module in the system shown.

[0127] Among them, the processor 401 is the control center of the device, and can connect various parts of the entire control device through various interfaces and lines. By running or executing the instructions stored in the memory 402 and calling the data stored in the memory 42, various functions of the device and process data, so as to monitor the device as a whole.

[0128] In a possible design, the processor 401 may include one or more processing units. The processor 401 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor may not be integrated into the processor 401. In some embodiments, the processor 401 and the memory 402 can be implemented on the same chip, and in some embodiments, they can also be implemented on separate chips respectively.

[0129] The processor 401 can be a general-purpose processor, such as a CPU, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of a multi-modal fusion question-and-answer method disclosed in combination with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0130] The memory 402, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The memory 402 can include at least one type of storage medium. For example, it can include flash memory, hard disks, multimedia cards, card-type memories, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memories, magnetic disks, optical disks, and so on. The memory 402 is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 402 in the embodiments of the present application can also be a circuit or any other device capable of implementing a storage function, for storing program instructions and / or data.

[0131] By designing and programming the processor 401, the code corresponding to the multi-modal fusion question-and-answer method introduced in the foregoing embodiments can be solidified into the chip, so that the chip can execute Figure 1 the steps of the multi-modal fusion question-and-answer method of the embodiment shown. How to design and program the processor 401 is a well-known technology to those skilled in the art and will not be elaborated here.

[0132] Based on the same inventive concept, the embodiments of the present application also provide a storage medium that stores computer instructions. When the computer instructions run on a computer, the computer is caused to execute a multi-modal fusion question-and-answer method discussed above.

[0133] In some possible implementation manners, the present application also provides that various aspects of a multi-modal fusion question-and-answer method can also be implemented in the form of a program product, which includes program code. When the program product runs on a device, the program code is used to cause the control device to execute the steps in a multi-modal fusion question-and-answer method according to various exemplary embodiments of the present application described above in this specification.

[0134] It should be noted that although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more of the above-described units may be embodied in one unit. Conversely, the features and functions of one unit described above may be further divided and embodied by multiple units.

[0135] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the shown operations must be performed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution.

[0136] Those skilled in the art should understand that the embodiments of the present application may be provided as a method, a system, or a computer program product. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0137] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program instructions. These computer program instructions can be provided to the processors of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a server, such that the instructions executed by the processors of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one or more of the flows or multiple flows and / or blocks Figure 1 one or more of the blocks or multiple blocks.

[0138] The program code for performing the operations of the present application can be written using any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, executed as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0139] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one process or a plurality of processes and / or blocks Figure 1 one process or a plurality of processes and / or blocks Figure 1 steps for implementing the functions specified in one block or a plurality of blocks.

[0140] Obviously, those skilled in the art can make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these modifications and variations.

Claims

1. A multi-modal fusion question-answering method, characterized in that, The method includes: Obtaining the text data, video data, and voice data input by the user at the current time, and respectively obtaining the first emotional feature corresponding to the text data, the second emotional feature corresponding to the video data, and the third emotional feature corresponding to the voice data; Using an attention mechanism to perform feature fusion on the first emotional feature, the second emotional feature, and the third emotional feature, and outputting a fused emotional label; Generating a Q&A result based on the fused emotional label and the historical user profile, and sequentially verifying the Q&A result step by step according to multiple preset rules. When any level of preset rule is verified to pass, the Q&A result is output.

2. The method according to claim 1, wherein The using an attention mechanism to perform feature fusion on the first emotional feature, the second emotional feature, and the third emotional feature, and outputting a fused emotional label includes: Respectively determining the weight parameters corresponding to the first emotional feature, the second emotional feature, and the third emotional feature; Based on the weight parameters corresponding to the first emotional feature, the second emotional feature, and the third emotional feature, performing feature fusion on the first emotional feature, the second emotional feature, and the third emotional feature to obtain a fused emotional feature; Inputting the fused emotional feature into a preset classifier to obtain the fused emotional label.

3. The method according to claim 1, characterized in that, The sequentially verifying the Q&A result step by step according to multiple preset rules includes: In response to executing the first preset rule to verify the Q&A result, extracting keywords from the Q&A result, and matching the keywords with the preset keywords in the preset keyword library. If the matching confidence is less than the preset threshold, the verification passes and the Q&A result is output. If the verification fails, the second preset rule is executed; In response to executing the second preset rule to verify the Q&A result, using dependency syntactic analysis to extract the sentence backbone in the Q&A result, and matching it with the preset violation sentences. If the matching confidence is less than the preset threshold, the verification passes and the Q&A result is output. If the verification fails, the third preset rule is executed; In response to executing the third preset rule to verify the Q&A result, determining the user intention in combination with the context content. If the user intention matches the Q&A result and there are no violation words, the verification passes and the Q&A result is output.

4. The method according to claim 1 or 3, characterized in that If the Q&A result fails to pass the verification, it further includes: If there are violation words in the Q&A result, matching a compliance word corresponding to the violation word in the preset replacement word library; wherein, the preset replacement word library stores compliance words corresponding to each violation word one by one; Based on the compliance word, correcting the Q&A result to generate a compliant Q&A result.

5. The method according to claim 1, wherein The generating a Q&A result based on the fused emotional label and the historical user profile includes: Based on the fused emotional label and the historical user profile, determining the risk level corresponding to the user; Based on the risk level and the context content of the current conversation, determining a conversation strategy, and generating the Q&A result according to the conversation strategy.

6. The method according to claim 4, wherein After the correcting the Q&A result based on the compliance word to generate a compliant Q&A result, it further includes: Generate an error report based on the unpassed Q&A results of the verification and the adjusted compliant Q&A results; wherein, the error report at least includes the unpassed Q&A results of the verification and the corresponding correction methods, and the error report is used to update the multi-level preset rules.

7. A multi-modal fusion question answering system, characterized in that, The system includes: A processing module, configured to obtain text data, video data, and voice data input by a user at the current time, and respectively obtain a first emotional feature corresponding to the text data, a second emotional feature corresponding to the video data, and a third emotional feature corresponding to the voice data; A fusion module, configured to fuse the first emotional feature, the second emotional feature, and the third emotional feature by using an attention mechanism, and output a fused emotional label; A generation module, configured to generate Q&A results based on the fused emotional label and a historical user profile, and sequentially verify the Q&A results level by level according to multi-level preset rules, and output the Q&A results when any level of preset rules passes the verification.

8. The system according to claim 7, wherein When fusing the first emotional feature, the second emotional feature, and the third emotional feature by using the attention mechanism and outputting a fused emotional label, the fusion module is specifically configured to: Determine the respective weight parameters corresponding to the first emotional feature, the second emotional feature, and the third emotional feature; Based on the respective weight parameters corresponding to the first emotional feature, the second emotional feature, and the third emotional feature, perform feature fusion on the first emotional feature, the second emotional feature, and the third emotional feature to obtain a fused emotional feature; Input the fused emotional feature into a preset classifier to obtain the fused emotional label.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, the method described in any one of claims 1-6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the method described in any one of claims 1-6 is implemented.

Images