An interaction method, device and electronic equipment

Abstract

The application provides an interaction method and device and electronic equipment, and relates to the technical field of artificial intelligence, wherein the interaction method comprises: receiving input first information, the first information comprising at least one of a picture, a video or voice; determining information content contained in the first information, analyzing the information content contained in the first information to obtain fusion features; using a natural language processing algorithm to identify and process the fusion features to obtain an answer corresponding to the first information, and outputting the answer. The embodiment of the application can improve the interaction experience.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an interaction method, device, and electronic device. Background Technology

[0002] With the continuous development of artificial intelligence technology, people's demand for more convenient, faster, and more efficient intelligent interaction methods is becoming stronger, and their requirements for interactive experience are getting higher and higher. For example, people can input text into a customer service robot, which will parse the input text and reply, thus realizing interaction between people and customer service robots.

[0003] However, existing intelligent interaction methods are basically limited to text-based interaction, resulting in a relatively simple interaction method and a poor user experience. Summary of the Invention

[0004] This invention provides an interaction method, device, and electronic device to solve the problem that existing intelligent interaction methods are basically limited to text-based interaction, resulting in a relatively simple interaction method and poor interaction experience.

[0005] To solve the above-mentioned technical problems, the present invention is implemented as follows:

[0006] In a first aspect, embodiments of the present invention provide an interaction method, the method comprising:

[0007] Receive input first information, the first information including at least one of an image, video, or voice;

[0008] Determine the information content contained in the first information, parse the information content contained in the first information, and obtain the fusion feature;

[0009] The fused features are processed using a natural language processing algorithm to obtain the answer corresponding to the first information, and the answer is output.

[0010] Optionally, parsing the information content contained in the first information to obtain the fusion features includes:

[0011] If the information content of the first information includes images and text, the images are parsed to obtain image category and image description information, and the fusion feature includes the image category, the image description information and the text;

[0012] If the information content included in the first information includes video and text, the video is parsed to obtain video action description information, and the fusion feature includes the video action description information and the text.

[0013] If the information content included in the first information includes speech, speech recognition is performed on the speech to obtain recognized text, and the fusion feature includes the recognized text.

[0014] Optionally, after performing speech recognition on the speech, the method further includes:

[0015] Extract voiceprint features from the speech, and determine whether the user corresponding to the speech is a preset user based on the voiceprint features;

[0016] If the user corresponding to the voice is the preset user, then the fusion feature includes the recognized text and the user information corresponding to the preset user.

[0017] Optionally, before performing recognition processing on the fused features using a natural language processing algorithm, the method further includes:

[0018] Search the historical issue log database to see if the preset user-associated historical issue exists;

[0019] The process of using natural language processing algorithms to identify the fused features includes:

[0020] If the historical issues associated with the preset user exist in the historical issue log database, then the fusion features are identified and processed using a natural language processing algorithm based on the historical issues associated with the preset user.

[0021] Optionally, before performing recognition processing on the fused features using a natural language processing algorithm, the method further includes:

[0022] Based on the first information, determine the target domain corresponding to the first information;

[0023] The fused features are processed using natural language processing algorithms to obtain the answer corresponding to the first information, including:

[0024] The fused features are identified and processed using a natural language processing algorithm corresponding to the target domain to obtain the answer corresponding to the first information.

[0025] Optionally, the target domain is a domain among multiple domains, and the first domain among the multiple domains maintains a knowledge graph;

[0026] The step of using a natural language processing algorithm corresponding to the target domain to identify and process the fused features to obtain the answer corresponding to the first information includes:

[0027] If the target domain belongs to the first domain, the fusion features are identified and processed using a knowledge graph question answering algorithm based on a first language model according to the knowledge graph corresponding to the target domain.

[0028] If the knowledge graph question-answering algorithm identifies the first answer, then the first answer is determined as the answer corresponding to the first information;

[0029] If the first answer is not identified using the knowledge graph question answering algorithm, then the fusion features are identified and processed using an intent recognition algorithm and a second language model based on the knowledge base corresponding to the target domain to obtain the answer corresponding to the first information.

[0030] If the target domain does not belong to the first domain, the fusion features are identified and processed using an intent recognition algorithm and a second language model based on the knowledge base corresponding to the target domain to obtain the answer corresponding to the first information.

[0031] Optionally, the step of using an intent recognition algorithm and a second language model to identify and process the fused features based on the knowledge base corresponding to the target domain to obtain the answer corresponding to the first information includes:

[0032] The intent of the fusion features is identified using an intent recognition algorithm based on the knowledge base corresponding to the target domain.

[0033] If the intent recognition algorithm identifies the first question, then the second language model is used to rank the first question by relevance, and the answer corresponding to the first information is determined based on the ranking result.

[0034] If the intent recognition algorithm fails to identify the first question, a natural language processing algorithm for multi-turn dialogue is used to identify the fused features and obtain the answer corresponding to the first information.

[0035] Optionally, the answer corresponding to the first information includes a text answer, which is associated with an image answer, a video answer, and an audio answer;

[0036] The output of the answer includes:

[0037] Output the text answer, and output at least one of the image answer, video answer, and audio answer associated with the text answer.

[0038] Secondly, embodiments of the present invention provide an interactive device, the device comprising:

[0039] A receiving module is configured to receive input first information, the first information including at least one of an image, video, or voice;

[0040] The parsing module is used to determine the information content contained in the first information, parse the information content contained in the first information, and obtain the fusion features.

[0041] The processing module is used to identify and process the fused features using a natural language processing algorithm to obtain the answer corresponding to the first information and output the answer.

[0042] Optionally, the parsing module includes:

[0043] The first parsing unit is used to parse the image when the information content included in the first information includes images and text, to obtain image category and image description information, wherein the fusion feature includes the image category, the image description information and the text;

[0044] The second parsing unit is used to parse the video when the information content included in the first information includes video and text, to obtain video action description information, and the fusion feature includes the video action description information and the text;

[0045] The third parsing unit is used to perform speech recognition on the speech when the information content included in the first information includes speech, and obtain recognized text, wherein the fusion feature includes the recognized text.

[0046] Optionally, the third parsing unit is further configured to:

[0047] Extract voiceprint features from the speech, and determine whether the user corresponding to the speech is a preset user based on the voiceprint features;

[0048] If the user corresponding to the voice is the preset user, then the fusion feature includes the recognized text and the user information corresponding to the preset user.

[0049] Optionally, the device further includes:

[0050] The search module is used to search the historical issue log library for whether the historical issues associated with the preset user exist.

[0051] The processing module is specifically used for:

[0052] If the historical question log database contains a historical question associated with the preset user, then based on the historical question associated with the preset user, a natural language processing algorithm is used to identify and process the fused features to obtain the answer corresponding to the first information, and the answer is output.

[0053] Optionally, the device further includes:

[0054] The determining module is used to determine the target domain corresponding to the first information based on the first information;

[0055] The processing module is specifically used for:

[0056] The fused features are identified and processed using a natural language processing algorithm corresponding to the target domain to obtain the answer corresponding to the first information, and the answer is output.

[0057] Optionally, the target domain is a domain among multiple domains, and the first domain among the multiple domains maintains a knowledge graph;

[0058] The processing module is specifically used for:

[0059] If the target domain belongs to the first domain, the fusion features are identified and processed using a knowledge graph question answering algorithm based on a first language model according to the knowledge graph corresponding to the target domain.

[0060] If the knowledge graph question-answering algorithm identifies the first answer, then the first answer is determined as the answer corresponding to the first information;

[0061] If the first answer is not identified using the knowledge graph question answering algorithm, then the fusion features are identified and processed using an intent recognition algorithm and a second language model based on the knowledge base corresponding to the target domain to obtain the answer corresponding to the first information.

[0062] If the target domain does not belong to the first domain, the fusion features are identified and processed using an intent recognition algorithm and a second language model based on the knowledge base corresponding to the target domain to obtain the answer corresponding to the first information.

[0063] Optionally, the processing module is further configured to:

[0064] The intent of the fusion features is identified using an intent recognition algorithm based on the knowledge base corresponding to the target domain.

[0065] If the intent recognition algorithm identifies the first question, then the second language model is used to rank the first question by relevance, and the answer corresponding to the first information is determined based on the ranking result.

[0066] If the intent recognition algorithm fails to identify the first question, a natural language processing algorithm for multi-turn dialogue is used to identify the fused features and obtain the answer corresponding to the first information.

[0067] Optionally, the answer corresponding to the first information includes a text answer, which is associated with an image answer, a video answer, and an audio answer;

[0068] The processing module is specifically used for:

[0069] The fused features are identified and processed using a natural language processing algorithm to obtain the answer corresponding to the first information. The text answer is output, and at least one of the image answer, video answer, and voice answer associated with the text answer is also output.

[0070] Thirdly, embodiments of the present invention provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the interaction method described in the first aspect.

[0071] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the interaction method described in the first aspect.

[0072] In this embodiment of the invention, first information is received, which includes at least one of an image, video, or audio. The information content contained in the first information is determined, and the information content is parsed to obtain fusion features. A natural language processing algorithm is used to identify the fusion features to obtain the answer corresponding to the first information, and the answer is output. Thus, compared to a single text-based interaction mode, the parsing, fusion, and recognition of multiple types of information content improves the interactive experience. Attached Figure Description

[0073] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 This is a flowchart of an interaction method provided in an embodiment of the present invention;

[0075] Figure 2 This is a schematic diagram of the structure of an interactive device provided in an embodiment of the present invention;

[0076] Figure 3 This is a schematic diagram of another interactive device provided in an embodiment of the present invention;

[0077] Figure 4 This is a schematic diagram of another interactive device provided in an embodiment of the present invention;

[0078] Figure 5This is a schematic diagram of another interactive device provided in an embodiment of the present invention;

[0079] Figure 6 This is a schematic diagram of another interactive device provided in an embodiment of the present invention;

[0080] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0081] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0082] In this embodiment of the invention, an interaction method, device, and electronic device are proposed to solve the problem that in existing intelligent interaction methods, interaction can basically only be carried out through text, resulting in a relatively simple interaction method and a poor interaction experience.

[0083] See Figure 1 , Figure 1 This is a flowchart of an interaction method provided in an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0084] Step 101: Receive the first input information, which includes at least one of an image, video, or voice.

[0085] The first information may be user-inputted information. For example, the first information may include images and text; or, the first information may include video and text; or, the first information may include voice; or, the first information may include images, videos, voice, and text, etc. This embodiment does not limit the content of the first information. The interaction method can be applied to electronic devices, where the first information can be input on the display interface of the electronic device. If the first information includes voice, the voice can be input through the microphone of the electronic device.

[0086] Step 102: Determine the information content contained in the first information, parse the information content contained in the first information, and obtain the fusion feature.

[0087] Specifically, the first information can be analyzed to determine the information content contained in the first information. For example, if the first information includes information content in image format, it can be determined that the first information includes an image; if the first information includes information content in video format, it can be determined that the first information includes a video.

[0088] Furthermore, the step of parsing the information content contained in the first information to obtain fusion features may include: if the information content contained in the first information includes an image, parsing the image to obtain image category and image description information, wherein the fusion feature includes the image category and the image description information; if the information content contained in the first information includes a video, parsing the video to obtain video action description information, wherein the fusion feature includes the video action description information; and if the information content contained in the first information includes speech, performing speech recognition on the speech to obtain recognized text, wherein the fusion feature includes the recognized text.

[0089] Step 103: Use a natural language processing algorithm to identify the fused features, obtain the answer corresponding to the first information, and output the answer.

[0090] The natural language processing algorithm may include a knowledge graph question-answering algorithm based on a first language model, an intent recognition algorithm based on a second language model, a natural language processing algorithm for multi-turn dialogue, etc. The step of using a natural language processing algorithm to identify the fused features may include: using a knowledge graph question-answering algorithm based on a first language model to identify the fused features; or it may also include: using an intent recognition algorithm based on a second language model to identify the fused features; or it may also include: using a natural language processing algorithm for multi-turn dialogue to identify the fused features; etc., this embodiment does not limit this.

[0091] For example, a knowledge graph question-answering algorithm based on a first language model can be used to identify the fused features. If the knowledge graph question-answering algorithm identifies a first answer, then the first answer is determined as the answer corresponding to the first information. If the knowledge graph question-answering algorithm does not identify the first answer, an intent recognition algorithm can be used to identify the intent of the fused features based on the knowledge base. If the intent recognition algorithm identifies a first question, a second language model is used to rank the first question by relevance, and the answer corresponding to the first information is determined based on the ranking result. If the intent recognition algorithm does not identify the first question, a natural language processing algorithm for multi-turn dialogue can be used to identify the fused features to obtain the answer corresponding to the first information.

[0092] In this embodiment of the invention, first information is received, which includes at least one of an image, video, or audio. The information content contained in the first information is determined, and the information content is parsed to obtain fusion features. A natural language processing algorithm is used to identify the fusion features to obtain the answer corresponding to the first information, and the answer is output. Thus, compared to a single text-based interaction mode, the parsing, fusion, and recognition of multiple types of information content improves the interactive experience.

[0093] Optionally, parsing the information content contained in the first information to obtain the fusion features includes:

[0094] If the information content of the first information includes images and text, the images are parsed to obtain image category and image description information, and the fusion feature includes the image category, the image description information and the text;

[0095] If the information content included in the first information includes video and text, the video is parsed to obtain video action description information, and the fusion feature includes the video action description information and the text.

[0096] If the information content included in the first information includes speech, speech recognition is performed on the speech to obtain recognized text, and the fusion feature includes the recognized text.

[0097] The step of parsing the image to obtain image category and image description information can be achieved by using the RESNET network model to classify the image target, identifying the image category, and then using the fine-grained controllable image description ASG2Caption network model to convert the image into text-type image description information. The image may not contain the answer to the question, but rather supplement the text. The step of parsing the video can be achieved by using the 3D decomposition network FstCN model to identify actions in the video and obtain video action description information.

[0098] In addition, when the information content included in the first information includes images and text, the fusion feature can be a feature obtained by concatenating the image category, the image description information, and the text; when the information content included in the first information includes videos and text, the fusion feature can be a feature obtained by concatenating the video action description information and the text.

[0099] In this embodiment, by determining the type of information content contained in the first information, the images, videos and voice are parsed respectively, and the parsing results are fused with the text. Natural language processing algorithms are used to identify the fused features, which can realize intelligent interaction that can understand and recognize multimodal inputs.

[0100] Optionally, after performing speech recognition on the speech, the method further includes:

[0101] Extract voiceprint features from the speech, and determine whether the user corresponding to the speech is a preset user based on the voiceprint features;

[0102] If the user corresponding to the voice is the preset user, then the fusion feature includes the recognized text and the user information corresponding to the preset user.

[0103] The preset user can be a user from the historical user list. A historical user list can be maintained, and user information can be stored for each user in the historical user list. The user information may include user name, user gender, user address, or areas of interest for the user, etc. The fusion feature can be a feature obtained by concatenating the features of the identified text and the user information corresponding to the preset user.

[0104] In this embodiment, if the user corresponding to the voice is the preset user, the user information corresponding to the preset user is used as a feature to obtain the answer corresponding to the first information, which can improve the accuracy of the obtained answer corresponding to the first information and thus improve the interactive experience.

[0105] Optionally, before performing recognition processing on the fused features using a natural language processing algorithm, the method further includes:

[0106] Search the historical issue log database to see if the preset user-associated historical issue exists;

[0107] The process of using natural language processing algorithms to identify the fused features includes:

[0108] If the historical issues associated with the preset user exist in the historical issue log database, then the fusion features are identified and processed using a natural language processing algorithm based on the historical issues associated with the preset user.

[0109] The step of using a natural language processing (NLP) algorithm to identify the fused features based on the preset user-associated historical questions can be achieved by using the preset user-associated historical questions as questions in a question candidate set and then using the NLP algorithm to identify the fused features. For example, taking a knowledge graph question-answering algorithm based on a first language model as an example, a knowledge graph can be constructed using questions in the question candidate set, and the knowledge graph question-answering algorithm can be implemented based on the knowledge graph. Alternatively, taking an intent recognition algorithm as an example, an intent recognition algorithm can be used to identify the intent of the fused features, identifying a first question. This first question can be used as a question candidate set, and a second language model can be used to rank the questions in the question candidate set by relevance. The answer corresponding to the first information can then be determined based on the ranking result.

[0110] In this embodiment, based on the historical questions associated with the preset user, a natural language processing algorithm is used to identify the fused features. The historical questions associated with the preset user are highly likely to match the first information, thereby improving the accuracy of the answers corresponding to the first information and thus enhancing the interactive experience.

[0111] Optionally, before performing recognition processing on the fused features using a natural language processing algorithm, the method further includes:

[0112] Based on the first information, determine the target domain corresponding to the first information;

[0113] The fused features are processed using natural language processing algorithms to obtain the answer corresponding to the first information, including:

[0114] The fused features are identified and processed using a natural language processing algorithm corresponding to the target domain to obtain the answer corresponding to the first information.

[0115] The step of determining the target domain corresponding to the first information based on the first information may be to extract the target domain from the first information, wherein the first information includes the target domain; or it may be to determine the target domain from the information content contained in the first information. For example, if the first information includes "purchase", then the target domain can be determined to be the shopping domain.

[0116] Furthermore, the target domain is one of multiple domains, and the first domain among these multiple domains may maintain a knowledge graph. The step of using a natural language processing algorithm corresponding to the target domain to identify the fusion feature may include: when the target domain belongs to the first domain, using a knowledge graph question-answering algorithm based on a first language model to identify the fusion feature according to the knowledge graph corresponding to the target domain; when the target domain does not belong to the first domain, using an intent recognition algorithm and a second language model to identify the fusion feature according to the knowledge base corresponding to the target domain.

[0117] In this embodiment, a natural language processing algorithm corresponding to the target domain is used to identify the fused features. Different natural language processing algorithms can be used for different domains, which can improve the accuracy of identification and thus improve the accuracy of the answer corresponding to the first information obtained.

[0118] Optionally, the target domain is a domain among multiple domains, and the first domain among the multiple domains maintains a knowledge graph;

[0119] The step of using a natural language processing algorithm corresponding to the target domain to identify and process the fused features to obtain the answer corresponding to the first information includes:

[0120] If the target domain belongs to the first domain, the fusion features are identified and processed using a knowledge graph question answering algorithm based on a first language model according to the knowledge graph corresponding to the target domain.

[0121] If the knowledge graph question-answering algorithm identifies the first answer, then the first answer is determined as the answer corresponding to the first information;

[0122] If the first answer is not identified using the knowledge graph question answering algorithm, then the fusion features are identified and processed using an intent recognition algorithm and a second language model based on the knowledge base corresponding to the target domain to obtain the answer corresponding to the first information.

[0123] If the target domain does not belong to the first domain, the fusion features are identified and processed using an intent recognition algorithm and a second language model based on the knowledge base corresponding to the target domain to obtain the answer corresponding to the first information.

[0124] The first domain can be a subset of the multiple domains. A knowledge graph can be maintained for the first domain, and a multi-channel knowledge base can be maintained for the multiple domains. The knowledge graph question-answering algorithm of the first language model can be the KBQA algorithm based on the BERT model. The KBQA algorithm based on the BERT model can be used to perform question-answer matching on the fused features to obtain a first answer, which is then determined as the answer corresponding to the first information. The step of using an intent recognition algorithm and a second language model to identify the fused features based on the knowledge base corresponding to the target domain can be as follows: using an intent recognition algorithm to identify the intent of the fused features based on the knowledge base corresponding to the target domain to obtain a first question; using a second language model to rank the first question by relevance; and determining the answer corresponding to the first information based on the ranking result.

[0125] In this implementation, different knowledge organization forms are maintained in different domains, including knowledge bases and knowledge graphs. Different natural language processing algorithms are used to identify and process the fusion features for different domains. Compared with interaction that is only for a certain vertical domain, it can realize intelligent interaction that can switch freely between multiple domains.

[0126] Optionally, the step of using an intent recognition algorithm and a second language model to identify and process the fused features based on the knowledge base corresponding to the target domain to obtain the answer corresponding to the first information includes:

[0127] The intent of the fusion features is identified using an intent recognition algorithm based on the knowledge base corresponding to the target domain.

[0128] If the intent recognition algorithm identifies the first question, then the second language model is used to rank the first question by relevance, and the answer corresponding to the first information is determined based on the ranking result.

[0129] If the intent recognition algorithm fails to identify the first question, a natural language processing algorithm for multi-turn dialogue is used to identify the fused features and obtain the answer corresponding to the first information.

[0130] The step of using an intent recognition algorithm to identify the intent of the fused features based on the knowledge base corresponding to the target domain can be as follows: The intent recognition algorithm is used to identify the intent of the fused features, guiding inputs of different intents to the scene corresponding to the intent in the target domain. A first question is then matched against the first information in the knowledge base corresponding to the first information within that scene. The first question can include multiple questions. The second language model can be a Roberta model. The Roberta model can be used to rank the first questions by relevance. The question ranked first, indicating the strongest relevance, is identified as the question matching the first information, and the answer corresponding to the question matching the first information is identified as the answer corresponding to the first information. The knowledge base stores the correspondence between questions and answers.

[0131] Furthermore, the step of using a natural language processing algorithm for multi-turn dialogue to identify the fused features and obtain the answer corresponding to the first information can be achieved by using a dialogue state tracking and slot recognition algorithm to identify the fused features and obtain the answer corresponding to the first information. The step of using a dialogue state tracking and slot recognition algorithm to identify the fused features and obtain the answer corresponding to the first information can include: using a dialogue state tracking and slot recognition algorithm to identify the fused features; if a second answer is identified, the identified second answer can be determined as the answer corresponding to the first information; if the second answer is not identified, information indicating that no answer was identified can be output.

[0132] In this embodiment, when the intent recognition algorithm fails to identify a question that matches the first information, a natural language processing algorithm for multi-turn dialogue is used to identify and process the fusion features. By identifying and processing the fusion features in multiple ways, the probability of matching the answer corresponding to the first information can be increased, thereby improving the interactive experience.

[0133] Optionally, the answer corresponding to the first information includes a text answer, which is associated with an image answer, a video answer, and an audio answer;

[0134] The output of the answer includes:

[0135] Output the text answer, and output at least one of the image answer, video answer, and audio answer associated with the text answer.

[0136] Specifically, when the first information includes an image, a text answer and an image answer associated with the text answer may be output; when the first information includes a video, a text answer and a video answer associated with the text answer may be output; when the first information includes audio, a text answer and an audio answer associated with the text answer may be output; or the text answer, the image answer associated with the text answer, the video answer, and the audio answer may be output simultaneously, etc. This embodiment does not limit this.

[0137] In this implementation, the text answer is output, along with at least one of the image answer, video answer, and voice answer associated with the text answer, thereby providing a multimodal answer output method and improving the interactive experience.

[0138] As a specific implementation method, this embodiment of the invention provides an interactive device that can be used to implement the interactive method described in this embodiment of the invention, such as... Figure 2 As shown, the interactive device can implement image-text question-and-answer, video-text question-and-answer, and voice-text question-and-answer mechanisms. Users can input text questions, image + text questions, video + text questions, or voice questions into the interactive device. The interactive device then parses the user's input (image, video, or voice) and outputs the corresponding answer. In this embodiment, the interactive device parses multimodal input information, performs feature fusion on the parsing results, and, based on knowledge graphs, knowledge bases, and historical question logs, uses natural language processing algorithms corresponding to different domains to identify and process the fused features, outputting the answer. The output answer can be presented in various formats such as images, videos, and voice. This achieves a multimodal intelligent interactive system that can flexibly switch between different domains, possessing a very broad application and promotion prospect.

[0139] See Figure 3 , Figure 3 This is a schematic diagram of the structure of an interactive device provided in an embodiment of the present invention, such as... Figure 3 As shown, the interactive device 200 includes:

[0140] The receiving module 201 is used to receive input first information, the first information including at least one of an image, video or voice;

[0141] The parsing module 202 is used to determine the information content contained in the first information, parse the information content contained in the first information, and obtain the fusion feature;

[0142] The processing module 203 is used to perform recognition processing on the fused features using a natural language processing algorithm to obtain the answer corresponding to the first information and output the answer.

[0143] Optional, such as Figure 4 As shown, the parsing module 202 includes:

[0144] The first parsing unit 2021 is used to parse the image when the information content included in the first information includes images and text, to obtain image category and image description information, wherein the fusion feature includes the image category, the image description information and the text;

[0145] The second parsing unit 2022 is used to parse the video when the information content included in the first information includes video and text, to obtain video action description information, and the fusion feature includes the video action description information and the text;

[0146] The third parsing unit 2023 is used to perform speech recognition on the speech when the information content included in the first information includes speech, and obtain recognized text, wherein the fusion feature includes the recognized text.

[0147] Optionally, the third parsing unit 2023 is further configured to:

[0148] Extract voiceprint features from the speech, and determine whether the user corresponding to the speech is a preset user based on the voiceprint features;

[0149] If the user corresponding to the voice is the preset user, then the fusion feature includes the recognized text and the user information corresponding to the preset user.

[0150] Optional, such as Figure 5 As shown, the device 200 further includes:

[0151] The search module 204 is used to search the historical issue log database for whether the preset user-associated historical issue exists.

[0152] The processing module 203 is specifically used for:

[0153] If the historical question log database contains a historical question associated with the preset user, then based on the historical question associated with the preset user, a natural language processing algorithm is used to identify and process the fused features to obtain the answer corresponding to the first information, and the answer is output.

[0154] Optional, such as Figure 6 As shown, the device 200 further includes:

[0155] The determining module 205 is used to determine the target area corresponding to the first information based on the first information;

[0156] The processing module 203 is specifically used for:

[0157] The fused features are identified and processed using a natural language processing algorithm corresponding to the target domain to obtain the answer corresponding to the first information, and the answer is output.

[0158] Optionally, the target domain is a domain among multiple domains, and the first domain among the multiple domains maintains a knowledge graph;

[0159] The processing module 203 is specifically used for:

[0160] If the target domain belongs to the first domain, the fusion features are identified and processed using a knowledge graph question answering algorithm based on a first language model according to the knowledge graph corresponding to the target domain.

[0161] If the knowledge graph question-answering algorithm identifies the first answer, then the first answer is determined as the answer corresponding to the first information;

[0162] If the first answer is not identified using the knowledge graph question answering algorithm, then the fusion features are identified and processed using an intent recognition algorithm and a second language model based on the knowledge base corresponding to the target domain to obtain the answer corresponding to the first information.

[0163] If the target domain does not belong to the first domain, the fusion features are identified and processed using an intent recognition algorithm and a second language model based on the knowledge base corresponding to the target domain to obtain the answer corresponding to the first information.

[0164] Optionally, the processing module 203 is further configured to:

[0165] The intent of the fusion features is identified using an intent recognition algorithm based on the knowledge base corresponding to the target domain.

[0166] If the intent recognition algorithm identifies the first question, then the second language model is used to rank the first question by relevance, and the answer corresponding to the first information is determined based on the ranking result.

[0167] If the intent recognition algorithm fails to identify the first question, a natural language processing algorithm for multi-turn dialogue is used to identify the fused features and obtain the answer corresponding to the first information.

[0168] Optionally, the answer corresponding to the first information includes a text answer, which is associated with an image answer, a video answer, and an audio answer;

[0169] The processing module 203 is specifically used for:

[0170] The fused features are identified and processed using a natural language processing algorithm to obtain the answer corresponding to the first information. The text answer is output, and at least one of the image answer, video answer, and voice answer associated with the text answer is also output.

[0171] Interactive devices can achieve Figure 1 The various processes implemented in the method embodiments can achieve the same technical effect, and will not be described again here to avoid repetition.

[0172] like Figure 7 As shown, this embodiment of the invention also provides an electronic device 300, including: a processor 301, a memory 302, and a program stored in the memory 302 and executable on the processor 301. When the program is executed by the processor 301, it implements the various processes of the above-described interactive method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0173] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described interactive method embodiments and achieves the same technical effects. To avoid repetition, further details are omitted here. The computer-readable storage medium may be, for example, ROM, RAM, a magnetic disk, or an optical disk.

[0174] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0175] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0176] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. An interaction method, characterized in that, The method comprises: receiving input first information, the first information comprising at least one of a picture, a video or a voice; determining information content contained in the first information, and analyzing the information content contained in the first information to obtain fusion features; using a natural language processing algorithm to recognize and process the fusion features to obtain an answer corresponding to the first information, and outputting the answer; Before the fusion features are recognized and processed using the natural language processing algorithm, the method further comprises: determining a target field corresponding to the first information based on the first information; recognizing and processing the fusion features using a natural language processing algorithm to obtain an answer corresponding to the first information, comprising: recognizing and processing the fusion features using a natural language processing algorithm corresponding to the target field to obtain an answer corresponding to the first information; the target field is a field in a plurality of fields, and a knowledge graph is maintained in a first field in the plurality of fields; recognizing and processing the fusion features using a natural language processing algorithm corresponding to the target field to obtain an answer corresponding to the first information, comprising: if the target field belongs to the first field, recognizing and processing the fusion features using a knowledge graph question and answer algorithm based on a first language model according to the knowledge graph corresponding to the target field; the knowledge graph question and answer algorithm based on the first language model comprises a KBQA algorithm based on a BERT model; if a first answer is recognized using the knowledge graph question and answer algorithm, the first answer is determined as the answer corresponding to the first information; if the first answer is not recognized using the knowledge graph question and answer algorithm, an intent recognition algorithm and a second language model are used to recognize and process the fusion features according to a knowledge base corresponding to the target field to obtain an answer corresponding to the first information; if the target field does not belong to the first field, an intent recognition algorithm and a second language model are used to recognize and process the fusion features according to a knowledge base corresponding to the target field to obtain an answer corresponding to the first information.

2. The method of claim 1, wherein, The analysis of the information content contained in the first information to obtain fusion features comprises: if the information content contained in the first information comprises a picture and text, the picture is analyzed to obtain a picture category and picture description information, and the fusion features comprise the picture category, the picture description information and the text; if the information content contained in the first information comprises a video and text, the video is analyzed to obtain video action description information, and the fusion features comprise the video action description information and the text; if the information content contained in the first information comprises a voice, the voice is recognized to obtain recognized text, and the fusion features comprise the recognized text.

3. The method of claim 2, wherein, After the voice is recognized, the method further comprises: extracting a voiceprint feature from the voice, and determining whether a user corresponding to the voice is a preset user based on the voiceprint feature; The fusion feature comprises the recognized text and user information corresponding to the preset user if the user corresponding to the voice is the preset user.

4. The method of claim 3, wherein, Before the fusion feature is identified by using the natural language processing algorithm, the method further comprises: checking whether there is a historical problem associated with the preset user in a historical problem log library; The fusion feature is identified by using the natural language processing algorithm, comprising: If there is a historical problem associated with the preset user in the historical problem log library, the fusion feature is identified by using the natural language processing algorithm based on the historical problem associated with the preset user.

5. The method of claim 1, wherein, The fusion feature is identified by using the intent recognition algorithm and the second language model based on the knowledge base corresponding to the target domain to obtain the answer corresponding to the first information, comprising: The fusion feature is identified by using the intent recognition algorithm based on the knowledge base corresponding to the target domain; If the first question is identified by using the intent recognition algorithm, the second language model is used to sort the relevance of the first question, and the answer corresponding to the first information is determined according to the sorting result; If the first question is not identified by using the intent recognition algorithm, the fusion feature is identified by using the natural language processing algorithm for multi-round dialogue to obtain the answer corresponding to the first information.

6. The method of claim 1, wherein, The answer corresponding to the first information comprises a text answer, and the text answer is associated with at least one of a picture answer, a video answer and a voice answer; The answer is output, comprising: The text answer is output, and at least one of the picture answer, the video answer and the voice answer associated with the text answer is output.

7. An interactive device, characterized by The device comprises: A receiving module is configured to receive input first information, the first information comprising at least one of a picture, a video or a voice; An analyzing module is configured to determine information content contained in the first information, and analyze the information content contained in the first information to obtain fusion feature; A processing module is configured to identify the fusion feature by using a natural language processing algorithm to obtain an answer corresponding to the first information, and output the answer; The device further comprises: A determining module is configured to determine a target domain corresponding to the first information based on the first information; The processing module is specifically configured to: Identify the fusion feature by using a natural language processing algorithm corresponding to the target domain to obtain an answer corresponding to the first information, and output the answer; The target domain is a domain in a plurality of domains, and a knowledge graph is maintained in a first domain in the plurality of domains; The processing module is specifically configured to: In a case where the target domain belongs to the first domain, the fusion feature is identified by using a knowledge graph question and answer algorithm based on a first language model corresponding to the knowledge graph of the target domain; the knowledge graph question and answer algorithm based on the first language model comprises a KBQA algorithm based on a BERT model; If a first answer is identified by using the knowledge graph question and answer algorithm, the first answer is determined as the answer corresponding to the first information; If the first answer is not recognized by using the knowledge graph question and answer algorithm, an intent recognition algorithm and a second language model are used to recognize and process the fusion features according to the knowledge base corresponding to the target field, to obtain an answer corresponding to the first information. In a case where the target field does not belong to the first field, an intent recognition algorithm and a second language model are used to recognize and process the fusion features according to the knowledge base corresponding to the target field, to obtain an answer corresponding to the first information.

8. An electronic device, comprising: Comprise: A processor, a memory, and a program stored on the memory and executable on the processor, the program being executed by the processor to implement the steps of the interaction method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the interaction method according to any one of claims 1 to 6.

Images