Multimodal data dialogue robot interaction methods, configuration methods, terminals, media and products

Abstract

This application provides a multimodal data conversational robot interaction method, configuration method, terminal, medium, and product. The method is applied to a conversational robot including an interaction layer, a processing layer, and a model layer, comprising: the interaction layer responding to user dialogue information input by the current dialogue node and establishing a session connection with the processing layer; the processing layer completing initial configuration with the model layer and sending the user dialogue information to the model layer; the model layer returning interaction information and multimodal extended information to the processing layer based on preset prompts from the current dialogue node and the user dialogue information; and the processing layer formatting the interaction information and the multimodal extended information and streaming them to the interaction layer. This application expands the boundaries of the conversational robot's multimodal data interaction capabilities, supporting the interaction of various types of data and meeting the needs of different interaction scenarios.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a multimodal data dialogue robot interaction method, configuration method, terminal, medium and product. Background Technology

[0002] Current chatbots have significant limitations in functionality, supporting only pure text messaging. This single-function design exposes several problems in practical applications: First, users can only interact through text input, unable to send multimedia content such as images, videos, and audio, which severely impacts communication efficiency in scenarios requiring visual assistance or voice interaction. Second, the lack of multi-category support makes the dialogue process monotonous and dull, reducing user engagement and interactive experience. Third, the pure text interaction mode struggles to meet users' diverse expressive needs, especially when rapidly conveying complex information or emotions, where the limitations of text become particularly apparent. Introducing a multimodal model processing architecture would affect the processing speed of interactive text during chatbot interaction, significantly reducing the fluency of human-computer interaction. This functional deficiency directly leads to a decline in user experience, manifested in multiple dimensions such as reduced user engagement, increased dialogue interruption rate, and decreased user satisfaction. Summary of the Invention

[0003] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a multimodal data chatbot interaction method, configuration method, terminal, medium and product to solve the problem of the single interaction type of existing chatbots.

[0004] To achieve the above and other related objectives, a first aspect of this application provides a multimodal data dialogue robot interaction method, applied to a dialogue robot including an interaction layer, a processing layer, and a model layer. The method includes: the interaction layer responding to user dialogue information input by the current dialogue node and establishing a session connection with the processing layer; the processing layer completing initialization configuration with the model layer and sending the user dialogue information to the model layer; the model layer returning interaction information and multimodal extended information to the processing layer based on preset prompt information of the current dialogue node and the user dialogue information; and the processing layer formatting the interaction information and the multimodal extended information respectively and streaming them to the interaction layer.

[0005] In some embodiments of the first aspect of this application, the model layer returns interaction information to the processing layer, including: after the user dialogue information triggers the function of the model layer, the function configures corresponding pre-response information; the model layer sends the pre-response information as interaction information to the processing layer.

[0006] In some embodiments of the first aspect of this application, the multimodal extended information carries a corresponding data assembly structure; the processing layer performs format assembly on the interaction information and the multimodal extended information respectively, and then streams them to the interaction layer, including: the processing layer performs format assembly on the multimodal extended information according to the data assembly structure; after the processing layer sends the interaction information in text format, it streams the assembled multimodal extended information.

[0007] In some embodiments of the first aspect of this application, the multimodal extension information includes preset variable configuration slots; after the model layer assigns values ​​to variables based on the pre-configured API interface in the variable configuration slots, it returns interaction information and the multimodal extension information to the processing layer.

[0008] In some embodiments of the first aspect of this application, the model layer includes a pre-configured node knowledge base; the model layer returns interaction information and multimodal extension information to the processing layer based on the node knowledge base, preset prompt information of the current dialogue node, and the user dialogue information.

[0009] In some embodiments of the first aspect of this application, before the processing layer sends the user dialogue information to the model layer, the model layer returns corresponding guidance information to the processing layer based on the prompt information; the processing layer then sends the guidance information to the interaction layer.

[0010] To achieve the above and other related objectives, a second aspect of this application provides a configuration method for a multimodal data chatbot, applied to a chatbot including an interaction layer, a processing layer, and a model layer, comprising: the chatbot receiving prompt information configured at the current dialogue node; the interaction layer responding to user dialogue information input at the current dialogue node and establishing a session connection with the processing layer; the processing layer completing initialization configuration with the model layer and sending the user dialogue information to the model layer; the model layer returning interaction information and multimodal extended information to the processing layer based on the prompt information and the user dialogue information; and the processing layer formatting the interaction information and the multimodal extended information respectively and streaming them to the interaction layer.

[0011] In some embodiments of the second aspect of this application, the configuration method of a multimodal data dialogue robot further includes: the dialogue robot receiving function configuration information of the current dialogue node, the function configuration information including pre-response information; and the model layer sending the pre-response information as interaction information to the processing layer.

[0012] In some embodiments of the second aspect of this application, the function configuration information further includes a data assembly structure. After the processing layer performs format assembly on the interaction information and the multimodal extended information respectively, it streams the data assembly structure to the interaction layer. This includes: the processing layer performs format assembly on the multimodal extended information according to the data assembly structure; after the processing layer sends the interaction information in text format, it streams the assembled multimodal extended information.

[0013] In some embodiments of the second aspect of this application, the function configuration information further includes preset variable configuration slots; after the model layer assigns values ​​to variables based on the pre-configured API interface in the variable configuration slots, it returns interaction information and the multimodal extension information to the processing layer.

[0014] In some embodiments of the second aspect of this application, the function configuration information further includes preset guidance information; before the processing layer sends the user dialogue information to the model layer, the model layer returns corresponding guidance information to the processing layer based on the prompt information; the processing layer sends the guidance information to the interaction layer.

[0015] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal data dialogue robot interaction method. To achieve the above and other related objectives,

[0016] A fourth aspect of this application provides a computer program product including computer program code, which, when run on a computer, enables the computer to implement the multimodal data dialogue robot interaction method.

[0017] To achieve the above and other related objectives, a fifth aspect of this application provides an electronic terminal, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the multimodal data dialogue robot interaction method.

[0018] As described above, the multimodal data dialogue robot interaction method, configuration method, terminal, medium, and product of this application have the following beneficial effects:

[0019] This application expands the boundaries of multimodal data interaction capabilities for chatbots, enabling the interaction of various data types and meeting the needs of different interaction scenarios. Attached Figure Description

[0020] Figure 1 The diagram shown is a flowchart illustrating a multimodal data dialogue robot interaction method according to an embodiment of this application.

[0021] Figure 2 The diagram shown is a schematic representation of a multimodal data dialogue robot according to an embodiment of this application.

[0022] Figure 3 The diagram shown is a schematic representation of the working process of a multimodal data dialogue robot in one embodiment of this application.

[0023] Figure 4 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of this application. Detailed Implementation

[0024] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0025] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0026] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0027] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0028] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0029] <1> Streaming transmission is a data transmission technology that allows data to be continuously sent and received while it is being generated, without waiting for the entire file to download completely. Features of streaming transmission include: real-time performance: data is transmitted and consumed simultaneously with its generation, resulting in low latency; continuity: data is continuously transmitted as a stream, rather than being sent all at once; on-demand loading: users can start using the file without waiting for it to download completely; and resource conservation: it reduces the need for local storage, as data can be discarded after playback or processing.

[0030] <2> Chatbot Service is an application service based on artificial intelligence technology. Through technologies such as Natural Language Processing (NLP) and machine learning, computer programs can engage in human-like conversational interactions with users.

[0031] <3> Dialogue Management Call: This is the method used in a chatbot to invoke the dialogue management module. Dialogue management is the core of a dialogue system, responsible for controlling the dialogue flow, understanding user intent, managing dialogue state, and generating appropriate responses.

[0032] <4> Assembly refers to combining and arranging information of different types or sources according to certain format requirements to make it conform to a specific structure or standard;

[0033] <5> A Q&A-style database is a database system specifically designed for storing and managing question-and-answer pairs. It is primarily used for quickly retrieving and providing content relevant to user questions.

[0034] <6> An API (Application Programming Interface) is a way for computer software to interact and communicate. It allows different software programs or systems to share data, functions, and services.

[0035] To facilitate understanding of the embodiments of this application, firstly, in conjunction with Figure 1 Detailed explanation. Figure 1This document illustrates a flowchart of a multimodal data-based chatbot interaction method according to an embodiment of the present invention. The multimodal data-based chatbot interaction method in this embodiment is applied to a chatbot (Large Language Model) comprising an interaction layer, a processing layer, and a model layer. It should be understood that the Large Language Model includes, but is not limited to, large language models such as Llamabot, GPT, and BERT. To clearly demonstrate the interaction between the Large Language Model and other functional layers of the chatbot, Llamabot is used as an example in this specification. Those skilled in the art will understand that the technical content of this application is not limited to using Llamabot as the model layer of the chatbot. The multimodal data-based chatbot interaction method mainly includes the following steps:

[0036] Step S11: The interaction layer responds to the user dialogue information input by the current dialogue node and establishes a session connection with the processing layer.

[0037] Specifically, such as Figure 2 as well as Figure 3 As shown, the interaction layer is Chatbot Service, and the processing layer is DMCall. The interaction layer includes a message layer. The model layer mainly interacts with the interaction layer and the large model through the Llamabot interface. The interaction layer interacts with the user through interaction messages in the application layer and message layer. When a user initiates a conversation, the interaction layer starts and obtains the chatbot's ID. Then, the interaction layer establishes a session connection with the conversation layer. A session connection refers to the communication between two parties negotiating and determining communication parameters, rules, etc., through a series of interactive processes, thereby forming a session that can exchange data.

[0038] In one embodiment, such as Figure 2 As shown, the types of user dialogue information output or input through the message layer can be any one or more of the following: text, list, button, image, file, video, audio, interactive, location, and contact.

[0039] Step S12: The initialization configuration between the processing layer and the model layer is completed, and the user dialogue information is sent to the model layer.

[0040] Specifically, such as Figure 3 As shown, after the session between the processing layer and the interaction layer is established, the processing layer and the model layer perform initialization configuration. Initialization configuration includes, but is not limited to, determining the data format, configuring the interface, and configuring the model.

[0041] Step S13: Based on the preset prompts and user dialogue information of the current dialogue node, the model layer returns the interaction information and multimodal extension information to the processing layer in sequence.

[0042] Step S14: The processing layer assembles the interaction information and multimodal extended information into their respective formats and then streams them to the interaction layer.

[0043] In the overall process of a chatbot interacting with a user, the customer configures the chatbot's prompts. The model layer then generates modal information such as text, lists, buttons, images, files, videos, audio, interactions, location, and contacts based on these pre-configured prompts. To adapt to the chatbot's smooth interaction, interactive information and multimodal extended information are generated in a streaming manner and sent to the processing layer. The processing layer then performs format assembly, and the information is fed back to the user through the interaction layer. This avoids the problems of inaccurate and unfriendly interactive information when the processing layer selects interactive content, and the excessive time spent on NLP and NER processes that limit the smoothness of the interaction. It also solves the problem that the model layer cannot directly generate interactive formats that conform to the current scenario, resulting in an inability to provide intuitive interactive information to the user.

[0044] For example, when a chatbot provides users with list-based services such as a list of supported services or a list of options, the menu descriptions should display different content depending on the scenario and the number of questions asked. The Llamabot interface returns text-based interactive information and list-based modal configuration information to the processing layer. The processing layer then assembles the text-based interactive information and the list-based modal configuration information into list-based interactive information, which is then fed back to the user by the interaction layer through the corresponding interactive application in the application layer (such as WhatsApp, Facebook, Instagram, Line, Viber, Telegram).

[0045] In one embodiment, the multimodal extended information carries a corresponding data assembly structure; the processing layer performs format assembly on the interaction information and the multimodal extended information respectively, and then streams it to the interaction layer, including: the processing layer performs format assembly on the multimodal extended information according to the data assembly structure; after the processing layer sends the interaction information in text format, it streams the assembled multimodal extended information.

[0046] Specifically, such as Figure 2As shown, the model layer's function calls include preset data assembly structures such as List, button, image, text, and flies. After generating multimodal extended information, the model layer's function calls configure the corresponding data assembly structure and return both the multimodal extended information and the corresponding data assembly structure to the processing layer. The processing layer assembles the multimodal extended information according to the data assembly structure. After streaming the interaction information to the interaction layer in text format, the processing layer streams the assembled multimodal extended information back to the interaction layer. Finally, the interaction layer sends the interaction information and the multimodal extended information to the user.

[0047] For example, if the data assembly structure is "markdown", then the multimodal extended information is assembled into text format information; if the data assembly structure is "", then the multimodal extended information is assembled into image format information. In one embodiment, such as Figure 3 As shown, after the initial configuration is completed between the processing layer and the model layer, before the processing layer sends the user dialogue information to the model layer, the model layer returns the corresponding guidance information to the processing layer based on the preset prompt information of the current dialogue node; the processing layer then sends the guidance information to the interaction layer.

[0048] Specifically, the model layer has multiple preset guidance dialogue templates. The preset prompts for the current dialogue node are used to clarify the dialogue intent or scenario of the current dialogue node. Based on the preset prompts for the current dialogue node, the model layer matches the corresponding guidance dialogue template from the multiple preset guidance dialogue templates and transmits the matched guidance dialogue template as guidance information stream to the interaction layer.

[0049] It should be understood that introductory information, also known as Entrance script, refers to the opening remarks or guiding words when a user interacts with a robot. It aims to attract the user's attention, clarify the robot's functions, and guide the user to the next step.

[0050] During the dialogue interaction between the chatbot and the user, if the streaming transmission of multimodal interaction information is involved, it may be affected by the data processing speed of the model layer and the processing layer, causing interaction lag. Therefore, in this embodiment, guidance information is introduced as the interaction content for the first round of interaction with the user. The model layer configuration and other steps are completed during the first round of interaction through guidance information, which improves the smoothness of the chatbot's interaction. At the same time, in some interaction nodes, the chatbot needs to clarify the corresponding information to the user in the first round of interaction (such as some nodes need to first confirm the user's identity information, address information, financial information, etc.). Therefore, introducing preset guidance information in such nodes can reduce the configuration requirements of this part of the prompt information and the number of interaction rounds.

[0051] In one embodiment, the model layer returns interaction information to the processing layer, including: after the user dialogue information triggers the function of the model layer, the function configures the corresponding pre-response information; the model layer sends the pre-response information as interaction information to the processing layer.

[0052] Specifically, to avoid uncertain output from the LLM when triggering function calls and to ensure that the LLM provides definite output when executing the current function call, pre-response information is configured within the function call. After the processing layer sends the user dialogue information to the model layer, it triggers the function call in the model layer. The model layer ignores its own generated interaction information, and the function call selects a corresponding pre-response information from a pre-configured list. This selected pre-response information is sent to the processing layer as the interaction information. The interaction information is generally text-based, used to respond to questions, prevent excessive user waiting time, and enhance interactivity. For example, "Regarding this issue, I suggest you try…". It should be noted that the interaction information generated by the model layer itself is uncertain, while the function call configuration ensures that the output of definite interaction information is provided.

[0053] It should be understood that the function is a configuration item in the LLM-BOT (Model Interface) of the model layer, specific to each dialogue node. The configuration information of the function includes name, description, function type (mainly including API call functions), API interface information, function request information, function response information, pre-response information, and data assembly structure. In one embodiment, the interaction information may have preset variable configuration slots. After the model layer fills the corresponding variables into the variable configuration slots of the interaction information, it returns the filled interaction information to the processing layer.

[0054] In one embodiment, the model layer includes a large language model. The model layer generates corresponding multimodal additional data based on prompts and user dialogue information. This multimodal additional data is JSON data. The types of multimodal additional data can be any one or more of the following: text, list, button, image, file, video, audio, interactive, location, and contacts.

[0055] In one embodiment, the multimodal extension information includes preset variable configuration slots; after the model layer assigns values ​​to variables based on the pre-configured API interface in the variable configuration slots, it returns interaction information and multimodal extension information to the processing layer.

[0056] Specifically, the variable configuration slots are used to populate the results returned by the API interface. For example... Figure 2As shown, after the API calls return results, the APIStore functions in the model layer can use the Response converter's arguments to post-process the obtained output parameters and assign them to specified variables, and then configure the variables in the Response data.

[0057] For example, in a logistics scenario, the variable is `waybill_number`. After an API call, the value of `waybill_number` is obtained as `XX123456789`. The variable `waybill_number` is configured on the button in the multimodal extended information. When the model layer returns multimodal extended information, it assigns `XX123456789` to the slot corresponding to the variable `waybill_number` in the multimodal extended information. That is, the button displays "XX123456789".

[0058] In one embodiment, a pre-configured node knowledge base (a database for Q&A patterns) can be introduced into the model layer. The node knowledge base can be configured in direct mode (original) and model-generated mode (llm_generated). When the node knowledge base is configured in direct mode, the corresponding answer information is selected from the node knowledge base and transmitted to the user; when the node knowledge base is configured in model-generated mode, the node knowledge base, along with the user's dialogue information, is input into the large language model built into the model layer to generate more accurate and complex information.

[0059] In one embodiment, such as Figure 2 As shown, after receiving interactive information and multimodal extension information, the user can make a selection or reply. Upon receiving the user's selection / reply information, the interaction layer sends it to the processing layer. The processing layer then sends a request corresponding to the selection / reply information to the model layer and performs corresponding result analysis (Dialogue Analysis). Dialogue Analysis at the model layer involves extracting key information from the dialogue, uncovering potential causes, and developing solutions to optimize the dialogue system or improve the user experience. After the result analysis is complete, the model layer finishes its work and transmits the execution result information to the processing layer. Upon receiving the execution result information, the processing layer notifies the interaction layer that the dialogue has ended. The interaction layer sends a dialogue evaluation to the user, and the dialogue process ends after the user completes the evaluation.

[0060] Similar to the embodiments described above, the present invention also provides a configuration method for a multimodal data dialogue robot, applicable to a dialogue robot including an interaction layer, a processing layer, and a model layer, comprising:

[0061] The chatbot receives prompts configured in the current dialogue node;

[0062] The interaction layer responds to the user dialogue information input by the current dialogue node and establishes a session connection with the processing layer;

[0063] The processing layer completes the initial configuration with the model layer and sends the user dialogue information to the model layer;

[0064] Based on prompts and user dialogue information, the model layer returns interaction information and multimodal extension information to the processing layer;

[0065] The processing layer assembles the interactive information and multimodal extended information into their respective formats and then streams them to the interaction layer.

[0066] In one embodiment, the configuration method of the multimodal data dialogue robot further includes: the dialogue robot receiving the function configuration information of the current dialogue node, the function configuration information including pre-response information; the model layer sending the pre-response information as interaction information to the processing layer.

[0067] In one embodiment, the function configuration information further includes a data assembly structure. After the processing layer performs format assembly on the interaction information and the multimodal extension information respectively, it sends them to the interaction layer in a streaming manner. This includes: the processing layer performs format assembly on the multimodal extension information according to the data assembly structure; after the processing layer sends the interaction information in text format, it sends the assembled multimodal extension information in a streaming manner.

[0068] In one embodiment, the function configuration information also includes preset variable configuration slots; after the model layer assigns values ​​to variables based on the pre-configured API interfaces in the variable configuration slots, it returns interaction information and multimodal extension information to the processing layer.

[0069] In one embodiment, the function configuration information further includes preset guidance information; before the processing layer sends the user dialogue information to the model layer, the model layer returns the corresponding guidance information to the processing layer based on the prompt information; the processing layer sends the guidance information to the interaction layer.

[0070] It should be noted that the prompts, function configurations (including pre-response information, data assembly structure, variable configuration slots, and API interfaces) and guidance information of the current dialogue node can be configured with reference to the above embodiments, and will not be repeated here.

[0071] Figure 4 This is a schematic block diagram of the electronic terminal provided in an embodiment of this application. Figure 4As shown, the electronic terminal includes at least one processor 401, a memory 402, at least one network interface 403, and a user interface 405. The various components in the device are coupled together via a bus system 404. It is understood that the bus system 404 is used to implement communication between these components. In addition to a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 4 The general will label all buses as bus systems.

[0072] The user interface 405 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0073] It is understood that memory 402 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0074] In this embodiment of the invention, the memory 402 is used to store various types of data to support the operation of the electronic terminal 400. Examples of this data include: any executable program for operation on the electronic terminal 400, such as the operating system 4021 and application program 4022; the operating system 4021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 4022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The multimodal data dialogue robot interaction method provided in this embodiment of the invention can be included in the application program 4022.

[0075] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 401 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0076] In an exemplary embodiment, the electronic terminal 400 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to execute the aforementioned method.

[0077] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute... Figure 1 The multimodal data dialogue robot interaction method shown in the embodiment.

[0078] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when executed on a computer, causes the computer to perform... Figure 1 The multimodal data dialogue robot interaction method shown in the embodiment.

[0079] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0080] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0081] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0082] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0083] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0085] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0086] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0087] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0088] In summary, this application provides a multimodal data conversational robot interaction method, configuration method, terminal, medium, and product. This method is applied to a conversational robot including an interaction layer, a processing layer, and a model layer, comprising: the interaction layer responding to user dialogue information input by the current dialogue node and establishing a session connection with the processing layer; the processing layer completing initial configuration with the model layer and sending the user dialogue information to the model layer; the model layer returning interaction information and multimodal extended information to the processing layer based on preset prompts from the current dialogue node and the user dialogue information; and the processing layer formatting the interaction information and the multimodal extended information and streaming them to the interaction layer. This application expands the boundaries of the conversational robot's multimodal data interaction capabilities, supporting the interaction of various types of data and meeting the needs of different interaction scenarios. Therefore, this application effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0089] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A multimodal data dialogue robot interaction method, characterized in that, The method, applied to a chatbot comprising an interaction layer, a processing layer, and a model layer, includes: The interaction layer responds to the user dialogue information input by the current dialogue node and establishes a session connection with the processing layer; The processing layer and the model layer complete the initialization configuration, and the user dialogue information is sent to the model layer; The model layer returns interaction information and multimodal extension information to the processing layer based on the preset prompt information of the current dialogue node and the user dialogue information. Specifically, the interaction information returned by the model layer includes: after the user dialogue information triggers the function of the model layer, the function configures corresponding pre-response information; the model layer sends the pre-response information as interaction information to the processing layer; the multimodal extension information includes preset variable configuration slots; the model layer assigns values ​​to variables based on the pre-configured API interface in the variable configuration slots, and then returns the interaction information and the multimodal extension information to the processing layer; after generating the multimodal extension information, the function configures a corresponding data assembly structure for it, and returns the multimodal extension information and the corresponding data assembly structure together to the processing layer; the function is a configuration item in the model interface of the model layer for each dialogue node; the configuration information of the function includes name, description, function category, API interface information, function request information, function response information, pre-response information, and data assembly structure. The processing layer formats and assembles the interaction information and the multimodal extended information respectively, and then streams them to the interaction layer; wherein, the multimodal extended information carries a corresponding data assembly structure; the processing layer formats and assembles the interaction information and the multimodal extended information respectively, and then streams them to the interaction layer, including: the processing layer formats and assembles the multimodal extended information according to the data assembly structure; after sending the interaction information in text format, the processing layer streams the assembled multimodal extended information.

2. The multimodal data dialogue robot interaction method according to claim 1, characterized in that, The model layer includes a pre-configured node knowledge base; The model layer returns interaction information and multimodal extension information to the processing layer based on the node knowledge base, the preset prompt information of the current dialogue node, and the user dialogue information.

3. The multimodal data dialogue robot interaction method according to claim 1, characterized in that, Before the processing layer sends the user dialogue information to the model layer, the model layer returns corresponding guidance information to the processing layer based on the prompt information; the processing layer then sends the guidance information to the interaction layer.

4. A configuration method for a multimodal data dialogue robot, characterized in that, Applications include chatbots with interaction, processing, and model layers, including: The chatbot receives prompts configured in the current chat node; The interaction layer responds to the user dialogue information input by the current dialogue node and establishes a session connection with the processing layer; The processing layer and the model layer complete the initialization configuration, and the user dialogue information is sent to the model layer; The model layer returns interaction information and multimodal extension information to the processing layer based on the prompt information and the user dialogue information. Specifically, the chatbot receives the function configuration information of the current dialogue node, which includes pre-response information. The model layer sends the pre-response information as interaction information to the processing layer. The function configuration information also includes a data assembly structure and preset variable configuration slots. After assigning values ​​to variables based on the pre-configured API interfaces in the variable configuration slots, the model layer returns the interaction information and the multimodal extension information to the processing layer. After generating the multimodal extension information, the model layer configures the corresponding data assembly structure for the function and returns the multimodal extension information and the corresponding data assembly structure to the processing layer. The function is a configuration item in the model interface of the model layer for each dialogue node. The configuration information of the function includes name, description, function category, API interface information, function request information, function response information, pre-response information, and data assembly structure. The processing layer assembles the interaction information and the multimodal extended information into their respective formats and then streams them to the interaction layer. This streaming transmission to the interaction layer includes: the processing layer assembling the multimodal extended information into its format according to the data assembly structure; and the processing layer sending the interaction information in text format followed by streaming the assembled multimodal extended information.

5. The configuration method for a multimodal data dialogue robot according to claim 4, characterized in that, The function configuration information also includes preset guidance information; Before the processing layer sends the user dialogue information to the model layer, the model layer returns corresponding guidance information to the processing layer based on the prompt information; The processing layer sends the guidance information to the interaction layer.

6. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to implement the method as described in any one of claims 1 to 3.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 3.

8. An electronic terminal, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1 to 3.

Images