Multimodal agent interaction method, device and equipment

Abstract

The multimodal intelligent agent conference information interaction method, device and equipment provided by the embodiment of the present disclosure include: sending a communication request to a signaling server, so that the signaling server obtains the first signaling information of the WebRTC client after receiving the communication request, and determines the target intelligent agent according to the communication request, and obtains the second signaling information of the target intelligent agent; after receiving the indication information sent by the signaling server that the communication connection relationship between the WebRTC client and the target intelligent agent is established, a data passage channel with the target intelligent agent is established, and in response to receiving the input data input by the target object, the input data is sent to the target intelligent agent through the data communication channel. Real-time interaction between users and intelligent agents through the WebRTC client is realized.

Description

Technical Field

[0001] The present invention relates to the field of computer technology and related technical fields, and in particular, to a method, device and equipment suitable for multimodal intelligent agent interaction. Background Art

[0002] With the rapid development of artificial intelligence technology, multimodal AI agent systems have gradually become a research hotspot. Traditional AI systems are mainly based on text interaction and have difficulty processing input and output of multiple modalities such as voice, image, and video. At the same time, most existing AI systems use asynchronous communication methods and cannot achieve real-time and smooth human-computer interaction experience.

[0003] WebRTC (Web Real-Time Communication) is a technology that supports web browsers for real-time voice or video conversations. It allows web applications to directly connect between browsers to achieve functions such as video conferencing and file transfer without the need for an intermediary. Due to the widespread use of this real-time communication technology, it has now been extended to other client applications that support WebRTC technology. However, WebRTC technology is currently mainly used for communication between people, and its application in combination with multimodal AI agents has not yet been fully explored. Summary of the invention

[0004] The embodiments described in this article provide a multimodal intelligent agent interaction method, device and equipment to solve the problems existing in the prior art.

[0005] In a first aspect, according to the content of the present disclosure, a multimodal agent interaction method is provided, which is applied to a WebRTC client, comprising:

[0006] Sending a communication request to a signaling server, so that after receiving the communication request, the signaling server obtains first signaling information of the WebRTC client, determines a target intelligent entity according to the communication request, and obtains second signaling information of the target intelligent entity, wherein the communication request includes a communication subject, the first signaling information includes at least first device information and first address information, and the second signaling information includes at least second device information and second address information;

[0007] After receiving the indication information of establishing the communication connection relationship between the WebRTC client and the target intelligent agent sent by the signaling server, a data communication channel with the target intelligent agent is established, and in response to receiving the input data input by the target object, the input data is sent to the target intelligent agent through the data communication channel, wherein the communication connection relationship is the communication connection relationship between the WebRTC client and the target intelligent agent, the data type of the input data includes at least text information, voice information, image information and video information, the data type supported by the WebRTC client is related to the first device information, the data type supported by the target intelligent agent is related to the second device information, and the indication information includes the data communication channel determined by the signaling server.

[0008] In some embodiments of the present disclosure, the method further includes:

[0009] Receive the response data sent by the target agent through the data communication channel to present the response data to the target object.

[0010] In some embodiments of the present disclosure, the method further includes:

[0011] The input data input by the target object is sent to the signaling server, so that the signaling server stores the input data in a dialog form.

[0012] In a second aspect, according to the content of the present disclosure, a multimodal agent interaction method is provided, which is applied to a signaling server, including:

[0013] After receiving the communication request sent by the WebRTC client, obtaining first signaling information of the WebRTC client, and determining the target agent according to the communication request, wherein the first signaling information at least includes first device information and first address information;

[0014] Acquire second signaling information of the target agent, wherein the second signaling information at least includes second device information and a second IP address;

[0015] According to the first signaling information and the second signaling information, the communication connection relationship and the data communication channel between the WebRTC client and the target intelligent entity are determined, and the communication connection relationship and the data communication channel between the WebRTC client and the target intelligent entity are established, and after the communication connection relationship between the WebRTC client and the target intelligent entity is established, an indication information of the establishment of the communication connection relationship between the WebRTC client and the target intelligent entity is sent to the WebRTC server, wherein the indication information includes the determined data communication channel.

[0016] In some embodiments of the present disclosure, determining the communication connection relationship and the data communication channel between the WebRTC client and the target agent according to the first signaling information and the second signaling information includes:

[0017] Determine and establish a communication connection relationship between the WebRTC client and the target intelligent entity according to the first IP address included in the first signaling information and the second IP address included in the second signaling information;

[0018] According to the first device information included in the first signaling information and the second device information included in the second signaling information, a data communication channel for the WebRTC client to communicate with the target agent is determined.

[0019] In some embodiments of the present disclosure, the method further includes:

[0020] Receiving input data input by the target object sent by the WebRTC client, and response data corresponding to the input data generated by the target agent;

[0021] Determine the association relationship between each input data and each response data and the session order of each input data;

[0022] According to the association relationship between each input data and each response data and the conversation sequence of each input data, each input data and each response data are stored in the form of context conversation.

[0023] In a third aspect, according to the content of the present disclosure, a multimodal agent interaction method is provided, which is applied to a target agent, comprising:

[0024] Receiving input data sent by the WebRTC client through a data communication channel;

[0025] After the input data is identified, response data corresponding to the input data is generated.

[0026] In some embodiments of the present disclosure, after identifying the input data, generating response data corresponding to the input data includes:

[0027] Based on a multimodal generative AI model, the input data is identified to generate response data of different data types;

[0028] Send response data to the WebRTC client and the signaling server.

[0029] In a fourth aspect, according to the content of the present disclosure, a multimodal agent interaction device is provided, comprising:

[0030] A communication request sending module, used to send a communication request to a signaling server, so that the signaling server obtains the first signaling information of the WebRTC client after receiving the communication request, determines the target intelligent entity according to the communication request, and obtains the second signaling information of the target intelligent entity, wherein the communication request includes a communication subject, the first signaling information includes at least first device information and first address information, and the second signaling information includes at least second device information and second address information;

[0031] A data sending module is used to establish a data communication channel with the target intelligent entity after receiving the indication information of establishing the communication connection relationship between the WebRTC client and the target intelligent entity sent by the signaling server, and in response to receiving the input data input by the target object, send the input data to the target intelligent entity through the data communication channel, wherein the communication connection relationship is the communication connection relationship between the WebRTC client and the target intelligent entity, the data type of the input data includes at least text information, voice information, image information and video information, the data type supported by the WebRTC client is related to the first device information, the data type supported by the target intelligent entity is related to the second device information, and the indication information includes the data communication channel determined by the signaling server.

[0032] In a fifth aspect, according to the present disclosure, there is provided a computer device, including:

[0033] one or more processors;

[0034] a storage device for storing one or more programs,

[0035] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of the first aspect, the second aspect or the third aspect.

[0036] The multimodal agent interaction method, device and medium provided by the embodiment of the present disclosure send a communication request to the signaling server, so that after receiving the communication request, the signaling server obtains the first signaling information of the WebRTC client, determines the target agent according to the communication request, and obtains the second signaling information of the target agent; after receiving the indication information of the establishment of the communication connection relationship sent by the signaling server, in response to receiving the input data input by the target object, the input data is sent to the target agent through the data communication channel. On the one hand, the user can interact with the agent in real time through the WebRTC client, and realize functions such as multi-party conference and collaboration. On the other hand, the data communication channel between the WebRTC client and the target agent allows the direct transmission of binary data, reducing the overhead of data conversion and encoding and decoding.

[0037] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly described below. It should be noted that the drawings described below only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure, wherein:

[0039] Figure 1 is a flowchart of a multimodal agent interaction method provided by an embodiment of the present disclosure;

[0040] Figure 2 is an interaction schematic diagram of a multimodal agent interaction method provided by an embodiment of the present disclosure;

[0041] Figure 3 is a flowchart of another multimodal agent interaction method provided by an embodiment of the present disclosure;

[0042] Figure 4 is a flowchart of another multimodal agent interaction method provided by an embodiment of the present disclosure;

[0043] Figure 5 is a structural schematic diagram of a multimodal agent interaction device provided by an embodiment of the present disclosure;

[0044] Figure 6 It is a structural diagram of a computer device provided in an embodiment of the present disclosure.

[0045] In the drawings, reference numerals with the same last two digits correspond to the same elements. It should be noted that the elements in the drawings are schematic and not drawn to scale. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work also fall within the scope of protection of the present disclosure.

[0047] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by a person skilled in the art to which the subject matter of the present disclosure belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the specification and the relevant art, and will not be interpreted in an idealized or overly formal form unless otherwise explicitly defined herein. As used herein, a statement that two or more parts are "connected" or "coupled" together shall mean that the parts are joined together directly or through one or more intermediate components.

[0048] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase "embodiments" in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0049] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists, A and B exist at the same time, and B exists. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0050] Furthermore, in all embodiments of the present disclosure, terms such as “first” and “second” are used only to distinguish one component (or a part of a component) from another component (or another part of a component).

[0051] In the description of the present application, unless otherwise specified, “plurality” means more than two (including two), and similarly, “plurality groups” means more than two (including two).

[0052] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0053] Based on the problems existing in the prior art, the present disclosure provides a multimodal agent interaction method, which is applied to a WebRTC client. Figure 1 is a flow chart of a multimodal agent interaction method provided by an embodiment of the present disclosure, Figure 2 is an interactive schematic diagram of a multimodal agent interaction method provided by an embodiment of the present disclosure, combined with Figure 1 and Figure 2 , the multimodal agent interaction method includes:

[0054] S110, sending a communication request to the signaling server, so that after receiving the communication request, the signaling server obtains the first signaling information of the WebRTC client, determines the target intelligent agent according to the communication request, and obtains the second signaling information of the target intelligent agent.

[0055] The communication request includes a communication subject, the first signaling information includes at least first device information and first address information, and the second signaling information includes at least second device information and second address information.

[0056] In a specific implementation, a WebRTC communication connection relationship is established between the WebRTC client and the signaling server, and a WebRTC communication connection relationship is established between the signaling server and multiple intelligent entities. When the WebRTC client sends a communication request to the signaling server, the signaling server obtains the first signaling information of the WebRTC client through the WebRTC communication protocol. The first signaling information includes first device information and first address information, wherein the first device information includes the data types supported by the WebRTC client, the ability of the WebRTC client to process data corresponding to different data types, and the number of WebRTC communication channels, etc., and the first address information includes the communication IP address of the WebRTC client.

[0057] In a specific implementation, a signaling server stores second signaling information of multiple agents, and the second signaling information of the agents includes second device information and second address information. The agents register the second signaling information to the signaling server through the WebRTC communication protocol, wherein the second device information includes the data types supported by the agents, the ability of the agents to process data corresponding to different data types, and the number of communication channels of the agents, etc., and the second address information includes the communication IP address of the agents.

[0058] In a specific implementation, when the target object sends a communication request to the signaling server through the WebRTC client, the signaling server determines the target intelligent entity based on the communication subject included in the communication request, and after determining the target intelligent entity, obtains the second signaling information of the target intelligent entity stored in the signaling server.

[0059] In a specific implementation, the specific process of the signaling server determining the target intelligent agent based on the communication subject included in the communication request includes: determining the target professional field to which the communication request belongs based on the communication subject; obtaining the label identification of each intelligent agent, the response efficiency of each intelligent agent and the status information of each intelligent agent, wherein the label identification is used to identify the professional field to which the intelligent agent belongs, and the status information includes an idle state and a busy state; determining the target intelligent agent based on the matching degree between the label content corresponding to the label identification of each intelligent agent and the target professional field, the response efficiency of each intelligent agent and the status information of each intelligent agent.

[0060] Among them, the target intelligent agent is determined according to the matching degree between the label content corresponding to the label identification of each intelligent agent and the target professional field, the response efficiency of each intelligent agent and the status information of each intelligent agent, including: sequentially obtaining the matching value corresponding to the matching degree between the label content corresponding to the label identification of each intelligent agent and the target professional field, the efficiency value corresponding to the response efficiency of each intelligent agent and the status value corresponding to the status information of each intelligent agent; the scoring parameter of each intelligent agent is determined according to the matching value corresponding to the matching degree between the label content corresponding to the label identification of each intelligent agent and the target professional field, the efficiency value corresponding to the response efficiency of each intelligent agent and the status value corresponding to the status information of each intelligent agent; after the scoring parameters of each intelligent agent are sorted in forward order, the first target number of intelligent agents are selected to form a target intelligent agent group, wherein the target number is the same as the number of participants.

[0061] Specifically, when the signaling server receives a communication request sent by the target object, it first determines the target professional field to which the meeting belongs based on the communication topic included in the communication request, and then obtains the label identification of each intelligent agent, the response efficiency of each intelligent agent and the status information of each intelligent agent, and then determines the matching value corresponding to the matching degree between the label content corresponding to the label identification of each intelligent agent and the target professional field, the efficiency value corresponding to the response efficiency of each intelligent agent and the status value corresponding to the status information of each intelligent agent, and then obtains the scoring parameters of each intelligent agent through the matching value, efficiency value and status value, and finally determines the target intelligent agent through the scoring parameters.

[0062] Exemplarily, the agents stored in the signaling server include agent 1, agent 2, agent 3, ..., agent N, the tag identifier corresponding to agent 1 includes tag identifier 1, tag identifier 2 and tag identifier 3, the efficiency value corresponding to the response efficiency of agent 1 is a%, and the status value corresponding to the status information of agent 1 is 1, the second tag identifier corresponding to agent 2 includes tag identifier 1, tag identifier 3 and tag identifier 4, the efficiency value corresponding to the response efficiency of agent 2 is b%, and the status value corresponding to the status information of agent 1 is 1, ..., the tag identifier corresponding to agent N includes tag identifier 1, tag identifier 2 and tag identifier i, the efficiency value corresponding to the response efficiency of agent N is c%, and the status value corresponding to the status information of agent N is 1. At this time, first, the label content corresponding to the label identifier of agent 1 is matched with the target professional field, that is, if the label identifier of agent 1 includes label identifier 1, label identifier 2 and label identifier 3, the label content corresponding to label identifier 1 is content 1, the label content corresponding to label identifier 2 is content 2, and the label content corresponding to label identifier 3 is content 3. Content 1 is matched with the target professional field in turn to obtain a first matching value, content 2 is matched with the target professional field to obtain a second matching value, and content 3 is matched with the target professional field to obtain a third matching value. The first matching value, the second matching value and the third matching value are then averaged to obtain the matching value of the label content corresponding to the label identifier of agent 1 and the target professional field, and so on, so as to obtain the matching value of the label content corresponding to the label identifier of other agents and the target professional field; then, the matching value, efficiency value and state value of agent 1 are weighted and summed to obtain the scoring parameters of each agent.

[0063] Exemplarily, the scoring parameters of Agent 1 satisfy:

[0064] ,

[0065] in, is the weighted value of the matching value, is the weighted value of the efficiency value, is the weighted value of the state value, is the first matching value between content 1 and the target professional field, is the second matching value between content 2 and the target professional field, is the third matching value between content 3 and the target professional field, is the matching value corresponding to the matching degree between the tag content corresponding to each tag identification included in agent 1 and the target professional field, is the efficiency value corresponding to the response efficiency of agent 1, is the state value corresponding to the state information of agent 1.

[0066] If the state information of the agent is an idle state, the state value corresponding to the state information of the agent is 1; if the state information of the agent is a busy state, the state value corresponding to the state information of the agent is 0.

[0067] After obtaining the scoring parameters of each agent, the scoring parameters of each agent are sorted in forward order, that is, after sorting the agent scoring parameters from large to small, the agents with the top target number in the forward sorting are selected to form the target agent, ensuring that the target agents included in the determined target agent group can have a better reference basis for the response data generated based on the input data when communicating with the target object.

[0068] It should be noted that in the embodiment of the present disclosure, the target intelligent agent for communicating with WebRTC is determined to be one by way of example. In other possible implementations, the target intelligent agent for communicating with WebRTC may also be multiple. The embodiment of the present disclosure does not specifically limit this. When the number of target intelligent agents for communicating with WebRTC is multiple, the WebRTC client establishes a point-to-point communication connection with each target intelligent agent respectively.

[0069] In a specific implementation, the target object can access the signaling server through a WebRTC client such as a Web browser. After receiving the communication request sent by the WebRTC client, the signaling server obtains the first signaling information of the WebRTC client, and determines the data type supported by the WebRTC client based on the first device information included in the first signaling information of the WebRTC client. In addition, after determining the target intelligent agent, the signaling server determines the data type supported by the target intelligent agent based on the second device information included in the second signaling information of the target intelligent agent obtained, and determines the data communication channel between the target intelligent agent and the WebRTC client based on the first device information included in the first signaling information of the WebRTC client obtained and the second device information included in the second signaling information of the target intelligent agent.

[0070] S120. After receiving the indication information sent by the signaling server indicating the establishment of the communication connection relationship between the WebRTC client and the target intelligent agent, a data communication channel with the target intelligent agent is established, and in response to receiving the input data input by the target object, the input data is sent to the target intelligent agent through the data communication channel.

[0071] Among them, the communication connection relationship is the communication connection relationship between the WebRTC client and the target intelligent entity, the data type of the input data includes at least text information, voice information, image information and video information, the data type supported by the WebRTC client is related to the first device information, the data type supported by the target intelligent entity is related to the second device information, and the indication information includes the data communication channel determined by the signaling server.

[0072] The signaling server obtains the first signaling information of the WebRTC client and the second signaling information of the target intelligent entity, determines the data communication channel between the WebRTC client and the target intelligent entity according to the first device information included in the first signaling information of the WebRTC client and the second device information included in the second signaling information of the target intelligent entity, and determines and establishes the communication connection relationship between the WebRTC client and the target intelligent entity according to the first address information included in the first signaling information of the WebRTC client and the second address information included in the second signaling information of the target intelligent entity.

[0073] After receiving the input data input by the target object, the WebRTC client establishes a data communication channel with the target agent according to the data communication channel between the target agent and the WebRTC client determined by the signaling server, sends the input data input by the target object to the target agent through the data communication channel, and receives response data corresponding to each input data sent by the target agent through the data communication channel, thereby realizing point-to-point communication between the WebRTC client and the target agent.

[0074] The multimodal agent interaction method provided by the embodiment of the present disclosure sends a communication request to a signaling server, so that after receiving the communication request, the signaling server obtains the first signaling information of the WebRTC client, determines the target agent according to the communication request, and obtains the second signaling information of the target agent; after receiving the indication information of the successful establishment of the communication connection relationship sent by the signaling server, in response to receiving the input data input by the target object, the input data is sent to the target agent through the data communication channel. On the one hand, the user can interact with the agent in real time through the WebRTC client, and realize functions such as multi-party conference and collaboration. On the other hand, the data communication channel between the WebRTC client and the target agent allows the direct transmission of binary data, reducing the overhead of data conversion and encoding and decoding.

[0075] On the basis of the above-mentioned embodiments, the multimodal agent interaction method provided by the embodiments of the present disclosure further includes: receiving response data sent by the target agent through the data communication channel to present the response data to the target object.

[0076] After the signaling server determines the data communication channel and communication connection relationship between the target intelligent agent and the WebRTC client, the signaling server establishes a communication connection relationship between the target intelligent agent and the WebRTC client, and after the communication connection relationship between the target intelligent agent and the WebRTC client is successfully established, the signaling server sends an indication of the establishment of the communication connection relationship to the WebRTC client. After receiving the indication of the establishment of the communication connection relationship, the WebRTC client establishes a data communication channel between the WebRTC client and the target intelligent agent according to the data communication channel between the target intelligent agent and the WebRTC client determined by the signaling server, and sends the input data entered by the target object in the WebRTC client through the data communication channel. At this time, after receiving the input data sent by the WebRTC client, the target intelligent agent generates response data and sends it to the WebRTC client, so that the WebRTC client presents the response data to the target object on the display terminal.

[0077] It should be noted that the data types of the input data entered by the target object in the WebRTC client include text type, voice type, image type and video type, and the data type supported by the WebRTC client is related to the first device information, and the data type supported by the target agent is related to the second device information. After the signaling server establishes a communication connection relationship between the WebRTC client and the target agent, the signaling server will send the input data of the data type supported by the WebRTC client to the target agent through the data communication channel. After receiving the input data sent by the WebRTC client, the target agent converts the input data into the data type supported by the target agent and generates response data corresponding to each input data. The data type of the response data is the data type supported by the target agent.

[0078] On the basis of the above-mentioned embodiments, the multimodal agent interaction method provided by the embodiments of the present disclosure further includes: sending input data input by the target object to a signaling server, so that the signaling server stores the input data in the form of a conversation.

[0079] In addition, in the method provided by the embodiment of the present disclosure, the WebRTC client also sends input data to the signaling server, so that the signaling server generates a context session according to the input data and the response data sent by the target intelligent agent.

[0080] Based on the above embodiments, the present disclosure also provides another multimodal agent interaction method, which is applied to a signaling server. Figure 3 is a flowchart of another multimodal agent interaction method provided by an embodiment of the present disclosure, combined with Figure 2 and Figure 3 , the multimodal agent interaction method includes:

[0081] S210. After receiving a communication request sent by a WebRTC client, obtain first signaling information of the WebRTC client, and determine a target agent according to the communication request.

[0082] The first signaling information at least includes first device information and first address information.

[0083] After receiving the communication request sent by the WebRTC client, the signaling server obtains the first signaling information of the WebRTC client, determines the data type supported by the WebRTC client based on the first device information included in the first signaling information of the WebRTC client, and determines the target intelligent agent according to the communication request.

[0084] S220, obtaining second signaling information of the target agent.

[0085] The second signaling information at least includes second device information and a second IP address.

[0086] After determining the target intelligent agent, the signaling server obtains the second device information included in the second signaling information of the target intelligent agent.

[0087] S230. Determine the communication connection relationship and data communication channel between the WebRTC client and the target intelligent entity based on the first signaling information and the second signaling information, and establish the communication connection relationship and data communication channel between the WebRTC client and the target intelligent entity. After the communication connection relationship between the WebRTC client and the target intelligent entity is successfully established, send indication information of the establishment of the communication connection relationship between the WebRTC client and the target intelligent entity to the WebRTC server.

[0088] The indication signal includes a determined data communication channel.

[0089] As a specific implementation, the communication connection relationship and data communication channel between the WebRTC client and the target intelligent entity are determined according to the first signaling information and the second signaling information, including: determining and establishing the communication connection relationship between the WebRTC client and the target intelligent entity according to the first IP address included in the first signaling information and the second IP address included in the second signaling information; determining the data communication channel for communication between the WebRTC client and the target intelligent entity according to the first device information included in the first signaling information and the second device information included in the second signaling information.

[0090] In a specific implementation, in the specific process of determining the data communication channel for communication between the WebRTC client and the target intelligent entity based on the first device information included in the first signaling information and the second device information included in the second signaling information, the data communication channel for the WebRTC client to send input data to the target intelligent entity is determined based on the first device information included in the first signaling information, that is, based on the data type supported by the WebRTC client, and the data communication channel for the WebRTC client to receive response data sent by the target intelligent entity is determined based on the second device information included in the second signaling information, that is, based on the data type supported by the target intelligent entity.

[0091] In addition, it should be noted that the indication information sent by the signaling server for establishing a communication connection relationship between the WebRTC client and the target intelligent entity includes a data communication channel determined by the signaling server. At this time, after receiving the indication information sent by the signaling server for establishing a communication connection relationship between the WebRTC client and the target intelligent entity, the WebRTC client establishes a data communication channel with the target intelligent entity according to the data communication channel included in the indication information.

[0092] The multimodal agent interaction method provided by the embodiment of the present disclosure obtains the first signaling information of the WebRTC client after receiving the communication request sent by the WebRTC client, and determines the target agent according to the communication request; and obtains the second signaling information of the target agent; establishes a communication connection relationship and a data communication channel between the WebRTC client and the target agent according to the first signaling information and the second signaling information, and after the communication connection relationship between the WebRTC client and the target agent is successfully established, sends an indication information of the successful establishment of the communication connection relationship to the WebRTC server and the target agent, so as to establish a point-to-point communication connection relationship between the WebRTC client and the target agent based on the signaling server, thereby realizing real-time interaction between the user and the agent through the WebRTC client, and realizing functions such as multi-party conferencing and collaboration.

[0093] On the basis of the above-mentioned embodiments, the multimodal agent interaction method provided by the embodiments of the present disclosure further includes: receiving input data input by the target object sent by the WebRTC client, and response data generated by the target agent corresponding to the input data; determining the association relationship between each input data and each response data and the conversation order of each input data; storing each input data and each response data in the form of a context conversation according to the association relationship between each input data and each response data and the conversation order of each input data.

[0094] In addition, in the multimodal agent interaction method provided by the embodiment of the present disclosure, the signaling server obtains input data from the WebRTC client and response data from the target agent, and then forms and stores session text data by associating the input data obtained from the WebRTC client and the response data obtained from the target agent, as well as the session order of the input data entered by the WebRTC client, to ensure the accuracy of the subsequent response to the communication request sent by the WebRTC client, thereby determining the target agent.

[0095] On the basis of the above embodiments, the present disclosure also provides another multimodal agent interaction method, which is applied to a signaling server. Figure 4 is a flowchart of another multimodal agent interaction method provided by the embodiment of the present disclosure, combined with Figure 2 and Figure 4 , the multimodal agent interaction method includes:

[0096] S310: Receive input data sent by the WebRTC client through the data communication channel.

[0097] Since the second signaling information of the agent is stored in the signaling server, after receiving the communication request sent by the WebRTC client, the signaling server obtains the first signaling information of the WebRTC client from the WebRTC client on the one hand, and directly obtains the stored second signaling information of the target agent on the other hand. After the signaling server establishes a point-to-point data communication channel between the WebRTC client and the target agent based on the first signaling information and the second signaling information, the target agent receives the input data sent by the WebRTC client through the data communication channel.

[0098] S320: After identifying the input data, generate response data corresponding to the input data.

[0099] In a specific implementation, after the input data is identified, response data corresponding to the input data is generated, including: based on a multimodal generative AI model, the input data is identified to generate response data of different data types; and the response data is sent to the WebRTC client and the signaling server.

[0100] Specifically, the target agent includes a multimodal input processing module and a multimodal output generation module. The multimodal input processing module of the target agent parses the input data and submits the processed input data to the model for reasoning based on the settings. The multimodal output generation module converts the response into the corresponding output form, and then transmits it back to the WebRTC client through the data communication channel and presents it to the user.

[0101] The multimodal agent interaction method provided by the embodiment of the present disclosure receives the indication information of successful establishment of the communication connection relationship sent by the signaling server, receives the input data sent by the WebRTC client through the data communication channel, and generates response data corresponding to the input data after identifying the input data. The user can interact with the agent in real time through the WebRTC client, and realize functions such as multi-party conference and collaboration.

[0102] It should be noted that in the agent multimodal interaction method provided in the above embodiment, during the data interaction between the WebRTC client and the target agent, the WebRTC DataChannel is used for efficient binary data transmission to reduce delays, and a streaming processing mode is adopted to achieve real-time processing and response of audio and video data, and the audio and video encoding and decoding capabilities of WebRTC are utilized to optimize transmission efficiency.

[0103] In addition, DTLS-SRTP is used to encrypt the communication between the WebRTC client and the target agent, achieving end-to-end data encryption between the WebRTC client and the target agent to protect user privacy.

[0104] Based on the above embodiments, the present disclosure also provides a multi-modal agent interaction device. Figure 5 is a schematic diagram of the structure of a multimodal agent interaction device provided by an embodiment of the present disclosure, such as Figure 5 As shown, the multimodal intelligent agent interaction device comprises:

[0105] The communication request sending module 510 is used to send a communication request to the signaling server, so that the signaling server obtains the first signaling information of the WebRTC client after receiving the communication request, and determines the target intelligent agent according to the communication request, and obtains the second signaling information of the target intelligent agent, wherein the communication request includes a communication subject, the first signaling information includes at least the first device information and the first address information, and the second signaling information includes at least the second device information and the second address information;

[0106] The data sending module 520 is used to establish a data communication channel with the target intelligent entity after receiving the indication information of establishing the communication connection relationship between the WebRTC client and the target intelligent entity sent by the signaling server, and in response to receiving the input data input by the target object, send the input data to the target intelligent entity through the data communication channel, wherein the communication connection relationship is the communication connection relationship between the WebRTC client and the target intelligent entity, the data type of the input data includes at least text information, voice information, image information and video information, the data type supported by the WebRTC client is related to the first device information, the data type supported by the target intelligent entity is related to the second device information, and the indication information includes the data communication channel determined by the signaling server.

[0107] The multimodal agent interaction device provided by the embodiment of the present disclosure sends a communication request to the signaling server, so that after receiving the communication request, the signaling server obtains the first signaling information of the WebRTC client, determines the target agent according to the communication request, and obtains the second signaling information of the target agent; after receiving the indication information of the successful establishment of the communication connection relationship sent by the signaling server, in response to receiving the input data input by the target object, the input data is sent to the target agent through the data communication channel. On the one hand, the user can interact with the agent in real time through the WebRTC client, and realize functions such as multi-party conference and collaboration. On the other hand, the data communication channel between the WebRTC client and the target agent allows the direct transmission of binary data, reducing the overhead of data conversion and encoding and decoding.

[0108] In a specific embodiment, it also includes:

[0109] Receive response data sent by the target agent through the data communication channel to present the response data to the target object.

[0110] In a specific embodiment, it also includes:

[0111] The input data input by the target object is sent to the signaling server so that the signaling server stores the input data in a dialog form.

[0112] On the basis of the above embodiments, the present disclosure also provides another multimodal agent interaction device, the multimodal agent interaction device comprising:

[0113] After receiving a communication request sent by a WebRTC client, first signaling information of the WebRTC client is obtained, and a target agent is determined according to the communication request, wherein the first signaling information includes at least first device information and first address information;

[0114] Acquire second signaling information of the target agent, wherein the second signaling information at least includes second device information and a second IP address;

[0115] According to the first signaling information and the second signaling information, the communication connection relationship and the data communication channel between the WebRTC client and the target intelligent entity are determined, and the communication connection relationship and the data communication channel between the WebRTC client and the target intelligent entity are established, and after the communication connection relationship between the WebRTC client and the target intelligent entity is successfully established, an indication information of the establishment of the communication connection relationship between the WebRTC client and the target intelligent entity is sent to the WebRTC server and the target intelligent entity, wherein the indication information includes the determined data communication channel.

[0116] In a specific implementation, determining the communication connection relationship and data communication channel between the WebRTC client and the target agent according to the first signaling information and the second signaling information includes:

[0117] Determine and establish a communication connection relationship between the WebRTC client and the target intelligent entity according to the first IP address included in the first signaling information and the second IP address included in the second signaling information;

[0118] According to the first device information included in the first signaling information and the second device information included in the second signaling information, a data communication channel for communication between the WebRTC client and the target intelligent agent is determined.

[0119] In a specific embodiment, it also includes:

[0120] Receive input data sent by the target object from the WebRTC client, and response data generated by the target agent corresponding to the input data;

[0121] Determine the association between each input data and each response data and the session order of each input data;

[0122] According to the association relationship between each input data and each response data and the conversation sequence of each input data, each input data and each response data are stored in the form of context conversation.

[0123] On the basis of the above embodiments, the present disclosure also provides another multimodal agent interaction device, the multimodal agent interaction device comprising:

[0124] Receive input data sent by the WebRTC client through the data communication channel;

[0125] After the input data is recognized, response data corresponding to the input data is generated.

[0126] In a specific implementation, after the input data is identified, response data corresponding to the input data is generated, including:

[0127] Based on the multimodal generative AI model, the input data is identified and response data of different data types is generated;

[0128] Send response data to the WebRTC client and signaling server.

[0129] The present application also provides a computer device. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0130] The computer device includes a memory 610 and a processor 620 that are connected to each other through a system bus. It should be noted that the figure only shows a computer device with components 610-620, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field-programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0131] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0132] The memory 610 includes at least one type of readable storage medium, and the readable storage medium includes a non-volatile memory or a volatile memory, such as a flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory, etc.), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc., and the RAM may include a static RAM or a dynamic RAM. In some embodiments, the memory 610 may be an internal storage unit of a computer device, such as a hard disk or a memory of the computer device. In other embodiments, the memory 610 may also be an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, or a flash card (Flash Card), etc., equipped on the computer device. Of course, the memory 610 may also include both an internal storage unit of the computer device and an external storage device thereof. In this embodiment, the memory 610 is generally used to store an operating system and various application software installed on the computer device, such as the program code of the above method, etc. In addition, the memory 610 may also be used to temporarily store various data that have been output or are to be output.

[0133] The processor 620 is generally used to perform the overall operation of the computer device. In this embodiment, the memory 610 is used to store program codes or instructions, the program code includes computer operation instructions, and the processor 620 is used to execute the program codes or instructions stored in the memory 610 or process data, such as running the program code of the above method.

[0134] In this article, the bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus system can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0135] Another embodiment of the present application also provides a computer-readable medium, which may be a computer-readable signal medium or a computer-readable medium. A processor in a computer reads a computer-readable program code stored in the computer-readable medium, so that the processor can execute the functional actions specified in each step or a combination of steps in the above method; and generate a device for implementing the functional actions specified in each block or a combination of blocks in the block diagram.

[0136] Computer-readable media include but are not limited to electronic, magnetic, optical, electromagnetic, infrared memory or semiconductor systems, devices or apparatuses, or any appropriate combination of the foregoing, the memory is used to store program codes or instructions, the program codes include computer operating instructions, and the processor is used to execute the program codes or instructions of the above methods stored in the memory.

[0137] For the definitions of memory and processor, please refer to the description of the aforementioned computer device embodiment and will not be repeated here.

[0138] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0139] Each functional unit or module in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0140] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., and other media that can store program codes.

[0141] Unless the context clearly indicates otherwise, the singular form of the words used herein and in the appended claims includes the plural and vice versa. Thus, when referring to the singular, the plural form of the corresponding term is generally included. Similarly, the words "comprise" and "include" are to be interpreted as inclusive rather than exclusive. Likewise, the terms "include" and "or" should be interpreted as inclusive unless such interpretation is expressly prohibited herein. Where the term "example" is used herein, particularly when it is located after a group of terms, the "example" is merely exemplary and illustrative and should not be considered exclusive or comprehensive.

[0142] Further aspects and scopes of adaptability become apparent from the description provided herein. It should be understood that various aspects of the present application can be implemented individually or in combination with one or more other aspects. It should also be understood that the description and specific embodiments herein are intended for purposes of illustration only and are not intended to limit the scope of the present application.

[0143] Several embodiments of the present disclosure are described in detail above, but it is obvious that those skilled in the art can make various modifications and variations to the embodiments of the present disclosure without departing from the spirit and scope of the present disclosure. The protection scope of the present disclosure is defined by the attached claims.

Claims

1. A multimodal agent interaction method, applied to a WebRTC client, characterized in that: include: Sending a communication request to a signaling server, so that after receiving the communication request, the signaling server obtains first signaling information of the WebRTC client, determines a target intelligent entity according to the communication request, and obtains second signaling information of the target intelligent entity, wherein the communication request includes a communication subject, the first signaling information includes at least first device information and first address information, and the second signaling information includes at least second device information and second address information; After receiving the indication information of establishing the communication connection relationship between the WebRTC client and the target intelligent agent sent by the signaling server, a data communication channel with the target intelligent agent is established, and in response to receiving the input data input by the target object, the input data is sent to the target intelligent agent through the data communication channel, wherein the communication connection relationship is the communication connection relationship between the WebRTC client and the target intelligent agent, the data type of the input data includes at least text information, voice information, image information and video information, the data type supported by the WebRTC client is related to the first device information, the data type supported by the target intelligent agent is related to the second device information, and the indication information includes the data communication channel determined by the signaling server.

2. The method according to claim 1, characterized in that The method further comprises: Receive the response data sent by the target agent through the data communication channel to present the response data to the target object.

3. The method according to claim 1, characterized in that The method further comprises: The input data input by the target object is sent to the signaling server, so that the signaling server stores the input data in a dialog form.

4. A multimodal agent interaction method, applied to a signaling server, characterized in that: include: After receiving a communication request sent by a WebRTC client, obtaining first signaling information of the WebRTC client, and determining a target agent according to the communication request, wherein the first signaling information includes at least first device information and first address information; Acquire second signaling information of the target agent, wherein the second signaling information at least includes second device information and a second IP address; According to the first signaling information and the second signaling information, the communication connection relationship and the data communication channel between the WebRTC client and the target intelligent entity are determined, and the communication connection relationship and the data communication channel between the WebRTC client and the target intelligent entity are established, and after the communication connection relationship between the WebRTC client and the target intelligent entity is established, indication information of the establishment of the communication connection relationship between the WebRTC client and the target intelligent entity is sent to the WebRTC client, wherein the indication information includes the determined data communication channel.

5. The method according to claim 4, characterized in that The determining, according to the first signaling information and the second signaling information, a communication connection relationship and a data communication channel between the WebRTC client and the target intelligent agent includes: Determine and establish a communication connection relationship between the WebRTC client and the target intelligent entity according to the first IP address included in the first signaling information and the second IP address included in the second signaling information; According to the first device information included in the first signaling information and the second device information included in the second signaling information, a data communication channel for the WebRTC client to communicate with the target agent is determined.

6. The method according to claim 4, characterized in that The method further comprises: Receiving input data input by the target object sent by the WebRTC client, and response data corresponding to the input data generated by the target agent; Determine the association relationship between each input data and each response data and the session order of each input data; According to the association relationship between each input data and each response data and the conversation sequence of each input data, each input data and each response data are stored in the form of context conversation.

7. A multimodal agent interaction device, characterized in that: include: A communication request sending module, used to send a communication request to a signaling server, so that the signaling server obtains first signaling information of the WebRTC client after receiving the communication request, determines a target intelligent entity according to the communication request, and obtains second signaling information of the target intelligent entity, wherein the communication request includes a communication subject, the first signaling information includes at least first device information and first address information, and the second signaling information includes at least second device information and second address information; A data sending module is used to establish a data communication channel with the target intelligent entity after receiving the indication information of establishing the communication connection relationship between the WebRTC client and the target intelligent entity sent by the signaling server, and in response to receiving the input data input by the target object, send the input data to the target intelligent entity through the data communication channel, wherein the communication connection relationship is the communication connection relationship between the WebRTC client and the target intelligent entity, the data type of the input data includes at least text information, voice information, image information and video information, the data type supported by the WebRTC client is related to the first device information, the data type supported by the target intelligent entity is related to the second device information, and the indication information includes the data communication channel determined by the signaling server.

8. A computer device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 3 or 4 to 6.

Images