Multi-modal information interaction method and device, intelligent agent, equipment and medium

Abstract

The invention provides a multi-modal information interaction method and device, an intelligent agent, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical field of large models and human-computer interaction. According to the specific implementation scheme, intention recognition is conducted on a media resource request from a terminal, an intention recognition result is obtained, and the intention recognition result represents whether the media resource request hits a preset processing mode or not; calling a first multi-mode processing module to determine a media resource address corresponding to the media resource request in response to the media resource request hitting a preset processing mode; and calling a second multi-mode processing module to render the media resource in the media resource address, and outputting the rendered media stream to the terminal.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and more particularly to the fields of large models and human-computer interaction technology. More specifically, the present disclosure provides a multimodal information interaction method, apparatus, intelligent agent, electronic device, storage medium, and computer program product. Background Art

[0002] With the gradual popularization of large models and intelligent agent applications, people's demand for multimodal interaction of intelligent agents is increasing. However, currently, intelligent agents generally still use voice and text interaction methods, and other media formats are output by providing links. Summary of the Invention

[0003] The present disclosure provides a multimodal information interaction method, apparatus, intelligent agent, electronic device, storage medium, and computer program product.

[0004] According to a first aspect, a multimodal information interaction method is provided. The method includes: performing intent recognition on a media resource request from a terminal to obtain an intent recognition result, where the intent recognition result indicates whether the media resource request hits a predetermined processing method; in response to the media resource request hitting the predetermined processing method, invoking a first multimodal processing module to determine a media resource address corresponding to the media resource request; and invoking a second multimodal processing module to render the media resource in the media resource address and output the rendered media stream to the terminal.

[0005] According to a second aspect, an intelligent agent is provided, which is configured to execute the above multimodal information interaction method.

[0006] According to a third aspect, a multimodal information interaction apparatus is provided. The apparatus includes: an intent recognition module for performing intent recognition on a media resource request from a terminal to obtain an intent recognition result, where the intent recognition result indicates whether the media resource request hits a predetermined processing method; a first invocation module for, in response to the received media resource request from the terminal hitting the predetermined processing method, invoking a first multimodal processing module to determine a media resource address corresponding to the media resource request; and a second invocation module for invoking a second multimodal processing module to render the media resource in the media resource address and output the rendered media stream to the terminal.

[0007] According to a fourth aspect, an electronic device is provided, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method provided by the present disclosure.

[0008] According to a fifth aspect, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method provided according to the present disclosure.

[0009] According to a sixth aspect, there is provided a computer program product including a computer program, the computer program being stored on at least one of a readable storage medium and an electronic device, and the computer program implementing the method provided according to the present disclosure when executed by a processor.

[0010] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them:

[0012] Figure 1 is a schematic diagram of an exemplary system architecture to which the multi-modal information interaction method and apparatus according to an embodiment of the present disclosure can be applied;

[0013] Figure 2 is a flowchart of a multi-modal information interaction method according to an embodiment of the present disclosure;

[0014] Figure 3 is a schematic diagram of a system to which the multi-modal information interaction method according to an embodiment of the present disclosure can be applied;

[0015] Figure 4 is a schematic diagram of a first multi-modal processing module according to an embodiment of the present disclosure;

[0016] Figure 5 is a schematic diagram of a second multi-modal processing module according to an embodiment of the present disclosure;

[0017] Figure 6 is a flowchart of a multi-modal information interaction method according to another embodiment of the present disclosure;

[0018] Figure 7 is a block diagram of a multi-modal information interaction apparatus according to an embodiment of the present disclosure; and

[0019] Figure 8 is a block diagram of an electronic device for a multi-modal information interaction method according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] The following describes exemplary embodiments of the present disclosure with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, descriptions of well-known functions and structures are omitted in the following description for clarity and conciseness.

[0021] With the gradual popularization of large models and agent applications, people's demand for multi-modal interaction of agents is increasing. People hope that the agent can directly output multi-modal content (pictures, videos, audio, web pages, documents, maps) according to the user's request and directly display and present it. For example: when the user inputs "Please play a song by a certain singer", "Please play the movie XXX", "Please open my XX summary PPT", the corresponding media content will be directly played on the user terminal, rather than giving some text descriptions or media link information.

[0022] However, the current agents generally use the basic intelligent interaction methods of voice and text. The agent recognizes the user's input voice, outputs text after being processed by the large model, or outputs voice after text-to-speech conversion, and returns it to the user side. Such an agent only supports the output of text and audio formats, and other media formats are output by providing links.

[0023] The current agent interaction method requires various media format renderings to be implemented on the user side, or other tools to be called on the user side to open the media. This requires the user to make multiple jumps, affecting the continuous interaction experience. In addition, it is necessary to integrate various media tool plugins on the user side, which greatly increases the volume of the user side SDK (Software Development Kit), making it unfriendly to user access, especially in scenarios such as browser and applet access, increasing the user access cost.

[0024] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information involved all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0025] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0026] Figure 1 It is a schematic diagram of an exemplary system architecture to which a multi-modal information interaction method and apparatus according to an embodiment of the present disclosure can be applied. It should be noted that Figure 1The figure shown is only an example of the system architecture to which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0027] As Figure 1 shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0028] Users can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The terminal devices 101, 102, 103 may be various electronic devices, including but not limited to smartphones, tablets, laptop computers, etc.

[0029] The server 105 may be a server that provides various services, such as a background management server (only an example) that supports the websites browsed by users using the terminal devices 101, 102, 103. The background management server may analyze and process data such as user requests received, and feedback the processing results to the terminal devices.

[0030] The multi-modal information interaction method provided by the embodiments of the present disclosure can generally be executed by the server 105. Correspondingly, the multi-modal information interaction device provided by the embodiments of the present disclosure can generally be set in the server 105.

[0031] Figure 2 is a flowchart of a multi-modal information interaction method according to an embodiment of the present disclosure.

[0032] As Figure 2 shown, the multi-modal information interaction method 200 includes operations S210 to S230.

[0033] The execution subject of this embodiment may be an agent, and the agent may be integrated with a large language model.

[0034] In operation S210, the intent of the media resource request from the terminal is recognized to obtain an intent recognition result, and the intent recognition result indicates whether the media resource request hits a predetermined processing method.

[0035] After the agent receives the media resource request sent by the terminal, the large language model in the agent can analyze and understand the media resource request to identify the user's intention. The result of the user's intention recognition may include whether the user's media resource request hits a predetermined processing method. The predetermined processing method may be a method that requires rendering the media resource requested by the user in the cloud.

[0036] For example, the user's media resource request is "Play a funny video for me". The large language model in the agent can perform intention recognition on this request and determine that what the user needs the agent to return is "Play a funny video", that is to say, it is not provided in the form of a link, but the video content is directly played in the form of a video stream. Therefore, according to the result of the intention recognition, it can be determined that the user's media resource request hits the predetermined processing method, that is, it is necessary to render the media resource requested by the user in the cloud and return it to the terminal in the form of a video stream.

[0037] In operation S220, in response to the media resource request hitting the predetermined processing method, the first multimodal processing module is called to determine the media resource address corresponding to the media resource request.

[0038] After determining that the media resource request hits the predetermined processing method, the agent can call the first multimodal processing module to determine the address of the media resource requested by the media resource request.

[0039] For example, the first multimodal processing module may include a search module. The search module can search according to the media resource request to obtain the media resource and the media resource address corresponding to the media resource request. For example, one or more "funny videos" and the address links of the "funny videos" can be searched.

[0040] In operation S230, the second multimodal processing module is called to render the media resource in the media resource address and output the rendered media stream to the terminal.

[0041] After the first multimodal processing module determines the media resource address corresponding to the media resource request, it can return the media resource address to the agent, and the agent can send the media resource address to the second multimodal processing module.

[0042] The second multimodal processing module may be a module for cloud rendering of media resources. After receiving the media resource address, the second multimodal processing module can obtain the media resource based on the media resource address, perform rendering, and then send the rendered media stream to the terminal.

[0043] According to an embodiment of the present disclosure, the intent of a media resource request from a terminal is recognized to determine whether the media resource request hits a predetermined processing method. In the case of hitting the predetermined processing method, the first multimodal processing module is called to determine the media resource address corresponding to the media resource request, the second multimodal processing module is called to render the media resource in the multimedia resource address, and the rendered media stream is sent to the terminal. Since the media resource can be output to the terminal in the form of a media stream, the user can intuitively obtain the media content, which can improve the interaction experience.

[0044] Compared with the related art where the media needs to be rendered on the terminal side or the terminal needs to call a media tool to open the media, the embodiment of the present disclosure can avoid multiple jumps during the interaction process and maintain the continuity of the interaction. Moreover, the multimedia rendering is implemented in the cloud, which can minimize the volume of the intelligent agent application program on the terminal side and reduce the user access cost.

[0045] The embodiment of the present disclosure outputs the media content in the form of a media stream. As a new interaction method, it provides more application scenarios for large model interaction applications and further improves the user interaction experience.

[0046] Figure 3 It is a schematic diagram of a system to which a multimodal information interaction method can be applied according to an embodiment of the present disclosure.

[0047] As Figure 3 shown, the system of this embodiment includes a terminal side and a cloud side. The terminal side includes an AI interaction application program 310, and the cloud side includes an intelligent agent 320, a multimodal media search component 330, a cloud rendering subsystem 340, an intelligent agent management platform 350, and an application service module 360. Among them, the multimodal media search component 330 can be the first multimodal processing module of the embodiment of the present disclosure, and the cloud rendering subsystem 340 can be the second multimodal processing module of the embodiment of the present disclosure.

[0048] The AI interaction application program 310 is used to provide a page for the user to interact with the intelligent agent 320. The user enters a request on the page, and the terminal can send the request to the intelligent agent 320 through a real-time communication network. A real-time communication protocol is established between the AI interaction application program 310 and the intelligent agent 320.

[0049] The agent 320 can be integrated with a speech recognition module, a large language model, a text-to-speech module, a real-time communication module, etc. The user's request can be a media resource request. After the agent 320 receives the user's media resource request, if the user's media resource request is in speech, the speech can be converted into text through the speech recognition module, and then the text is sent to the large language model. The large language model can analyze and understand the text to determine the user's intention. If the user's intention is an ordinary interaction, such as asking the large model to summarize the media resources of multiple requests, asking the large model to return the link of the media resources, etc., that is, the user's media resource request does not hit the predetermined processing method, the large language model can generate a response content and send the response content to the text-to-speech module. The text-to-speech module can convert the response information into audio and then return it to the AI interaction application 310 through the real-time communication network.

[0050] If the user's media resource request hits the predetermined processing method, that is, the user's intention is to ask the large model to return the media resource in the form of a media stream, the agent 320 will call the multimodal media search component 330 for processing.

[0051] For example, if the large language model recognizes the intention of "want to play a funny video" from the user's request, it can determine that the user's intention is to ask the agent 320 to return the played video instead of providing the link of the video. Therefore, it can be determined that the user's media resource request hits the predetermined processing method.

[0052] For another example, if the user's media resource request contains a predetermined prefix, it can also be determined that the media resource request hits the predetermined processing method. The predetermined prefix includes keywords such as "cloud rendering" and "cloud playback". If the user's media resource request is "cloud play a certain movie", then the media resource request hits the predetermined processing method.

[0053] The application service module 360 is used to configure and manage the above-mentioned predetermined processing methods. For example, the application service module 360 can configure a predetermined processing function Function Call in the agent 320 to process the media resource request that hits the predetermined processing method. When the large language model determines through intention recognition that the user's media resource request hits the predetermined processing method, the agent 320 can call the Function Call function to process the media resource request, and Function Call will then call the multimodal media search component 330 for processing.

[0054] The multimodal media search component 330 can search for media resources corresponding to a media resource request and send the addresses of the searched media resources to the agent management platform 350. In addition, the intention of the user's multimedia resource request may also include the processing type of the media resource request, and the processing type may include search, generation, etc. For example, the media resources requested by the user may not be existing media resources in the network, but need to be generated by a large model. In this case, the multimodal media search component 330 can call the multimodal large model to generate the media resources required by the user, such as generating images, videos, texts, audios, documents, etc. Then, the generated media resources are stored, and the storage addresses are sent to the agent management platform 350.

[0055] The agent management platform 350 is the management platform of the agent 320 and is responsible for the real-time communication between the agent 320, the multimodal media search component 330, and the cloud rendering subsystem 340. For example, after receiving the media resource address sent by the multimodal media search component 330, the agent management platform 350 can send the media resource address to the cloud rendering subsystem 340.

[0056] According to an embodiment of the present disclosure, the cloud rendering subsystem 340 is used to obtain a media resource from a media resource address; render the media resource onto a virtual screen; and collect the content on the virtual screen to obtain a media stream.

[0057] The cloud rendering subsystem 340 may include a media rendering assistant and a streaming service module. The media resource address may be a media resource link, and the media rendering assistant can open the link, obtain the media resource, and render the media resource. For example, the media rendering assistant can render the media resource onto a virtual screen, and the rendered media stream may be included on the virtual screen. The streaming service module can collect the content on the virtual screen to obtain the rendered media stream. Next, the streaming service module can output the collected media stream and send the media stream to the A1 interaction application 310 on the terminal side through a real-time communication network.

[0058] In addition, the agent management platform 350 is also used to create and manage cloud rendering tasks in the cloud rendering subsystem 340, and create and manage agent tasks in the agent 320.

[0059] According to an embodiment of the present disclosure, in response to an agent call request from the terminal, the agent is started and a cloud rendering task is assigned to the second multimodal processing module; in response to an agent shutdown request from the terminal, the agent is shut down and the cloud rendering task is released.

[0060] For example, before the user interacts with the agent 320 based on the A1 interactive application 310, the user first calls the agent to start the agent. Specifically, the agent management platform 350 responds to the request to call the agent and starts the agent. Starting the agent, for example, means creating an agent instance. The agent instance is the agent task, and the user's interaction with the agent is performed in the task. After the user initiates a request to close the agent, the agent management platform 350 responds to the request to close the agent and closes the agent instance, and no longer interacts.

[0061] After starting the agent, the agent management platform 350 can pre-create a cloud rendering instance, so that in the process of interacting with the user, after receiving a media resource request that hits the predetermined processing method, cloud rendering can be quickly performed to improve the response speed. The agent management platform 350 creates a cloud rendering instance, that is, assigns a cloud rendering task to the cloud rendering subsystem 340. Specifically, after starting the agent, the agent management platform 350 pre-assigns an idle instance to the cloud rendering subsystem 340. After obtaining the media resource address, the agent management platform 350 sends the media resource address to the pre-assigned instance. The cloud rendering subsystem 340 starts the instance and performs cloud rendering operations in the instance.

[0062] According to the embodiments of the present disclosure, the intelligent agent management platform pre-allocates a cloud rendering instance to the cloud rendering subsystem after the intelligent agent is started. After the user's media resource request hits the predetermined processing method, the multimedia element content can be quickly streamed and output directly to the terminal side through the cloud rendering instance, thereby improving the interaction efficiency.

[0063] According to an embodiment of the present disclosure, the multimodal information interaction method also includes: receiving an interactive operation event from a terminal, wherein the interactive operation event is generated by the terminal in response to an interactive operation performed by a target object on a media stream on the terminal; and calling a second multimodal processing module to perform an interactive operation on the media stream on the virtual screen according to the interactive operation event, so that the terminal displays the interactive result.

[0064] After the cloud rendering subsystem sends the media stream to the terminal, the terminal can display the media stream. The user can perform interactive operations on the media stream displayed on the terminal, such as clicking, sliding, etc. The terminal can collect the user's interactive operations such as clicking and sliding, and generate interactive operation events. The terminal can send the interactive operation event to the cloud rendering instance of the second multimodal processing module.

[0065] The cloud rendering instance of the second multimodal processing module can perform interactive operations such as clicking and swiping on the media stream on the virtual screen based on the received interactive operation events, so that the media stream on the virtual screen presents interactive effects such as pausing, playing, turning pages of the broadcast file, and pulling down the output page network, etc., and then the media stream on the terminal also correspondingly presents interactive effects such as pausing, playing, turning pages of the broadcast file, and pulling down the output page network, etc.

[0066] According to an embodiment of the present disclosure, by sending the interactive operation event to the cloud rendering instance, and the cloud rendering instance performs interactive operations on the media stream on the virtual screen, and then the media stream on the terminal presents interactive effects, it can enable users to interact with the intelligent agent at a deeper level and improve the user interaction experience.

[0067] Figure 4 It is a schematic diagram of the first multimodal processing module according to an embodiment of the present disclosure.

[0068] As Figure 4 shown, the first multimodal processing module 410 may be the above-mentioned multimodal media search component. It may include a multimodal media resource search unit 411, a multimodal media resource return unit 412, and a multimodal media resource generation unit 413. Different units correspond to different processing types of media resource requests.

[0069] According to an embodiment of the present disclosure, the intent recognition result also characterizes the processing type of the media resource request; the first multimodal processing module, in response to the processing type being search, searches for the media resource corresponding to the media resource request and obtains the media resource address corresponding to the media resource request; in response to the processing type being return, determines the predetermined address as the media resource address corresponding to the media resource request; in response to the processing type being generation, generates the media resource corresponding to the media resource request and the media resource address.

[0070] For example, the large language model in the intelligent agent performs intent recognition on the media resource request. The intent recognition result, in addition to characterizing whether the media resource request hits the predetermined processing method, also characterizes the processing type for the media resource request in the case of hitting the predetermined processing method. The processing type may include search, return, generation, etc.

[0071] For example, if the user's intent is to search for one or more media resources, the processing type of the media resource request may be "search". The media resource request may include the conditions that the media resource to be searched satisfies, such as name, time, keywords, etc. The multimodal media resource search unit 411 can search according to the media resource request to obtain the media resource that meets the user's needs and the address of the media resource. The multimodal media resource search unit 411 can send the searched media resource to the intelligent agent management platform.

[0072] For another example, if the user's intention is to have the agent return a predetermined media resource, such as a correspondence relationship between a specified intention and a link to a predetermined media resource is pre-configured. At this time, the multimodal media resource sending unit 412 can directly send the address of the predetermined media resource to the agent management platform.

[0073] For another example, if the user's intention is to generate a custom media resource, the media resource request contains the user's custom generation information. The multimodal media resource generation unit 413 can call a multimodal large model to generate a media resource that meets the user's needs based on the custom generation information, and then store the generated media resource to obtain a media resource address. Then the multimodal media resource generation unit 413 can send the generated media resource address to the agent management platform.

[0074] According to an embodiment of the present disclosure, by performing intention recognition on a media resource request, determining the processing type of the media resource request, and performing corresponding processing for different processing types to obtain a media resource address, it is possible to meet the diverse needs of users and improve the user experience.

[0075] Figure 5 It is a schematic diagram of a second multimodal processing module according to an embodiment of the present disclosure.

[0076] As Figure 5 shown, the second multimodal processing module 510 may be a cloud rendering subsystem. The second multimodal processing module 510 may include a cloud rendering management platform 511 and multiple cloud rendering instances 512. Each cloud rendering instance may include a media rendering assistant and a streaming service module. The cloud rendering management platform 511 is used to communicate with the agent management platform and manage the cloud rendering instances.

[0077] For example, after the agent is started, the agent management platform may allocate one or more idle instances to the cloud rendering subsystem as the cloud rendering instances 512. The agent management platform may pre-launch a specified cloud rendering instance through the interface of the cloud rendering management platform and may also pre-start the media rendering assistant. In this way, after receiving the media resource address sent by the agent management platform, the cloud rendering instance 512 can quickly perform media rendering.

[0078] For example, after the cloud rendering management platform 511 receives the media resource address sent by the agent management platform, it can send a command to the media rendering assistant. The media rendering assistant opens the media link to perform media rendering and can render the media resource onto a virtual screen. The streaming service module captures the media stream on the virtual screen and pushes the media stream to the user side.

[0079] The intelligent body management platform may send one or more media resource addresses. When there are multiple media resource addresses, the media rendering assistant may open and render them in sequence according to the list of media resource addresses.

[0080] According to the embodiments of the present disclosure, by rendering the media resource in the media resource address and collecting the rendered media stream and pushing it to the terminal, the user can intuitively obtain the content of the media resource and provide an interactive experience.

[0081] Figure 6 is a flowchart of a multimodal information interaction method according to another embodiment of the present disclosure.

[0082] like Figure 6 As shown, this embodiment includes operations S601 to S615.

[0083] In operation S601, the terminal sends a request to start an agent to the agent management platform through the A1 interactive application.

[0084] In operation S602, the agent management platform starts the agent. Starting the agent, for example, means creating an agent instance.

[0085] In operation S603, the intelligent agent management platform allocates a cloud rendering instance to the cloud rendering subsystem and starts a streaming service, such as starting a media rendering assistant in the cloud rendering subsystem.

[0086] In operation S604, the agent management platform returns the agent instance ID and agent context to the A1 interactive application.

[0087] In operation S605, the user sends a media resource request to the agent management platform through the AI ​​interactive application.

[0088] In operation S606, the agent identifies the intent of the media resource request and determines whether the media resource request matches a predetermined processing method.

[0089] In operation S607, when the media resource request hits a predetermined processing method, the agent sends the media resource request to the multimodal media search component.

[0090] In operation S608, the multimodal media search component searches for a media resource address according to the media resource request and sends the media resource address to the intelligent agent management platform.

[0091] In operation S609, the intelligent agent management platform sends a cloud rendering request to the cloud rendering subsystem, wherein the cloud rendering request includes a media resource address.

[0092] In operation S610, the cloud rendering subsystem calls the media rendering assistant to render the media resource and calls the streaming service to collect the rendered media stream.

[0093] In operation S611, the cloud rendering subsystem sends the rendered media stream to the AI interaction application.

[0094] In operation S612, the user sends a request to the agent management platform to close the agent through the AI interaction application.

[0095] In operation S613, the agent management platform closes the agent, such as closing the agent instance.

[0096] In operation S614, the agent management platform closes the streaming service and releases the cloud rendering instance.

[0097] In operation S615, the agent management platform sends a notification to the AI interaction application to close the agent.

[0098] According to an embodiment of the present disclosure, the present disclosure also provides a multimodal information interaction device.

[0099] Figure 7 is a block diagram of a multimodal information interaction device according to an embodiment of the present disclosure.

[0100] As Figure 7 shown, the multimodal information interaction device 700 includes an intent recognition module 710, a first call module 720, and a second call module 730.

[0101] The intent recognition module 710 is configured to perform intent recognition on a media resource request from a terminal to obtain an intent recognition result, where the intent recognition result indicates whether the media resource request hits a predetermined processing method.

[0102] The first call module 720 is configured to, in response to the media resource request received from the terminal hitting a predetermined processing method, call a first multimodal processing module to determine a media resource address corresponding to the media resource request.

[0103] The second call module 730 is configured to call a second multimodal processing module to render the media resource in the media resource address and output the rendered media stream to the terminal.

[0104] According to an embodiment of the present disclosure, the second call module 730 is configured to call the second multimodal processing module to perform the following operations: obtain the media resource from the media resource address; render the media resource onto a virtual screen; and collect the content on the virtual screen to obtain a media stream.

[0105] According to an embodiment of the present disclosure, the multimodal information interaction device 700 further includes a receiving module and a third call module.

[0106] The receiving module is configured to receive an interactive operation event from a terminal, where the interactive operation event is generated by the terminal in response to an interactive operation on a media stream on the terminal.

[0107] The third calling module is configured to call the second multimodal processing module to perform an interactive operation on the media stream on the virtual screen according to the interactive operation event, so that the terminal displays the interactive result.

[0108] According to an embodiment of the present disclosure, the intent recognition result also characterizes the processing type of the media resource request. The first calling module 720 is configured to call the second multimodal processing module to perform one of the following operations: in response to the processing type being search, search for a media resource corresponding to the media resource request to obtain a media resource address corresponding to the media resource request; in response to the processing type being return, determine a predetermined address as the media resource address corresponding to the media resource request; in response to the processing type being generation, generate a media resource corresponding to the media resource request and a media resource address.

[0109] According to an embodiment of the present disclosure, the multimodal information interaction device 700 further includes a first response module and a second response module.

[0110] The first response module is configured to, in response to an agent call request from the terminal, start the agent and allocate a cloud rendering task to the second multimodal processing module.

[0111] The second response module is configured to, in response to an agent shutdown request from the terminal, shut down the agent and release the cloud rendering task.

[0112] According to an embodiment of the present disclosure, the second calling module 730 is configured to call the second multimodal processing module to start a cloud rendering task to perform an operation of rendering the media resource in the media resource address.

[0113] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0114] Figure 8 A schematic block diagram of an exemplary electronic device 800 that can be used to implement the embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0115] As shown Figure 8 in FIG. 800, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0116] A plurality of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disc, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0117] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 executes the various methods and processes described above, such as the multi-modal information interaction method. For example, in some embodiments, the multi-modal information interaction method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the multi-modal information interaction method described above can be executed. Alternatively, in other embodiments, the computing unit 801 can be configured to execute the multi-modal information interaction method in any other appropriate manner (e.g., by means of firmware).

[0118] The various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), systems on a chip (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0119] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowchart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0120] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include electrical connections based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0121] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0122] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), and the Internet.

[0123] A computer system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other.

[0124] It should be understood that various forms of the processes shown above can be used, steps can be reordered, added, or deleted. For example, the steps recited in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this is not limited herein.

[0125] The above specific implementation manners do not constitute a limitation on the protection scope of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the protection scope of this disclosure.

Claims

1. A multimodal information interaction method, comprising: Performing intent recognition on a media resource request from a terminal to obtain an intent recognition result, wherein the intent recognition result indicates whether the media resource request matches a predetermined processing method; In response to the media resource request hitting a predetermined processing mode, calling a first multimodal processing module to determine a media resource address corresponding to the media resource request; The second multimodal processing module is called to render the media resource in the media resource address, and the rendered media stream is output to the terminal.

2. The method according to claim 1, wherein: The calling of the second multimodal processing module to render the media resource in the media resource address includes: calling the second multimodal processing module to perform the following operations: Acquire media resources from the media resource address; Rendering the media resource onto a virtual screen; and The content on the virtual screen is collected to obtain a media stream.

3. The method according to claim 2, further comprising: receiving an interactive operation event from the terminal, wherein the interactive operation event is generated by the terminal in response to an interactive operation performed by a target object on a media stream on the terminal; and The second multimodal processing module is called to perform an interactive operation on the media stream on the virtual screen according to the interactive operation event, so that the terminal displays the interactive result.

4. The method according to claim 1, wherein: The intent recognition result also characterizes the processing type of the media resource request; In response to the media resource request hitting a predetermined processing mode, calling the first multimodal processing module to determine a media resource address corresponding to the media resource request includes: calling the second multimodal processing module to perform one of the following operations: In response to the processing type being search, searching for a media resource corresponding to the media resource request to obtain a media resource address corresponding to the media resource request; In response to the processing type being a return, determining the predetermined address to be a media resource address corresponding to the media resource request; In response to the processing type being generation, a media resource and a media resource address corresponding to the media resource request are generated.

5. The method according to claim 1, further comprising: In response to an agent call request from a terminal, starting the agent and assigning a cloud rendering task to the second multimodal processing module; as well as In response to an agent shutdown request from the terminal, the agent is shut down and the cloud rendering task is released.

6. According to the method of claim 5, the calling of the second multimodal processing module to render the media resource in the media resource address comprises: The second multimodal processing module is called to start the cloud rendering task to perform an operation of rendering the media resource in the media resource address.

7. An agent configured to execute the method according to any one of claims 1 to 6.

8. A multimodal information interaction device, comprising: An intent recognition module, used to perform intent recognition on a media resource request from a terminal and obtain an intent recognition result, wherein the intent recognition result indicates whether the media resource request matches a predetermined processing method; A first calling module, configured to, in response to a media resource request received from a terminal hitting a predetermined processing mode, call a first multimodal processing module to determine a media resource address corresponding to the media resource request; The second calling module is used to call the second multimodal processing module to render the media resource in the media resource address, and output the rendered media stream to the terminal.

9. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

11. A computer program product, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images