Interaction method and apparatus, electronic device, storage medium and product

Abstract

An interaction method and apparatus, an electronic device, a storage medium and a product, relating to the technical field of terminals. The interaction method comprises: displaying multimedia content on a playback interface (S102); on the basis of the multimedia content, determining a called object (S104); in response to the called object being an agent, by means of a message control, displaying on the playback interface a message sent by the agent, the message being obtained by comprehending the multimedia content (S106); and, in response to a trigger operation of a user for the message control, displaying a dialogue interface for the user and the agent (S108).

Description

Interaction methods, devices, electronic devices, storage media and products Technical Field

[0001] This disclosure relates to the field of terminal technology, and in particular to an interaction method, apparatus, electronic device, storage medium, and product. Background Technology

[0002] In short video and other multimedia applications, users can browse multimedia content such as videos and text, and switch between multiple multimedia content through operations such as switching. For example, if a user is not interested in the current multimedia content, they can quickly switch to the next recommended video. If a user is interested in the current multimedia content, they can interact with the author of the multimedia content by posting comments, liking, etc., to express their feelings or learn more information related to the video.

[0003] Summary of the Invention

[0004] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] According to some embodiments of this disclosure, an interaction method is provided, including: displaying multimedia content on a playback interface; determining a invoked object based on the multimedia content; in response to the invoked object being an intelligent agent, displaying a message sent by the intelligent agent in the playback interface through a message control, the message being obtained by understanding the multimedia content; and displaying a dialogue interface between the user and the intelligent agent in response to a user's triggering operation on the message control.

[0006] According to other embodiments of this disclosure, an interactive device is provided, comprising: a first display module configured to display multimedia content on a playback interface; a determination module configured to determine a invoked object based on the multimedia content; a second display module configured to, in response to the invoked object being an intelligent agent, display a message sent by the intelligent agent in the playback interface via a message control, the message being obtained by understanding the multimedia content; and a third display module configured to, in response to a user's triggering operation on the message control, display a dialogue interface between the user and the intelligent agent.

[0007] According to some embodiments of the present disclosure, an electronic device is provided, including: a memory; and a processor coupled to the memory, the processor being configured to execute an interaction method of any embodiment described in the present disclosure based on instructions stored in the memory.

[0008] According to some embodiments of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, performs the interactive method of any embodiment described in the present disclosure.

[0009] According to some embodiments of the present disclosure, a computer program product is provided that, when the computer program product is run on a computer, enables the computer to implement the interaction method of any embodiment described in the present disclosure.

[0010] According to some embodiments of the present disclosure, a computer program is provided, comprising: instructions that, when executed by a processor, cause the processor to perform an interactive method according to any embodiment of the present disclosure.

[0011] Other features, aspects, and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0012] Preferred embodiments of the present disclosure are described below with reference to the accompanying drawings. The accompanying drawings, which are included to provide a further understanding of the present disclosure, and which, together with the following detailed description, are incorporated in and form a part of this specification and are used to explain the present disclosure. It should be understood that the drawings described below only relate to some embodiments of the present disclosure and are not intended to limit the present disclosure. In the drawings:

[0013] Figure 1 shows a flowchart illustrating an interaction method according to some embodiments of the present disclosure.

[0014] Figures 2A and 2B show schematic diagrams of playback interfaces according to some embodiments of the present disclosure.

[0015] Figure 3 shows a flowchart illustrating a method for determining the invoked object according to some embodiments of the present disclosure.

[0016] Figure 4 shows a flowchart illustrating a method for generating messages according to some embodiments of the present disclosure.

[0017] Figure 5 shows a flowchart illustrating an interaction method according to other embodiments of the present disclosure.

[0018] Figure 6 shows a schematic diagram of the structure of an interactive device according to some embodiments of the present disclosure.

[0019] Figure 7 shows a schematic diagram of the structure of an electronic device according to some embodiments of the present disclosure.

[0020] Figure 8 shows a schematic diagram of the structure of a computer system according to some embodiments of the present disclosure.

[0021] It should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not necessarily drawn to actual scale. The same or similar reference numerals are used in the various drawings to denote the same or similar parts. Therefore, once an item is defined in one drawing, it may not be discussed further in subsequent drawings. Detailed Implementation

[0022] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. However, it is obvious that the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of the embodiments is merely illustrative and is in no way intended to limit this disclosure or its application or use. It should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein.

[0023] It should be understood that the various steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of components and steps set forth in these embodiments should be interpreted as merely exemplary and do not limit the scope of this disclosure.

[0024] As used in this disclosure, the term "comprising" and its variations are open-ended terms that include at least the following elements / features but do not exclude other elements / features, i.e., "including but not limited to". Furthermore, as used in this disclosure, the term "including" and its variations are open-ended terms that include at least the following elements / features but do not exclude other elements / features, i.e., "including but not limited to". Therefore, "comprising" and "including" are synonymous. The term "based on" means "at least partially based on".

[0025] Throughout this specification, the terms "one embodiment," "some embodiments," or "embodiment" mean that a specific feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. For example, the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; and the term "some embodiments" means "at least some embodiments." Furthermore, the appearance of the phrases "in one embodiment," "in some embodiments," or "in an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment, but may refer to the same embodiment.

[0026] It should be noted that the concepts of "first," "second," etc., used in this disclosure are used only to distinguish different devices, modules, or units, and are not intended to define the order of functions performed by these devices, modules, or units or their interdependencies. Unless otherwise specified, the concepts of "first," "second," etc., are not intended to imply that the objects described herein must be in a given temporal, spatial, rank, or any other given order.

[0027] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0028] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0029] The embodiments of this disclosure are described in detail below with reference to the accompanying drawings; however, this disclosure is not limited to these specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. Furthermore, in one or more embodiments, specific features, structures, or characteristics can be combined in any suitable manner that will be apparent to those skilled in the art from this disclosure.

[0030] It should be understood that this disclosure does not limit how the image to be applied / processed is obtained. In one embodiment of this disclosure, it can be obtained from a storage device, such as internal or external memory; in another embodiment, it can be captured by a camera component. It should be noted that the acquired image can be a single captured image or a frame from a captured video, and is not particularly limited to these methods.

[0031] In the context of this disclosure, "image" can refer to any of a variety of images, such as color images, grayscale images, etc. It should be noted that the type of image is not specifically limited in the context of this specification. Furthermore, an image can be any suitable image, such as a raw image obtained by a camera device, or an image from which specific processing has been performed, such as preliminary filtering, dealiasing, color adjustment, contrast adjustment, normalization, etc. It should be noted that preprocessing operations may also include other types of preprocessing operations known in the art, which will not be described in detail here.

[0032] When browsing multimedia content, users who are interested can search for related information or leave messages for the content's author to learn more. However, these actions require users to initiate them. Furthermore, some users are unfamiliar with the process or lack the willingness to actively search or interact, resulting in relatively low information acquisition efficiency.

[0033] This disclosure proposes an interactive method that, during the display of multimedia content, shows the user messages sent by an intelligent agent, generated based on the multimedia content. This automatically provides the user with information related to the interactive multimedia content.

[0034] Figure 1 shows a flowchart of an interaction method according to some embodiments of the present disclosure. As shown in Figure 1, the interaction method of this embodiment includes steps S102 to S108.

[0035] In step S102, multimedia content is displayed on the playback interface.

[0036] The playback interface may include a display window for multimedia content and controls. The display window is used to hold the multimedia content. Controls may be located on top of the display window or arranged side-by-side with the display window in the playback interface. Controls may include interactive controls such as like, favorite, comment, share, etc.

[0037] Users can control the playback of multimedia content using designated interactive gestures. For example, clicking on multimedia content controls playback and pause, swiping up and down controls switching between different content, and swiping left and right switches between channels. Of course, these operations can also be triggered by controls within the playback interface, which can be selected as needed by those skilled in the art.

[0038] In some embodiments, the multimedia content is content within a recommended multimedia content stream. A media stream refers to a stream of multimedia content recommended to a user based on a specified recommendation strategy. The multimedia content in the media stream can be displayed in an immersive manner, such as full-screen. Users browse recommended videos sequentially by switching between multimedia content.

[0039] In step S104, the object to be invoked is determined based on the multimedia content.

[0040] The object being invoked is a function provided by the application for user use, such as at least one of the following: an agent, a sub-application, etc.

[0041] Intelligent agents, including robots, digital humans, intelligent assistants, and virtual proxies of machine learning models, are intelligent objects capable of automatically responding to user input; for example, they could be chatbots. Intelligent agents can generate corresponding content based on dialogues sent by other subjects in a conversational scenario (other users or other intelligent agents participating in the dialogue). Intelligent agents can be implemented in software, hardware, or a combination of both. Intelligent agents can be implemented using machine learning models, such as Large Language Models (LLMs) or Foundation Models. Machine learning models can be generative models, used to output target content based on input information. The input information of a generative model includes the processing criteria used during the generation process, such as which information to refer to during the generation process and the requirements for the target output content. Generative models include, for example, models that generate data based on text or images, and the output of a generative model can include text, images, or a combination of both. Of course, the input or output of a generative model can also be data of other modalities, such as audio, video, or a combination of multiple types of data. Generative models can be single-modal models, such as text-to-text models (referred to as "text-to-text models") or image-to-image models (referred to as "image-to-image models"); or, generative models can be cross-modal models, that is, models whose inputs and outputs belong to different modalities, such as text-to-image models (referred to as "text-to-image models"); or, the inputs of generative models can include multiple modalities, and the outputs can also include multiple modalities.

[0042] Sub-applications are objects within an application that run based on specified logic, including mini-programs and plugins. Taking mini-programs as an example, these could include weather mini-programs, quick note-taking mini-programs, reading mini-programs, and so on. Sub-applications allow users to quickly access more functions within the same application.

[0043] Based on multimedia content, the invoked object that matches the multimedia content can be identified. For example, the invoked object can be identified through machine-understood information of the multimedia content. Machine-understood information refers to semantic information obtained through computer processing. For instance, multimedia content can be processed using a machine learning model, and the invoked object that matches the multimedia content can be identified based on the processing results of the machine learning model.

[0044] In some embodiments, multimedia content can be parsed to obtain understanding information about the multimedia content (i.e., machine understanding information), and then the object to be invoked can be determined based on the understanding information. For example, when candidate objects include agents and sub-applications, it can be first determined whether to invoke an object of the agent type or an object of the sub-application type, and then further determined which object to invoke.

[0045] In some embodiments, multimedia content is processed by at least one of image processing and audio processing to obtain at least one of image semantic information and audio semantic information of the multimedia content. The image processing and audio processing described above can be performed using a machine learning model capable of processing multimedia data. Then, understanding information about the multimedia content can be generated based on at least one of image semantic information, audio semantic information, descriptive text of the multimedia content, and tags of the multimedia content. For example, at least one of image semantic information, audio semantic information, descriptive text of the multimedia content, and tags of the multimedia content can be fused to generate a summary of the multimedia content as understanding information; or, keywords of the multimedia content can be extracted from at least one of image semantic information, audio semantic information, descriptive text of the multimedia content, and tags of the multimedia content as understanding information; or, the type of the multimedia content can be determined by considering the types involved in at least one of image semantic information, audio semantic information, descriptive text of the multimedia content, and tags of the multimedia content as understanding information.

[0046] In step S106, in response to the fact that the called object is an intelligent agent, a message sent by the intelligent agent is displayed in the playback interface via a message control. This message is obtained through understanding the multimedia content. This understanding refers to machine understanding.

[0047] After identifying the target object, a message to be sent by the agent can be generated based on the multimedia content. For example, the message can be generated based on the understanding information of the multimedia content. In some embodiments, a machine learning model for processing text (e.g., a "text-to-text model") is used to process the understood information and obtain the text output by the machine learning model, which is then used to generate the agent's message. The processing objects of the machine learning model for processing text may include not only the understood information but also the agent's information (e.g., configuration information) and information of the user authorized by the user (e.g., user preferences), so that the generated message is more closely matched to the interaction style of the agent and the user. For example, a life assistant-type agent can send messages in everyday language, while a professional knowledge-type agent can express itself in more formal language.

[0048] The message control may include only the aforementioned message, or it may include the agent's identifier (e.g., name, avatar, etc.) and the message. The message control may float above the playback interface and display in response to a generated message; alternatively, it may be fixed within the playback interface and display the message thereafter. In some embodiments, the message control may include a dialog box, icon, or overlay.

[0049] In step S108, in response to the user's triggering operation on the message control, the dialogue interface between the user and the intelligent agent is displayed.

[0050] After receiving a message from the agent, the user can continue the conversation by triggering a message control. Of course, the user can also choose not to trigger the message control if they do not wish to continue interacting with the agent. In some embodiments, where the message control is not a fixed control in the playback interface, it can be closed in response to the user not triggering it and the message control's display duration reaching a specified threshold, allowing the user to continue focusing on browsing multimedia content. For example, the message control could be a pop-up window that closes if the user does not trigger it within the specified threshold duration.

[0051] In the above embodiments, during the playback of multimedia content to a user, an intelligent agent is invoked based on the multimedia content, and messages related to the multimedia content are sent through the intelligent agent. This allows for the automatic push of information related to the multimedia content to the user, enabling them to gain a more efficient understanding of the content. Furthermore, the user can easily continue interacting with the intelligent agent to learn more information of interest. Therefore, the embodiments of this disclosure can improve the efficiency of information acquisition during the user's browsing of multimedia content.

[0052] Figures 2A and 2B illustrate schematic diagrams of a playback interface according to some embodiments of the present disclosure. As shown in Figure 2A, the multimedia content playback interface 2 includes displayed multimedia content 21, such as a clip from a movie. Additionally, a message control 22 floats above the playback interface 2, including an avatar 221 of agent X and the message 222, "This movie was directed by director A in 2020 and tells a story of...". The message 222 is obtained through machine understanding of the multimedia content. Thus, the user can efficiently obtain information related to the currently viewed multimedia content. The message control 22 can be manifested in other forms as needed; for example, it can be an icon that, when triggered, further displays the message content; or it can be a floating layer, a dialog box, or a waiting area.

[0053] In some embodiments, the playback interface 2 may further include an input control 23, which may be a text input box, a voice input control, etc. Content entered into the input control 23 is sent to the intelligent agent for processing. That is, the user can send messages to the intelligent agent through the input control 23. The intelligent agent can reply to the user based on the received messages. For example, the intelligent agent can generate a message based on the received message and the currently playing multimedia content.

[0054] In response to the user's triggering operation on message control 22, playback interface 2 can be as shown in Figure 2B. In Figure 2B, a dialogue interface 24 between the user and the agent is displayed above the multimedia content 21. The dialogue interface includes the content of messages sent by the agent to the user (such as the content of message 222) and input control 241. The user can continue to communicate with the agent by triggering input control 241, such as continuing to ask questions about relevant information about multimedia content 21, or sending messages on other topics to the agent.

[0055] The process of understanding multimedia content and generating messages can be automatically triggered or triggered in response to user instructions. These two triggering strategies can be used individually or in combination. For example, one triggering method can be used for all multimedia content, or an automatic triggering method can be used for some multimedia content and a manual triggering method can be used for another part of the multimedia content.

[0056] In some embodiments, the multimedia content is understood in response to its display. That is, the understanding of the multimedia content can be initiated automatically without waiting for user instructions. Depending on the needs, this strategy can be applied to all multimedia content, or it can be used to select a portion of the multimedia content based on its tags. Taking Figure 2A as an example, message control 22 and message 222 can be generated and displayed in response to the display of multimedia content 21, thus eliminating the need for active user interaction.

[0057] In some embodiments, messages sent by the user to the agent are obtained through input controls; multimedia content is understood based on the instructions in the user-sent messages. That is, the understanding of multimedia content is initiated only after the user provides an instruction. For example, the user can send instructions to the agent through input control 23, such as "Who directed this movie?" or "What is the ending of this movie?", thereby triggering the understanding of multimedia content in response to the user's instructions and determining what kind of message to send to the user based on the user's instructions. This reduces the processing load on the system and allows for more targeted generation of message content.

[0058] When determining which type of object to invoke, it can be based on whether the user intends to continue consuming multimedia content. An embodiment of the method for determining the invoked object in this disclosure is described below with reference to Figure 3.

[0059] Figure 3 shows a flowchart illustrating a method for determining the invoked object according to some embodiments of the present disclosure. As shown in Figure 3, the determination method of this embodiment includes steps S302 to S304.

[0060] In step S302, based on the multimedia content, it is determined whether the user intends to continue consuming the multimedia content.

[0061] Continued consumption of multimedia content refers to performing operations related to the multimedia content after browsing it, such as searching for information related to the multimedia content, asking questions to the agent about the multimedia content, etc.

[0062] In some embodiments, a user's intention to continue consuming multimedia content can be determined by the amount of information, complexity, and associated information of the multimedia content. Information volume can be determined, for example, by attributes of the multimedia content, such as duration, amount of text involved, etc., or by identification based on a model used to determine information volume. Complexity can be determined, for example, by the number and type of topics involved in the multimedia content. The number of topics is positively correlated with complexity; that is, the more topics, the more complex the multimedia content. Certain specific types of topics have higher complexity; for example, topics involving mathematics and physics are generally more complex than everyday life topics. Information associated with the multimedia content can be determined through searching, and the intention to continue consuming can be determined based on the amount of information, complexity, etc., of the searched information. Generally speaking, users are more likely to have the intention to continue consuming multimedia content with a large amount of information, high complexity, and many associated information.

[0063] In some embodiments, data on the continued consumption of multimedia content of certain themes or types can be pre-collected to determine whether the user intends to continue consuming the current multimedia content. For example, if a user in the application exhibits a high degree of continued consumption behavior for a certain theme or type of multimedia content (e.g., the number of times or the percentage of continued consumption exceeds a specified threshold), then, in response to the fact that the currently viewed multimedia content belongs to that theme or type, it can also be determined that the user intends to continue consuming the currently viewed multimedia content.

[0064] In step S304, in response to the user's intention to continue consuming, an agent is selected from one or more candidate agents as the object to be invoked, based on the understanding information of the multimedia content and the matching results of one or more candidate agents.

[0065] If the user intends to continue consuming content, it indicates that the user may need to understand the currently viewed multimedia content more deeply or learn more about information related to it. In this case, an intelligent agent can be invoked. Thus, in addition to understanding the current multimedia content and sending messages, the intelligent agent can also receive further input from the user and respond to that input.

[0066] One or more candidate agents can include agents of different types, such as agents created by the current user, agents created by other users, the default agent provided by the application, or agent-type agents such as emotion-based agents, tool-based agents, knowledge-based agents, etc. Furthermore, agents can be selected based on multimedia content to match it, or the default agent can be selected directly.

[0067] In some embodiments, step S306 may also be included: in response to the user's lack of intention to continue consuming content, selecting a sub-application from one or more candidate sub-applications as the object to be invoked. That is, when the user has no intention to continue consuming content, a mini-program related to the multimedia content can be invoked to provide the user with more information related to the multimedia content in another way. For example, for a video describing extreme weather in a certain place, a weather mini-program can be invoked for the user, so that the user can quickly check whether there are any abnormalities in the local weather.

[0068] The above embodiments can invoke objects based on whether the user intends to continue consuming, thereby invoking objects that are more compatible with the current multimedia content, in order to improve the user's interactive experience.

[0069] After browsing multimedia content, users can further consume it. The types of multimedia content consumption can include in-depth consumption, extended consumption, and comprehension-aid consumption. In-depth consumption refers to recommending other functions based on the understanding of the current multimedia content. Extended consumption refers to recommending other related information based on the understanding of the current video. Comprehension-aid consumption refers to assisting users in understanding multimedia content with a large amount of information. Depending on the consumption type, corresponding messages can be generated. An embodiment of the message generation method of this disclosure is described below with reference to Figure 4.

[0070] Figure 4 shows a flowchart illustrating a message generation method according to some embodiments of the present disclosure. As shown in Figure 4, the message generation method of this embodiment includes steps S402 to S404.

[0071] In step S402, based on the understanding information of the multimedia content, the consumption type of the multimedia content is determined, and the consumption type includes at least one of in-depth consumption, extended consumption, and assisted understanding consumption.

[0072] Based on the understanding of multimedia content, various types of objects associated with the multimedia content can be identified, and then the consumption type can be determined based on these associated objects. Each associated object can correspond to a consumption type; for example, associated functions correspond to in-depth consumption, associated information to extended consumption, and associated browsing history to aid understanding consumption. In some embodiments, it is necessary to determine whether the multimedia content can provide a certain consumption type, for example, whether the multimedia content has a certain type of associated object, and the consumption type that the multimedia content can provide is taken as the consumption type of the multimedia content. If the multimedia content can provide multiple consumption types, the object with the highest compatibility with the multimedia content can be determined from among the various associated objects based on the degree of association between the associated objects and the multimedia content, or the amount of information in the associated objects, thereby determining the most suitable consumption type for that multimedia content.

[0073] In some embodiments, based on the understanding information of the multimedia content, one or more associated objects of the multimedia content are determined, wherein the one or more associated objects include at least one of associated functions in the application, searched associated information, and user-authorized associated browsing history; a target associated object is determined from the one or more associated objects according to the degree of association between each type of associated object and the understanding information; a consumption type corresponding to the target associated object is determined, wherein associated functions correspond to in-depth consumption, associated information to extended consumption, and associated browsing history to auxiliary understanding consumption. The degree of association can be determined based on a machine learning model used to determine the degree of association, or it can be determined by determining the similarity between the associated object and the multimedia content. By selecting an associated object of a certain type according to the degree of association and then determining the corresponding consumption type, the determined consumption type can be more closely matched with the multimedia content.

[0074] In step S404, a message is generated based on the understanding of the multimedia content and the consumption type.

[0075] In cases of in-depth consumption, messages are generated based on the understanding of multimedia content to recommend related functions. These related functions include tools, sub-applications, other agents, etc. Users can then choose to continue interacting with the current agent or use other functions recommended by that agent. This enables users to utilize functions within the application related to the current multimedia content, increasing the overall usage rate of the application's features.

[0076] When the consumption type is extended consumption, messages describing related information are generated based on the understanding of multimedia content. This allows users to quickly learn more about the multimedia content, improving their information acquisition efficiency. For example, if the multimedia content being played is a video introducing a TV series, the messages generated by the agent could include other film and television works recently starred by the lead actor of that series.

[0077] When the consumption type is to aid in understanding consumption, key information about the multimedia content can be generated, such as a summary of the multimedia content. In some embodiments, messages can also be generated by combining previously viewed multimedia content, such as generating a summary or comparison of the current multimedia content and previously viewed multimedia content, thereby helping users quickly obtain the main information in the multimedia content.

[0078] In some embodiments, summary information on the multimedia content in the historical browsing history and the displayed multimedia content is generated based on the understanding information of the multimedia content in the historical browsing history and the similarity information in the understanding information of the displayed multimedia content; based on the summary information, a message to be sent by the intelligent agent is generated. For example, if the user is currently browsing a video of a certain food being made, other videos of the same food being made that the user has browsed can be summarized together to extract the key elements of making that food.

[0079] In some embodiments, comparison information between the multimedia content in the historical browsing history and the displayed multimedia content is generated based on the differences between the understanding information of the multimedia content in the historical browsing history and the understanding information of the displayed multimedia content; based on the comparison information, a message to be sent by the intelligent agent is generated. For example, if the user is currently browsing a video about making a food dish with a lot of content, this video can be compared with other videos of making the same dish that the user has browsed, to highlight the special techniques used in the current video.

[0080] The above embodiments generate different messages based on the consumption type of multimedia content, thereby enabling the generated messages to have a high degree of matching with the multimedia content, thus improving the user's multimedia content browsing experience.

[0081] As mentioned in the foregoing embodiments, users can actively send messages to the agent while browsing multimedia content. Besides instructions related to understanding the multimedia content, users can also send other types of information to the agent. For example, users can use the agent to provide feedback on the content recommended in the recommendation stream. An embodiment of the interaction method for adjusting recommendation strategies disclosed herein is described below with reference to FIG5.

[0082] Figure 5 shows a flowchart illustrating an interaction method according to some other embodiments of the present disclosure. As shown in Figure 5, the interaction method of this embodiment includes steps S502 to S504.

[0083] In step S502, it is determined whether the message sent by the user includes an intent to adjust the recommendation strategy. This step can be obtained through semantic analysis, for example, by processing the message sent by the user using a machine learning model with semantic analysis capabilities.

[0084] In step S504, in response to a message sent by the user that includes an intention to adjust the recommendation strategy, the recommendation strategy for multimedia content is adjusted based on the message sent by the user.

[0085] When a user sends a message that includes the intention to adjust the recommendation strategy, the message itself may directly reveal the user's intention, such as "I want to watch more pet videos"; or it may require combining the currently playing multimedia content to determine the user's intention, such as "I like watching this type of video, recommend more to me." In this case, "this type" needs to be determined by combining the analysis results of the currently playing multimedia content to determine which type of video the user likes to watch.

[0086] In some embodiments, in response to a user-sent message including an intent to adjust the recommendation strategy, it is determined whether the user-sent message includes a reference to the displayed multimedia content; in response to the user-sent message including a reference to the played multimedia content, the recommendation strategy for the multimedia content is adjusted based on the user-sent message and the understanding information regarding the displayed multimedia content. The reference to the displayed multimedia content is, for example, various pronouns such as "this," "this kind," etc., thereby enabling it to be determined that the user's feedback is based on the currently displayed multimedia content. In this embodiment, the adjustment benchmark can be determined based on the understanding information, such as the type or tag of the currently displayed multimedia content, and then the adjustment direction can be determined based on the user-sent message. For example, "send more videos like this" means adding multimedia content with the type or tag of the currently displayed multimedia content.

[0087] Through the above embodiments, users can adjust the recommendation strategy of the recommendation stream by conversing with the intelligent agent. Therefore, users can provide more flexible and detailed feedback on recommendation needs, thereby improving the efficiency of user feedback and user experience.

[0088] The various method embodiments of this disclosure have been described above by way of example. The apparatus for implementing the methods of the above embodiments is further described below.

[0089] Figure 6 shows a schematic diagram of the structure of an interactive device according to some embodiments of the present disclosure. As shown in Figure 6, the interactive device 60 of this embodiment includes: a first display module 601 configured to display multimedia content on a playback interface; a determination module 602 configured to determine the invoked object based on the multimedia content; a second display module 603 configured to, in response to the invoked object being an intelligent agent, display a message sent by the intelligent agent in the playback interface through a message control, the message being obtained by understanding the multimedia content; and a third display module 604 configured to, in response to a user's triggering operation on the message control, display a dialogue interface between the user and the intelligent agent.

[0090] In some embodiments, the determining module 602 is further configured to: determine whether the user intends to continue consuming the multimedia content based on the multimedia content; and in response to the user's intention to continue consuming the multimedia content, select an agent from one or more candidate agents as the object to be invoked, based on the matching result of the understanding information of the multimedia content and one or more candidate agents.

[0091] In some embodiments, the determining module 602 is further configured to: select a sub-application from one or more candidate sub-applications as the object to be invoked in response to the user's lack of intention to continue consuming.

[0092] In some embodiments, the interactive device 60 further includes a generation module 605, configured to: determine the consumption type of the multimedia content based on the understanding information of the multimedia content, the consumption type including at least one of in-depth consumption, extended consumption, and assisted understanding consumption; and generate a message based on the understanding information of the multimedia content and the consumption type.

[0093] In some embodiments, the generation module 605 is further configured to: determine one or more associated objects of the multimedia content based on the understanding information of the multimedia content, wherein the one or more associated objects include at least one of associated functions in the application, searched associated information, and user-authorized associated browsing history; determine a target associated object from the one or more associated objects according to the degree of association between each type of associated object and the understanding information; and determine the consumption type corresponding to the target associated object, wherein the associated function corresponds to in-depth consumption, the associated information corresponds to extended consumption, and the associated browsing history corresponds to assisted understanding consumption.

[0094] In some embodiments, the consumption type is in-depth consumption, and the generation module 605 is further configured to generate a message for recommending associated functions based on the understanding information of the multimedia content.

[0095] In some embodiments, the consumption type is extended consumption, and the generation module 605 is further configured to generate a message describing the associated information based on the understanding information of the multimedia content.

[0096] In some embodiments, the consumption type is assisted understanding consumption, and the generation module 605 is further configured to: generate summary information on the multimedia content in the historical browsing record and the multimedia content displayed, based on the understanding information of the multimedia content in the historical browsing record and the similarity information in the understanding information of the displayed multimedia content; and generate a message to be sent by the intelligent agent based on the summary information.

[0097] In some embodiments, the consumption type is assisted understanding consumption, and the generation module 605 is further configured to: generate comparison information between the multimedia content in the historical browsing record and the displayed multimedia content based on the difference information between the understanding information of the multimedia content in the historical browsing record and the understanding information of the displayed multimedia content; and generate a message to be sent by the intelligent agent based on the comparison information.

[0098] In some embodiments, the interactive device 60 further includes an understanding module 606.

[0099] In some embodiments, the understanding module 606 is configured to understand the multimedia content in response to its display.

[0100] In some embodiments, the playback interface further includes an input control, and the understanding module 606 is configured to: obtain messages sent by the user to the intelligent agent through the input control; and understand the multimedia content based on the indication information in the messages sent by the user.

[0101] In some embodiments, the second display module 603 is further configured to display messages sent by the robot to the user via dialog boxes, icons, or overlays.

[0102] In some embodiments, the multimedia content is content in the recommendation stream of multimedia content, and the interactive device 60 further includes: an adjustment module 607, configured to: determine whether a message sent by a user includes an intention to adjust the recommendation strategy; and, in response to the user sending a message including an intention to adjust the recommendation strategy, adjust the recommendation strategy of the multimedia content based on the message sent by the user.

[0103] In some embodiments, the adjustment module 607 is further configured to: in response to a message sent by a user including an intent to adjust the recommendation strategy, determine whether the message sent by the user includes a reference to the displayed multimedia content; in response to a message sent by the user including a reference to the played multimedia content, adjust the recommendation strategy for the multimedia content based on the message sent by the user and the understanding information of the displayed multimedia content.

[0104] It should be noted that the above-described units are merely logical modules divided according to their specific functions, and are not intended to limit the specific implementation method. For example, they can be implemented in software, hardware, or a combination of both. In actual implementation, the above-described units can be implemented as independent physical entities, or they can be implemented by a single entity (e.g., a processor (CPU or DSP, etc.), integrated circuit, etc.). Furthermore, the units shown in the accompanying drawings with dashed lines indicate that these units may not actually exist, and the operations / functions they perform can be implemented by the processing circuitry itself.

[0105] In addition, although not shown, the device may also include a memory that can store various information generated by the device and its constituent units during operation, programs and data used for operation, data to be transmitted by the communication unit, etc. The memory can be volatile memory and / or non-volatile memory. For example, the memory may include, but is not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. Of course, the memory may also be located outside the device. Optionally, although not shown, the device may also include a communication unit that can be used to communicate with other devices. In one example, the communication unit can be implemented in a manner known in the art, such as including communication components such as antenna arrays and / or radio frequency links, various types of interfaces, communication units, etc. These will not be described in detail here. Furthermore, the device may also include other components not shown, such as radio frequency links, baseband processing units, network interfaces, processors, controllers, etc. These will not be described in detail here.

[0106] Some embodiments of this disclosure also provide an electronic device. Figure 7 shows a schematic diagram of the structure of an electronic device according to some embodiments of this disclosure. For example, in some embodiments, the electronic device 7 can be various types of devices, such as mobile terminals including but not limited to mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. For example, the electronic device 7 may include a display panel for displaying data and / or execution results utilized in the scheme according to this disclosure. For example, the display panel can be of various shapes, such as a rectangular panel, an elliptical panel, or a polygonal panel. In addition, the display panel can be not only a planar panel, but also a curved panel, or even a spherical panel.

[0107] As shown in FIG. 7, the electronic device 7 of this embodiment includes a memory 71 and a processor 72 coupled to the memory 71. It should be noted that the components of the electronic device 7 shown in FIG. 7 are merely exemplary and not limiting; the electronic device 7 may also have other components depending on the actual application requirements. The processor 72 can control other components in the electronic device 7 to perform desired functions.

[0108] In some embodiments, memory 71 is used to store one or more computer-readable instructions. When processor 72 executes the computer-readable instructions, the computer-readable instructions are executed by processor 72 to implement the method according to any of the above embodiments. For specific implementations and related explanations of the various steps of the method, please refer to the above embodiments; repeated details will not be elaborated here.

[0109] For example, processor 72 and memory 71 can communicate with each other directly or indirectly. For example, processor 72 and memory 71 can communicate via a network. The network can include wireless networks, wired networks, and / or any combination of wireless and wired networks. Processor 72 and memory 71 can also communicate with each other via a system bus, which is not limited in this disclosure.

[0110] For example, processor 72 can be embodied in various suitable processors, processing devices, such as central processing unit (CPU), graphics processing unit (GPU), network processor (NP), etc.; it can also be digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The central processing unit (CPU) can be an x86 or ARM architecture, etc. For example, memory 71 can include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Memory 71 can include, for example, system memory, which stores, for example, the operating system, application programs, boot loader, database, and other programs. Various application programs and various data can also be stored in the storage medium.

[0111] Furthermore, according to some embodiments of this disclosure, various operations / processes according to this disclosure, implemented via software and / or firmware, can install programs constituting the software from a storage medium or network onto a computer system with a dedicated hardware architecture, such as the computer system 80 shown in FIG. 8. When various programs are installed, the computer system is capable of performing various functions, including those described above. FIG. 8 shows a schematic diagram of the structure of a computer system according to some embodiments of this disclosure.

[0112] In Figure 8, the Central Processing Unit (CPU) 801 performs various processes according to a program stored in the Read-Only Memory (ROM) 802 or a program loaded from the storage portion 808 into the Random Access Memory (RAM) 803. The RAM 803 also stores data required as needed when the CPU 801 performs various processes, etc. The CPU is merely exemplary and can also be other types of processors, such as the various processors described above. The ROM 802, RAM 803, and storage portion 808 can be various forms of computer-readable storage media, as described below. It should be noted that although the ROM 802, RAM 803, and storage device 808 are shown separately in Figure 8, one or more of them can be combined or located in the same or different memories or storage modules.

[0113] CPU 801, ROM 802 and RAM 803 are interconnected via bus 804. Input / output interface 805 is also connected to bus 804.

[0114] The following components are connected to the input / output interface 805: input section 806, such as a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output section 807, including displays such as cathode ray tube (CRT), liquid crystal display (LCD), speakers, vibrators, etc.; storage section 808, including hard disks, magnetic tapes, etc.; and communication section 809, including network interface cards such as LAN cards, modems, etc. The communication section 809 allows communication processing to be performed via a network such as the Internet. It is readily understood that although the various devices or modules in the computer system 80 shown in Figure 8 communicate via bus 804, they can also communicate via a network or other means, wherein the network can include wireless networks, wired networks, and / or any combination of wireless and wired networks.

[0115] As needed, drive 810 is also connected to input / output interface 805. Removable media 811, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 810 as needed, so that computer programs read from them can be installed into storage section 808 as needed.

[0116] When the above series of processes are implemented through software, the program constituting the software can be installed from a network such as the Internet or from a storage medium such as removable media 811.

[0117] According to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the CPU 801, it performs the functions defined in the methods of embodiments of this disclosure.

[0118] It should be noted that, in the context of this disclosure, a computer-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0119] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0120] In some embodiments, a computer program is also provided, comprising: instructions that, when executed by a processor, cause the processor to perform the method of any of the above embodiments. For example, the instructions may be embodied in computer program code.

[0121] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0122] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0123] The modules, components, or units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.

[0124] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0125] The above description is merely an embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0126] Many specific details are set forth in the description provided herein. However, it is understood that embodiments of the invention may be practiced without these specific details. In other instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of the description.

[0127] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0128] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. An interaction method, comprising: Display multimedia content on the playback interface; Based on the multimedia content, determine the object to be invoked; In response to the fact that the invoked object is an agent, the message sent by the agent is displayed in the playback interface through a message control. The message is obtained by understanding the multimedia content. In response to the user's triggering operation on the message control, the dialogue interface between the user and the intelligent agent is displayed.

2. The interaction method according to claim 1, wherein, The determination of the invoked object based on the multimedia content includes: Based on the multimedia content, determine whether the user intends to continue consuming the multimedia content; In response to the user's intention to continue consuming, an agent is selected as the invoked object from one or more candidate agents based on the matching results of the understanding information of the multimedia content and the one or more candidate agents.

3. The interaction method according to claim 2 further includes: In response to the user's lack of intent to continue consuming, a sub-application is selected from one or more candidate sub-applications as the object to be invoked.

4. The interaction method according to any one of claims 1 to 3, further comprising: Based on the understanding information of the multimedia content, the consumption type of the multimedia content is determined, and the consumption type includes at least one of in-depth consumption, extended consumption, and comprehension-aid consumption; The message is generated based on the understanding of the multimedia content and the consumption type.

5. The interaction method according to claim 4, wherein, Determining the consumption type of the multimedia content based on the understanding information of the multimedia content includes: Based on the understanding information of the multimedia content, one or more associated objects of the multimedia content are determined, wherein the one or more associated objects include at least one of the following: associated functions in the application, associated information found through search, and associated browsing history authorized by the user. Based on the degree of association between each type of associated object and the understood information, a target associated object is determined from the one or more associated objects; Determine the consumption type corresponding to the target associated object, wherein the associated function corresponds to in-depth consumption, the associated information corresponds to extended consumption, and the associated browsing history corresponds to consumption aided in understanding.

6. The interaction method according to claim 5, wherein, The consumption type is in-depth consumption, and the generation of the message based on the understanding information of the multimedia content and the consumption type includes: Based on the understanding of the multimedia content, a message is generated to recommend the associated functions.

7. The interaction method according to claim 5 or 6, wherein, The consumption type is extended consumption, and the generation of the message based on the understanding information of the multimedia content and the consumption type includes: Based on the understanding of the multimedia content, a message is generated to describe the associated information.

8. The interaction method according to any one of claims 5 to 7, wherein, The consumption type is assisted understanding consumption. Generating the message based on the understanding information of the multimedia content and the consumption type includes: Based on the understanding information of the multimedia content in the browsing history and the similarity information in the understanding information of the displayed multimedia content, a summary information of the multimedia content in the browsing history and the displayed multimedia content is generated. Based on the summarized information, the message to be sent by the intelligent agent is generated.

9. The interaction method according to any one of claims 5 to 8, wherein, The consumption type is assisted understanding consumption. Generating the message based on the understanding information of the multimedia content and the consumption type includes: Based on the understanding information of the multimedia content in the browsing history and the difference information in the understanding information of the displayed multimedia content, comparison information of the multimedia content in the browsing history and the displayed multimedia content is generated. Based on the comparison information, the message to be sent by the intelligent agent is generated.

10. The interaction method according to any one of claims 1 to 9, further comprising: In response to the display of the multimedia content, the multimedia content is understood.

11. The interaction method according to any one of claims 1 to 10, wherein, The playback interface also includes input controls, and further includes: The user sends a message to the intelligent agent through the input control; The multimedia content is understood based on the instructions in the messages sent by the user.

12. The interaction method according to claim 10 or 11, wherein, The message sent by the agent is displayed via a message control, including: The message sent by the robot to the user is displayed through a dialog box, icon, or overlay.

13. The interaction method according to claim 11 or 12, wherein, The multimedia content is content from the multimedia content recommendation stream, and the interaction method further includes: Determine whether the message sent by the user includes an intention to adjust the recommendation strategy; In response to a message sent by the user that includes the intent to adjust the recommendation strategy, the recommendation strategy for multimedia content is adjusted based on the message sent by the user.

14. The interaction method according to claim 13, wherein, The response to the message sent by the user includes the intent to adjust the recommendation strategy. Adjusting the recommendation strategy for multimedia content based on the message sent by the user includes: In response to the user-sent message including the intent to adjust the recommendation strategy, determine whether the user-sent message includes a reference to the displayed multimedia content; In response to a message sent by the user that includes a reference to the multimedia content being played, the recommendation strategy for the multimedia content is adjusted based on the message sent by the user and the understanding information regarding the displayed multimedia content.

15. An interactive device, comprising: The first display module is configured to display multimedia content on the playback interface; The determination module is configured to determine the invoked object based on the multimedia content; The second display module is configured to, in response to the called object being an agent, display a message sent by the agent in the playback interface via a message control, the message being obtained by understanding the multimedia content; The third display module is configured to display the dialogue interface between the user and the intelligent agent in response to the user's triggering operation on the message control.

16. An electronic device comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the interaction method as described in any one of claims 1 to 14 based on instructions stored in the memory.

17. A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the interactive method of any one of claims 1 to 14.

18. A computer program product, when run on a computer, causes the computer to implement the interactive method of any one of claims 1 to 14.

19. A computer program comprising: Instructions, which, when executed by a processor, cause the processor to perform the interaction method according to any one of claims 1 to 14.

Images