Interaction method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to an interaction method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence and computers. The interaction method comprises the steps that in response to visual media content input by a user, the visual media content is understood, one or more pieces of recommendation demand information matched with the visual media content are determined, and the one or more pieces of recommendation demand information are used for representing one or more intentions for the visual media content; displaying one or more items of recommendation demand information; in response to a trigger operation of a user on target recommendation demand information in the one or more items of recommendation demand information, determining a working link corresponding to the target recommendation demand information; and based on the visual media content and the working link, generating answer content and displaying the answer content.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence and computer technology, and in particular to an interaction method, device, electronic device, storage medium and program product. Background Art

[0002] With the development of artificial intelligence (AI) technology, AI-based applications or agents can provide users with more and more diverse functions. For example, applications or agents can generate text, generate videos, conduct question-and-answer sessions, etc. based on images input by users. Summary of the invention

[0003] According to some embodiments of the present disclosure, an interaction method is provided, including: in response to user input of visual media content, understanding the visual media content, determining one or more recommendation demand information matching the visual media content, wherein the one or more recommendation demand information are used to characterize one or more intentions for the visual media content; displaying one or more recommendation demand information; in response to the user's triggering operation on target recommendation demand information in the one or more recommendation demand information, determining a working link corresponding to the target recommendation demand information; based on the visual media content and the working link, generating answer content and displaying the answer content.

[0004] According to some other embodiments of the present disclosure, an interactive device is provided, including: a first determination module, configured to respond to user input of visual media content, understand the visual media content, and determine one or more recommendation demand information matching the visual media content, wherein the one or more recommendation demand information are used to characterize one or more intentions for the visual media content; a first display module, configured to display one or more recommendation demand information; a second determination module, configured to respond to a user triggering operation on a target recommendation demand information in the one or more recommendation demand information, and determine a working link corresponding to the target recommendation demand information; a generation module, configured to generate answer content based on the visual media content and the working link; and a second display module, configured to display the answer content.

[0005] According to some further embodiments of the present disclosure, an electronic device is provided, comprising: a processor; and a memory coupled to the processor, for storing instructions, which, when executed by the processor, causes the processor to execute an interaction method as in any embodiment of the present disclosure.

[0006] According to some further embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the program is executed by a processor, the interactive method of any embodiment of the present disclosure is implemented.

[0007] According to some further embodiments of the present disclosure, a computer program product is provided, comprising: instructions, which, when executed by a processor, cause the processor to execute an interaction method as described in any embodiment of the present disclosure.

[0008] Other features, aspects and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The following is an explanation of the embodiments of the present disclosure with reference to the accompanying drawings. It should be understood that the drawings described below only relate to some embodiments of the present disclosure and do not constitute a limitation to the present disclosure. In the accompanying drawings:

[0010] Figure 1 A schematic diagram showing a flow chart of an interaction method according to some embodiments of the present disclosure;

[0011] Figures 2 to 7 A schematic diagram showing an interface of some embodiments of the present disclosure;

[0012] Figure 8 A schematic diagram showing the structure of an interactive device in some embodiments of the present disclosure;

[0013] Fig. 9 A schematic diagram showing the structure of an electronic device according to some embodiments of the present disclosure;

[0014] Fig.10 A schematic diagram showing the structure of an electronic device according to some other embodiments of the present disclosure. DETAILED DESCRIPTION

[0015] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present disclosure. It should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein.

[0016] It should be understood that the various steps described in the method embodiments of the present disclosure can be performed in different orders and / or performed in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement of the steps set forth in these embodiments should be interpreted as being merely exemplary and not limiting the scope of the present disclosure.

[0017] The term “including” and its variations used in the present disclosure are intended to be open terms that include at least the following elements / features but do not exclude other elements / features, that is, “including but not limited to.” The term “based on” means “at least partly based on.”

[0018] It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. Unless otherwise specified, the concepts of "first", "second", etc. are not intended to imply that the objects described in this way must be in a given order in time, space, ranking, or any other manner.

[0019] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0020] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0021] It should be understood that the present disclosure does not limit how to obtain the image to be applied / processed. In some embodiments of the present disclosure, it can be obtained from a storage device, such as an internal memory or an external storage device. In other embodiments of the present disclosure, a photographic component can be mobilized to shoot. It should be noted that the acquired image can be a captured image or a frame of an image in a captured video, and is not particularly limited to this.

[0022] In the context of the present disclosure, an image may refer to any of a variety of images, such as a color image, a grayscale image, etc. It should be noted that in the context of the present specification, the type of image is not specifically limited. In addition, the image may be any appropriate image, such as an original image obtained by a camera device, or an image that has been subjected to specific processing, such as preliminary filtering, anti-aliasing, color adjustment, contrast adjustment, normalization, etc. It should be noted that the preprocessing operation may also include other types of preprocessing operations known in the art, which will not be described in detail here.

[0023] The embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings, but the present disclosure is not limited to these specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. In addition, in one or more embodiments, specific features, structures or characteristics can be combined in any suitable manner that will be clear from the present disclosure by a person of ordinary skill in the art.

[0024] At present, some applications or intelligent agents can provide users with a variety of image processing functions. However, if users want to use a certain function, they need to enter the corresponding instructions (prompt information). For example, if a user wants to beautify an image, he needs to enter instructions to beautify the image, such as "Please help me retouch the picture." In many cases, the way of entering instructions to call a specific processing function will make users feel cumbersome and inefficient, and users may find it difficult to accurately express instructions, making it difficult for the image processing results to meet the user's needs.

[0025] Based on the above problems, the present disclosure provides an interactive method, in response to a user inputting visual media content, through understanding the visual media content, determining one or more recommended demand information matching the visual media content, and displaying one or more recommended demand information. One or more recommended demand information is used to characterize one or more intentions for the visual media content. Since one or more recommended demand information is determined based on the understanding of the visual media content, the one or more recommended demand information is closer to the user's real needs, which is convenient for the user to quickly locate the target recommended demand information, and for different visual media content, the one or more recommended demand information is dynamically adjusted and changed. In response to the user's triggering operation on the target recommended demand information in the one or more recommended demand information, the working link corresponding to the target recommended demand information is determined, and then based on the visual media content and the working link, the answer content is generated and displayed. In the whole interactive process, the user does not need to input instructions by himself, but only needs to trigger the target recommended demand information to obtain the answer content, thereby improving the interaction efficiency from the visual media content to the answer content. And based on the target recommended demand information, the corresponding working link is called to generate the answer content, thereby improving the accuracy and efficiency of generating the answer content.

[0026] Combine the following Figures 1 to 7 The interactive method of the present disclosure is described. The interactive method of the present disclosure can be executed by at least one of an application, an agent, a robot (Bot), a multimedia content generation device, and an electronic device, but is not limited to the examples given.

[0027] Figure 1 Flowcharts of some embodiments of the interactive method disclosed herein. Figure 1 As shown, the method of this embodiment includes: steps S102 to S108.

[0028] In step S102, in response to the user inputting visual media content, the visual media content is understood and one or more pieces of recommendation demand information matching the visual media content are determined.

[0029] For example, the user may input visual media content in an interactive interface, and the interactive interface may be an interactive interface provided by an application, an intelligent agent, a robot, a multimedia content generating device, an electronic device, etc. The visual media content may be obtained from an internal storage device or an external storage device of an electronic device (such as a terminal) used by the user, or may be captured by calling a capturing component in the electronic device used by the user, and is not limited to the examples given.

[0030] For example, visual media content includes images, videos, and other content that transmit information through visual channels. For example, a machine learning model can be used to understand the visual media content and determine one or more pieces of recommended demand information that match the visual media content. The one or more pieces of recommended demand information determined may be different for different visual media contents.

[0031] One or more pieces of recommended demand information correspond to one or more functions that can be realized. Recommended demand information can be understood as instructions recommended to users, and users can select a certain piece of recommended demand information as an instruction (or prompt information) to instruct applications or agents to perform corresponding processing. One or more pieces of recommended demand information are used to represent one or more intentions for visual media content, that is, one or more pieces of recommended demand information can represent one or more intentions of users predicted based on visual media content.

[0032] For example, one or more recommendation demand information may include different types of recommendation demand information, may include recommendation demand information for modifying visual media, may include recommendation demand information for asking questions to visual media content, may include recommendation demand information for generating answer information based on visual media content and feeding it back into the visual media content, etc., without limitation to the examples cited. Different types of recommendation demand information may be further subdivided, as will be described in subsequent embodiments.

[0033] In step S104, one or more pieces of recommended demand information are displayed.

[0034] The visual media content input by the user may be displayed in the interactive interface or in the preview interface. One or more pieces of recommended demand information may be displayed in the interactive interface or the preview interface. The one or more pieces of recommended demand information may be interactive elements in the form of controls, items, menus, etc., and are not limited to the examples given. The display interface and display method of the one or more pieces of recommended demand information are not limited to the examples given.

[0035] In step S106, in response to the user's triggering operation on the target recommended requirement information among the one or more recommended requirement information, a working link corresponding to the target recommended requirement information is determined.

[0036] The user can trigger the target recommendation requirement information by clicking, selecting, etc. The work link corresponding to the target recommendation requirement information may include one or more work nodes, and the one or more work nodes include at least one of a model, an application, a plug-in, and an intelligent agent. Different types of target recommendation requirement information may correspond to different work links.

[0037] In step S108, answer content is generated based on the visual media content and the working link and the answer content is displayed.

[0038] One or more working nodes in the working link can be used to process the visual media content in a working order, and finally generate answer content for the target recommendation requirement information. The answer content can be displayed in an interactive interface, in the visual media content, or in a variety of forms, not limited to the examples given.

[0039] The interactive method of the above embodiment, in response to the user inputting visual media content, determines one or more recommended demand information matching the visual media content through understanding the visual media content, and displays one or more recommended demand information. One or more recommended demand information is used to characterize one or more intentions for the visual media content. Since one or more recommended demand information is determined based on the understanding of the visual media content, the one or more recommended demand information is closer to the user's real needs, which is convenient for the user to quickly locate the target recommended demand information, and for different visual media content, one or more recommended demand information is dynamically adjusted and changed. In response to the user's triggering operation on the target recommended demand information in the one or more recommended demand information, the working link corresponding to the target recommended demand information is determined, and then based on the visual media content and the working link, the answer content is generated and displayed. In the whole interactive process, the user does not need to input instructions by himself, but only needs to trigger the target recommended demand information to obtain the answer content, which improves the interaction efficiency from the visual media content to the answer content. And based on the target recommended demand information, the corresponding working link is called to generate the answer content, which improves the accuracy and efficiency of generating the answer content.

[0040] In addition, displaying one or more recommended demand information based on the understanding of visual media content is more in line with the user's intentions and needs than displaying some fixed recommended demand information, improves the effectiveness of the recommended demand information, and also enables users to locate target recommended demand information more quickly.

[0041] The following describes how a user inputs visual media content, how to trigger understanding of the visual media content, how to determine one or more pieces of recommended demand information matching the visual media content, and how to display one or more pieces of recommended demand information.

[0042] In some embodiments, a camera control is displayed; in response to a user triggering the camera control, a preview interface is displayed; in response to a user completing a shooting operation of visual media content, the visual media content is understood, and one or more pieces of recommended demand information matching the visual media content are determined, wherein the one or more pieces of recommended demand information are displayed in the preview interface.

[0043] For example, a camera control may be displayed in the interactive interface, and a user triggering the camera control in the interactive interface may display a preview interface, in which a picture captured by the camera lens may be displayed. A shooting control may be displayed in the preview interface, and a user may shoot visual media content by triggering the shooting control. In response to the user completing shooting of the visual media content, the visual media content may be displayed in the preview interface.

[0044] One or more recommended demand information can also be displayed in the shooting and reading interface. Figure 2 As shown in FIG. 1 , it is a preview interface that can display an image (visual media content) 201 and multiple recommendation demand information 202 to 204. Figure 2 The preview interface shown may also display a selection control 205. In response to the user triggering the selection control 205, multiple candidate visual media contents may be displayed. In response to the user selecting a certain visual media content from the multiple candidate visual media contents, the selected visual media content may be displayed in the preview interface. The preview interface may also display a download 206, which may be used to download the visual media content. The preview interface may also include a rotation control 207, which may be used to rotate the visual media content to a specific angle for display. The preview interface may also include a backoff control 208, which may be used to cancel the display of the current visual media content and redisplay the image captured by the camera lens. The preview interface may also include an input control 210, which may display an input area in response to the user triggering the input control 210. In response to the user inputting prompt information in the input area and confirming it, a reply information is generated and displayed according to the visual media content and the prompt information. The user may also choose to input the prompt information (or instruction) himself to trigger the processing of the visual media content.

[0045] Based on the method of the above embodiment, the user can directly shoot the visual media content, automatically determine one or more recommended demand information and display them without the user having to input instructions, thereby improving the display efficiency of one or more recommended demand information and enhancing the user experience.

[0046] In some embodiments, a selection control is displayed; in response to a user triggering the selection control, one or more candidate visual media content is displayed; in response to a user selecting a visual media content from the one or more candidate visual media content, the visual media content is displayed in a preview interface, the visual media content is understood, and one or more recommendation demand information matching the visual media content is determined, wherein the one or more recommendation demand information are displayed in the preview interface.

[0047] For example, you can display a selection control in the interactive interface, or you can display a selection control in the preview interface. Figure 3 As shown in FIG. 3 , it is a preview interface that can display an image (visual media content) 301 and multiple recommendation demand information 302-303. Figure 3 The preview interface shown may also display a selection control 304, which may be used to reselect visual media content. The preview interface may also display a download 305, which may be used to download visual media content. The preview interface may also include a rotation control 306, which may be used to rotate the visual media content to a specific angle for display. The preview interface may also include a backoff control 307, which may be used to cancel the display of the current visual media content and redisplay multiple candidate visual media content. The preview interface may also include an input control 309, which displays an input area in response to a user triggering the input control 309, and generates and displays reply information based on the visual media content and the prompt information in response to the user inputting and confirming the prompt information in the input area.

[0048] Based on the method of the above embodiment, the user can automatically determine and display one or more recommended demand information by selecting existing visual media content, without the user having to input instructions, thereby improving the display efficiency of one or more recommended demand information and enhancing the user experience. The user can input visual media content in different ways to facilitate user use.

[0049] In addition to displaying one or more visual media content in the preview interface, one or more visual media content may also be displayed in the interactive interface. In some embodiments, a confirmation control is displayed in the preview interface; in response to the user triggering the confirmation control, the interactive interface is displayed, and one or more recommendation demand information is displayed in the interactive interface.

[0050] like Figure 2 As shown, a confirmation control 209 is displayed in the preview interface, such as Figure 3 As shown, the preview interface displays a confirmation control 308. For example, in response to the user's triggering operation on the confirmation control 308, the following may be displayed: Figure 4 The interactive interface shown. Multiple pieces of recommended demand information 403, 404 may be displayed in the interactive interface. The recommended demand information displayed in the interactive interface may be the same as or different from that in the preview interface. Figure 4As shown, a thumbnail 402 of the visual media content and multiple pieces of recommended demand information 403 and 404 can be displayed in the input display area. An input box 405 can also be displayed in the input display area, and the prompt information (or instruction) input by the user can be displayed in the input box 405. After the confirmation control 209 is triggered, a Figure 4 Similar interactive interfaces will not be described here.

[0051] One or more recommendation demand information can be displayed in multiple interfaces and multiple distribution scenarios, which improves the convenience of users in selecting recommendation demand information, and by setting multiple display and interaction forms, it is convenient for users to use different interaction methods to trigger the processing of visual media content.

[0052] In addition to displaying the visual media content and one or more pieces of recommended demand information in the preview interface, annotation information of the content related to one or more pieces of recommended demand information may also be displayed. Figure 5 As shown, the preview interface displays annotation information 501-504 of multiple questions in the image, as well as recommendation requirement information 505.

[0053] By displaying the annotation information, users can more clearly and intuitively determine the recognition results of the visual media content, as well as the association between the recommended demand information and the visual media content, thereby improving the display effect.

[0054] After receiving the visual media content input by the user, the visual media content is understood to determine one or more recommendation demand information matching the visual media content. The following describes how to determine one or more recommendation demand information in conjunction with some embodiments.

[0055] In some embodiments, a machine learning model is used to understand visual media content to determine at least one of an entity or a scene in the visual media content; based on at least one of the entity or the scene, one or more recommendation demand information matching the visual media content is determined.

[0056] Entities and scenes in visual media content are the main elements. For visual media content that includes different types of entities, users may have different needs. For example, for visual media content that includes people, users may prefer to beautify it, for visual media content that includes objects, users may prefer to ask relevant questions about the objects, and for visual media content that includes text, users may prefer to process the text. For visual media content that includes different types of scenes, users may also have different needs. For example, for landscape scenes, users may prefer to generate matching text, and for indoor scenes, users may prefer to identify and ask questions about the objects therein, etc., without limitation to the examples given.

[0057] For example, one or more pieces of recommended requirement information matching the entity type may be determined, and / or one or more pieces of recommended requirement information matching the scene type may be determined.

[0058] The method of the above embodiment determines at least one of the entities and scenes in the visual media content by understanding the visual media content, and then determines one or more pieces of recommendation demand information, which can improve the accuracy of determining one or more pieces of recommendation demand information.

[0059] In some embodiments, a machine learning model is used to understand visual media content to determine at least one of the entities and scenes in the visual media content; based on at least one of the entities and scenes, one or more of the first type of recommendation demand information, the second type of recommendation demand information, and the third type of recommendation demand information are determined, wherein the first type of recommendation demand information includes recommendation demand information for modifying the visual media content, the second type of recommendation demand information includes recommendation demand information for raising questions about the visual media content, and the third type of recommendation demand information includes recommendation demand information that generates reply information based on the visual media content and feeds back into the visual media content.

[0060] Different types of recommendation demand information (or recommendation instructions) can be divided according to the specific processing methods of visual media content. The first type and the second type of recommendation demand information can be further subdivided. For example, the first type of recommendation demand information can be further divided into recommendation demand information such as modifying the visual media content based on one or more specific beautification methods, generating another type of visual media content based on the current type of visual media content, etc. based on the modification method. For example, one or more specific beautification methods include: beautifying characters, adding filters, changing the style of painting, adding special effects, changing colors, etc., one or more combinations thereof, not limited to the examples given. For example, the current type of visual media content is an image, and the generated another type of visual media content is a video.

[0061] For example, the second type of recommendation demand information can be further differentiated into recommendation demand information such as asking questions associated with entities in visual media content, asking questions associated with scenes in visual media content, asking questions about generating specific text based on visual media content, asking questions about generating music based on visual media content, etc., without being limited to the examples given.

[0062] The type of one or more recommendation demand information can be determined according to at least one of the entities and scenes in the visual media content. For example, in response to determining that the entity in the visual media content includes a person, one or more recommendation demand information in the first type of recommendation demand information is determined. For example, in response to determining that the entity in the visual media content includes at least one entity of food, articles, animals, plants, and buildings, one or more recommendation demand information in the second type of recommendation demand information is determined. For example, in response to determining that the entity in the visual media content includes a character, one or more recommendation demand information in the third type of recommendation demand information is determined. For example, in response to determining that the scene in the visual media content is a landscape, one or more recommendation demand information in the first type of recommendation demand information and the second type of recommendation demand information are determined. For example, in response to determining that the scene in the visual media content is a learning scene, and the entity includes a character, one or more recommendation demand information in the third type of recommendation demand information is determined. Specifically, the type of one or more recommendation demand information and the specific one or more recommendation demand information can be determined by the understanding of the visual media content by the machine learning model, which is not limited to the above examples.

[0063] After determining the type of one or more pieces of recommendation demand information, one or more pieces of recommendation demand information may be determined based on at least one of an entity and a scene in the visual media content.

[0064] For example, Figure 2 As shown, by understanding the image and determining that the entity is an object, and recognizing that it is chocolate, the second type of recommended demand information can be determined, and the recommended demand information is determined to be "what kind of chocolate is this". By recognizing that there is text on the object, the third type of recommended demand information can be determined, and the recommended demand information is determined to be "translation", and it is implicitly indicated that the translated text is directly displayed in the image. In response to the user selecting "Translate", a prompt message can be sent to the work chain. For example, the prompt message can indicate "translate the text in the image, and display the translated text in the image at the position corresponding to the original text". As shown Figure 3 As shown, by understanding the image, determining that the entity is a book, and identifying the characters involved in the book, the second type of recommendation demand information can be determined, and the recommendation demand information can be determined as "Which innovative ideas of XXX are mentioned in the book" and "What kind of book is this".

[0065] The method of the above embodiment determines at least one of the entities and scenes in the visual media content based on the understanding of the visual media content, and then determines one or more of the first type of recommendation demand information, the second type of recommendation demand information, and the third type of recommendation demand information. This can improve the accuracy of determining the recommendation demand information, and different types of recommendation demand information can be determined for different visual media content, thereby improving the matching degree between the recommendation demand information and the visual media content, and better meeting the needs of users.

[0066] In some embodiments, the visual media content includes multiple images, and at least one of the entities and scenes in the visual media content includes at least one of the entities and scenes of each of the multiple images; determining one or more of the first type of recommendation demand information, the second type of recommendation demand information, and the third type of recommendation demand information based on at least one of the entities and scenes includes: determining themes and key elements of the multiple images based on at least one of the entities and scenes of each image; determining one or more of the first type of recommendation demand information, the second type of recommendation demand information, and the third type of recommendation demand information based on themes and key elements of the multiple images.

[0067] The visual media content may include multiple images, and themes and key elements need to be determined for the multiple images. The key elements may be entities, scenes, features related to entities, and / or features related to scenes, etc. For example, the theme of the multiple images is determined to be a set of fitness movements, and the key elements include the movements of the characters. Further, the second type of recommendation demand information may be determined, for example, the recommendation demand information may be "what is the effect of this set of fitness movements", "what should be paid attention to in training", etc.

[0068] The method of the above-mentioned embodiment, in the case of multiple images, associates and understands the multiple images, extracts the themes and key elements of the multiple images, and then determines one or more recommendation demand information of the first type, the second type, and the third type. It can provide users with one or more recommendation demand information for multiple images as a whole, improve the degree of matching between the recommendation demand information and the multiple images, and improve the accuracy of determining the recommendation demand information.

[0069] In some embodiments, the visual media content includes a video, and at least one of the entities and scenes in the visual media content includes at least one of the entities and scenes of key frames in the video. Determining one or more of the first type of recommendation demand information, the second type of recommendation demand information, and the third type of recommendation demand information based on at least one of the entities and scenes includes: determining the theme and key elements of the video based on at least one of the entities and scenes of the key frames; determining one or more of the first type of recommendation demand information, the second type of recommendation demand information, and the third type of recommendation demand information based on the theme and key elements of the video.

[0070] The visual media content may include a video, and the key frames of the video are determined, and at least one of the entities and scenes in each key frame is identified, thereby determining the theme and key elements of the video, and determining one or more of the first type of recommendation demand information, the second type of recommendation demand information, and the third type of recommendation demand information based on the theme and key elements of the video. The key elements may be entities, scenes, features related to entities, and / or features related to scenes, etc. For example, if the theme of the video is a makeup tutorial, and the key elements are the cosmetics used by the characters in the video, the second type of recommendation demand information may be determined, and the recommendation demand information may be "the name and model of the cosmetics", etc.

[0071] The method of the above embodiment can determine the theme and key elements of the video input by the user based on the entities and scenes of the key frames, and then determine one or more recommendation demand information of the first type, the second type, and the third type. It can provide the user with one or more recommendation demand information for the video as a whole, improve the matching degree between the recommendation demand information and the video, and improve the accuracy of determining the recommendation demand information.

[0072] In the above embodiments, one or more recommendation demand information is determined mainly based on the visual media content itself, and one or more recommendation demand information may also be determined with reference to the user's historical interaction information and behavior habits.

[0073] In some embodiments, understanding the visual media content and determining one or more recommendation demand information matching the visual media content includes: determining one or more recommendation demand information matching the visual media content based on at least one of the user's corresponding historical interaction information and the user's behavioral habits, as well as the understanding of the visual media content.

[0074] For example, semantic understanding is performed on the user's historical interaction information to determine whether there is information associated with the visual media content, and one or more recommendation demand information is determined based on the associated information and the understanding of the visual media content. For example, if the user mentioned in the historical interaction information that he likes to travel, and the image input by the user is a landscape photo, the recommendation demand information may be "where is this" or "make a travel guide", etc.

[0075] For example, the user's behavior habits include the type of recommendation demand information selected by the user in the past and at least one of the entities and scenes in the corresponding visual media content. For certain visual media content, the user may tend to select a certain type of recommendation demand information, and one or more recommendation demand information can be determined based on the user's behavior habits and understanding of the visual media content.

[0076] The method of the above embodiment refers to at least one of the user's historical interaction information and behavioral habits, combined with the understanding of the visual media content, to determine one or more recommendation demand information, which is closer to the user's true intention and improves the accuracy and effectiveness of determining the recommendation demand information.

[0077] The following describes how to determine the working link corresponding to the target recommendation requirement information in conjunction with some embodiments.

[0078] In some embodiments, determining a working link corresponding to target recommendation requirement information includes: understanding the target recommendation requirement information, determining one or more working nodes and the order of one or more working nodes required to generate answer content corresponding to the target recommendation requirement information, wherein the one or more working nodes include at least one of a model, an application, a plug-in, and an intelligent agent; determining the working link based on the one or more working nodes and the order of one or more working nodes.

[0079] In order to enable users to better understand the recommended demand information, the displayed recommended demand information is usually relatively brief. Each recommended demand information may correspond to prompt information. For example, if the recommended demand information is "translation", the corresponding prompt information is "translate the text in the image, and display the translated text in the position corresponding to the original text in the image". Therefore, understanding the target recommended demand information may be understanding the prompt information corresponding to the target recommended demand information.

[0080] The machine learning model can be used to understand the target recommendation requirement information and determine one or more work nodes and the order of one or more work nodes required to generate the answer content corresponding to the target recommendation requirement information. The work nodes can be in different forms and need to be determined according to the specific method of generating the answer content.

[0081] The method of the above embodiment can dynamically generate working links in real time according to different target recommendation demand information, thereby improving the accuracy of determining the working links and further improving the accuracy of generating reply content.

[0082] The following describes how to determine one or more working nodes according to the different types of target recommendation requirement information.

[0083] In some embodiments, the target recommendation requirement information is understood, and one or more working nodes required to generate answer content corresponding to the target recommendation requirement information are determined, including: in response to the target recommendation requirement information being the first type of recommendation requirement information, one or more working nodes are determined to include a visual understanding node and a modification node, wherein the visual understanding node is used to understand the visual media content and obtain understanding information, and the modification node is used to modify the visual media content according to the understanding information and the target recommendation requirement information.

[0084] The target recommendation requirement information is the first type of recommendation requirement information, that is, the target recommendation requirement information is the recommendation requirement information for modifying the visual media content. Based on the visual media content and the working link, in the process of generating the answer content, the visual media content can be input into the visual understanding node to obtain the understanding information, and the understanding information is further input into the modification node, and the visual media content is modified according to the understanding information and the target recommendation requirement information. The modification nodes can be different for different modification methods. For example, the modification node for beautifying the image is different from the modification node for generating the video based on the image.

[0085] The method of the above embodiment can determine that the working link includes a visual understanding node and a modification node when the user chooses to modify the visual media content, so that the working link can accurately modify the visual media content and improve the accuracy of the answer content.

[0086] In some embodiments, understanding the target recommendation requirement information and determining one or more working nodes required to generate answer content corresponding to the target recommendation requirement information include: in response to the target recommendation requirement information being the second type of recommendation requirement information, determining one or more working nodes including a visual understanding node, a search node and a generation node, wherein the visual understanding node is used to understand the visual media content and obtain understanding information, the search node is used to search for information related to the question of the visual media content based on the understanding information and the target recommendation requirement information, and the generation node is used to generate an answer based on the information related to the question of the visual media content.

[0087] The target recommendation demand information is the second type of recommendation demand information, that is, the target recommendation demand information is the recommendation demand information that raises questions about the visual media content. One or more working nodes may also only include visual understanding nodes and search nodes or only include visual understanding nodes and generation nodes. For example, the target recommendation demand information is to raise questions associated with entities in the visual media content, to raise questions associated with scenes in the visual media content, and multiple working nodes may only include visual understanding nodes and search nodes. For example, the target recommendation demand information is to raise questions about generating specific texts based on visual media content, to raise questions about generating music based on visual media content, and other recommendation demand information, and multiple working nodes may only include visual understanding nodes and generation nodes. The generation node is used to generate answers based on the target recommendation information and understanding information.

[0088] In the process of generating answer content based on visual media content and working links, the visual media content can be input into the visual understanding node to obtain understanding information, and the understanding information can be further input into the search node to obtain information related to the question of the visual media content, and the information related to the question of the visual media content can be input into the generation node to generate an answer. Figure 2 As shown, the image includes chocolate, and the target recommendation demand information is "What kind of chocolate is this?" Based on the understanding information, information related to chocolate can be searched to obtain the name and introduction information of the chocolate, and the answer content can be generated based on the name and introduction information of the chocolate.

[0089] The method of the above embodiment can determine that the working link includes a visual understanding node, a search node, and a generation node when the user chooses to ask questions about the visual media content, so that the working link can accurately answer questions about the visual media content and improve the accuracy of the answer content.

[0090] In some embodiments, in response to the target recommendation requirement information being the third type of recommendation requirement information, one or more working nodes are determined to include a visual understanding node, a reply information determination node, and an adding node, wherein the visual understanding node is used to understand the visual media content to obtain understanding information, the reply information determination node is used to determine the reply information based on the understanding information and the target recommendation requirement information, and the adding node is used to add the reply information to the visual media content.

[0091] The target recommendation requirement information is the third type of recommendation requirement information, that is, the target recommendation requirement information is the recommendation requirement information that generates reply information based on the visual media content and feeds back into the visual media content. In the process of generating reply content based on the visual media content and the working link, the visual media content can be input into the visual understanding node to obtain understanding information, and the understanding information can be further input into the reply information determination node to determine the reply information, and the reply information can be input into the adding node to be added to the visual media content.

[0092] For example, the reply information determination node can be a search node, a generation node, a translation node, etc., which needs to be determined according to the target recommendation requirement information. For example, if the target recommendation requirement information is "translation", the reply information determination node is a translation node, and if the target recommendation requirement information is "correcting homework", the reply information determination node is a search node and / or a generation node.

[0093] The method of the above embodiment, when the user chooses to generate reply information based on visual media content and feed back recommended demand information in the visual media content, can determine that the working link includes a visual understanding node, a reply information determination node and an adding node, so that the working link can accurately generate reply information and add it to the visual media content, thereby improving the accuracy of the answer content.

[0094] The following describes how to generate answer content and display the answer content.

[0095] In some embodiments, generating answer content and displaying the answer content based on the visual media content and the work link includes at least one of the following: in response to the target recommendation requirement information being the first type of recommendation requirement information, generating modified visual media content based on the visual media content and the work link, and displaying the modified visual media content in the interactive interface; in response to the target recommendation requirement information being the second type of recommendation requirement information, generating answers to questions posed to the visual media content based on the visual media content and the work link, and displaying the answers in the interactive interface; in response to the target recommendation requirement information being the third type of recommendation requirement information, generating reply information corresponding to the visual media content based on the visual media content and the work link, and adding and displaying the reply information in the visual media content.

[0096] The modified visual media content and answers can be displayed directly in the interactive interface. Figure 2 As shown, in response to the user selecting the recommended demand information 202 “What kind of chocolate is this”, the following may be displayed: Figure 6 The interactive interface shown in FIG. 6 shows an image 601 , recommended requirement information 602 and answer content 603 .

[0097] Response information can also be added and displayed in the visual media content. For example, Figure 5 As shown, in response to the user's selection of the recommended requirement information 505 "correction assignment", the following may be displayed: Figure 7 The interactive interface shown in Figure 1 is as follows. Figure 7 As shown, the visual media content after adding the reply information can be displayed in the interactive interface, the visual media content is 701, and the other content is the reply information added to the visual media content, such as check marks, question numbers, relevant content for identifying the questions, and answers.

[0098] For different types of target recommendation demand information, different working links are used to generate answer content, and different display methods are used to display it, which can present better visual effects to users and facilitate users to locate, view and understand the answer content.

[0099] The present disclosure also provides an interactive device, Figure 8 Give a description.

[0100] Figure 8 FIG. 1 is a structural diagram of some embodiments of the interactive device disclosed in the present invention. Figure 8 As shown, the interaction device 80 of this embodiment includes: a first determination module 810 , a first display module 820 , a second determination module 830 , a generation module 840 , and a second display module 850 .

[0101] The first determination module 810 is configured to respond to user input of visual media content, understand the visual media content, and determine one or more recommendation requirement information matching the visual media content, wherein the one or more recommendation requirement information is used to characterize one or more intentions for the visual media content.

[0102] The first display module 820 is configured to display one or more pieces of recommended requirement information.

[0103] The second determination module 830 is configured to determine a working link corresponding to the target recommended requirement information in response to a user's triggering operation on the target recommended requirement information among one or more recommended requirement information.

[0104] The generation module 840 is configured to generate answer content based on the visual media content and the working link.

[0105] The second display module 850 is configured to display the answer content.

[0106] The interactive device of the above embodiment, in response to the user inputting visual media content, determines one or more recommended demand information matching the visual media content through understanding the visual media content, and displays one or more recommended demand information. One or more recommended demand information is used to characterize one or more intentions for the visual media content. Since one or more recommended demand information is determined based on the understanding of the visual media content, the one or more recommended demand information is closer to the user's real needs, which is convenient for the user to quickly locate the target recommended demand information, and for different visual media content, one or more recommended demand information is dynamically adjusted and changed. In response to the user's triggering operation on the target recommended demand information in the one or more recommended demand information, the working link corresponding to the target recommended demand information is determined, and then based on the visual media content and the working link, the answer content is generated and displayed. In the whole interactive process, the user does not need to input instructions by himself, but only needs to trigger the target recommended demand information to obtain the answer content, which improves the interaction efficiency from the visual media content to the answer content. And based on the target recommended demand information, the corresponding working link is called to generate the answer content, which improves the accuracy and efficiency of generating the answer content.

[0107] In some embodiments, the second determination module 830 is configured to understand the target recommendation requirement information, determine one or more work nodes and the order of one or more work nodes required to generate the answer content corresponding to the target recommendation requirement information, wherein the one or more work nodes include at least one of a model, an application, a plug-in, and an intelligent agent; determine the work link based on the one or more work nodes and the order of one or more work nodes.

[0108] In some embodiments, the first determination module 810 is configured to use a machine learning model to understand the visual media content and determine at least one of the entities and scenes in the visual media content; based on at least one of the entities and scenes, determine one or more of the first type of recommendation demand information, the second type of recommendation demand information, and the third type of recommendation demand information, wherein the first type of recommendation demand information includes recommendation demand information for modifying the visual media content, the second type of recommendation demand information includes recommendation demand information for raising questions about the visual media content, and the third type of recommendation demand information includes recommendation demand information that generates reply information based on the visual media content and feeds back into the visual media content.

[0109] In some embodiments, the second determination module 830 is configured to determine, in response to the target recommendation requirement information being the first type of recommendation requirement information, one or more working nodes including a visual understanding node and a modification node, wherein the visual understanding node is used to understand the visual media content and obtain understanding information, and the modification node is used to modify the visual media content according to the understanding information and the target recommendation requirement information.

[0110] In some embodiments, the second determination module 830 is configured to determine, in response to the target recommendation requirement information being the second type of recommendation requirement information, one or more working nodes including a visual understanding node, a search node and a generation node, wherein the visual understanding node is used to understand the visual media content and obtain understanding information, the search node is used to search for information related to the question of the visual media content based on the understanding information and the target recommendation requirement information, and the generation node is used to generate an answer based on the information related to the question of the visual media content.

[0111] In some embodiments, the second determination module 830 is configured to determine one or more working nodes including a visual understanding node, a reply information determination node and an adding node in response to the target recommendation requirement information being the third type of recommendation requirement information, wherein the visual understanding node is used to understand the visual media content and obtain understanding information, the reply information determination node is used to determine the reply information based on the understanding information and the target recommendation requirement information, and the adding node is used to add the reply information to the visual media content.

[0112] In some embodiments, the generation module 840 is configured to perform at least one of the following: in response to the target recommendation requirement information being the first type of recommendation requirement information, based on the visual media content and the working link, generating modified visual media content, and displaying the modified visual media content in the interactive interface; in response to the target recommendation requirement information being the second type of recommendation requirement information, based on the visual media content and the working link, generating answers to questions posed to the visual media content, and displaying the answers in the interactive interface; in response to the target recommendation requirement information being the third type of recommendation requirement information, based on the visual media content and the working link, generating reply information corresponding to the visual media content, and adding and displaying the reply information in the visual media content.

[0113] In some embodiments, the first determination module 810 is configured to determine one or more recommendation demand information matching the visual media content according to at least one of the historical interaction information corresponding to the user and the user's behavior habits and the understanding of the visual media content.

[0114] In some embodiments, the visual media content includes multiple images, and at least one of the entities and scenes in the visual media content includes at least one of the entities and scenes of each of the multiple images. The first determination module 810 is configured to determine themes and key elements of the multiple images based on at least one of the entities and scenes of each image; and determine one or more of the first type of recommendation demand information, the second type of recommendation demand information, and the third type of recommendation demand information based on the themes and key elements of the multiple images.

[0115] In some embodiments, the visual media content includes a video, and at least one of the entities and scenes in the visual media content includes at least one of the entities and scenes of key frames in the video. The first determination module 810 is configured to determine the theme and key elements of the video based on at least one of the entities and scenes of the key frames; and determine one or more of the first type of recommendation demand information, the second type of recommendation demand information, and the third type of recommendation demand information based on the theme and key elements of the video.

[0116] In some embodiments, the first display module 820 is configured to display camera controls; in response to the user triggering the camera controls, a preview interface is displayed; the first determination module 810 is configured to understand the visual media content in response to the user completing the shooting operation of the visual media content, and determine one or more recommended demand information matching the visual media content, wherein the one or more recommended demand information are displayed in the preview interface.

[0117] In some embodiments, the first display module 820 is configured to display a confirmation control in the preview interface; in response to the user triggering the confirmation control, display an interactive interface, and display one or more recommended requirement information in the interactive interface.

[0118] In some embodiments, the first display module 820 is configured to display a selection control; in response to a user triggering the selection control, one or more candidate visual media content is displayed; the first determination module 810 is configured to display the visual media content in a preview interface in response to a user selection of visual media content from one or more candidate visual media content, understand the visual media content, and determine one or more recommended demand information matching the visual media content, wherein the one or more recommended demand information are displayed in the preview interface.

[0119] The present disclosure also provides an electronic device, comprising: a processor; and a memory coupled to the processor, for storing instructions, wherein when the instructions are executed by the processor, the processor executes an interaction method as in any embodiment of the present disclosure.

[0120] The interactive device of the disclosed embodiment, in response to the user inputting visual media content, determines one or more recommended demand information matching the visual media content through understanding the visual media content, and displays one or more recommended demand information. The one or more recommended demand information is used to characterize one or more intentions for the visual media content. Since the one or more recommended demand information is determined based on the understanding of the visual media content, the one or more recommended demand information is closer to the real needs of the user, which is convenient for the user to quickly locate the target recommended demand information, and for different visual media content, the one or more recommended demand information is dynamically adjusted and changed. In response to the user's triggering operation on the target recommended demand information in the one or more recommended demand information, the working link corresponding to the target recommended demand information is determined, and then based on the visual media content and the working link, the answer content is generated and displayed. In the whole interactive process, the user does not need to input instructions by himself, but only needs to trigger the target recommended demand information to obtain the answer content, which improves the interaction efficiency from the visual media content to the answer content. And based on the target recommended demand information, the corresponding working link is called to generate the answer content, which improves the accuracy and efficiency of generating the answer content.

[0121] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the interactive method of any embodiment of the present disclosure is implemented.

[0122] The computer-readable storage medium of the disclosed embodiment, in response to the user inputting visual media content, determines one or more recommendation demand information matching the visual media content through understanding the visual media content, and displays one or more recommendation demand information. One or more recommendation demand information is used to characterize one or more intentions for the visual media content. Since one or more recommendation demand information is determined based on the understanding of the visual media content, the one or more recommendation demand information is closer to the real needs of the user, which is convenient for the user to quickly locate the target recommendation demand information, and for different visual media content, the one or more recommendation demand information is dynamically adjusted and changed. In response to the user's triggering operation on the target recommendation demand information in the one or more recommendation demand information, the working link corresponding to the target recommendation demand information is determined, and then based on the visual media content and the working link, the answer content is generated and displayed. In the whole interactive process, the user does not need to input instructions by himself, but only needs to trigger the target recommendation demand information to obtain the answer content, which improves the interaction efficiency from the visual media content to the answer content. And based on the target recommendation demand information, the corresponding working link is called to generate the answer content, which improves the accuracy and efficiency of generating the answer content.

[0123] The present disclosure also provides a computer program product, including: instructions, which, when executed by a processor, enable the processor to execute an interactive method as described in any embodiment of the present disclosure.

[0124] The computer program product of the disclosed embodiment, in response to the user inputting visual media content, determines one or more recommendation demand information matching the visual media content through understanding the visual media content, and displays one or more recommendation demand information. One or more recommendation demand information is used to characterize one or more intentions for the visual media content. Since one or more recommendation demand information is determined based on the understanding of the visual media content, the one or more recommendation demand information is closer to the real needs of the user, which is convenient for the user to quickly locate the target recommendation demand information, and for different visual media content, the one or more recommendation demand information is dynamically adjusted and changed. In response to the user's triggering operation on the target recommendation demand information in the one or more recommendation demand information, the working link corresponding to the target recommendation demand information is determined, and then based on the visual media content and the working link, the answer content is generated and displayed. In the whole interactive process, the user does not need to input instructions by himself, but only needs to trigger the target recommendation demand information to obtain the answer content, which improves the interaction efficiency from the visual media content to the answer content. And based on the target recommendation demand information, the corresponding working link is called to generate the answer content, which improves the accuracy and efficiency of generating the answer content.

[0125] Combine the following Fig. 9 and Fig.10 An electronic device of the present disclosure is described.

[0126] Fig. 9 A block diagram of an electronic device according to some embodiments of the present disclosure is shown.

[0127] The memory 91 is used to store one or more computer-readable instructions. The memory 91 may include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory, including but not limited to random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. The memory 91 may store, for example, an operating system, an application, a boot loader (BootLoader), a database, and other programs, and may also store various applications and various data.

[0128] The processor 92 is used to run computer-readable instructions to implement the interactive method of any of the above embodiments. The specific implementation of each step of the method can be referred to the above embodiments, and the repeated parts are not repeated here.

[0129] The processor 92 may be embodied as various processing devices, such as a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The central processing unit (CPU) may be an X86 or ARM architecture, etc.

[0130] The processor 92 and the memory 91 may communicate with each other directly or indirectly. For example, the processor 92 and the memory 91 may communicate with each other via a network. The network may include a wireless network, a wired network, and / or any combination of a wireless network and a wired network. The processor 92 and the memory 91 may also communicate with each other via a system bus, which is not limited in the present disclosure.

[0131] It should be noted that Fig. 9 The components of the electronic device 9 shown are only exemplary and non-restrictive. The electronic device 9 may also have other components according to actual application requirements. The processor 92 may control other components in the electronic device 9 to perform desired functions.

[0132] The electronic device 9 may be implemented by software, firmware and / or hardware, and may be integrated into a device installed with relevant application programs.

[0133] Fig.10 A block diagram of an electronic device according to some other embodiments of the present disclosure is shown.

[0134] Fig.10 The electronic device 10 shown may be a computer system with a dedicated hardware structure, which can execute corresponding functions when a relevant application program is installed.

[0135] Electronic devices include, but are not limited to, mobile terminals such as smart phones, laptops, personal digital assistants (PDA), tablet personal computers (Tablet PC), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable devices, etc., and fixed terminals such as digital televisions, desktop computers, etc.

[0136] like Fig.10As shown, a central processing unit (CPU) 101 performs various processes according to a program stored in a read-only memory (ROM) 102 or a program loaded from a storage section 108 to a random access memory (RAM) 103. In the RAM 103, data required when the CPU 101 performs various processes, etc., is stored as needed. The central processing unit is merely exemplary, and it may also be other types of processors, such as the various processors described above. The ROM 102, the RAM 103, and the storage section 108 may be various forms of computer-readable storage media. It should be noted that although Fig.10 ROM 102, RAM 103 and storage section 108 are shown separately in FIG. 1 , but one or more of them may be combined or located in the same or different memory or storage modules.

[0137] The CPU 101, the ROM 102, and the RAM 103 are connected to each other via a bus 104. To the bus 104, an input / output interface 105 is also connected.

[0138] The following components are connected to the input / output interface 105: an input section 106, such as a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output section 107, including a display, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage section 108, including a hard disk, a magnetic tape, etc.; and a communication section 109, including a network interface card such as a LAN card, a modem, etc. The communication section 109 allows communication processing to be performed via a network such as the Internet. It is easy to understand that although Fig.10 Parts of the electronic device 10 are shown to communicate via the bus 104, but they may also communicate via a network or other means, wherein the network may include a wireless network, a wired network, and / or any combination of wireless networks and wired networks.

[0139] A drive 1010 is also connected to the input / output interface 105 as needed. A removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 1010 as needed so that a computer program read therefrom is installed into the storage section 108 as needed.

[0140] When the above-described series of processing is realized by software, a program constituting the software can be installed from a network such as the Internet or a storage medium such as the removable medium 1011 .

[0141] According to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which, when the computer program product is run on a computer, enables the computer to implement the method described in any of the aforementioned embodiments. The computer program product includes a computer instruction carried on a computer-readable medium, containing a program code for executing the method shown in the flowchart. In such an embodiment, the computer instruction can be downloaded and installed from the network through the communication part 109, or installed from the storage part 108, or installed from the ROM 102. When the computer program is executed by the CPU 101, the method of the embodiment of the present disclosure is executed.

[0142] It should be noted that, in the context of the present disclosure, a computer-readable medium may be a tangible medium that may contain or store a program for use by an instruction execution system, apparatus, or device or for use in conjunction with an instruction execution system, apparatus, or device.

[0143] The computer readable medium may be a computer readable storage medium, or a computer readable signal medium, or any combination of the two.

[0144] Computer-readable storage media include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device. Computer instructions are stored on a computer-readable storage medium, and when the instructions are executed by a processor, the method described in any of the foregoing embodiments is implemented.

[0145] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer readable program codes. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0146] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0147] In some embodiments, a computer program is further provided, comprising: instructions, which, when executed by a processor, cause the processor to execute the method described in any of the above embodiments. For example, the instructions may be embodied as computer program codes.

[0148] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In situations involving a remote computer, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0149] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0150] The functions described above may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0151] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. An interactive method, comprising: In response to a user inputting visual media content, the visual media content is understood, and one or more pieces of recommendation demand information matching the visual media content are determined, wherein the one or more pieces of recommendation demand information are used to represent one or more intentions for the visual media content; Displaying the one or more recommended demand information; In response to the user's triggering operation on target recommended requirement information among the one or more recommended requirement information, determining a working link corresponding to the target recommended requirement information; Based on the visual media content and the working link, answer content is generated and the answer content is displayed.

2. The interactive method according to claim 1, wherein: The determining of the working link corresponding to the target recommendation requirement information includes: Understanding the target recommendation requirement information, determining one or more work nodes and the order of the one or more work nodes required to generate the answer content corresponding to the target recommendation requirement information, wherein the one or more work nodes include at least one of a model, an application, a plug-in, and an intelligent agent; The working link is determined according to the one or more working nodes and the order of the one or more working nodes.

3. The interactive method according to claim 2, wherein: The understanding of the visual media content and determining one or more pieces of recommendation demand information matching the visual media content includes: Using a machine learning model to understand the visual media content, and determining at least one of an entity and a scene in the visual media content; Based on at least one of the entities and the scenes, one or more recommendation demand information of a first type, a second type, and a third type is determined, wherein the first type of recommendation demand information includes recommendation demand information for modifying the visual media content, the second type of recommendation demand information includes recommendation demand information for raising questions to the visual media content, and the third type of recommendation demand information includes recommendation demand information for generating reply information based on the visual media content and feeding back into the visual media content.

4. The interactive method according to claim 3, wherein: The understanding of the target recommendation requirement information and determining one or more work nodes required to generate the answer content corresponding to the target recommendation requirement information includes: In response to the target recommendation requirement information being the first type of recommendation requirement information, it is determined that the one or more working nodes include a visual understanding node and a modification node, wherein the visual understanding node is used to understand the visual media content to obtain understanding information, and the modification node is used to modify the visual media content according to the understanding information and the target recommendation requirement information.

5. The interactive method according to claim 3, wherein: The understanding of the target recommendation requirement information and determining one or more work nodes required to generate the answer content corresponding to the target recommendation requirement information includes: In response to the target recommendation requirement information being the second type of recommendation requirement information, it is determined that the one or more working nodes include a visual understanding node, a search node and a generation node, wherein the visual understanding node is used to understand the visual media content to obtain understanding information, the search node is used to search for information related to the question of the visual media content based on the understanding information and the target recommendation requirement information, and the generation node is used to generate an answer based on the information related to the question of the visual media content.

6. The interactive method according to claim 3, wherein: The understanding of the target recommendation requirement information and determining one or more work nodes required to generate the answer content corresponding to the target recommendation requirement information includes: In response to the target recommendation requirement information being the third type of recommendation requirement information, it is determined that the one or more working nodes include a visual understanding node, a reply information determination node and an adding node, wherein the visual understanding node is used to understand the visual media content to obtain understanding information, the reply information determination node is used to determine reply information based on the understanding information and the target recommendation requirement information, and the adding node is used to add the reply information to the visual media content.

7. The interactive method according to claim 3, wherein: Based on the visual media content and the working link, generating answer content and displaying the answer content comprises at least one of the following: In response to the target recommendation requirement information being the first type of recommendation requirement information, generating modified visual media content based on the visual media content and the working link, and displaying the modified visual media content in an interactive interface; In response to the target recommendation requirement information being the second type of recommendation requirement information, based on the visual media content and the working link, generating an answer to a question posed to the visual media content, and displaying the answer in an interactive interface; In response to the target recommendation requirement information being the third type of recommendation requirement information, based on the visual media content and the working link, reply information corresponding to the visual media content is generated, and the reply information is added and displayed in the visual media content.

8. The interactive method according to claim 1, wherein: The understanding of the visual media content and determining one or more pieces of recommendation demand information matching the visual media content includes: One or more recommendation demand information matching the visual media content is determined based on at least one of the historical interaction information corresponding to the user and the user's behavior habits, as well as the understanding of the visual media content.

9. The interactive method according to claim 3, wherein: The visual media content includes a plurality of images, and at least one of an entity and a scene in the visual media content includes at least one of an entity and a scene in each of the plurality of images, and determining one or more of the first type of recommendation demand information, the second type of recommendation demand information, and the third type of recommendation demand information according to the at least one of the entity and the scene includes: Determining the themes and key elements of the plurality of images based on at least one of the entity and the scene of each image; According to the themes and key elements of the plurality of images, one or more recommendation requirement information of the first type, the second type, and the third type is determined.

10. The interactive method according to claim 3, wherein: The visual media content includes a video, at least one of an entity and a scene in the visual media content includes at least one of an entity and a scene of a key frame in the video, and determining one or more of the first type of recommendation demand information, the second type of recommendation demand information, and the third type of recommendation demand information according to the at least one of the entity and the scene includes: Determine the theme and key elements of the video according to at least one of the entity and the scene of the key frame; According to the theme and key elements of the video, one or more recommendation demand information of the first type, the second type, and the third type is determined.

11. The interactive method according to any one of claims 1 to 10, wherein: In response to the user inputting visual media content, understanding the visual media content, and determining one or more pieces of recommendation demand information matching the visual media content includes: Display camera controls; In response to the user triggering the camera control, displaying a preview interface; In response to the user completing the shooting operation of the visual media content, the visual media content is understood, and one or more pieces of recommended demand information matching the visual media content are determined, wherein the one or more pieces of recommended demand information are displayed in the preview interface.

12. The interactive method according to claim 11, wherein: The displaying of the one or more pieces of recommended demand information includes: Displaying a confirmation control in the preview interface; In response to the user triggering the confirmation control, an interactive interface is displayed, and the one or more pieces of recommended requirement information are displayed in the interactive interface.

13. The interactive method according to any one of claims 1 to 10, wherein: In response to the user inputting visual media content, understanding the visual media content, and determining one or more pieces of recommendation demand information matching the visual media content includes: Display selection controls; In response to the user triggering the selection control, displaying one or more candidate visual media content; In response to the user's selection of the visual media content from the one or more candidate visual media content, the visual media content is displayed in a preview interface, the visual media content is understood, and one or more recommended demand information matching the visual media content is determined, wherein the one or more recommended demand information is displayed in the preview interface.

14. An interactive device, comprising: A first determination module is configured to respond to a user inputting visual media content, understand the visual media content, and determine one or more pieces of recommendation demand information matching the visual media content, wherein the one or more pieces of recommendation demand information are used to represent one or more intentions for the visual media content; A first display module is configured to display the one or more pieces of recommended demand information; A second determining module is configured to determine a working link corresponding to the target recommended requirement information in response to a triggering operation of the user on the target recommended requirement information among the one or more recommended requirement information; A generating module, configured to generate answer content based on the visual media content and the working link; The second display module is configured to display the answer content.

15. An electronic device comprising: processor; as well as A memory coupled to the processor, for storing instructions, wherein when the instructions are executed by the processor, the processor executes the interaction method according to any one of claims 1 to 13.

16. A computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the interactive method according to any one of claims 1 to 13 is implemented.

17. A computer program product comprising: Instructions, when executed by a processor, cause the processor to perform the interaction method as described in any one of claims 1 to 13.

Images