Interaction method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides an interaction method and device, equipment, a storage medium and a program product. The method includes, in response to receiving a user input, generating a first text based on the user input. And obtaining first rich media content corresponding to at least one part of the first text at least based on the first text. The first text and the first rich media content are presented as at least part of a reply to the user input. In this way, the rich media content corresponding to the reply content can be added based on the reply content to respond to the user input, so that the expected rich media content can be provided as far as possible when the user expects to respond to the rich media content, and the user experience and satisfaction are improved.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly to interactive methods, apparatuses, devices, computer-readable storage media, and computer program products. Background Art

[0002] With the rapid development of machine learning technology, intelligent question answering is gradually becoming an important development direction of the search industry. Intelligent search uses machine learning models (for example, language models) to improve search algorithms and the presentation of search results, thereby improving search efficiency and accuracy. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for interacting with a user is provided. The method includes: in response to receiving a user input, generating a first text based on the user input. At least based on the first text, obtaining a first rich media content corresponding to at least a portion of the first text. Presenting the first text and the first rich media content as part of a reply.

[0004] In a second aspect of the present disclosure, an interactive device is provided. The device includes: a text generation module configured to generate a first text based on the user input in response to receiving the user input; a rich media content acquisition module configured to acquire first rich media content corresponding to at least a portion of the first text based on at least the first text; and a presentation module configured to present the first text and the first rich media content as at least a portion of a response to the user input.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the electronic device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, which includes a computer program, wherein when the computer program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0008] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 2 A flowchart illustrating an example process of interaction according to some embodiments of the present disclosure;

[0012] Figure 3A-3D A schematic diagram showing an interactive interface according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A flowchart showing an example process of obtaining rich media content according to some embodiments of the present disclosure;

[0014] Figure 5 A flowchart showing an interaction method according to some embodiments of the present disclosure is shown;

[0015] Figure 6 An exemplary structural block diagram showing an interaction device according to some embodiments of the present disclosure; and

[0016] Figure 7 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0019] Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0020] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0021] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scopes of use, usage scenarios, etc. of the information involved in the present disclosure should be informed to relevant users and their authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations. The relevant users may include any type of right holders, such as individuals, enterprises, and groups.

[0022] For example, in response to receiving an active request from a user, a prompt message is sent to the relevant user to clearly prompt the relevant user that the operation requested to be performed will require obtaining and using the information of the relevant user, so that the relevant user can independently choose whether to provide information to software or hardware such as an electronic device, application, server or storage medium that executes the operation of the technical solution of the present disclosure based on the prompt message.

[0023] As an optional but non-limiting implementation, in response to receiving an active request from a relevant user, a prompt message is sent to the relevant user, for example, in the form of a pop-up window, in which the prompt message may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide information to the electronic device.

[0024] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0025] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0026] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs, and typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes input from the previous layer.

[0027] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. In the example environment 100, a target application 112 is installed in a client device 110. A user 130 can interact with the application service component target application 112 via the client device 110 and / or an attached device of the client device 110.

[0028] In some embodiments, the target application 112 can be downloaded and installed on the client device 110. In some embodiments, the target application 112 can also be accessed through other means, such as through a web page. Figure 1 In the environment 100, in response to the target application 112 being started, the client device 110 can present an interface 140 of the target application 112. The interface 140 can be, for example, an interactive interface of the target application 112.

[0029] In some embodiments, the target application 112 may have intelligent dialogue and information processing capabilities. For example, the target application 112 may be a user's digital assistant or an application configured with a digital assistant. In an embodiment of the present disclosure, the target application 112 is used to interact with the user 130 to assist the user 130 in using the terminal or processing and searching for information. In some embodiments, an interaction window with the target application 112 may be presented in the interface 140. In the interaction window, the user 130 can have a dialogue with the target application 112 by inputting natural language, pictures, audio files, video files, web page files, etc., to instruct the target application 112 to assist in completing various tasks, including operations on content entities; or to instruct the target application 112 to complete questions and answers or searches. In some embodiments, the interaction between the user 130 and the target application 112 may include interaction between the user 130 and the digital assistant. For example, the user's input to the target application 112 may be issued to the digital assistant, and the target application 112's reply to the user 130 may be issued to the user 130 by the digital assistant.

[0030] In some embodiments, multiple interaction modes between the user 130 and the target application 112 may be provided, and the multiple interaction modes may be flexibly switched. When a certain interaction mode is triggered, a corresponding interaction area is presented to facilitate the interaction between the user 130 and the target application 112. The user 130 interacts with the target application 112 in different interaction modes in different ways, so that the interaction needs in different application scenarios can be flexibly adapted.

[0031] In some embodiments, information processing services specific to the user 130 can be provided based on historical interaction information between the user 130 and the target application 112 and / or a data range specific to the user 130. In some embodiments, historical interaction information of the user 130 interacting with the target application 112 in multiple interaction modes can be stored in association with the user 130. In this way, in one interaction mode (any one or a designated one interaction mode) among the multiple interaction modes, the target application 112 can provide services for the user 130 based on the historical interaction information stored in association with the user 130.

[0032] The target application 112 may be called or awakened by an appropriate method (e.g., a shortcut key, a button, or voice) to present an interactive window with the user 130. By selecting the target application 112, an interactive window with the target application 112 may be opened. The interactive window may include interface elements for information interaction, such as an input box, a message list, a message bubble, and the like. In other embodiments, the target application 112 may be awakened by an entry control or menu provided in the interface 140, or by inputting a preset instruction.

[0033] The interactive window between the target application 112 and the user 130 may include a conversation window, such as a conversation window in an instant messaging application or an instant messaging module of a specific application. In the conversation window, the interaction between the target application 112 and the user 130 may be presented in the form of a conversation message. Alternatively or additionally, the interactive window between the target application 112 and the user 130 may also include other types of windows, such as a window in a floating window mode, in which the user 130 may trigger the target application 112 to perform a corresponding operation by inputting an instruction, selecting a shortcut instruction, etc.

[0034] In some embodiments, the target application 112 may support an interactive mode of a conversation window, also referred to as a conversation mode. In this interactive mode, a conversation window between the user 130 and the target application 112 is presented, in which the user 130 and the target application 112 interact through conversation messages. In the conversation mode, the target application 112 may perform tasks according to the conversation messages in the conversation window. In the interactive window, the user 130 inputs an interactive message, and the target application 112 provides a reply message in response to the user input. By selecting the target application 112, a conversation window with the target application 112 may be opened. The conversation window may include interface elements for information interaction, such as an input box, a message list, a message bubble, and the like.

[0035] In some embodiments, the client device 110 communicates with the server 120 to provide services for the target application 112. The client device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the client device 110 can also support any type of interface for the user (such as a "wearable" circuit, etc.). The server 120 can be various types of computing systems / servers that can provide computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like. The server 120 can be deployed with various machine learning models to provide services for the target application 112.

[0036] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0037] As mentioned above, intelligent search based on machine learning technology has become a popular new search method. In intelligent search, search results often need to provide rich media content to better meet user needs. However, currently, whether to introduce rich media content in the reply is determined only based on the user's question. As a result, although the user does not express the intention of needing rich media content in the question asked, if the answer does not provide corresponding rich media resources, the user's satisfaction is often not high.

[0038] In view of this, according to an embodiment of the present disclosure, an improved scheme of interaction is provided. Specifically, in response to receiving user input, a first text is generated based on the user input. At least based on the first text, a first rich media content corresponding to at least a portion of the first text is obtained. The first text and the first rich media content are presented as at least a portion of a response to the user input.

[0039] According to the scheme of the present disclosure, rich media content corresponding to the reply content is added based on the reply content in response to the user input. In this way, rich media content related to the generated content can be provided on demand according to the actual content generated. In this way, rich media content can be intelligently added to various parts of the content based on the reply content generated for the user input, thereby improving the reply quality and the user experience and satisfaction.

[0040] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0041] Figure 2 A flowchart of an example process 200 of interaction according to some embodiments of the present disclosure is shown. For ease of discussion, reference will be made to Figure 1 The process 200 is described with reference to the environment 100. The process 200 may be implemented at the client device 110 and / or the server 120. For the convenience of description, the process 200 is implemented at the server 120 as an example.

[0042] It should be noted that if the process 200 is implemented at the client device 110, some operations described with reference to the client device 110 may require the assistance of the server 120. It should be noted that the operations performed by the client device 110 may be performed by the relevant application and / or the target application installed on the client device 110.

[0043] In box 210, the server 120 provides the user input to the processing model in response to the target application 112 receiving the user input. The user input can be various types of information, such as information related to questions and queries. For example, the user input can be an interactive message sent to a digital assistant. In some embodiments, the user input can be text input or voice input. The processing model can be various machine learning models, such as a language model. The processing model can generate response information as a reply to the user input based on the user input. Exemplarily, the server 120 can present the response information to the user on the interface 140 through the target application 112.

[0044] At block 220, the server 120 generates a first text based on the user input. For example, the first text may be part of a digital assistant's response to the user input.

[0045] In some embodiments, the processing model can divide the response information into multiple texts according to preset rules, for example, into a first text, a second text, a third text, etc. In some embodiments, the processing model can divide the response information into multiple texts according to the subject type of the content included in the response information. For example, the user input is "What fun places are recommended in X city", and the response information is "X city has A, B, C and other attractions worth visiting, description of A attraction, description of B attraction, description of C attraction". At this time, the processing model can divide the content describing the three attractions A, B, and C in the response information into three texts, and the server 120 can present them to the user in the order of the three texts. It should be understood that the way the processing model divides the response information is not limited to the above-mentioned example of dividing by subject, but can adopt any suitable way. In addition, although the division of text is described relative to the processing model, the division of text may not depend on the model.

[0046] In some embodiments, the processing model can generate multiple texts as replies in sequence based on user input. For example, the user input is "What news has actor D had recently?", and the processing model first generates the following reply text "Actor D recently attended an awards ceremony. The event description of the awards ceremony. Server 120 can first present this information to the user. Then, processing model 220 continues to generate response information based on the user input, and server 120 presents it to the user.

[0047] In block 230 , the server 120 determines whether to obtain rich media content corresponding to at least a portion of the first text. If it is determined to obtain rich media content, it is determined what type of rich media content to obtain.

[0048] In some embodiments, the server 120 determines whether to obtain rich media content of a target type among multiple predetermined types based at least on the first text. In other words, the server 120 determines whether to obtain rich media content corresponding to at least a portion of the first text, and the type of the rich media content, based at least on the first text. In the present disclosure, the predetermined types include at least one of the following: image, audio, video, structured card, etc. In the present disclosure, structured card refers to a card message similar to an icon that can display information changes, such as a weather card, a text writing process card, etc.

[0049] In some embodiments, the server 120 first determines the theme described by the first text by analyzing the semantics of the first text. Then, a predetermined type related to the theme is selected from multiple predetermined types as the target type. For example, if the theme of the first text is related to visual elements, the target type includes images. If the theme of the first text is related to auditory elements, the target type includes audio. If the theme of the first text is related to dynamic scenes, the target type includes videos. If the theme of the first text is related to change trends, the target type includes structured cards. As an example, if the first text is a description of a certain person, the theme of the first text can be regarded as related to visual elements, and it can be determined that the target type includes images. In this case, the server 120 can obtain a photo of the person as the rich media content corresponding to the first text. For another example, if the first text is about or includes a description of a certain song, the theme of the first text can be regarded as related to auditory elements, and it can be determined that the target type includes audio. In this case, the server 120 can obtain the audio content of the song as the rich media content corresponding to the first text. For another example, if the first text is a description of the scene of someone receiving an award at an award ceremony, the theme of the first text can be regarded as related to a dynamic scene, and the target type includes video. In this case, the server 120 may obtain a video clip of someone receiving a prize as the rich media content corresponding to the first text.

[0050] In some embodiments, if the first text includes or relates to content related to weather, stocks, exchange rates, calendars, or text reading and writing, the first text is considered to be related to the change trend, and the target type includes a structured card. For example, if the first text relates to content related to weather, the server 120 obtains the corresponding weather card as the rich media content corresponding to the first text. Figure 3D , which shows an example of such an interaction. Figure 3D , response information 340 is provided for user input 330. Response information 340 includes first text 341. In the first text 341, the weather change trend is mentioned, and the server 120 obtains a weather card 342 as the first rich media content corresponding to the first text 341, and presents the corresponding weather card 342 as part of the reply.

[0051] In some embodiments, the server 120 first determines the target text portion of at least one of the following objects from the first text by analyzing the semantics of the first text: visual elements, auditory elements, dynamic scenes, or changing trends. Then, a predetermined type related to the object described in the target text portion is selected from multiple predetermined types as the target type. That is, the type of rich media content is determined based on part of the content in the first text. For example, if the first text mentions the melody of song A, the target type may include audio. In this case, song A or a portion thereof may be presented to the user as part of the reply.

[0052] As described above, in an embodiment of the present application, whether to obtain the first rich media content and the type of the first rich media content can be determined based on the entire content or a portion of the content of the first text. For the first text, one or more first rich media contents can be obtained. In the case of obtaining multiple first rich media contents, the multiple first rich media contents can be rich media contents of the same type or different types. For example, for the first text, an image and a video can be obtained at the same time as part of the reply.

[0053] If in block 230, the server 120 determines not to obtain the corresponding rich media content of the first text, for example, when the subject of the first text does not include or involve vision, hearing, dynamic scenes, trend changes, etc., then the process 200 returns to between blocks 210 and 220, and the server 120 continues to generate a reply, such as a second text, for the user input 210. In a question-asking or question-answering scenario, this process may also be referred to as the server 120 continuing to answer.

[0054] If, at block 230, the server 120 determines to obtain corresponding rich media content for the first text, the process 200 proceeds to blocks 240 and 250. At block 240, the server 120 obtains rich media content of a target type.

[0055] In some embodiments, the server 120 uses the search reference information generated based on the first text to obtain the target type of rich media content. In some embodiments, the server 120 first generates search reference information that matches the target type of rich media content based on at least the first text using a machine learning model, and then obtains the target type of rich media content based on the search reference information. For example, the first text involves weather, and the machine learning model can generate search reference information based on the first text. In some embodiments, the search reference information may include some search parameters. The search parameters may, for example, be: search card, card type is a weather card, date on which weather information needs to be presented, and so on. In some embodiments, the search reference information may include search instructions generated by the machine learning model. The search instructions are, for example, corresponding tool or function calls. That is, the machine learning model can obtain the target type of rich media content by calling the corresponding tool or function. The machine learning model here and the aforementioned processing model may be the same model or different models.

[0056] In some embodiments, the server 120 first uses the search reference information to obtain multiple candidate rich media content of the target type; then based on the user input, the server 120 selects the candidate rich media content that matches the user input from the multiple candidate rich media content as the rich media content corresponding to the first text. For example, the user input is "How is actor D doing recently", the first text is "D actor recently attended the YY awards ceremony", the search reference information is: search for YY awards ceremony pictures, the search results of the server 120 are multiple YY awards ceremony pictures, then the server 120 selects a picture related to XX from the multiple YY awards ceremony pictures as the rich media content corresponding to the first text.

[0057] In some embodiments, the first rich media content includes structured cards, and the server 120 obtains the corresponding structured cards in the following manner. First, based on the first text and multiple card types, a machine learning model is used to determine the card type and one or more input parameters of the structured card. Then the one or more input parameters are provided to the card generation function block of the card type, and the first rich media content is obtained from the card generation function block. For example, in the case where the first text mentions the weather, based on the first text and multiple card types, a machine learning model is used to determine to obtain a weather card and related weather parameters (such as temperature change trends, time, location, etc.). The relevant weather parameters are then input into a weather card generation module, and the weather card is obtained from the weather card generation module as the first rich media content.

[0058] In some embodiments, the server 120 may select candidate rich media content that matches the user input from multiple candidate rich media content based on preset rules as the rich media content corresponding to the first text. For example, multiple candidate rich media content all match the user input. In this case, one of the candidate content may be selected as the rich media content corresponding to the first text based on the time sequence, quality, and other information of the multiple candidate rich media content. For example, the candidate rich media content with the closest time or the rich media content with the highest quality may be selected as the rich media content corresponding to the first text. The quality of rich media content refers to the quality of indicators such as resolution (for images), clarity (for videos), and sound quality (for audio) of the rich media.

[0059] In block 250, the server 120 incorporates the target type of rich media content into the reply and presents it to the user as part of the reply. In other words, the server 120 presents the first rich media content corresponding to the first text as part of the reply.

[0060] Continue to refer Figure 2The above description is explained by taking the example of whether to add rich media content to the first text generated by user input. It should be understood that after receiving the user input, process 200 can continuously repeat box 220 to box 250 until the server 120 completes all responses to the user input.

[0061] For example, in some embodiments, after the server 120 presents the first text and / or the first rich media content, the process 200 may also include: the server 120 presents a second text generated based on the user input as part of a response to the user input, wherein the first media content is visually presented between the first text and the second text.

[0062] In some embodiments, process 200 may further include: presenting second rich media content corresponding to the second text as part of the reply, the second rich media content being determined based on at least the second text and being visually presented after the second text. The determination and acquisition of the second rich media content is similar to the determination and acquisition of the first rich media content, and will not be described in detail herein.

[0063] like Figure 3A-3C As shown, in some embodiments, in response to receiving user input 310 "How about XXX cake", the server 120 presents response information 320 generated for the user input 310 on the interface. The response information 320 includes a first text 321, a first rich media content 322, a second text 323, and a second rich media content 324. The first text 321 generally describes the XXX cake. The first rich media content 322 is a picture of the XXX cake. The second text 323 is a description of the first step of making the XXX cake. The second rich media content is a picture of the cream involved in the second text 323.

[0064] like FIG. 3A to FIG. 3C As shown, in some embodiments, the server 120 presents a first text 321 generated based on the user input in response to receiving the user input 310. Then, based on the first text 321, it is determined that a picture of a certain cake is to be obtained. In the process of obtaining the picture of the certain cake, a prompt is displayed in the to-be-displayed area of ​​the first rich media content 322 to inform the user that the picture of the certain cake is being searched. After presenting the first text 321 and the first rich media content 322, the server 120 presents the second text 323 and the second rich media content 324.

[0065] Figure 4 FIG. 4 is a flowchart showing an example process 400 for obtaining rich media content according to some embodiments of the present disclosure. Figure 1The process 400 is described with reference to the environment 100. The process 400 may be implemented at the client device 110 and / or the server 120. For the convenience of description, the process 400 is implemented at the server 120 as an example.

[0066] It should be noted that if the process 400 is implemented at the client device 110, some operations described with reference to the client device 110 may require the assistance of the server 120. It should be noted that the operations performed by the client device 110 may be performed by the relevant application and / or the target application installed on the client device 110.

[0067] In some embodiments, after the server 120 generates a first text for the user input using the machine learning model 420, it can generate prompt information 410 based on the first text and multiple predetermined types of descriptive information. The prompt information 410 is used to instruct the machine learning model 420 to determine whether to obtain at least one type of rich media content among multiple predetermined types of rich media content based at least on the first text. The prompt information may include at least one of the following: indication information, which indicates that the machine learning model determines whether additional processing is required for the first text, such as whether additional tools need to be called; reference information for the machine learning model to refer to when performing additional processing on the first text, such as what types of rich media content are there; processing information, which indicates how the machine learning model should perform additional processing on the first text, such as how to call a tool or function; how the machine learning model should process the return information obtained by performing additional processing on the first text, such as directly presenting the information or determining whether to continue calling the tool or function based on the return information.

[0068] After the server 120 generates the prompt information 410, it provides it to the machine learning model 420. It should be understood that the machine learning model 420 and the aforementioned processing model can be the same model or different models.

[0069] In some embodiments, the machine learning model 420 determines to obtain at least one type of rich media content from a plurality of predetermined types of rich media content based on the first text. Figure 2 The description will not be repeated here.

[0070] In some embodiments, the machine learning model 420 determines to acquire at least one type of rich media content among a plurality of predetermined types of rich media content based on the user input and the first text. For example, when the semantics of the user input expresses the need to acquire an image, the image content corresponding to the first text is acquired based on the first text.

[0071] In some embodiments, after the machine learning model 420 determines that at least one type of rich media content among multiple predetermined types of rich media content is to be obtained, an output is generated, and the output obtains at least one type of rich media content among multiple predetermined types of rich media content. As an example, the output is a tool or function call indication. That is, the machine learning model 420 indicates calling a tool or a function to obtain at least one type of rich media content among multiple predetermined types of rich media content. For example, the machine learning model 420 output indicates calling a tool or function to obtain the weather. It should be understood that the machine learning model 420 also outputs the input required to provide to the tool to be called or the parameters required by the function to be called. For example, for the tool or function to obtain the weather, the machine learning model 420 also outputs parameters or information related to time and place.

[0072] In some embodiments, the called tool or function 440 obtains at least one type of rich media content from a plurality of predetermined types of rich media content based on the parameters or information output by the machine learning model 420. For example, a picture search tool searches for picture information about a person based on information such as a person's name and time. The called tool or function 440 then returns the result to the machine learning model 420. It should be understood that the called tool or function may be one, or two or more. For example, a picture search tool may be called, and a video search tool may also be called at the same time. Accordingly, the result obtained by the called tool or function 440 may include at least one type of rich media content from a plurality of predetermined types of rich media content.

[0073] Subsequently, the machine learning model 420 outputs a response to the user input based on the return result of the called tool or function 440, namely, the target type of rich media content 450. It should be understood that the result returned by the called tool or function 440 may include at least one target type of rich media content, or may include multiple target types of rich media content. The machine learning model 420 may select one from the multiple target types of rich media content as the final target type of rich media content 450 based on user input, preset rules, etc.

[0074] Figure 5 FIG. 5 is a flow chart showing an interactive process 500 according to some embodiments of the present disclosure. Figure 1 The process 500 is described with reference to the environment 100 of FIG. The process 500 may be implemented at the server 120 or the client device 110 , or may be implemented by the server 120 in cooperation with the client device 110 .

[0075] At block 510 , the server 120 generates a first text based on the user input in response to receiving the user input.

[0076] In block 520 , the server 120 obtains first rich media content corresponding to at least a portion of the first text based at least on the first text.

[0077] At block 530 , the first text and the first rich media content are presented as at least a portion of a response to the user input.

[0078] In some embodiments, the process 500 further includes: determining to obtain a target type of rich media content among a plurality of predetermined types based at least on the first text, and obtaining the target type of rich media content as the first rich media content using reference information generated based on the first text.

[0079] In some embodiments, determining to acquire a target type of rich media content from among the plurality of predetermined types includes: determining the subject described by the first text by analyzing the semantics of the first text, and selecting a predetermined type related to the subject from among the plurality of predetermined types as the target type.

[0080] In some embodiments, the target type satisfies at least one of the following: for a topic related to visual elements, the target type includes image. For a topic related to auditory elements, the target type includes audio. For a topic related to dynamic scenes, the target type includes video. For a topic related to changing trends, the target type includes structured cards.

[0081] In some embodiments, determining that a target type of rich media content among multiple predetermined types is to be acquired includes: generating prompt information based on the first text and the description information of the multiple predetermined types, the prompt information being used to instruct the machine learning model to determine whether to acquire at least one type of rich media content among the multiple predetermined types of rich media content based at least on the first text. Providing the prompt information to the machine learning model to obtain an output of the machine learning model, the output indicating that the target type of rich media content among the multiple predetermined types is to be acquired.

[0082] In some embodiments, determining to acquire rich media content of a target type from among a plurality of predetermined types comprises: determining, by analyzing the semantics of the first text, a target text portion describing at least one of the following objects from the first text: a visual element, an auditory element, a dynamic scene, or a change trend. Selecting a predetermined type related to the object described by the target text portion from the plurality of predetermined types as the target type.

[0083] In some embodiments, the first rich media content includes rich media content of a target type, and the first rich media content is obtained by: generating search reference information matching the rich media content of the target type using a machine learning model based at least on the first text. The first rich media content is obtained based on the search reference information.

[0084] In some embodiments, obtaining the first rich media content based on the search reference information includes: obtaining multiple candidate rich media content of the target type using the search reference information. Based on user input, selecting candidate rich media content matching the user input from the multiple candidate rich media content as the first rich media content.

[0085] In some embodiments, the first rich media content includes a structured card, and the first rich media content is obtained by: determining a card type and one or more input parameters of a structured card based on the first text and a plurality of card types using a machine learning model. The one or more input parameters are provided to a card generation functional block of the card type. The first rich media content is obtained from the card generation functional block. In some embodiments, process 500 also includes: presenting a second text generated based on the user input as part of a response to the user input, wherein the first media content is visually presented between the first text and the second text.

[0086] In some embodiments, process 500 further includes presenting second rich media content corresponding to the second text as part of the reply, the second rich media content being determined based on at least the second text and being visually presented after the second text.

[0087] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 6 An exemplary structural block diagram of an interactive apparatus 600 according to some embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in a client device 110 and / or a server 120. Each module / component in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0088] like Figure 6 As shown, the apparatus 600 includes a text generation module 610, configured to generate a first text based on a user input in response to receiving the user input. A rich media content acquisition module 620, configured to acquire a first rich media content corresponding to at least a portion of the first text based on at least the first text. A presentation module 630, configured to present the first text and the first rich media content as at least a portion of a response to the user input.

[0089] In some embodiments, the rich media content acquisition module is further configured to: determine to acquire rich media content of a target type among multiple predetermined types based at least on the first text, and acquire rich media content of the target type as first rich media content using search reference information generated based on the first text.

[0090] In some embodiments, the rich media content acquisition module is further configured to: determine the subject described by the first text by analyzing the semantics of the first text, and select a predetermined type related to the subject from a plurality of predetermined types as the target type.

[0091] In some embodiments, the target type satisfies at least one of the following: for a topic related to visual elements, the target type includes image. For a topic related to auditory elements, the target type includes audio. For a topic related to dynamic scenes, the target type includes video. For a topic related to changing trends, the target type includes structured cards.

[0092] In some embodiments, the rich media content acquisition module is further configured to: generate prompt information based on the first text and the description information of the plurality of predetermined types, the prompt information being used to instruct the machine learning model to determine whether to acquire at least one type of rich media content among the plurality of predetermined types of rich media content based at least on the first text, and provide the prompt information to the machine learning model to obtain an output of the machine learning model, the output indicating that the target type of rich media content among the plurality of predetermined types is to be acquired.

[0093] In some embodiments, the rich media content acquisition module is further configured to: determine, from the first text, a target text portion that describes at least one of the following objects: a visual element, an auditory element, a dynamic scene, or a change trend by analyzing the semantics of the first text, and select a predetermined type related to the object described by the target text portion from a plurality of predetermined types as the target type.

[0094] In some embodiments, the rich media content acquisition module is further configured to: generate search reference information matching the target type of rich media content using a machine learning model based at least on the first text, and acquire the first rich media content based on the search reference information.

[0095] In some embodiments, the rich media content acquisition module is further configured to: acquire multiple candidate rich media content of the target type using the search reference information, and select, based on user input, candidate rich media content matching the user input from the multiple candidate rich media content as the first rich media content.

[0096] In some embodiments, the rich media content acquisition module is further configured to: determine the card type and one or more input parameters of the structured card based on the first text and the plurality of card types using a machine learning model, provide the one or more input parameters to a card generation function block of the card type, and obtain the first rich media content from the card generation function block.

[0097] In some embodiments, the first presentation module 610 is further configured to present a second text generated based on the user input as part of a reply to the user input, wherein the first media content is visually presented between the first text and the second text.

[0098] In some embodiments, the second presentation module 620 is further configured to present second rich media content corresponding to the second text as part of the reply, where the second rich media content is determined based on at least the second text and is visually presented after the second text.

[0099] The units and / or modules included in the device 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units and / or modules in the device 600 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0100] It should be understood that one or more steps in the above method can be performed by a suitable electronic device or a combination of electronic devices. Such an electronic device or a combination of electronic devices may include, for example, a computer that runs Figure 1 The system manages the devices of the platform 110.

[0101] Figure 7 1 shows a block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 7 The electronic device 700 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 7 The electronic device 700 shown may include or be implemented as Figure 1 System management platform 110 or Figure 6 Device 600.

[0102] like Figure 7As shown, the electronic device 700 is in the form of a general electronic device. The components of the electronic device 700 may include, but are not limited to, one or more processors or processing units 710, a memory 720, a storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processing unit 710 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 720. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 700.

[0103] The electronic device 700 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 730 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 700.

[0104] The electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 7 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0105] The communication unit 740 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 700 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 700 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0106] The input device 750 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 760 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 700 may also communicate with one or more external devices (not shown) through the communication unit 740 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 700, or communicate with any device that allows the electronic device 700 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0107] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0108] According to an exemplary implementation of the present disclosure, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 5 The methods provided in the various optional methods are therefore not described in detail here.

[0109] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0110] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0111] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0112] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0113] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. An interactive method, comprising: In response to receiving a user input, generating a first text based on the user input; Based at least on the first text, obtaining first rich media content corresponding to at least a portion of the first text; as well as The first text and the first rich media content are presented as at least part of a response to the user input.

2. The method according to claim 1, further comprising: At least based on the first text, determining to obtain rich media content of a target type among a plurality of predetermined types; as well as The target type of rich media content is acquired as the first rich media content by using the reference information generated based on the first text.

3. The method according to claim 2, wherein determining to obtain a target type of rich media content among a plurality of predetermined types comprises: Determining the subject described by the first text by analyzing the semantics of the first text; as well as A predetermined type related to the subject is selected from the plurality of predetermined types as the target type.

4. The method according to claim 3, wherein the target type satisfies at least one of the following: For the topic related to visual elements, the target type includes images, For the topic related to auditory elements, the target type includes audio, For the subject matter related to dynamic scenes, the target type includes video, or For the topic related to changing trends, the target type includes structured cards.

5. The method according to claim 2, wherein determining to obtain a target type of rich media content among a plurality of predetermined types comprises: Generate prompt information based on the first text and the description information of the plurality of predetermined types, wherein the prompt information is used to instruct the machine learning model to determine whether to obtain at least one type of rich media content among the plurality of predetermined types of rich media content based at least on the first text; as well as The prompt information is provided to the machine learning model to obtain an output of the machine learning model, wherein the output indicates that rich media content of a target type among multiple predetermined types is to be obtained.

6. The method according to claim 2, wherein determining to obtain a target type of rich media content among a plurality of predetermined types comprises: By analyzing the semantics of the first text, determining from the first text a target text portion that describes at least one of the following objects: a visual element, an auditory element, a dynamic scene, or a change trend; as well as A predetermined type related to the object described by the target text portion is selected from the plurality of predetermined types as the target type.

7. The method according to claim 1, wherein the first rich media content comprises rich media content of a target type, and the first rich media content is obtained by: generating search reference information matching the target type of rich media content using a machine learning model based at least on the first text; and The first rich media content is acquired based on the search reference information.

8. The method according to claim 7, wherein acquiring the first rich media content based on the search reference information comprises: Using the search reference information, obtaining a plurality of candidate rich media contents of the target type; as well as Based on the user input, candidate rich media content matching the user input is selected from the plurality of candidate rich media content as the first rich media content.

9. The method according to claim 1, wherein the first rich media content comprises a structured card, and the first rich media content is obtained by: Determining a card type and one or more input parameters of the structured card using a machine learning model based on the first text and a plurality of card types; providing the one or more input parameters to a card generation function block of the card type; and The first rich media content is obtained from the card generation function block.

10. The method according to claim 1, further comprising: Second text generated based on the user input is presented as part of a response to the user input, wherein the first rich media content is visually presented between the first text and the second text.

11. The method according to claim 10, further comprising: Second rich media content corresponding to at least a portion of the second text is presented as part of the reply, the second rich media content being determined based on at least the second text and being visually presented after the second text.

12. A device for interaction, comprising: a text generation module, configured to generate a first text based on the user input in response to receiving the user input; a rich media content acquisition module, configured to acquire, based at least on the first text, first rich media content corresponding to at least a portion of the first text; as well as A presentation module is configured to present the first text and the first rich media content as at least a part of a response to the user input.

13. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 11 when executed by the at least one processing unit.

14. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, wherein the computer program implements the method according to any one of claims 1 to 11 when executed by a processor.

Images