Information processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides an information processing method and device, equipment, a storage medium and a program product. The method includes determining at least one document segment associated with a user input based on the user input, a target document segment in the at least one document segment including one or more media contents. And generating prompt information based on the text in the at least one document segment and the user input. The prompt information comprises one or more identifiers, and the identifiers correspond to media contents in the one or more media contents and are related to the positions of the corresponding media contents in the target document segment. And based on the prompt information, utilizing a machine learning model to generate a reply for user input. In this way, when the prompt information is generated, the identifier is used for replacing the media content in the document, so that the machine learning model can quote the identifier with a simpler expression mode when generating the output, and finally accurate media content quotation is achieved for the reply input by the user.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to information processing methods, apparatuses, devices, computer-readable storage media, and computer program products. Background Art

[0002] With the development of information technology, various terminal devices can provide various services to people in work and life. Applications that provide services can be deployed in terminal devices. Terminal devices present corresponding content and interact with users through the user interface of the application to meet various needs of users. In some cases, users may initiate queries or inquiries within the application. Therefore, how to use the knowledge base to enable the application to generate higher quality query results for user input is a problem of concern. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for interaction is provided. The method comprises: determining at least one document segment associated with the user input based on the user input, wherein a target document segment in the at least one document segment comprises one or more media contents. Based on the text in the at least one document segment and the user input, prompt information is generated, the prompt information comprising one or more identifiers, wherein an identifier in the one or more identifiers corresponds to a media content in the one or more media contents and is related to the position of the corresponding media content in the target document segment. Based on the prompt information, a reply to the user input is generated using a machine learning model, wherein in the case where the reply includes the content of the target text segment, at least one of the one or more media contents is associated with at least part of the text of the target text segment and displayed in the reply.

[0004] In a second aspect of the present disclosure, an interactive device is provided. The device includes: a document determination module, configured to determine at least one document segment associated with the user input based on the user input, wherein the target document segment in the at least one document segment includes one or more media contents. A prompt information generation module, configured to generate prompt information based on the text in the at least one document segment and the user input, wherein the prompt information includes one or more identifiers, wherein the identifier in the one or more identifiers corresponds to the media content in the one or more media contents and is related to the position of the corresponding media content in the target document segment. And a reply module, configured to generate a reply to the user input based on the prompt information using a machine learning model, wherein in the case where the reply includes the content of the target text segment, at least one of the one or more media contents is associated with at least part of the text of the target text segment and displayed in the reply.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the electronic device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, which includes a computer program, wherein when the computer program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0008] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 2 A diagram showing an example architecture for information processing according to some embodiments of the present disclosure;

[0012] Figure 3 A flowchart showing an example process of building a knowledge base according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A flowchart illustrating an example process of a question-and-answer process according to some embodiments of the present disclosure;

[0014] Figure 5 A flowchart of an information processing method according to some embodiments of the present disclosure is shown;

[0015] Figure 6 An exemplary structural block diagram of an information processing device according to some embodiments of the present disclosure is shown; and

[0016] Figure 7 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0019] Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0020] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0021] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scopes of use, usage scenarios, etc. of the information involved in the present disclosure should be informed to relevant users and their authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations. The relevant users may include any type of right holders, such as individuals, enterprises, and groups.

[0022] For example, in response to receiving an active request from a user, a prompt message is sent to the relevant user to clearly prompt the relevant user that the operation requested to be performed will require obtaining and using the information of the relevant user, so that the relevant user can independently choose whether to provide information to software or hardware such as an electronic device, application, server or storage medium that executes the operation of the technical solution of the present disclosure based on the prompt message.

[0023] As an optional but non-limiting implementation, in response to receiving an active request from a relevant user, a prompt message is sent to the relevant user, for example, in the form of a pop-up window, in which the prompt message may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide information to the electronic device.

[0024] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0025] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0026] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs, and typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes input from the previous layer.

[0027] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. In the example environment 100, a target application 112 is installed in a client device 110. A user 130 can interact with the application service component target application 112 via the client device 110 and / or an attached device of the client device 110.

[0028] In some embodiments, the target application 112 can be downloaded and installed on the client device 110. In some embodiments, the target application 112 can also be accessed through other means, such as through a web page. Figure 1 In the environment 100, in response to the target application 112 being started, the client device 110 can present an interface 140 of the target application 112. The interface 140 can be, for example, an interactive interface of the target application 112.

[0029] In some embodiments, the target application 112 may have intelligent dialogue and information processing capabilities. For example, the target application 112 may be a user's digital assistant or an application configured with a digital assistant. In an embodiment of the present disclosure, the target application 112 is used to interact with the user 130 to assist the user 130 in using the terminal or processing and searching for information. In some embodiments, an interaction window with the target application 112 may be presented in the interface 140. In the interaction window, the user 130 can have a dialogue with the target application 112 by inputting natural language, pictures, audio files, video files, web page files, etc., to instruct the target application 112 to assist in completing various tasks, including operations on content entities; or to instruct the target application 112 to complete questions and answers, queries or searches. In some embodiments, the interaction between the user 130 and the target application 112 may include interaction between the user 130 and the digital assistant. For example, the user's input to the target application 112 may be issued to the digital assistant, and the target application 112's reply to the user 130 may be issued to the user 130 by the digital assistant.

[0030] In some embodiments, multiple interaction modes between the user 130 and the target application 112 may be provided, and the multiple interaction modes may be flexibly switched. When a certain interaction mode is triggered, a corresponding interaction area is presented to facilitate the interaction between the user 130 and the target application 112. The user 130 interacts with the target application 112 in different interaction modes in different ways, so that the interaction needs in different application scenarios can be flexibly adapted.

[0031] In some embodiments, information processing services specific to the user 130 can be provided based on historical interaction information between the user 130 and the target application 112 and / or a data range specific to the user 130. In some embodiments, historical interaction information of the user 130 interacting with the target application 112 in multiple interaction modes can be stored in association with the user 130. In this way, in one interaction mode (any one or a designated one interaction mode) among the multiple interaction modes, the target application 112 can provide services for the user 130 based on the historical interaction information stored in association with the user 130.

[0032] The target application 112 may be called or awakened by an appropriate method (e.g., a shortcut key, a button, or voice) to present an interactive window with the user 130. By selecting the target application 112, an interactive window with the target application 112 may be opened. The interactive window may include interface elements for information interaction, such as an input box, a message list, a message bubble, and the like. In other embodiments, the target application 112 may be awakened by an entry control or menu provided in the interface 140, or by inputting a preset instruction.

[0033] The interactive window between the target application 112 and the user 130 may include a conversation window, such as a conversation window in an instant messaging application or an instant messaging module of a specific application. In the conversation window, the interaction between the target application 112 and the user 130 may be presented in the form of a conversation message. Alternatively or additionally, the interactive window between the target application 112 and the user 130 may also include other types of windows, such as a window in a floating window mode, in which the user 130 may trigger the target application 112 to perform a corresponding operation by inputting an instruction, selecting a shortcut instruction, etc.

[0034] In some embodiments, the target application 112 may support an interactive mode of a conversation window, also referred to as a conversation mode. In this interactive mode, a conversation window between the user 130 and the target application 112 is presented, in which the user 130 and the target application 112 interact through conversation messages. In the conversation mode, the target application 112 may perform tasks according to the conversation messages in the conversation window. In the interactive window, the user 130 inputs an interactive message, and the target application 112 provides a reply message in response to the user input. By selecting the target application 112, a conversation window with the target application 112 may be opened. The conversation window may include interface elements for information interaction, such as an input box, a message list, a message bubble, and the like.

[0035] In some embodiments, the client device 110 communicates with the server 120 to provide services for the target application 112. The client device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the client device 110 can also support any type of interface for the user (such as a "wearable" circuit, etc.). The server 120 can be various types of computing systems / servers that can provide computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like. The server 120 can be deployed with one or more various machine learning models 122 to provide services for the target application 112. In some embodiments, the model 122 can be a machine learning model, a deep learning model, a learning model, a neural network, etc. In some embodiments, the model can be based on a language model (LM). The language model can have the ability to answer questions by learning from a large amount of corpus. The model 122 can also be based on other appropriate models.

[0036] In some embodiments, the implementation of at least part of the functions of the target application 112 and / or the implementation of at least part of the functions of the digital assistant in the target application 112 can be implemented based on the model. During the creation or operation of the target application 112, one or more models 122 can be called, such as the capabilities of the model 122. In the target application 112, the digital assistant can use the model 122 to understand the user input and provide a reply to the user based on the output of the model 122.

[0037] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0038] As mentioned above, how to use the knowledge base to enable the application to generate higher quality query results for user input is a concern. In intelligent question-answering, the current general link itself does not have the ability to recall media content (such as pictures). However, in typical question-answering services, such as customer service duty numbers, official website help centers, and other question-answering scenarios, there is a strong demand for related pictures in the given replies.

[0039] At present, when using models, especially language models, to handle intelligent question-answering scenarios with media content recall demands, there are the following problems. First, when generating text content, large models often incorrectly quote media content (such as pictures), that is, irrelevant or incorrect media content (quotes and uses it as the final generated answer. This is usually due to misunderstandings in the model when associating text descriptions with media content, or the prompt information generated for the model is too complex, resulting in the model being unable to fully follow and understand the instructions. Second, currently, when using language models to generate text content, the model often becomes unstable when quoting or generating media content links (such as uniform resource locators URLs) in a predetermined format (such as Markdown format). This is mainly manifested in the frequent generation of incorrect or inaccurate URLs by the model when repeating or quoting media content links. This instability may be caused by inconsistencies in the URLs in the model training data, or the model fails to correctly Correctly understand and process the context of the linked text. Incorrect URL generation can lead to inaccurate content and affect users' trust in the generated content, especially in professional fields that require accurate document and data traceability, such as law and academic research. Third, when processing text in a predetermined format (such as Markdown format), media content is usually referenced through longer public network Hypertext Transfer Protocol (HTTP) links or file identifiers (FileID). These long links take up a lot of space in text slices, thereby reducing the density of effective information. When these texts are converted into vectors for information recall, the generated vectors often cannot accurately reflect the true semantic content of the text due to the presence of a large number of non-semantic elements (such as URLs). This phenomenon causes the accuracy of vector-based recall systems to be affected.

[0040] In view of this, according to an embodiment of the present disclosure, an improved scheme for information processing is provided. Specifically, at least one document fragment associated with the user input is determined based on the user input, and the target document fragment in the at least one document fragment includes one or more media contents. Based on the text in the at least one document fragment and the user input, prompt information is generated. The prompt information includes one or more identifiers, and an identifier in the one or more identifiers corresponds to a media content in the one or more media contents and is related to the position of the corresponding media content in the target document fragment. Based on the prompt information, a reply to the user input is generated using a machine learning model. In the case where the reply includes the content of the target text fragment, at least one of the one or more media contents is associated with at least part of the text of the target text fragment and displayed in the reply.

[0041] According to the scheme of the present disclosure, when generating prompt information, the identifier is used to replace the media content in the document. Compared with the media content, the identifier is more concise and easy to understand. This allows the machine learning model to accurately quote the identifier when generating the output, and finally realizes accurate media content reference in the reply to the user input. Therefore, when processing the intelligent question and answer scenario with the media content recall demand, the media content can be accurately recalled and correctly quoted in the reply and answer, so as to provide the user with higher quality query results or question and answer results. In the case where the media content is included in the reply, since the media content and the corresponding text content are displayed in association, the media content contained in the reply can be placed in a suitable or correct position. This improves the consistency of the media content and the text in the reply, and further improves the quality of the reply. Further, the associated display of the media content and the corresponding text helps the user understand the reply. The embodiment of the present disclosure can accurately place the media content in the reply, so as to avoid the interference caused to the user by the incorrect insertion of the media content.

[0042] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0043] Figure 2 2 shows a diagram of an example architecture 200 for information processing according to some embodiments of the present disclosure. Figure 2 As shown, the example architecture 200 includes a knowledge base 210, a target application 112, and one or more machine learning models 122. The knowledge base 210 is used to store document data required to provide responses to user inputs, such as user queries, inquiries, or searches. In the knowledge base 210, the document data is stored in the form of document fragments represented by vectorization, so as to facilitate efficient retrieval of data.

[0044] In some embodiments, the knowledge base 210 stores only the text content of the document data, but does not store the media content in the original document corresponding to the document data, such as images, charts, audio, video, structured cards, etc. This can improve the density of effective information in the knowledge base 210, so that when the required document data is recalled from the knowledge base 210, the interference of the media content can be avoided, and the text recall accuracy can be improved. The details of the knowledge base construction will be combined later in Figure 3 The description is given and no further elaboration is given here.

[0045] The target application 112 obtains one or more document fragments associated with the user input from the knowledge base 210 based on the user input 220. Then the target application 112 generates context information based on the one or more document fragments. In some embodiments, the original document fragment corresponding to the target document fragment in the one or more document fragments associated with the user input includes one or more media contents. In this case, the target application 112 uses an identifier to restore the one or more media contents to the target document fragment. Specifically, the target application 112 adds the identifier corresponding to the one or more media contents to the text of the target document fragment. And the position of the identifier corresponding to the one or more media contents corresponds to the position of the one or more media contents in the original document fragment.

[0046] The target application 112 generates context information based on one or more document fragments, and then generates prompt information based on the context information and user input. In some embodiments, the original document fragment corresponding to the target document fragment in the one or more document fragments associated with the user input includes one or more media contents. In this case, since the one or more document fragments have been processed as described above, the prompt information also includes one or more identifiers. The identifier in the one or more identifiers corresponds to the media content in the one or more media contents and is related to the position of the corresponding media content in the target document fragment. The details of adding identifiers to the target document fragment will be described later in conjunction with Figure 4 The description is given and no further elaboration is given here.

[0047] In the embodiment of the present disclosure, the media content in the document fragment is replaced by the identifier, thereby simplifying the prompt information, and the identifier can be used to accurately associate the text description with the media content. In this way, when the machine learning model generates a reply based on the user input 220 and the prompt information, there is no need to reference or generate a media content link in a predetermined format (such as markdown format), thereby improving the accuracy of the content generated by the machine learning model.

[0048] The target application 112 provides the prompt information to the machine learning model 122, and uses the machine learning model 122 to generate a response 230 to the user input 220. In the question-and-answer scenario, the response 230 to the user input is also called an answer. In the query scenario, the response 230 to the user input is also called a query result.

[0049] In some embodiments, the target application 112 generates an example process of a reply 230 for a user input 220 as follows. First, the target application 112 provides prompt information to the machine learning model 122 to obtain the output of the machine learning model 122. Then, in response to determining that the output of the machine learning model 122 includes a first identifier among the one or more identifiers, the target application 112 obtains media content corresponding to the first identifier. Then, the target application 112 adds the obtained media content to the reply 230 for the user input 220. The details of adding media content to the reply 230 will be described later in conjunction with Figure 4 The description is given and no further elaboration is given here.

[0050] Figure 3 FIG. 3 is a flowchart showing an example process 300 for building a knowledge base according to some embodiments of the present disclosure. Figure 1 The process 300 is described with reference to the environment 100. The process 300 may be implemented at the client device 110 and / or the server 120. For the convenience of description, the process 300 is implemented at the server 120 as an example.

[0051] It should be noted that if the process 200 is implemented at the client device 110, some operations described with reference to the client device 110 may require the assistance of the server 120. It should be noted that the operations performed by the client device 110 may be performed by the relevant application and / or the target application installed on the client device 110.

[0052] In block 310, the server 120 obtains documents for building a knowledge base and parses the documents. In some embodiments, the documents for building a knowledge base can be obtained by the server 120 from a public database or from the Internet based on user input or other information. In some embodiments, the documents for building a knowledge base can be internal document data of an entity or organization (such as a company or enterprise), and the server 120 can obtain the documents for building a knowledge base in response to data uploaded by a user. That is, the documents for building a knowledge base can be obtained by the server 120 based on a public data search, or can be provided by the user.

[0053] In certain embodiments, the target document in the document for building the knowledge base includes one or more media contents. After server 120 obtains the target document, server 120 parses the target document, obtains and stores one or more media contents in the target document. For example, server 120 stores one or more media contents in the target document in a predetermined memory location. The predetermined memory location can be local storage or cloud storage. In certain embodiments, server 120 identifies the one or more media contents with a resource identifier (UID). Like this, when the media contents in the one or more media contents need to be quoted in the reply for the user, the media contents can be obtained by the resource identifier of the media contents in the one or more media contents, and added to the reply.

[0054] In block 320, the server 120 divides the document used to construct the knowledge base into multiple document segments. For example, the document used to construct the knowledge base can be divided into multiple document segments based on the document semantics or other criteria used to construct the knowledge base by using a common text segmentation method in the art.

[0055] In some embodiments, the target document segment in the plurality of document segments includes one or more media contents. For each target document segment, the server 120 stores the corresponding information of the media contents in the one or more media contents included in the target document segment in the metadata of the target document segment to record the corresponding position information and corresponding resource identifier of the media contents in the one or more media contents in the target document segment. The corresponding position information of the media contents in the one or more media contents in the target document segment can be obtained by parsing the target document segment or the original document corresponding to the target document segment.

[0056] In box 330, the server 120 removes the media content included in the multiple document fragments to obtain multiple updated document fragments. In the present disclosure, the updated document fragment refers to the document fragment obtained by removing the media content in the document fragment, such as a picture, that is, the text in the document fragment. When the document fragment does not include media content, the document fragment and the updated document fragment are consistent. In the present disclosure, since the media content included in the document fragment is removed, there is no media content link in a predetermined format (such as markdown format) in the updated document fragment, which improves the effective information density of the document fragment. Therefore, when the document fragment (that is, the updated document fragment) is subsequently recalled, since non-semantic elements (such as URLs) are excluded, the generated text can reflect the true semantics. Therefore, the accuracy of document fragment recall is improved. It should be noted that the recalled document fragment refers to the updated document fragment.

[0057] At block 340, the server 120 stores the updated vectorized representations of the plurality of document segments. In some embodiments, the server 120 may convert the updated document segments into vector matrices using a predetermined embedding model. In some embodiments, the server 120 stores the updated vectorized representations of the plurality of document segments in a predetermined knowledge path, such as Figure 2 In the knowledge base 210.

[0058] Figure 4 FIG. 4 is a flowchart showing an example process 400 for information processing according to some embodiments of the present disclosure. For ease of discussion, reference will be made to Figure 1 The process 400 is described with reference to the environment 100. The process 400 may be implemented at the client device 110 and / or the server 120. For the convenience of description, the process 400 is implemented at the server 120 as an example.

[0059] It should be noted that if the process 400 is implemented at the client device 110, some operations described with reference to the client device 110 may require the assistance of the server 120. It should be noted that the operations performed by the client device 110 may be performed by the relevant application and / or the target application installed on the client device 110.

[0060] At block 410, the server 120 receives user input. For example, the user input provided by the user at the client device 110 may be sent to the server 120. At block 420, the server 120 determines at least one updated document segment associated with the user input based on the user input. In some embodiments, the server 120 obtains at least one updated document segment associated with the user input from the knowledge base through a predetermined retrieval method, such as similarity retrieval or full-text retrieval.

[0061] In block 430, the server 120 uses the identifier to restore the one or more media contents to at least one updated document segment to generate context information. In block 440, the server 120 generates prompt information based on the user input and the context information. That is, the server 120 injects the recalled document segment into the machine learning model to generate prompt information.

[0062] In some embodiments, the target document segment corresponding to the updated target document segment in the at least one updated document segment determined in block 420 includes one or more media contents. In this case, the server 120 restores the one or more media contents to the updated target document segment using the identifier.

[0063] In some embodiments, the identifier may be an identifier in a predetermined format. Taking a picture as an example, the identifier in the predetermined format may be, for example, “picture:<<image x-x> >", where "icon" is used to prompt that there is an image here in the model context information. Image identifies the type of media content as an image. The first x represents the sequence number of the image in the updated target document segment in at least one updated document segment. The second x represents the sequence number of the image in one or more images in the target document segment corresponding to the updated target document segment. For example, the server 120 determines two updated document segments associated with the user input based on the user input. The first document segment (i.e., the first target document segment) corresponding to the first updated document segment (i.e., the first updated target document segment) includes two images, then the first image can be the representation <<image 1-1> >, the second image can be represented as <<image 1-2> >. The second document fragment (i.e., the second target document fragment) corresponding to the second updated document fragment (i.e., the second updated target document fragment) includes three pictures, which can be represented as < <image2-1>>、<<image 2-2> > and <<image 2-3> >. It should be understood that, in the present disclosure, an identifier of a predetermined format is described by taking a picture as an example, but the present disclosure is not limited to a picture.

[0064] In some embodiments, an example process of restoring one or more media contents to at least one updated document fragment using an identifier to generate context information is as follows. The server 120 obtains corresponding position information of the one or more media contents from the metadata of the target document fragment corresponding to the updated target document fragment. Then, based on the corresponding position information of the one or more media contents, the server 120 adds the corresponding identifiers of the one or more media contents in a predetermined format to the updated target document fragment to generate context information. If the first document fragment (i.e., the first target document fragment) corresponding to the first updated document fragment (i.e., the first updated target document fragment) includes two pictures, then based on the corresponding position information of the first picture in the first document fragment, "Figure:<<image 1-1> >" to the first updated document fragment. For example, if the first image is after the fifth line of the first document fragment, then "Image:<<image 1-1> >" after the fifth line of the first updated document fragment.

[0065] After restoring one or more media contents to the updated target document segment using the identifier in a predetermined format, the server 120 generates prompt information based on the user input and context information. The prompt information is used to guide the model to generate a correct response based on the user input and context information, and in the case where the target document segment includes media content, attach the relevant media content and output the media content with the identifier. In this way, since the identifier is much shorter than the media content link represented in a format such as markdown, the probability of the machine learning model generating incorrect media content is significantly reduced.

[0066] Example prompts and model outputs are shown in Table 1.

[0067] Table 1

[0068]

[0069]

[0070] The upper part of the table in Table 1 is an example of the system prompt information of the model. The lower part is an example of the output of the model. It should be understood that the above prompt information and model output are only an example and do not constitute a limitation of the present disclosure. The prompt information may also include more or less content than the prompt information shown. For example, the prompt information may also include information related to the model role, glossary, model language style, output target or output constraint, etc. In some embodiments, the prompt information may include indication information, which indicates that the model needs to be associated with the corresponding content output in the case where the output of the model includes an identifier, such as the identifier is output immediately after the relevant content. Taking Table 1 as an example, the identifier corresponding to the lion picture is immediately after the summary description of the lion's living habits, rather than after the description of the difference in living habits between the lion and the giant panda. The identifier corresponding to the giant panda picture is immediately after the description of the difference in living habits between the lion and the giant panda, rather than in the lion description content part. Through this restriction, on the one hand, the position of the picture in the document (which is an example of media content) can be accurately restored, and on the other hand, it can be beneficial for the subsequent response generated based on the model output to have a higher media content-text consistency. Taking Table 1 as an example, in the response generated based on the model output, the picture of the lion is displayed in association with the overview of the lion's living habits, for example, the picture of the lion follows the overview of the lion's living habits.

[0071] In some embodiments, the prompt information may include constraint information, which for example instructs the model not to output identifiers with graphic characters, or not to output media content in some markup language or programming language, such as not to output images in markdown format.

[0072] In some embodiments, the identifier may be a sentence identifier. The sentence identifier identifies a sentence in at least one updated document fragment. The sentence identifier is, for example, a sentence ID. After the server 120 determines at least one updated document fragment associated with the user input in box 420, the prompt information generation process is as follows. The server 120 divides the at least one updated document fragment into multiple sentences according to the sentence granularity. Then the server 120 generates a sentence identifier for each sentence in the multiple sentences based on the order of the at least one updated document fragment and the order of the multiple sentences. Then the server 120 determines the prompt information based on the sentence identifiers generated for the multiple sentences, at least one updated document fragment, and the user input. In some embodiments, for example, at least one updated document fragment includes 4 sentences, and identifiers representing the first sentence, the second sentence, the third sentence, and the fourth sentence, such as sentence numbers, can be generated for the 4 sentences.

[0073] In some embodiments, the target document segment corresponding to the updated target document segment in at least one updated document segment includes one or more media contents, and the server 120 also associates the one or more media contents with the sentence identifier of the updated target document segment. The example association process is as follows. For a given media content in one or more media contents, the server 120 obtains the location information and resource identifier of the given media content from the metadata of the target document segment. Then, based on the location information of the given media content, the server 120 determines the first sentence adjacent to the given media content in the updated target document segment, such as the first adjacent sentence before the given media content, or the first adjacent sentence after the given media content. Then, the server 120 associates the resource identifier of the given media content with the first sentence identifier of the determined first sentence.

[0074] Continue to refer Figure 4 In box 450, the server 120 generates a response to the user input based on the prompt information using a machine learning model.

[0075] In some embodiments, the identifier is an identifier in a predetermined format. In this case, an example process for generating a reply to a user input is as follows. In response to the model output provided by the machine learning model based on the prompt information, the server 120 includes a second identifier in a predetermined format, and obtains media content corresponding to the second identifier. The server 120 then replaces the second identifier in the model output with the acquired media content to update the model output. The server 120 then determines a reply based on the updated model output. In this solution, through an identifier in a predetermined format, the model can accurately generate a reference to the media content, and by removing the media content information in a predetermined format, the text recall effect is improved, and the corresponding location information and corresponding resource identifier of the media content stored in the metadata can be restored to the original text losslessly in the final reply.

[0076] In some embodiments, in the case where the reply includes the target media content, the media content and the text corresponding to the media content are displayed in association in the reply. For example, the media content can be immediately followed by the text corresponding to the media content. For another example, the media content can be displayed side by side with the corresponding text. This can improve the consistency of the media content and the text in the reply.

[0077] In some embodiments, an example process of obtaining media content corresponding to the second identifier is as follows: Server 120 determines a resource identifier from metadata of the target document segment based on the second identifier. Server 120 then uses the determined resource identifier to obtain media content corresponding to the second identifier.

[0078] In some embodiments, the identifier is a sentence identifier in a predetermined format. In this case, an example process for generating a reply to the user input is as follows. The server 120 provides prompt information to the machine learning model 122 to obtain the model output of the machine learning model 122. Then, in response to determining that the second sentence identifier in the model output is associated with a resource identifier, the server 120 obtains the media content corresponding to the second sentence identifier based on the associated resource identifier. The server 120 then adds the obtained media content to a position adjacent to the second sentence identifier to generate a reply to the user input. For obtaining the media content corresponding to the second sentence identifier based on the associated resource identifier, refer to the above description and will not be repeated here. In this scheme, by reducing the understanding of the large model of the media content (such as pictures) in the context, the output picture reference task is converted into an output text reference task in which the model is better at, thereby reducing the probability of model errors and improving the stability of the overall system.

[0079] Figure 5 FIG. 5 is a flowchart of an information processing process 500 according to some embodiments of the present disclosure. For ease of discussion, reference will be made to Figure 1 The process 500 is described with reference to the environment 100 of FIG. The process 500 may be implemented at the server 120 or the client device 110 , or may be implemented by the server 120 in cooperation with the client device 110 .

[0080] In block 510 , the server 120 determines at least one document segment associated with the user input based on the user input, wherein a target document segment in the at least one document segment includes one or more media contents.

[0081] In box 520, the server 120 generates prompt information based on text in at least one document fragment and user input, the prompt information including one or more identifiers, the identifier in the one or more identifiers corresponding to media content in one or more media content and related to the position of the corresponding media content in the target document fragment.

[0082] In box 530, based on the prompt information, a response to the user input is generated using a machine learning model, wherein when the response includes content of the target text segment, at least one of the one or more media contents is associated with at least part of the text of the target text segment and displayed in the response.

[0083] In some embodiments, process 500 further includes: providing prompt information to the machine learning model to obtain an output of the machine learning model; in response to determining that the output includes a first identifier of the one or more identifiers, obtaining media content corresponding to the first identifier; and adding the obtained media content to the reply.

[0084] In some embodiments, an identifier among the one or more identifiers includes an identifier in a predetermined format, and the identifier in the predetermined format includes at least one of the following: information indicating the type of the corresponding media content, the sequence number of the target document fragment in at least one document fragment, and the sequence number of the corresponding media content in one or more media contents.

[0085] In some embodiments, generating prompt information includes: obtaining corresponding position information of one or more media contents from the metadata of the target document fragment; based on the corresponding position information of the one or more media contents, adding corresponding identifiers of a predetermined format of the one or more media contents to the text of the target document fragment to generate context information; and determining the prompt information based on the context information and user input.

[0086] In some embodiments, generating a response to user input includes: in response to a model output provided by the machine learning model based on prompt information including a second identifier in a predetermined format, obtaining media content corresponding to the second identifier; replacing the second identifier in the model output with the obtained media content to update the model output; and determining a response based on the updated model output.

[0087] In some embodiments, acquiring the media content corresponding to the second identifier includes: determining a resource identifier from metadata of the target document segment based on the second identifier; and acquiring the media content corresponding to the second identifier using the determined resource identifier.

[0088] In some embodiments, the identifier of the one or more identifiers includes a sentence identifier that identifies a sentence in the target document segment.

[0089] In some embodiments, generating prompt information includes: dividing at least one document fragment into multiple sentences according to sentence granularity; generating a sentence identifier for each of the multiple sentences based on the order of at least one document fragment and the order of the multiple sentences; and determining the prompt information based on the sentence identifiers generated for the multiple sentences, the text in at least one document fragment, and the user input.

[0090] In some embodiments, process 500 also includes: for given media content among one or more media content, obtaining location information and a resource identifier of the given media content from the metadata of the target document fragment; based on the location information of the given media content, determining the first sentence adjacent to the given media content in the target document fragment; and associating the resource identifier of the given media content with the first sentence identifier of the determined first sentence.

[0091] In some embodiments, generating a response to user input includes: providing prompt information to a machine learning model to obtain a model output of the machine learning model; in response to determining that a second sentence identifier in the model output is associated with a resource identifier, obtaining media content corresponding to the second sentence identifier based on the associated resource identifier; and adding the obtained media content to a position adjacent to the second sentence identifier to generate a response.

[0092] In some embodiments, process 500 also includes: dividing an original document including a target document fragment into multiple document fragments, the target document fragment being one of the multiple document fragments; removing media content included in the multiple document fragments to obtain updated multiple document fragments; and storing corresponding vectorized representations of the updated multiple document fragments.

[0093] In some embodiments, process 500 further includes: determining corresponding location information and corresponding resource identifiers of one or more media contents in the target document segment; and storing the corresponding location information and corresponding resource identifiers of the one or more media contents in metadata corresponding to the target document segment.

[0094] According to the scheme disclosed in the present invention, the error rate of the model referencing media content is reduced to, for example, 2.5%, and the end-to-end accuracy is increased from, for example, 20.6% to, for example, 92.5%. The scheme disclosed in the present invention no longer outputs the wrong media content URL in a predetermined format (such as markdown). The scheme disclosed in the present invention significantly improves the recall rate, and the recall rate of the fragments related to the user input reaches 95%. The scheme disclosed in the present invention solves the current problems related to the recall of media content in intelligent question and answering without introducing a large multimodal model, and increases the probability of the application generating higher quality replies to user input through the knowledge base.

[0095] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 6 An exemplary structural block diagram of an interactive apparatus 600 according to some embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in a client device 110 and / or a server 120. Each module / component in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0096] like Figure 6 As shown, the device 600 includes a document determination module 610, which is configured to determine at least one document segment associated with the user input based on the user input, and the target document segment in the at least one document segment includes one or more media contents. The device 600 includes a prompt information generation module 620, which is configured to generate prompt information based on the text in the at least one document segment and the user input, and the prompt information includes one or more identifiers, and the identifiers in the one or more identifiers correspond to the media content in the one or more media contents and are related to the position of the corresponding media content in the target document segment. The device 600 includes a reply module 630, which is configured to generate a reply to the user input based on the prompt information using a machine learning model, wherein in the case where the reply includes the content of the target text segment, at least one of the one or more media contents is associated with at least part of the text of the target text segment and displayed in the reply.

[0097] In some embodiments, the reply module 630 is also configured to: provide prompt information to the machine learning model to obtain the output of the machine learning model; in response to determining that the output includes a first identifier among one or more identifiers, obtain media content corresponding to the first identifier; and add the obtained media content to the reply.

[0098] In some embodiments, an identifier among the one or more identifiers includes an identifier in a predetermined format, and the identifier in the predetermined format includes at least one of the following: information indicating the type of the corresponding media content, the sequence number of the target document fragment in at least one document fragment, and the sequence number of the corresponding media content in one or more media contents.

[0099] In some embodiments, the prompt information generation module 620 is also configured to: obtain corresponding position information of one or more media contents from the metadata of the target document fragment; based on the corresponding position information of one or more media contents, add corresponding identifiers of a predetermined format of one or more media contents to the text of the target document fragment to generate context information; and determine the prompt information based on the context information and user input.

[0100] In some embodiments, the reply module 630 is further configured to: in response to the model output provided by the machine learning model based on the prompt information including a second identifier in a predetermined format, obtain media content corresponding to the second identifier, replace the second identifier in the model output with the obtained media content to update the model output, and determine a reply based on the updated model output.

[0101] In some embodiments, the reply module 630 is further configured to: determine a resource identifier from metadata of the target document segment based on the second identifier; and obtain media content corresponding to the second identifier using the determined resource identifier.

[0102] In some embodiments, the identifier of the one or more identifiers includes a sentence identifier that identifies a sentence in the target document segment.

[0103] In some embodiments, the prompt information generation module 620 is also configured to: divide at least one document fragment into multiple sentences according to sentence granularity; generate a sentence identifier for each of the multiple sentences based on the order of at least one document fragment and the order of the multiple sentences; determine the prompt information based on the sentence identifiers generated for the multiple sentences, the text in at least one document fragment and the user input.

[0104] In some embodiments, the prompt information generation module 620 is also configured to: for a given media content among one or more media contents, obtain the location information and resource identifier of the given media content from the metadata of the target document fragment; based on the location information of the given media content, determine the first sentence adjacent to the given media content in the target document fragment; and associate the resource identifier of the given media content with the first sentence identifier of the determined first sentence.

[0105] In some embodiments, the reply module 630 is also configured to: provide prompt information to the machine learning model to obtain a model output of the machine learning model; in response to determining that the second sentence identifier in the model output is associated with a resource identifier, obtain media content corresponding to the second sentence identifier based on the associated resource identifier; and add the obtained media content to a position adjacent to the second sentence identifier to generate a reply.

[0106] In some embodiments, the device 600 also includes an index creation module configured to: divide an original document including a target document fragment into multiple document fragments, the target document fragment being one of the multiple document fragments; remove media content included in the multiple document fragments to obtain updated multiple document fragments; and store corresponding vectorized representations of the updated multiple document fragments.

[0107] In some embodiments, the index creation module is further configured to: determine corresponding location information and corresponding resource identifiers of one or more media contents in the target document segment; and store the corresponding location information and corresponding resource identifiers of one or more media contents in the metadata corresponding to the target document segment.

[0108] The units and / or modules included in the device 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units and / or modules in the device 600 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0109] It should be understood that one or more steps in the above method can be performed by a suitable electronic device or a combination of electronic devices. Such an electronic device or a combination of electronic devices may include, for example, a computer that runs Figure 1 The system manages the devices of the platform 110.

[0110] Figure 7 1 shows a block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 7 The electronic device 700 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 7 The electronic device 700 shown may include or be implemented as Figure 1 System management platform 110 or Figure 6 Device 600.

[0111] like Figure 7 As shown, the electronic device 700 is in the form of a general electronic device. The components of the electronic device 700 may include, but are not limited to, one or more processors or processing units 710, a memory 720, a storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processing unit 710 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 720. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 700.

[0112] The electronic device 700 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 730 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 700.

[0113] The electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 7 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0114] The communication unit 740 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 700 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 700 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0115] The input device 750 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 760 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 700 may also communicate with one or more external devices (not shown) through the communication unit 740 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 700, or communicate with any device that allows the electronic device 700 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0116] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0117] According to an exemplary implementation of the present disclosure, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 5 The methods provided in the various optional methods are therefore not described in detail here.

[0118] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0119] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0120] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0121] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0122] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. An information processing method, comprising: Determining at least one document segment associated with the user input based on the user input, wherein a target document segment in the at least one document segment includes one or more media contents; Generate prompt information based on the text in the at least one document fragment and the user input, the prompt information comprising one or more identifiers, an identifier in the one or more identifiers corresponding to a media content in the one or more media contents and related to a position of the corresponding media content in the target document fragment; as well as Based on the prompt information, a response to the user input is generated using a machine learning model, wherein, when the response includes the content of the target text segment, at least one of the one or more media contents and at least part of the text of the target text segment are associated and displayed in the response.

2. The method of claim 1 , wherein generating a response to the user input comprises: Providing the prompt information to the machine learning model to obtain an output of the machine learning model; In response to determining that the output includes a first identifier of the one or more identifiers, obtaining media content corresponding to the first identifier; as well as The retrieved media content is added to the reply.

3. The method according to claim 1, wherein the identifier of the one or more identifiers comprises an identifier in a predetermined format, and the identifier in the predetermined format comprises at least one of the following: information indicating the type of the corresponding media content, the sequence number of the target document segment in the at least one document segment, or The sequence number of the corresponding media content in the one or more media contents.

4. The method according to claim 3, wherein generating the prompt information comprises: Acquire corresponding location information of the one or more media contents from metadata of the target document segment; Based on the corresponding position information of the one or more media contents, adding the corresponding identifiers of the one or more media contents in a predetermined format to the text of the target document fragment to generate context information; as well as The prompt information is determined based on the context information and the user input.

5. The method of claim 3, wherein generating a reply to the user input comprises: In response to a model output provided by the machine learning model based on the prompt information including a second identifier in the predetermined format, acquiring media content corresponding to the second identifier; replacing the second identifier in the model output with the acquired media content to update the model output; as well as The response is determined based on the updated model output.

6. The method according to claim 5, wherein obtaining the media content corresponding to the second identifier comprises: Based on the second identifier, determining a resource identifier from metadata of the target document fragment; as well as Using the determined resource identifier, media content corresponding to the second identifier is acquired. 7 . The method of claim 1 , wherein an identifier of the one or more identifiers comprises a sentence identifier that identifies a sentence in the target document segment.

8. The method according to claim 7, wherein generating the prompt information comprises: Dividing the at least one document segment into a plurality of sentences according to sentence granularity; generating the sentence identifier for each sentence in the plurality of sentences based on an order of the at least one document fragment and an order of the plurality of sentences; as well as The prompt information is determined based on the sentence identifiers generated respectively for the plurality of sentences, text in the at least one document segment, and the user input.

9. The method according to claim 8, further comprising: For a given media content among the one or more media content, Acquire location information and a resource identifier of the given media content from metadata of the target document fragment; Based on the position information of the given media content, determining a first sentence in the target document segment that is adjacent to the given media content; as well as A resource identifier of the given media content is associated with a first sentence identifier of the determined first sentence.

10. The method of claim 7, wherein generating a response to the user input comprises: Providing the prompt information to the machine learning model to obtain a model output of the machine learning model; In response to determining that a second sentence identifier in the model output is associated with a resource identifier, acquiring media content corresponding to the second sentence identifier based on the associated resource identifier; as well as The acquired media content is added to a position adjacent to the second sentence identifier to generate the reply.

11. The method according to claim 1, further comprising: Dividing an original document including the target document segment into a plurality of document segments, wherein the target document segment is one of the plurality of document segments; removing the media contents included in the plurality of document pieces to obtain a plurality of updated document segments; and The corresponding vectorized representations of the updated plurality of document segments are stored.

12. The method according to claim 11, further comprising: Determine corresponding location information and corresponding resource identifiers of the one or more media contents in the target document segment; as well as The corresponding location information and the corresponding resource identifier of the one or more media contents are stored in the metadata corresponding to the target document segment.

13. An information processing device, comprising: A document determination module, configured to determine, based on a user input, at least one document segment associated with the user input, wherein a target document segment in the at least one document segment includes one or more media contents; a prompt information generating module configured to generate prompt information based on the text in the at least one document segment and the user input, the prompt information comprising one or more identifiers, an identifier in the one or more identifiers corresponding to a media content in the one or more media contents and related to a position of the corresponding media content in the target document segment; as well as The reply module is configured to generate a reply to the user input based on the prompt information and using a machine learning model, wherein when the reply includes the content of the target text segment, at least one of the one or more media contents and at least part of the text of the target text segment are associated and displayed in the reply.

14. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 12 when executed by the at least one processing unit.

15. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 12.

16. A computer program product, comprising a computer program, wherein the computer program implements the method according to any one of claims 1 to 12 when executed by a processor.

Images