Session generation method, apparatus and device, and readable storage medium

Abstract

The invention discloses a session generation method and device, equipment and a readable storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: obtaining previous text information of a session process; the intention of the previous information is recognized, an intention recognition result is obtained, and the intention recognition result is used for indicating the generation relation between the data modality corresponding to the previous information and the data modality corresponding to the following information of the previous information; selecting an information generation model corresponding to the generation relation indicated by the intention recognition result from a plurality of candidate generation relation models; and generating the following text information according to the previous text information through an information generation model. Through the method, the generation relation between the data modals can be identified in a fine-grained manner, so that accurate following information can be generated through the model corresponding to the generation relation, and the session quality is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a session generation method, apparatus, device, and readable storage medium. Background Technology

[0002] With the development of artificial intelligence technology, its research results have been applied to various industries. Among them, intelligent customer service is an important application scenario. Intelligent customer service is a scenario in which intelligent robots replace humans in conducting conversations. Through intelligent customer service, labor costs can be reduced and response speed can be improved.

[0003] In related technologies, intelligent customer service systems need to obtain the preceding context information during a conversation, and then use a trained network model to generate the following context information based on that context. However, the preceding context information includes multiple conversation data points, and the conversation data contains various modalities such as text, images, audio, and video, which makes the accuracy of the following context information generated by the network model poor, affecting the quality of the conversation. Summary of the Invention

[0004] This application provides a session generation method, apparatus, device, and readable storage medium, which can be used to accurately generate the following information and improve session quality. The technical solution includes the following contents.

[0005] On the one hand, a session generation method is provided, the method comprising:

[0006] Retrieve context information from the previous part of the conversation;

[0007] The intent of the preceding information is identified to obtain an intent identification result, which is used to indicate the generation relationship between the data modality corresponding to the preceding information and the data modality corresponding to the following information of the preceding information;

[0008] From multiple candidate generation relationship models, select the information generation model corresponding to the generation relationship indicated by the intent recognition result;

[0009] The aforementioned information is generated based on the aforementioned preceding information using the information generation model.

[0010] On the other hand, a session generation apparatus is provided, the apparatus comprising:

[0011] The acquisition module is used to retrieve the context information of the conversation process;

[0012] An intent recognition module is used to recognize the intent of the preceding information and obtain an intent recognition result. The intent recognition result is used to indicate the generation relationship between the data modality corresponding to the preceding information and the data modality corresponding to the following information of the preceding information.

[0013] The selection module is used to select the information generation model corresponding to the generation relationship indicated by the intent recognition result from multiple candidate generation relationship models;

[0014] The generation module is used to generate the following information based on the preceding information using the information generation model.

[0015] In one possible implementation, the data modality corresponding to the above information includes a text modality and a non-text modality;

[0016] The intent recognition module is used to segment the preceding information to obtain first session data corresponding to the text modality and second session data corresponding to the non-text modality; add first prompt information to the first session data to obtain third session data, wherein the first prompt information is used to indicate at least one of the following: the non-text modality corresponding to the second session data, the corresponding acquisition method, the session object to which it belongs, and the knowledge domain involved; and perform intent recognition on the second session data and the third session data to obtain intent recognition results.

[0017] In one possible implementation, the intent recognition module is configured to add the first prompt information at an addition location in the first session data to obtain third session data, wherein the addition location is determined based on at least one of the following:

[0018] The positional relationship between the second session data and the first session data mentioned above; the session object of the second session data and the dialogue object of the first session data.

[0019] In one possible implementation, the intent recognition module is configured to extract first data features of the third session data through a text feature extraction network; extract second data features of the second session data through the non-text modality feature extraction network; and determine the intent recognition result based on the first and second data features through the intent recognition network.

[0020] In one possible implementation, the text feature extraction network includes a first attention layer and a first feedforward layer;

[0021] The intent recognition module is configured to perform attention processing on the third session data through the first attention layer to obtain a first attention processing result; perform at least one of linear mapping or nonlinear mapping on the first attention processing result through the first feedforward layer to obtain a first mapping processing result; and determine the first data feature based on the first mapping processing result.

[0022] In one possible implementation, the non-textual modality feature extraction network includes an encoding layer, a second attention layer, and a second feedforward layer;

[0023] The intent recognition module is used to divide the second session data into multiple data parts; encode each data part through the encoding layer to obtain the encoding result of each data part; concatenate the encoding results of each data part to obtain a concatenated result; perform attention processing on the concatenated result through the second attention layer to obtain a second attention processing result; perform at least one of linear mapping or nonlinear mapping on the second attention processing result through the second feedforward layer to obtain a second mapping processing result; and determine the second data feature based on the second mapping processing result.

[0024] In one possible implementation, the intent recognition module is configured to acquire second prompt information, which instructs the execution of a task to identify the generation relationship between the data modality corresponding to the preceding information and the data modality corresponding to the following information; and to perform intent recognition on the preceding information based on the second prompt information to obtain an intent recognition result.

[0025] In one possible implementation, the information generation model includes a classification model and a generation model for multiple candidate knowledge domains;

[0026] The generation module is configured to classify the preceding information using the classification model to obtain a classification result, wherein the classification result indicates the knowledge domain involved in the preceding information; select the generation model corresponding to the knowledge domain indicated by the classification result from the multiple candidate knowledge domain generation models; and generate the following information based on the preceding information using the generation model corresponding to the knowledge domain indicated by the classification result.

[0027] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement any of the session generation methods described above.

[0028] On the other hand, a computer-readable storage medium is also provided, wherein at least one computer program is stored therein, the at least one computer program being loaded and executed by a processor to enable an electronic device to implement any of the session generation methods described above.

[0029] On the other hand, a computer program is also provided, wherein the computer program is at least one, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-described session generation methods.

[0030] On the other hand, a computer program product is also provided, which stores at least one computer program, which is loaded and executed by a processor to enable an electronic device to implement any of the above-described session generation methods.

[0031] The technical solution provided in this application brings at least the following beneficial effects:

[0032] The technical solution provided in this application achieves fine-grained identification of the generative relationship between the data modalities of the preceding and following information by performing intent recognition on the preceding information in the conversation process. Subsequently, the preceding information can be accurately understood and the following information can be output accurately through the information generation model corresponding to this generative relationship, thereby improving the semantic relevance between the preceding and following information and thus improving the quality of the conversation. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of a computer system provided in an embodiment of this application;

[0035] Figure 2 This is a schematic diagram of a session generation method provided in an embodiment of this application;

[0036] Figure 3 This is a flowchart of a session generation method provided in an embodiment of this application;

[0037] Figure 4 This is a schematic diagram illustrating an intent recognition method provided in an embodiment of this application;

[0038] Figure 5 This is a schematic diagram of the structure of a Transformer encoding module provided in an embodiment of this application;

[0039] Figure 6 This is a schematic diagram of the structure of an image feature extraction network provided in an embodiment of this application;

[0040] Figure 7 This is a schematic diagram of another Transformer encoding module provided in an embodiment of this application;

[0041] Figure 8 This is a flowchart illustrating another session generation method provided in an embodiment of this application;

[0042] Figure 9 This is a schematic diagram illustrating the generation of the following information provided in an embodiment of this application;

[0043] Figure 10 This is a schematic diagram of the structure of a session generation device provided in an embodiment of this application;

[0044] Figure 11 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0045] Figure 12 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0047] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0048] First, the abbreviations and key terms involved in the embodiments of this application are defined.

[0049] Conversation: In intelligent customer service scenarios, a conversation refers to the process from initiating to ending a conversation. Generally, at least two objects participate in a conversation, and these objects can be either users or intelligent chatbots. At least one round of dialogue data is generated during the conversation, which can include questions and answers. Typically, one question corresponds to at least one answer. The object asking the question and the object providing the answer are different objects, but the objects providing different answers can be the same object or different objects.

[0050] Data modality: In data processing and analysis, data modality refers to the form in which data is presented. Common data modalities include text modality, audio modality, image modality, and video modality.

[0051] Intent recognition: This aims to identify the intent expressed by the input information. For example, if the input information is "I want to book a flight from city A to city B", then the intent expressed by the input information is to book a flight.

[0052] In intelligent customer service scenarios, it is necessary to first obtain the preceding context of the conversation, then generate the following context based on the preceding context using a trained network model, and finally display the following context. However, the preceding context includes multiple conversation data points, and the conversation data contains various modalities such as text, images, audio, and video, which makes the accuracy of the following context generated by the network model poor, affecting the quality of the conversation.

[0053] In response, this application provides a session generation method that can identify the generation relationship between the data modal of the preceding information and the data modal of the following information. By using the information generation model corresponding to the generation relationship, the following information is generated based on the preceding information, thereby improving the accuracy of the following information and thus improving the session quality.

[0054] like Figure 1 As shown, Figure 1 This is a schematic diagram of a computer system for a session generation method provided in this application embodiment. The computer system includes a terminal device 101 and a server 102. The terminal device 101 has a client installed and running, and the server 102 provides background services to the client. The client includes, but is not limited to, game clients, instant messaging clients, and shopping clients. An object 103 can use the terminal device 101 to realize games, instant messaging, shopping, etc., through data transmission between the client and the server 102. The session generation method provided in this application embodiment can be executed by the terminal device 101, by the server 102, or by both the terminal device 101 and the server 102; this application embodiment does not limit this.

[0055] In one possible implementation, server 102 undertakes the primary computational work, while terminal device 101 undertakes the secondary computational work. Alternatively, server 102 undertakes the secondary computational work, while terminal device 101 undertakes the primary computational work. Or, terminal device 101 and server 102 collaborate on computation using a distributed computing architecture.

[0056] Optionally, the terminal device 101 can be any electronic device product capable of human-computer interaction with the user through one or more methods such as a keyboard, touchpad, remote control, voice interaction, or handwriting device. For example, the terminal device 101 can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, PC (Personal Computer), mobile phone, PDA (Personal Digital Assistant), wearable device, PPC (Pocket PC), smart car system, smart TV, etc.

[0057] Terminal device 101 can refer to one of a plurality of terminal devices. This embodiment uses terminal device 101 as an example. Those skilled in the art will know that the number of terminal devices 101 can be more or less. For example, there may be only one terminal device 101, or there may be dozens or hundreds of terminal devices 101, or more. This application embodiment does not limit the number or type of terminal devices 101.

[0058] Server 102 can be a single server, a server cluster consisting of multiple servers, or any of the following: a cloud computing platform or a virtualization center. This embodiment of the application does not limit this. Server 102 communicates directly or indirectly with terminal device 101 via a wired or wireless network. Server 102 has data receiving, data processing, and data sending functions. Of course, server 102 may also have other functions, which are not limited in this embodiment of the application.

[0059] Those skilled in the art should understand that the terminal device 101 and server 102 described above are merely illustrative examples. Other existing or future terminal devices or servers that are applicable to this application should also be included within the scope of protection of this application, and are hereby incorporated by reference.

[0060] In an exemplary embodiment, the session generation method of this application embodiment is executed through terminal device 101 to implement a session. For example... Figure 2 As shown, the session process includes the following steps.

[0061] Step 1: Preprocess the preceding information. During the conversation between conversation objects 201 and 202, terminal device 101 can display the conversation data of the two conversation objects in real time through display page 200. Optionally, conversation object 201 first sends the conversation data "Describe this picture" and uploads the picture. Then, conversation object 202 replies with the conversation data "This is a cartoon image of a little boy. The little boy is wearing overalls, his face is full of smiles, and he is enjoying the fun of running." After that, conversation object 201 sends the conversation data "Change the character to a little girl." In this case, terminal device 101 can obtain the preceding information 203 displayed on display page 200 and perform preprocessing on the preceding information 203 to obtain the processed preceding information. The processed preceding information includes text data 204-1 and image data 204-2. The text data 204-1 includes the prompt message "{Upload Image}", which indicates the modality and acquisition method of the image data 204-2. This prompt message can reflect that the image data 204-2 and the session data "Describe this image" correspond to the same session object, and that the image data 204-2 and the session data "Describe this image" together constitute the session object.

[0062] Step 2, Intent Recognition. Terminal device 101 can acquire the trained intent recognition model 205. Text data 204-1 and image data 204-2 are input into the intent recognition model 205, which performs intent recognition and outputs an intent recognition result 206. The intent recognition result 206 indicates any one of several candidate generation relationships, such as text-to-text 206-1, text-to-image 206-2, image-to-text 206-3, and image-to-image 206-4.

[0063] Step 3: Model Selection. Each candidate generation relation corresponds to a network model. For example, text-to-text 206-1 corresponds to network model 207-1 for text-to-text generation, text-to-image 206-2 corresponds to network model 207-2 for text-to-image generation, image-to-text 206-3 corresponds to network model 207-3 for image-to-text generation, and image-to-image 206-4 corresponds to network model 207-4 for image-to-image generation. The terminal device 101 selects the network model corresponding to the generation relation indicated by the intent recognition result 206 from the network models corresponding to each candidate generation relation. For example, if the intent recognition result 206 indicates image-to-image 206-4, then the terminal device 101 selects network model 207-4 for image-to-image generation.

[0064] Step 4: Generate the following information. Terminal device 101 inputs text data 204-1 and image data 204-2 into the selected network model. For example, it inputs text data 204-1 and image data 204-2 into a network model 207-4 that generates images based on images, and the network model generates the following information 208. Afterwards, terminal device 101 can display the following information 208 on display page 200.

[0065] This application provides a session generation method, which can be applied to the aforementioned computer system and executed by terminal device 101 or server 102. For ease of description, terminal device 101 or server 102 is collectively referred to as an electronic device; that is, the session generation method of this application is executed by an electronic device. Figure 3 As shown, the method includes the following steps.

[0066] Step 301: Obtain the context information of the conversation.

[0067] A conversation is a process in which at least two conversation objects participate in a conversation, during which each conversation object can send at least one conversation data. Each conversation data corresponds to a data modality, which includes, but is not limited to, text, images, audio, and video. Optionally, a conversation data can be a question, an answer, or data other than questions and answers. For example, conversation data can be "Good morning," "Nice to meet you," or other data that is not a question or answer.

[0068] In this example, the electronic device can receive session data input by the user during a session and determine the preceding context based on that session data. Optionally, the preceding context may also include session data preceding the initial session data. Alternatively, the electronic device can receive preceding context sent by other devices. The preceding context includes at least one session data, and any two session data can correspond to the same or different data modalities; that is, the preceding context corresponds to at least one data modality. Figure 2 As shown in the above information 203, the above information 203 includes session data of three text modalities and session data of one image modality.

[0069] Step 302: Identify the intent of the preceding information and obtain the intent identification result. The intent identification result is used to indicate the generation relationship between the data modality corresponding to the preceding information and the data modality corresponding to the following information.

[0070] Electronic devices can acquire a trained intent recognition model, perform intent recognition on the preceding information using the intent recognition model, obtain the intent recognition result, and use the intent recognition result to indicate the generation relationship between the data modal of the preceding information and the data modal of the following information. In this example, the preceding information corresponds to at least one data modal, and the following information corresponds to at least one data modal. For example, the preceding information corresponds to both text and image modalities, and the following information corresponds to only one text modality.

[0071] Assuming there are N data modalities, there are at least N×N combinations of generation relationships between data modalities, where N is a positive integer. Generally, data modalities include text, image, audio, and video modalities, resulting in 16 possible generation relationships between them.

[0072] Taking data modalities including text and image modalities as an example, the generative relations between data modalities include the following four combinations: text-to-text (text-to-text), text-to-image (text-to-image), image-to-text (image-to-text), and image-to-image (image-to-image). It should be noted that multi-turn dialogues may occur during a session. In multi-turn dialogue scenarios, each turn involves at least one generative relation, and adjacent turns may involve different generative relations (i.e., cross-modal generative relations). Taking the above four generative relations as examples, there are at least 4 × 4 = 16 types of cross-modal generative relations. Table 1 below shows several cross-modal generative relations that may occur during a session.

[0073] Table 1

[0074]

[0075] In Table 1, Q1 represents the question in the previous round of dialogue, and Q2 represents the question in the next round of dialogue. In "AB", "A" represents the generative relation involved in the previous round of dialogue, and "B" represents the generative relation involved in the next round of dialogue. For example, in the first example in Table 1, the dialogue "Q1: How's the weather today?" involves text-to-text, and the dialogue "Q2: (User uploaded image) Describe the weather in this photo" involves image-to-text. Explanations of the other examples in Table 1 can be found in the explanation of the first example, and will not be repeated here.

[0076] In this example, there are multiple generative relationships between data modalities. For ease of description, these relationships are referred to as candidate generative relationships, meaning there are multiple candidate generative relationships. Optionally, the intent recognition result includes multiple sub-results, each corresponding to a candidate generative relationship. Each sub-result represents the probability that the generative relationship between the preceding and following information belongs to a corresponding candidate generative relationship; the larger the sub-result, the higher the probability. Optionally, the sub-result is a probability value between 0 and 1, representing the probability that the generative relationship between the preceding and following information belongs to a corresponding candidate generative relationship. When the sub-result is not less than a reference value, the generative relationship between the preceding and following information can be considered to belong to a corresponding candidate generative relationship. Based on this, the intent recognition result can represent that the generative relationship between the preceding and following information belongs to at least one candidate generative relationship. This example does not limit the reference value. For example, the reference value is a value set based on human experience, or it is one of the multiple sub-results, such as the sub-result with the largest value among the multiple sub-results.

[0077] In one possible implementation, the data modality corresponding to the above information includes a text modality and a non-text modality. Step 302 includes steps 3021 to 3023 (not shown in the figure).

[0078] Step 3021: Divide the preceding information to obtain the first session data corresponding to the text modality and the second session data corresponding to the non-text modality.

[0079] Non-text modalities include at least one data modality, such as at least one of image modalities, audio modalities, and video modalities. The preceding information includes at least one session data, with each session data corresponding to a data modality. For each session data, if the session data corresponds to a text modality, then the session data is classified as first session data; if the session data corresponds to a data modality other than text, such as an image modality, then the session data is classified as second session data. There are at least one first session data and at least one second session data, and any two second session data can correspond to the same data modality or different data modalities.

[0080] Step 3022: Add a first prompt message to the first session data to obtain the third session data. The first prompt message is used to indicate at least one of the following: the non-text modality corresponding to the second session data, the corresponding acquisition method, the session object to which it belongs, and the knowledge domain involved.

[0081] There is at least one second session data, and each second session data corresponds to a first prompt message. The first prompt message can indicate the non-text modality corresponding to the second session data. For example, one first prompt message can indicate an image modality, and another first prompt message can indicate an audio modality. The first prompt message can also indicate the acquisition method of the second session data. For example, if the second session data is an image uploaded by a user, the first prompt can indicate that the acquisition method of the second session data is uploading. Or, if the second session data is an image generated by an image generation model, the first prompt can indicate that the acquisition method of the second session data is generation. The first prompt message can also indicate the session object to which the second session data belongs. For example, when a user and an intelligent robot participate in a session, if the user sends the second session data, the first prompt message can indicate the session object "user"; if the intelligent robot sends the second session data, the first prompt message can indicate the session object "intelligent robot". In addition, the first prompt message can also indicate the knowledge domain involved in the second session data, which includes, but is not limited to, calendar, weather, code, travel, and other fields. It should be noted that the first prompt message can also indicate other information such as the data volume and data format of the second session data, which is not limited in this embodiment.

[0082] In practical applications, the first prompt can indicate at least one of the information exemplified above, such as non-text modality, acquisition method, session object, or knowledge domain. For example, the first prompt could be: {Upload Image}, indicating the acquisition method as "upload" and the non-text modality as "image".

[0083] This application does not limit the position where the first prompt information is added to the first session data. For example, the first prompt information can be added after or before the first session data to obtain the third session data. By inserting first prompt information indicating the data modality, acquisition method, session object, knowledge domain, etc., of the second session data into the first session data of the text modality, the first session data and the second session data are associated. This allows for a better understanding of the first and second session data during subsequent intent recognition, improving the accuracy of intent recognition results, thereby improving the accuracy of subsequent information and enhancing session quality.

[0084] In an exemplary embodiment, step 3022 includes: adding a first prompt message at the addition position in the first session data to obtain third session data, wherein the addition position is determined based on at least one of the following: the positional relationship between the second session data and the first session data in the preceding information; the session object of the second session data and the dialogue object of the first session data.

[0085] Since the above information consists of first session data and second session data, it can reflect the positional relationship between the first session data and the second session data. Therefore, based on this positional relationship, a corresponding addition position can be determined in the first session data, and the first prompt information corresponding to the second session data can be added at that addition position to obtain the third session data. This ensures that the positional relationship between the first session data and the second session data reflected in the third session data is the same as the positional relationship between the first session data and the second session data reflected in the above information.

[0086] For example, the above information includes three session data, namely, text-modal session data 1 (belonging to the first session data), image-modal session data 2 (belonging to the second session data), and text-modal session data 3 (belonging to the first session data). Then, based on the positional relationship between the second session data and the first session data in the above information, it can be determined that the first prompt information corresponding to session data 2 is added after session data 1 or before session data 3, so that the positional relationship remains unchanged.

[0087] The above information includes session data sent by at least one session object, and each session object can send at least one session data. That is, the first session data corresponds to at least one session object, and the second session data also corresponds to at least one session object. The session objects corresponding to the first session data and the second session data can be the same or different. Based on this, according to the session objects of the first and second session data, a corresponding addition position can be determined in the first session data. At this addition position, the first prompt information corresponding to the second session data is added to obtain the third session data, ensuring that the session data sent by each session object reflected in the third session data is the same as the session data sent by each session object reflected in the above information.

[0088] For example, the above information includes three session data, namely session data 1 sent by session object 1 (belonging to the first session data), session data 2 sent by session object 2 (belonging to the second session data), and session data 3 sent by session object 2 (belonging to the first session data). Then, based on the session object of the first session data and the session object of the second session data, it can be determined that the first prompt information corresponding to session data 2 is added before or after session data 3, so that the relationship between the session object and the session data remains unchanged.

[0089] Optionally, based on the positional relationship between the second session data and the first session data in the above information, the session object of the second session data and the dialogue object of the first session data, an addition position is determined in the first session data, such that the third session data obtained after adding the first prompt information at the addition position reflects the same positional relationship between the first session data and the second session data as reflected in the above information, and the session data sent by each session object reflected in the third session data is the same as the session data sent by each session object as reflected in the above information.

[0090] For example, the above information includes three conversation data: conversation data 1 in text mode sent by conversation object 1 (belonging to the first conversation data), conversation data 2 in image mode sent by conversation object 2 (belonging to the second conversation data), and conversation data 3 in text mode sent by conversation object 2 (belonging to the first conversation data). Then, based on the positional relationship between the second conversation data and the first conversation data in the above information, the conversation object of the first conversation data and the conversation object of the second conversation data, it can be determined that the first prompt information corresponding to conversation data 2 is added before conversation data 3, so that the positional relationship remains unchanged and the relationship between the conversation object and the conversation data remains unchanged.

[0091] Based on the positional relationship between the second session data and the first session data in the above information, and / or the session object of the second session data and the dialogue object of the first session data, the addition position is determined in the first session data, and the first prompt information is added at the addition position, so that the relationship between the first session data and the second session data remains unchanged, so that the first session data and the second session data can be accurately understood during subsequent intent recognition, thereby improving the accuracy of intent recognition results.

[0092] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating an intent recognition method provided in an embodiment of this application. Wherein, Figure 4 (A) shows the third session data 401-1 and the second session data 404-1. The third session data 401-1 includes the first prompt message "{Upload Image}". This first prompt message indicates that the second session data 404-1 is obtained by uploading and that the data modality is an image. Furthermore, the insertion position of this first prompt message reflects the positional relationship and the relationship between the text modality session data and the image modality session data (i.e., the second session data 404-1).

[0093] Similarly, Figure 4 (B) shows the third session data 401-2 and the second session data 404-2. The third session data 401-2 includes the first prompt message "{Generate Image}". This first prompt message indicates that the second session data 404-2 was obtained by generation, that the data modality is an image, and that the insertion position of the first prompt message reflects the positional relationship and the relationship between the text modality session data and the image modality session data (i.e., the second session data 404-2).

[0094] Figure 4 (C) shows the third session data 401-3 and the second session data 404-3. The third session data 401-3 includes the first prompt message "{Generate Image}". This first prompt message indicates that the second session data 404-3 was obtained by generation, that the data modality is an image, and that the insertion position of the first prompt message reflects the positional relationship and the relationship between the text modality session data and the image modality session data (i.e., the second session data 404-3).

[0095] Step 3023: Perform intent recognition on the second session data and the third session data to obtain intent recognition results.

[0096] In this embodiment, the second and third session data can be input into the intent recognition model, and intent recognition can be performed through the intent recognition model to obtain the intent recognition result. The structure, type, and network layers included in the intent recognition model are not limited here. For example, the intent recognition model may include at least one of the following: convolutional layer, attention layer, normalization layer, activation layer, and feedforward layer. It is understood that different intent recognition models will result in different intent recognition processes, which will not be elaborated upon here.

[0097] Optionally, step 3023 includes steps A1 to A3 (not shown in the figure).

[0098] Step A1: Extract the first data features of the third session data using a text feature extraction network.

[0099] In this embodiment, the intent recognition model includes a text feature extraction network. Third session data is input into the text feature extraction network, which then performs text feature extraction on the third session data to obtain first data features. The first data features are text features used to characterize the third session data and can exist in matrix form. The structure, type, and network layers included in the text feature extraction network are not limited here. Optionally, the text feature extraction network includes at least one of the following: convolutional layers, attention layers, normalization layers, activation layers, and feedforward layers. Different structures of text feature extraction networks result in different text feature extraction methods.

[0100] For example, the text feature extraction network includes a first attention layer and a first feedforward layer. Step A1 includes: performing attention processing on the third session data through the first attention layer to obtain a first attention processing result; performing at least one of linear mapping or nonlinear mapping on the first attention processing result through the first feedforward layer to obtain a first mapping processing result; and determining a first data feature based on the first mapping processing result.

[0101] Text feature extraction networks are networks used to extract text features. Typically, a text feature extraction network includes an input layer that receives third-party conversation data. This third-party conversation data consists of multiple words, and the input layer maps each word in the data into a word vector. Based on these word vectors, a text embedding is determined.

[0102] In this example, the text feature extraction network also includes a first attention layer concatenated after the input layer. This first attention layer performs attention processing on the text embedding based on an attention mechanism, yielding the first attention processing result. The attention mechanism is a method for processing input data in a neural network, enabling the model to automatically learn to focus on important parts of the input sequence, thereby improving model performance.

[0103] Optionally, the first attention layer is a self-attention layer. The self-attention layer can perform weighted processing on the text embedding using different weight matrices, obtaining a query (Q) matrix, a key (K) matrix, and a value (V) matrix. This weighted processing achieves matrix scaling of the text embedding. The query matrix represents the parts that need attention. The key matrix is ​​used to match the query matrix to determine the relevance between them. The value matrix is ​​used to generate the self-attention processing result based on the relevance between the key and query matrices. In other words, the self-attention layer can determine the relevance based on the query and key matrices, and determine the self-attention processing result based on the relevance and the value matrix. In this example, the self-attention processing result can be the result of the first attention processing.

[0104] Optionally, according to Determine the self-attention processing result. Here, Attention(Q,K,V) represents the self-attention processing result. Q represents the query matrix, K represents the key matrix, and V represents the value matrix. T is the transpose symbol, and d... k The dimension of the bond matrix is ​​represented by softmax, which is a normalized exponential function. The correlation between the key matrix and the query matrix is ​​obtained by calculating the dot product.

[0105] Optionally, the first attention layer is a multi-head attention layer. Each head attention layer can perform attention processing on the text embedding according to the principle of self-attention layers, obtaining the attention processing result of that head. The attention processing results of each head are weighted and summed to obtain the multi-head attention processing result. In this example, the multi-head attention processing result can be the first attention processing result.

[0106] Optionally, the first attention layer is a masked attention layer. Since the text embedding includes word vectors for multiple words, the masked attention layer can perform attention processing on the word vector of the first word, following the principle of self-attention layers, to obtain the attention processing result for the first word. Furthermore, the masked attention layer can perform attention processing on the word vectors of words other than the first word, as well as the word vectors of at least one word preceding it, following the principle of self-attention layers, to obtain the attention processing result for that word. The attention processing results for each word are then weighted and summed to obtain the masked attention processing result. In this example, the masked attention processing result can be the first attention processing result.

[0107] In this example, the text feature extraction network also includes a first feedforward layer concatenated after the first attention layer. A feedforward layer, also known as a fully connected layer, has a basic structure that includes at least one of a linear transformation and an activation function. The linear transformation is a linear mapping, and the activation function performs a non-linear mapping. In other words, the first feedforward layer can perform at least one of a linear or non-linear mapping on the result of the first attention processing to obtain the first mapping result.

[0108] Optionally, the result of the first mapping process is used as the first data feature. Alternatively, the text feature extraction network may also include other network layers, such as a normalization layer. The first data feature is determined by these other network layers based on the result of the first mapping process.

[0109] By performing attention processing on the third session data, the long-range dependencies within the third session data itself are efficiently captured, enabling the extraction of global features from the third session data and improving the representational power of the first data features. Mapping processing through the first feedforward layer enhances the network's expressive power, further improving the representational power of the first data features.

[0110] In practical applications, the structure of text feature extraction networks can be more complex. For example, other network layers can be connected in series in at least one of the following ways: before the first attention layer, after the first feedforward layer, or between the first attention layer and the first feedforward layer. Alternatively, text feature extraction networks can introduce residual connections between network layers.

[0111] For example, the text feature extraction network extracts features from a pre-trained GPT (Generative Pre-Trained Transformer). The GPT comprises multiple cascaded Transformer encoding modules, such as... Figure 5 As shown, each Transformer encoding module 500 includes a multi-head masked self-attention layer 504 and a feedforward layer 502. Both the multi-head masked self-attention layer 504 and the feedforward layer 502 apply residual connections, that is, the input and output of the network layer are added through the addition module 505. In addition, a layer normalization (LN) layer 503 is concatenated after the multi-head masked self-attention layer 504, and a layer normalization layer 501 is concatenated after the feedforward layer 502. The layer normalization layer 501 and layer normalization layer 503 accelerate model convergence, normalize the output of the network layer, and alleviate the problems of gradient vanishing and gradient exploding in the network.

[0112] In this example, the text feature extraction network is a series of Transformer coding modules in GPT, and the structure of each Transformer coding module is as follows: Figure 5 As shown. The first attention layer is a multi-head mask self-attention layer 504, and the first feedforward layer is a feedforward layer 502.

[0113] Optionally, the electronic device can acquire a pre-trained GPT, perform SFT (Supervised Fine-Tuning) on ​​the acquired GPT to obtain a trained GPT, and extract text feature extraction networks from the trained GPT. The parameters of the Transformer encoding module in the acquired GPT are shown in Table 2 below.

[0114] Table 2

[0115]

[0116] Optionally, the obtained GPT is subjected to SFT using sample text. Specifically, sample text is obtained, which includes multiple characters. For any given character in the sample text, the characters following it are masked, and only that character and at least one character preceding it are input into the GPT. The GPT is then used to predict the probability of the next character's occurrence. Optionally, the probability of the i-th character in the sample text can be expressed as: P(u i |u i-k ,…,u i-1 ;Θ). Where Θ represents GPT, u i-k ,…,u i-1 Representing the ikth to the (i-1th)th character, u i Let k represent the i-th character, and k be a positive integer. In this way, the probability of multiple characters appearing in the sample text can be obtained, according to L1(u) = ∑ i logP(u i |u i-k ,…,u i-1 ;Θ) Determine the loss value L1(u), and adjust the GPT using the loss value L1(u). Where u represents the sample text.

[0117] After performing the SFT on the GPT, a trained GPT is obtained, from which a text feature extraction network can be extracted. The text feature extraction network includes at least one concatenated feature set, such as... Figure 5 The Transformer encoding module shown.

[0118] Taking a text feature extraction network including a Transformer encoding module as an example, the text embedding of the third session data is input into the Transformer encoding module. Multi-head masked self-attention processing is performed on the text embedding through a multi-head masked self-attention layer 504 to obtain the first attention processing result. The text embedding and the first attention processing result are added together, and then the result is normalized through a layer normalization layer 503 to obtain the first layer normalization processing result. At least one of linear or nonlinear mapping is performed on the first layer normalization processing result through a feedforward layer 502 to obtain the first mapping processing result. The first layer normalization processing result and the first mapping processing result are added together, and then the result is normalized through a layer normalization layer 501 to obtain the second layer normalization processing result. The Transformer encoding module outputs the second layer normalization processing result, and this output is the first data feature.

[0119] Step A2: Extract the second data features of the second session data through a non-text modality feature extraction network.

[0120] In this embodiment, there is at least one non-textual modality, and the intent recognition model includes feature extraction networks for each non-textual modality. For example, if the non-textual modality includes an image modality, then the intent recognition model includes a feature extraction network for the image modality (hereinafter referred to as the image feature extraction network); if the non-textual modality includes an audio modality, then the intent recognition model includes a feature extraction network for the audio modality.

[0121] The second session data can be input into a feature extraction network corresponding to the data modality. This network then performs feature extraction on the second session data to obtain second data features. These second data features are used to characterize the second session data and can exist in matrix form. The structure, type, and layers of the feature extraction network are not limited here. Optionally, the feature extraction network may include at least one of the following: convolutional layers, attention layers, normalization layers, activation layers, and feedforward layers. Different feature extraction network structures result in different feature extraction methods.

[0122] For example, the feature extraction network for non-textual modalities includes an encoding layer, a second attention layer, and a second feedforward layer. Step A2 includes: dividing the second session data into multiple data parts; encoding each data part through the encoding layer to obtain the encoding result of each data part; concatenating the encoding results of each data part to obtain a concatenated result; performing attention processing on the concatenated result through the second attention layer to obtain a second attention processing result; performing at least one of linear mapping or nonlinear mapping on the second attention processing result through the second feedforward layer to obtain a second mapping processing result; and determining the second data features based on the second mapping processing result.

[0123] This application does not limit the method of dividing the second session data. For example, when the second session data is an image, the image can be divided into multiple image blocks of the same size; when the second session data is audio, the audio can be divided into multiple audio segments of the same length. The data portion can be a frame image, an image block, or an audio segment.

[0124] The feature extraction network for non-textual modalities includes an encoding layer, which is a linear mapping layer used to map each data part to a corresponding encoded result. The encoded result of each data part represents its features and exists in matrix form. The encoded results of each data part can be directly concatenated to obtain the concatenated result. Alternatively, the position vectors of each data part can be obtained, representing the position of that data part in the second session data. The position vectors of each data part and the encoded result can then be concatenated to obtain the concatenated result.

[0125] The non-text modality feature extraction network also includes a second attention layer concatenated after the encoding layer. This second attention layer performs attention processing on the concatenated result based on an attention mechanism, yielding a second attention processing result. The text feature extraction network also includes a second feedforward layer concatenated after the second attention layer. The second feedforward layer can perform at least one of linear or nonlinear mapping on the second attention processing result, yielding a second mapping processing result. This second mapping processing result can be used as a second data feature. Alternatively, the non-text modality feature extraction network may also include other network layers, which determine the second data feature based on the second mapping processing result.

[0126] By dividing the second session data into multiple data parts and determining the concatenation result based on each data part, the second session data is transformed into sequence data. Attention processing is applied to the concatenation result to efficiently capture the long-range dependencies within the sequence data itself, thereby extracting global features and improving the representational power of the second data features. Mapping processing through a second feedforward layer enhances the network's expressive power and further improves the representational power of the second data features.

[0127] The structure of non-text modal feature extraction networks is similar to that of text feature extraction networks. Therefore, the relevant descriptions of text feature extraction networks can be found in the section on non-text modal feature extraction networks, and will not be repeated here. The following section uses image modality as an example to introduce image modal feature extraction networks (i.e., image feature extraction networks).

[0128] In this example, such as Figure 6As shown, a trained VisionTransformer (ViT) model can be obtained, and an image feature extraction network 600 can be extracted from the trained ViT model. The image feature extraction network 600 includes a linear mapping layer 601 and a Transformer encoding module 602. The second session data of the image modality can be divided into multiple image patches 603. The resolution of the image patch 603 is not limited here; for example, the resolution of the image patch 603 is 224×224 or 896×896. The linear mapping layer 601 maps each image patch 603 to a corresponding encoding result. The position vectors of each image patch and the encoding result are concatenated to obtain the concatenated result. Optionally, in addition to concatenating the position vectors and encoding results of each image patch, a reference vector and its corresponding position vector can also be concatenated. This position vector represents the position of the reference vector in the concatenated result. For example, the position vector of the reference vector is "0", indicating that the reference vector is located at the beginning of the concatenated result. The stitching result is input into the Transformer encoding module 602, and the Transformer encoding module 602 outputs the second data feature, which is an image feature used to characterize the second session data.

[0129] Optionally, the structure of the Transformer encoding module 602 is as follows: Figure 7 As shown, the Transformer encoding module 602 includes L encoding blocks 700, where L is a positive integer. After the concatenation result is input into the Transformer encoding module 602, the first encoding block 700 processes the concatenation result to obtain the processing result of the non-first encoding block 700. The non-first encoding block 700 processes the processing result of the previous encoding block 700 to obtain the processing result of the non-first encoding block 700. The processing result of the last encoding block 700 is the second data feature.

[0130] Each encoding block 700 includes a normalization layer 701, a multi-head attention layer 702, a normalization layer 703, and a multilayer perceptron 704. The encoding block 700 connects network layers using residual connections. For example, an addition module 705 adds the input of the normalization layer 701 to the output of the multi-head attention layer 702, and another addition module 705 adds the input of the normalization layer 703 to the output of the multilayer perceptron 704, thus implementing residual connections. The encoding block 700 can first normalize the input through the normalization layer 701 to obtain the first normalized result, and then perform multi-head attention processing on the first normalized result through the multi-head attention layer 702 to obtain the multi-head attention result. The input of encoding block 700 and the multi-head attention processing are added together. The addition result is normalized by normalization layer 703 to obtain a second normalized result. The second normalized result is then mapped by multilayer perceptron 704 to obtain the mapping result. The addition result and the mapping result are then added together to obtain the output of encoding block 700.

[0131] In the example above, the linear mapping layer 601 is an encoding layer, the second attention layer is a multi-head attention layer 702, and the second feedforward layer is a multilayer perceptron 704.

[0132] Step A3: Determine the intent recognition result based on the first data feature and the second data feature through the intent recognition network.

[0133] In this embodiment, the intent recognition model further includes an intent recognition network. First data features and various second data features can be concatenated to obtain concatenated features. These concatenated features are then input into the intent recognition network, which determines the intent recognition result. The structure, type, and network layers of the intent recognition network are not limited here. For example, the intent recognition network is a Transformer decoding module, which decodes the concatenated features to obtain the intent recognition result.

[0134] By employing feature extraction networks for each data modality, features corresponding to that data modality are extracted, improving the accuracy of the first and second data features. Subsequently, based on the first and second data features, a more accurate intent recognition result can be determined, thereby improving the accuracy of the information presented below.

[0135] In an exemplary embodiment, step 302 includes: obtaining second prompt information, the second prompt information being used to instruct the execution of a task to identify the generation relationship between the data modality corresponding to the preceding information and the data modality corresponding to the following information; performing intent recognition on the preceding information based on the second prompt information to obtain an intent recognition result.

[0136] This application does not limit the specific content of the second prompt information; any data that can instruct the execution of a recognition task can serve as the second prompt information. The second prompt information is text-based data. For example... Figure 4 As shown, Figure 4 The text indicated by reference numeral 402-1 in (A) is the second prompt message. Similarly, Figure 4 The text indicated by reference 402-2 in (B) is the second prompt message. Figure 4 The text indicated by reference numeral 402-3 in (C) is also the second prompt message. The second prompt message is: Referring to the above multi-turn dialogue and the provided image, determine the generation relationship between the data modality of the preceding information and the data modality of the following information.

[0137] The second prompt can be appended before or after the preceding information. The appended information is input into the intent recognition model, which then outputs the intent recognition result. The process by which the intent recognition model determines the intent recognition result is described in steps A1 to A3; the underlying principles are similar and will not be repeated here. The second prompt guides the model to accurately perform intent recognition, improving the accuracy of the result and thus the accuracy of the following information.

[0138] like Figure 4 As shown in (A), the second conversation data 404-1, the third conversation data 401-1, and the second prompt information 402-1 are input into the intent recognition model. The intent recognition model determines the intent recognition result to be graph-to-text 403-1. From the third conversation data 401-1, it can be seen that the intent recognition result of question 1 is also graph-to-text. Therefore, the above two rounds of dialogue do not belong to a cross-modal generation relationship.

[0139] Similarly, such as Figure 4 As shown in (B), the second conversation data 404-2, the third conversation data 401-2, and the second prompt information 402-2 are input into the intent recognition model. The intent recognition model determines the intent recognition result to be graph-to-text 403-2. From the third conversation data 401-2, it can be seen that the intent recognition result of question 1 is text-to-graph. Therefore, the above two rounds of dialogue belong to a cross-modal generation relationship, and the generation relationship changes from text-to-graph to graph-to-text.

[0140] Similarly, such as Figure 4As shown in (C), the second conversation data 404-3, the third conversation data 401-3, and the second prompt information 402-3 are input into the intent recognition model. The intent recognition model determines the intent recognition result to be graph-to-graph 403-3. From the third conversation data 401-3, it can be seen that the intent recognition result of question 1 is text-to-graph. Therefore, the above two rounds of dialogue belong to a cross-modal generation relationship, and the generation relationship changes from text-to-graph to graph-to-graph.

[0141] Step 303: Select the information generation model corresponding to the generation relationship indicated by the intent recognition result from multiple candidate generation relationship models.

[0142] Candidate generation relations are used to indicate the generation of a second data modality based on a first data modality; that is, a candidate generation relation is a generation relationship between the first and second data modalities. The first and second data modalities can be the same or different. For example, candidate generation relations include text-to-text and text-to-image.

[0143] Assuming there are N data modalities, there are at least N×N candidate generation relations. Each candidate generation relation corresponds to a model, which is used to generate the context information of the second data modality based on the context information of the first data modality. This application does not limit the structure, type, or network layers included in the models corresponding to the candidate generation relations. For example, the models corresponding to text-to-text, text-to-image, text-to-audio, and text-to-video are all large language models, and the network parameters of each large language model differ, making them different large language models.

[0144] Since the intent recognition result is used to indicate one of the multiple candidate generative relations, the model of the generative relation indicated by the intent recognition result can be selected from the multiple candidate generative relation models. This model is the information generation model.

[0145] Step 304: Generate the following information based on the preceding information using the information generation model.

[0146] The preceding information can be input into the information generation model, which will then output the following information. It is understood that different information generation models will produce different ways of generating the following information from the preceding information; however, no specific limitations are made here.

[0147] For example, taking a data modality that includes text and image modalities, candidate generation relations include text-to-text, text-to-image, image-to-text, and image-to-image. Assuming the intent recognition result indicates text-to-image, the model corresponding to the text-to-image is selected as the information generation model. This model then generates the context information of the image modality based on the preceding context information of the text modality. Similarly, assuming the intent recognition result indicates image-to-image, the model corresponding to the image-to-image is selected as the information generation model. This model then generates the context information of the image modality based on the preceding context information of the image modality.

[0148] Optionally, the information generation model includes a classification model and generation models for multiple candidate knowledge domains. Step 304 includes: classifying the preceding information using the classification model to obtain a classification result, the classification result being used to indicate the knowledge domain involved in the preceding information; selecting the generation model corresponding to the knowledge domain indicated by the classification result from the multiple candidate knowledge domain generation models; and generating the following information based on the preceding information using the generation model corresponding to the knowledge domain indicated by the classification result.

[0149] Optionally, if the data modality of the preceding information is the first data modality and the data modality of the following information is the second data modality, then the classification model is used to classify the preceding information of the first data modality to obtain a classification result. This application does not limit the structure of the classification model; different classification models result in different classification methods, which will not be elaborated upon here.

[0150] Assuming there are N candidate knowledge domains, the classification result includes N sub-results, where N is a positive integer, and each sub-result corresponds to a different candidate knowledge domain. Each sub-result represents the probability that the preceding information indicates a corresponding candidate knowledge domain; the larger the sub-result, the higher the probability. Optionally, the sub-result is a probability value between 0 and 1, representing the probability that the preceding information indicates a corresponding candidate knowledge domain. When a sub-result is not less than a reference value, it can be considered that the preceding information indicates a corresponding candidate knowledge domain. Based on this classification result, at least one candidate knowledge domain indicated by the preceding information can be represented. This example does not limit the reference value. For example, the reference value may be a value set based on human experience, or it may be one of the multiple sub-results, such as the sub-result with the largest value among multiple sub-results.

[0151] Each candidate knowledge domain corresponds to a generative model, which is used to generate context information belonging to the corresponding candidate knowledge domain and with a second data modality, based on the context information of the first data modality. This application does not limit the structure, type, or network layers included in the generative model corresponding to the candidate knowledge domain.

[0152] For example, candidate knowledge domains include weather, calendar, travel, and code. If the classification result indicates weather, then a weather generation model is selected, and this model generates the following information based on the preceding information.

[0153] By classifying the preceding information, we can obtain classification results for the indicated knowledge domain. Then, by generating the following information through the generative model of the corresponding knowledge domain, we can improve the professionalism and accuracy of the following information, thereby improving the quality of the conversation.

[0154] Understandably, in practical applications, one can choose to perform a classification operation or not. For example... Figure 9 As shown, classification is performed only for text-to-text pairs; no classification is performed for image-to-text pairs, text-to-image pairs, or image-to-image pairs.

[0155] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant regions. For example, the information mentioned above and sample texts involved in this application were obtained with full authorization.

[0156] In the above method, by performing intent recognition on the preceding information in the conversation process, the generative relationship between the data modalities of the preceding and following information is obtained, achieving fine-grained identification of the generative relationship between data modalities. Subsequently, the preceding information can be accurately understood and the following information can be output accurately through the information generation model corresponding to this generative relationship, improving the semantic relevance between the preceding and following information, thereby improving the quality of the conversation.

[0157] The above-described session generation method can be applied to various scenarios such as instant messaging, intelligent customer service, and question-and-answer. The following example illustrates the session generation method of this application.

[0158] In a question-and-answer scenario, a user and an intelligent chatbot can engage in at least one round of dialogue. In each round, the user can ask a question, and the intelligent chatbot can generate an answer. If no dialogue has been completed prior to this round, the intelligent chatbot can generate an answer based on the question in this round. If at least one round of dialogue has been completed prior to this round, the intelligent chatbot can generate an answer for this round based on the completed at least one round and the question in this round. Specifically, as... Figure 8 As shown, the process includes the following steps.

[0159] Step 801: Based on the user's input question, obtain the context information, which includes first session data in the text modality and second session data in the image modality.

[0160] If the user inputs a question for the first round of dialogue, then the preceding information includes that question. If the user inputs a question for the i-th round of dialogue (where i is a positive integer greater than 1), then the preceding information includes the ik-th to (i-1)-th rounds of dialogue, as well as the question for the i-th round, where k is a positive integer and k is less than i. If the question-answering scenario supports both text and image data modalities, then the first session data is the text in the preceding information, and the second session data is the image in the preceding information. The implementation of step 801 can be found in the descriptions of steps 301 and 3021, and will not be repeated here.

[0161] Step 802: Insert the first prompt message into the first session data to obtain the third session data.

[0162] For example, if the second session data is an image uploaded by a user, a first prompt message "{Upload Image}" is inserted into the question of the first session data to indicate the data modality (i.e., image modality) and acquisition method (i.e., upload) of the second session data. If the second session data is an image generated by an intelligent robot, a first prompt message "{Generate Image}" is inserted into the answer of the first session data to indicate the data modality (i.e., image modality) and acquisition method (i.e., generation) of the second session data. The implementation of step 802 can be found in the description of step 3022, and will not be repeated here.

[0163] Step 803: Determine the intent recognition result based on the second session data and the third session data.

[0164] The second and third session data can be input into the intent recognition model, which then outputs the intent recognition result. Since the question-and-answer scenario supports both text and image data modalities, the generation relationships between question and answer data modalities include: text-to-text, text-to-image, image-to-text, and image-to-image. Based on this, the intent recognition result can indicate text-to-text, text-to-image, image-to-text, or image-to-image. The implementation of step 803 can be found in the description of step 3023, and will not be repeated here.

[0165] It is understandable that, such as Figure 9As shown, steps 801 to 803 above are equivalent to performing intent recognition on the context information 901 including the question, and obtaining intent recognition result 902. For question-and-answer scenarios where the data modality includes text modality and image modality, intent recognition result 902 can indicate any one of the following generative relationships: text-to-text 902-1, text-to-image 902-2, image-to-text 902-3, and image-to-image 902-4.

[0166] Step 804: Determine whether the intent recognition result indicates text-to-text. If the intent recognition result indicates text-to-text, proceed to steps 805 and 806; if the intent recognition result does not indicate text-to-text, proceed to step 807.

[0167] Step 805: Classify the third session data using a classification model to obtain the classification results for the indicated knowledge domain.

[0168] Step 806: Generate an answer based on the third conversation data using the information generation model corresponding to the knowledge domain.

[0169] Step 807: Generate an answer based on the third conversation data using the information generation model corresponding to the intent recognition result.

[0170] The following is combined Figure 9 Let's explain steps 804 to 807. In this example, there are multiple candidate knowledge domains, such as weather 903-1, calendar 903-2, code 903-3, and travel 903-4. If the intent recognition result 902 indicates textual information 902-1, then the preceding information 901 is first classified to obtain a classification result 903. The classification result 903 can indicate one of the multiple candidate knowledge domains; for example, the classification result can indicate weather 903-1 or calendar 903-2. Each candidate knowledge domain corresponds to an information generation model, used to generate an answer related to that candidate knowledge domain based on the preceding information. That is, weather 903-1, calendar 903-2, code 903-3, travel 903-4, etc., each correspond to an information generation model. From the information generation models corresponding to each candidate knowledge domain, the information generation model corresponding to the knowledge domain indicated by the classification result 903 can be selected, and the answer 904 of the question can be generated based on the preceding information using that information generation model.

[0171] Furthermore, text-to-image 902-2, image-to-text 902-3, and image-to-image 902-4 each correspond to an information generation model used to generate the answer for the corresponding data modality based on the contextual information. For example, the information generation model corresponding to text-to-image 902-2 is used to generate the answer for the image modality based on the contextual information of the text modality. Similarly, the information generation model corresponding to image-to-text 902-3 is used to generate the answer for the text modality based on the contextual information of the image modality. And the information generation model corresponding to image-to-image 902-4 is used to generate the answer for the image modality based on the contextual information of the image modality. If the intent recognition result 902 indicates text-to-image 902-2, image-to-text 902-3, or image-to-image 902-4, then the answer 904 for the question is directly generated based on the contextual information through the corresponding information generation model.

[0172] The implementation methods of steps 804 to 807 can be found in the descriptions of steps 303 and 304, and will not be repeated here.

[0173] Step 808, the answer is displayed.

[0174] Optionally, after generating the answer, the answer is displayed on a display page. If the user enters a new question on the display page, steps 801 to 808 can be repeated to generate the answer to that question. That is, in a question-and-answer scenario, steps 801 to 808 can be executed multiple times in a loop to generate the answer to each question.

[0175] It should be noted that any two rounds of dialogue can correspond to the same generative relationship or different generative relationships (i.e., generative relationships across data modalities). For example, the first round of dialogue corresponds to text-to-text, the second round to text-to-image, and the third round to text-to-text. In this example, on the one hand, by reflecting the data modality, acquisition method, and positional relationship between non-text data and text data through the first prompt, the accuracy of intent recognition can be improved. Especially for generative relationships across data modalities, the accuracy of intent recognition using the method in this example can reach 0.85, an improvement of approximately 0.05. On the other hand, by identifying the generative relationships between data modalities with fine granularity, the information generation model corresponding to the generative relationship can be used to accurately understand the preceding information and output accurate following information, improving the semantic relevance between the preceding and following information, thereby improving the quality of the conversation. Furthermore, by classifying to obtain the knowledge domain, the information generation model of the corresponding knowledge domain can be used to output more professional following information, improving the accuracy of the following information, thereby improving the quality of the conversation.

[0176] Figure 10 The diagram shown is a schematic representation of a session generation device provided in an embodiment of this application. Figure 10As shown, the device includes:

[0177] The acquisition module 1001 is used to acquire the context information of the conversation process;

[0178] The intent recognition module 1002 is used to perform intent recognition on the preceding information and obtain the intent recognition result. The intent recognition result is used to indicate the generation relationship between the data modality corresponding to the preceding information and the data modality corresponding to the following information of the preceding information.

[0179] The selection module 1003 is used to select the information generation model corresponding to the generation relationship indicated by the intent recognition result from multiple candidate generation relationship models.

[0180] The generation module 1004 is used to generate the following information based on the preceding information using the information generation model.

[0181] In one possible implementation, the data modality corresponding to the above information includes text modality and non-text modality;

[0182] The intent recognition module 1002 is used to divide the preceding information to obtain first session data corresponding to the text modality and second session data corresponding to the non-text modality; add first prompt information to the first session data to obtain third session data, the first prompt information being used to indicate at least one of the following: the non-text modality corresponding to the second session data, the corresponding acquisition method, the session object to which it belongs, and the knowledge domain involved; and perform intent recognition on the second session data and the third session data to obtain intent recognition results.

[0183] In one possible implementation, the intent recognition module 1002 is used to add a first prompt message at the addition position in the first session data to obtain the third session data. The addition position is determined based on at least one of the following: the positional relationship between the second session data and the first session data in the preceding information; the session object of the second session data and the dialogue object of the first session data.

[0184] In one possible implementation, the intent recognition module 1002 is used to extract first data features of third session data through a text feature extraction network; extract second data features of second session data through a non-text modality feature extraction network; and determine the intent recognition result based on the first and second data features through the intent recognition network.

[0185] In one possible implementation, the text feature extraction network includes a first attention layer and a first feedforward layer;

[0186] The intent recognition module 1002 is used to perform attention processing on the third session data through a first attention layer to obtain a first attention processing result; perform at least one of linear mapping or nonlinear mapping on the first attention processing result through a first feedforward layer to obtain a first mapping processing result; and determine a first data feature based on the first mapping processing result.

[0187] In one possible implementation, the non-textual modality feature extraction network includes an encoding layer, a second attention layer, and a second feedforward layer;

[0188] The intent recognition module 1002 is used to divide the second session data into multiple data parts; encode each data part through an encoding layer to obtain the encoding result of each data part; concatenate the encoding results of each data part to obtain a concatenated result; perform attention processing on the concatenated result through a second attention layer to obtain a second attention processing result; perform at least one of linear mapping or nonlinear mapping on the second attention processing result through a second feedforward layer to obtain a second mapping processing result; and determine the second data features based on the second mapping processing result.

[0189] In one possible implementation, the intent recognition module 1002 is used to obtain second prompt information, which is used to instruct the execution of a task to recognize the generation relationship between the data modality corresponding to the preceding information and the data modality corresponding to the following information; and to perform intent recognition on the preceding information according to the second prompt information to obtain the intent recognition result.

[0190] In one possible implementation, the information generation model includes a classification model and a generation model for multiple candidate knowledge domains;

[0191] The generation module 1004 is used to classify the preceding information using a classification model to obtain a classification result, which is used to indicate the knowledge domain involved in the preceding information; to select the generation model corresponding to the knowledge domain indicated by the classification result from multiple candidate knowledge domain generation models; and to generate the following information based on the preceding information using the generation model corresponding to the knowledge domain indicated by the classification result.

[0192] In the aforementioned device, by performing intent recognition on the preceding information in the conversation process, the generative relationship between the data modalities of the preceding and following information is obtained, achieving fine-grained identification of the generative relationship between data modalities. Subsequently, the preceding information can be accurately understood and the following information can be output accurately through the information generation model corresponding to this generative relationship, improving the semantic relevance between the preceding and following information, thereby improving the quality of the conversation.

[0193] It should be understood that the above Figure 10The provided device, in implementing its functions, is only illustrated by the division of the above-described functional modules. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments, and will not be repeated here.

[0194] Figure 11 A structural block diagram of a terminal device 1100 provided in an exemplary embodiment of this application is shown. The terminal device 1100 includes a processor 1101 and a memory 1102.

[0195] Processor 1101 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1101 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1101 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1101 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1101 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0196] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1102 are used to store at least one computer program, which is executed by the processor 1101 to implement the session generation method provided in the method embodiments of this application.

[0197] In some embodiments, the terminal device 1100 may also optionally include: a peripheral device interface 1103 and at least one peripheral device. The processor 1101, memory 1102, and peripheral device interface 1103 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1103 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of: a radio frequency circuit 1104, a display screen 1105, a camera assembly 1106, an audio circuit 1107, and a power supply 1108.

[0198] Peripheral device interface 1103 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1101 and memory 1102. In some embodiments, processor 1101, memory 1102 and peripheral device interface 1103 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1101, memory 1102 and peripheral device interface 1103 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0199] The radio frequency (RF) circuit 1104 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1104 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1104 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1104 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1104 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1104 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0200] Display screen 1105 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1105 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1101 for processing. In this case, display screen 1105 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1105, disposed on the front panel of terminal device 1100; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal device 1100 or in a folded design; in still other embodiments, display screen 1105 may be a flexible display screen, disposed on a curved or folded surface of terminal device 1100. Furthermore, display screen 1105 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1105 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0201] The camera assembly 1106 is used to acquire images or videos. Optionally, the camera assembly 1106 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1106 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0202] The audio circuit 1107 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1101 for processing, or input to the radio frequency circuit 1104 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal device 1100. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1101 or the radio frequency circuit 1104 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1107 may also include a headphone jack.

[0203] Power supply 1108 is used to supply power to the various components in terminal device 1100. Power supply 1108 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1108 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0204] In some embodiments, the terminal device 1100 further includes one or more sensors 1109. The one or more sensors 1109 include, but are not limited to: an acceleration sensor 1111, a gyroscope sensor 1112, a pressure sensor 1113, an optical sensor 1114, and a proximity sensor 1115.

[0205] Accelerometer 1111 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal device 1100. For example, accelerometer 1111 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 1101 can control display screen 1105 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1111. Accelerometer 1111 can also be used for games or for acquiring user motion data.

[0206] The gyroscope sensor 1112 can detect the orientation and rotation angle of the terminal device 1100. The gyroscope sensor 1112 can work in conjunction with the accelerometer sensor 1111 to collect the user's 3D movements on the terminal device 1100. Based on the data collected by the gyroscope sensor 1112, the processor 1101 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0207] The pressure sensor 1113 can be disposed on the side bezel of the terminal device 1100 and / or on the lower layer of the display screen 1105. When the pressure sensor 1113 is disposed on the side bezel of the terminal device 1100, it can detect the user's grip signal on the terminal device 1100, and the processor 1101 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1113. When the pressure sensor 1113 is disposed on the lower layer of the display screen 1105, the processor 1101 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1105. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0208] Optical sensor 1114 is used to collect ambient light intensity. In one embodiment, processor 1101 can control the display brightness of display screen 1105 based on the ambient light intensity collected by optical sensor 1114. Specifically, when the ambient light intensity is high, the display brightness of display screen 1105 is increased; when the ambient light intensity is low, the display brightness of display screen 1105 is decreased. In another embodiment, processor 1101 can also dynamically adjust the shooting parameters of camera assembly 1106 based on the ambient light intensity collected by optical sensor 1114.

[0209] The proximity sensor 1115, also known as a distance sensor, is typically located on the front panel of the terminal device 1100. The proximity sensor 1115 is used to detect the distance between the user and the front of the terminal device 1100. In one embodiment, when the proximity sensor 1115 detects that the distance between the user and the front of the terminal device 1100 is gradually decreasing, the processor 1101 controls the display screen 1105 to switch from a screen-on state to a screen-off state; when the proximity sensor 1115 detects that the distance between the user and the front of the terminal device 1100 is gradually increasing, the processor 1101 controls the display screen 1105 to switch from a screen-off state to a screen-on state.

[0210] Those skilled in the art will understand that Figure 11 The structure shown does not constitute a limitation on the terminal device 1100, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0211] Figure 12This is a schematic diagram of the server structure provided in the embodiments of this application. The server 1200 can vary considerably due to different configurations or performance. It may include one or more processors 1201 and one or more memories 1202. The one or more memories 1202 store at least one computer program, which is loaded and executed by the one or more processors 1201 to implement the session generation method provided in the above-described method embodiments. For example, the processor 1201 is a CPU. Of course, the server 1200 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 1200 may also include other components for implementing device functions, which will not be elaborated here.

[0212] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one computer program that is loaded and executed by a processor to enable an electronic device to implement any of the above-described session generation methods.

[0213] Optionally, the aforementioned computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0214] In an exemplary embodiment, a computer program is also provided, which is at least one such computer program, loaded and executed by a processor to enable an electronic device to implement any of the above-described session generation methods.

[0215] In an exemplary embodiment, a computer program product is also provided, which stores at least one computer program that is loaded and executed by a processor to enable an electronic device to implement any of the above-described session generation methods.

[0216] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0217] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0218] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A session generation method, characterized in that, The method includes: Retrieve context information from the previous part of the conversation; The intent of the preceding information is identified to obtain an intent identification result, which is used to indicate the generation relationship between the data modality corresponding to the preceding information and the data modality corresponding to the following information of the preceding information; From multiple candidate generation relationship models, select the information generation model corresponding to the generation relationship indicated by the intent recognition result; The aforementioned information is generated based on the aforementioned preceding information using the information generation model.

2. The method according to claim 1, characterized in that, The data modality corresponding to the above information includes text modality and non-text modality; the process of recognizing the intent of the above information and obtaining the intent recognition result includes: The above information is divided to obtain the first session data corresponding to the text modality and the second session data corresponding to the non-text modality; A first prompt message is added to the first session data to obtain the third session data. The first prompt message is used to indicate at least one of the following: the non-text modality corresponding to the second session data, the corresponding acquisition method, the session object to which it belongs, and the knowledge domain involved. Intent recognition is performed on the second session data and the third session data to obtain intent recognition results.

3. The method according to claim 2, characterized in that, The step of adding a first prompt message to the first session data to obtain the third session data includes: The first prompt message is added at the addition location in the first session data to obtain the third session data, wherein the addition location is determined based on at least one of the following: The positional relationship between the second session data and the first session data mentioned above; the session object of the second session data and the dialogue object of the first session data.

4. The method according to claim 2, characterized in that, The process of performing intent recognition on the second session data and the third session data to obtain intent recognition results includes: The first data feature of the third session data is extracted using a text feature extraction network. The second data features of the second session data are extracted using the non-text modality feature extraction network. The intent recognition result is determined by the intent recognition network based on the first data feature and the second data feature.

5. The method according to claim 4, characterized in that, The text feature extraction network includes a first attention layer and a first feedforward layer; the extraction of the first data features of the third session data through the text feature extraction network includes: Attention processing is performed on the third session data through the first attention layer to obtain the first attention processing result; The first attention processing result is obtained by performing at least one of linear or nonlinear mapping on the first attention processing result through the first feedforward layer; Based on the result of the first mapping process, the first data feature is determined.

6. The method according to claim 4, characterized in that, The non-textual modality feature extraction network includes an encoding layer, a second attention layer, and a second feedforward layer; the extraction of second data features from the second session data through the non-textual modality feature extraction network includes: The second session data is divided into multiple data parts; The encoding layer encodes each data part to obtain the encoding result of each data part; The encoding results of the various data parts are concatenated to obtain the concatenated result; Attention processing is performed on the splicing result through the second attention layer to obtain the second attention processing result; The second attention processing result is obtained by performing at least one of linear or nonlinear mapping on the second attention processing result through the second feedforward layer; Based on the result of the second mapping process, the second data feature is determined.

7. The method according to any one of claims 1 to 6, characterized in that, The intention to identify the aforementioned information, and to obtain the intention identification result, includes: Obtain a second prompt message, which is used to instruct the execution of a task to identify the generation relationship between the data modality corresponding to the preceding information and the data modality corresponding to the following information; Based on the second prompt information, intent recognition is performed on the above information to obtain the intent recognition result.

8. The method according to any one of claims 1 to 6, characterized in that, The information generation model includes a classification model and generation models for multiple candidate knowledge domains; the step of generating the following information based on the preceding information using the information generation model includes: The above information is classified using the classification model to obtain a classification result, which is used to indicate the knowledge domain involved in the above information. Select the generative model corresponding to the knowledge domain indicated by the classification result from the generative models of the multiple candidate knowledge domains; The following information is generated based on the preceding information using the generative model corresponding to the knowledge domain indicated by the classification results.

9. A session generation apparatus, characterized in that, The device includes: The acquisition module is used to retrieve the context information of the conversation process; An intent recognition module is used to recognize the intent of the preceding information and obtain an intent recognition result. The intent recognition result is used to indicate the generation relationship between the data modality corresponding to the preceding information and the data modality corresponding to the following information of the preceding information. The selection module is used to select the information generation model corresponding to the generation relationship indicated by the intent recognition result from multiple candidate generation relationship models; The generation module is used to generate the following information based on the preceding information using the information generation model.

10. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the session generation method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable an electronic device to implement the session generation method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product stores at least one computer program, which is loaded and executed by a processor to enable the electronic device to implement the session generation method as described in any one of claims 1 to 8.

Images