Human-computer interaction method, device and equipment

Through the pre-trained multimodal pre-trained model to identify expression theme information and provide corresponding expression image reply, the problems of low human-computer interaction efficiency and users' turn to manual services in the prior art are solved, and more efficient human-computer communication is achieved.

CN114637833BActive Publication Date: 2024-11-26ALIPAY (HANGZHOU) INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210296215.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-24
Publication Date
2024-11-26
Estimated Expiration
2042-03-24

AI Technical Summary

Technical Problem

The existing human-computer interaction is based on a plain text form, which leads to low communication efficiency and difficulty for users to accurately understand by computer equipment, resulting in frequent turnover to manual services, increasing service pressure.

Method used

A pre-trained multimodal pre-trained model is adopted to establish correspondence through emoticon images and character information, identify emoticon subject information, and provide it to users as a reply, improving the understanding ability and communication efficiency of computer equipment in human-computer interaction.

Benefits of technology

It improves the dialogue and communication efficiency of human-computer interaction, reduces the probability of users turning to manual services, and enhances user satisfaction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114637833B_ABST
    Figure CN114637833B_ABST
Patent Text Reader

Abstract

The embodiments of this specification disclose a human-computer interaction method, apparatus and equipment, the method comprising: obtaining reply information output by a pre-trained dialogue model in response to input information of a target user, if a first expression theme information matching the reply information exists in a pre-established correspondence between expression images and expression theme information, then obtaining an expression image corresponding to the first expression theme information, the expression subject information in the correspondence being obtained by recognizing the expression image and character information contained in the expression image through a pre-trained multimodal pre-training model, the multimodal pre-training model being obtained by model training using a training image containing the expression image and the character information contained in the training image, and the expression image corresponding to the first expression theme information can be provided to the target user as a reply to the input information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This document relates to the field of computer technology, and in particular to a human-computer interaction method, device and equipment. Background Art

[0002] At present, human-computer interaction is usually implemented in the form of plain text (i.e., users and computer devices interact through language and text). However, in actual network interaction, it is usually implemented in the form of a combination of text and emoticons. Emoticons can more vividly express the current user's emotions, and an interesting emoticon will also make the user want to continue communicating. For this reason, many users will specially collect emoticons for subsequent use. Since human-computer interaction is usually implemented in the form of plain text, this will inevitably reduce the efficiency of dialogue communication, and in many cases, computer devices may not be able to accurately understand the user's needs, causing a large number of users to skip human-computer interaction and turn to manual services, thereby increasing the processing pressure of manual services. Therefore, it is necessary to provide a human-computer interaction solution that can improve the efficiency of dialogue communication in human-computer interaction and reduce the probability of users turning to manual services. Summary of the invention

[0003] The purpose of the embodiments of this specification is to provide a human-computer interaction solution that can improve the efficiency of dialogue communication in human-computer interaction and reduce the probability of users turning to manual service.

[0004] In order to implement the above technical solution, the embodiments of this specification are implemented as follows:

[0005] An embodiment of the present specification provides a human-computer interaction method, the method comprising: obtaining reply information output by a pre-trained dialogue model in response to input information of a target user. If there is a first expression theme information matching the reply information in the pre-established correspondence between expression images and expression theme information, then obtaining an expression image corresponding to the first expression theme information, wherein the expression subject information in the correspondence is obtained by recognizing the expression image and the character information contained in the expression image through a pre-trained multimodal pre-training model, wherein the multimodal pre-training model is obtained by model training using a training image containing the expression image and the character information contained in the training image. The expression image corresponding to the first expression theme information is provided to the target user as a reply to the input information.

[0006] An embodiment of the present specification provides a human-computer interaction device, the device comprising: a reply information acquisition module, which acquires reply information output by a pre-trained dialogue model in response to input information of a target user. An expression determination module, which acquires an expression image corresponding to the first expression theme information if there is a first expression theme information matching the reply information in the pre-established correspondence between expression images and expression theme information, the expression subject information in the correspondence being obtained by recognizing the expression image and the character information contained in the expression image through a pre-trained multimodal pre-training model, the multimodal pre-training model being obtained by model training of a training image containing the expression image and the character information contained in the training image. An expression reply module, which provides the expression image corresponding to the first expression theme information to the target user as a reply to the input information.

[0007] The embodiment of the present specification provides a human-computer interaction device, the human-computer interaction device comprising: a processor; and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor: obtains the reply information output by the pre-trained dialogue model in response to the input information of the target user. If there is a first expression theme information matching the reply information in the pre-established correspondence between the expression image and the expression theme information, then obtain the expression image corresponding to the first expression theme information, the expression subject information in the correspondence is obtained by recognizing the expression image and the character information contained in the expression image through a pre-trained multimodal pre-training model, and the multimodal pre-training model is obtained by model training of a training image containing the expression image and the character information contained in the training image. Provide the expression image corresponding to the first expression theme information to the target user as a reply to the input information.

[0008] The embodiments of this specification also provide a storage medium for storing computer executable instructions, which implement the following process when executed by a processor: obtaining the reply information output by a pre-trained dialogue model in response to the input information of a target user. If there is a first expression theme information matching the reply information in the pre-established correspondence between expression images and expression theme information, then obtaining an expression image corresponding to the first expression theme information, wherein the expression subject information in the correspondence is obtained by recognizing the expression image and the character information contained in the expression image through a pre-trained multimodal pre-training model, wherein the multimodal pre-training model is obtained by model training of a training image containing the expression image and the character information contained in the training image. The expression image corresponding to the first expression theme information is provided to the target user as a reply to the input information. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0010] Figure 1A This is an embodiment of a human-computer interaction method of this specification;

[0011] Figure 1B This is a schematic diagram of a human-computer interaction process in this manual;

[0012] Figure 2 This is a schematic diagram of a human-computer interaction interface in this manual;

[0013] Figure 3A A schematic diagram of an expression image;

[0014] Figure 3B This is a schematic diagram of another human-computer interaction interface in this manual;

[0015] Figure 3C This is a schematic diagram of another human-computer interaction interface of this manual;

[0016] Figure 4 This is a schematic diagram of another human-computer interaction process in this manual;

[0017] Figure 5 This is a schematic diagram of a dialogue model training process in this specification;

[0018] Figure 6 This is an embodiment of a human-computer interaction device of this specification;

[0019] Figure 7 This is an embodiment of a human-computer interaction device in this specification. DETAILED DESCRIPTION

[0020] The embodiments of this specification provide a human-computer interaction method, device and equipment.

[0021] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0022] Embodiment 1

[0023] like Figure 1A and Figure 1B As shown, the embodiment of this specification provides a human-computer interaction method, the execution subject of the method can be a computer device, wherein the computer device can be a terminal device, a server or a corresponding machine, etc. The terminal device can be a mobile terminal device such as a mobile phone, a tablet computer, or a computer device such as a laptop or a desktop computer, or it can also be an IoT device (specifically such as a smart watch, a car device, etc.), etc. The machine can be an intelligent customer service machine of a certain business, etc., which can be set according to the actual situation. The method can specifically include the following steps:

[0024] In step S102, the response information output by the pre-trained dialogue model in response to the input information of the target user is obtained.

[0025] Among them, the dialogue model can be a model of the dialogue between the user and the computer device during the human-computer interaction process. The dialogue model can be constructed by a variety of different algorithms. For example, the dialogue model can be constructed by algorithms such as search engines, natural language understanding NLU (Natural Language Understanding), and dialogue strategies. The specific settings can be based on actual conditions. In addition, the currently commonly used dialogue model can also be used for retraining to obtain the trained dialogue model as the dialogue model used in this embodiment, etc., and this specification embodiment does not limit this. The computer device can be a mobile phone, a tablet computer, a smart device, a wearable device, etc., and the computer device can be installed with a corresponding application, through which a dialogue can be conducted with the user, such as an intelligent customer service system of a shopping application. The target user can be any user who interacts with the computer device. Input information may be information input into a computer device by a target user. The input information may include text information or emoticon images, etc. An emoticon image may be an image expressing emotion. In the era of mobile Internet, people use currently popular stars, quotations, cartoons, and film and television screenshots as materials, accompanied by a series of matching characters to express specific emotions. An emoticon image may be a static image or a dynamic image, or may be an image configured with a certain animation display effect (e.g., an emoticon image contains a tiger head. After the emoticon image is sent, a jumping tiger may be displayed within a preset page range, accompanied by animation display effects such as the tiger's roar). An emoticon image may only include images that can present a certain expression. For example, an emoticon image may include an image of an "OK" gesture, or it may include images and characters that can present a certain expression. For example, an emoticon image may include a laughing cartoon avatar and four characters "Ha".

[0026] In practice, currently, human-computer interaction is usually implemented in the form of plain text (i.e., the user and the computer device interact through language and text). However, in actual network interaction, it is usually implemented in the form of a combination of text and emoticons, where emoticons can more vividly express the current user's emotions, and an interesting emoticon will also make the user want to continue communicating. For this reason, many users will specifically collect emoticons for subsequent use. Since human-computer interaction is usually implemented in the form of plain text, this is bound to reduce the efficiency of dialogue communication, and in many cases, the computer device may not be able to accurately understand the user's needs, causing a large number of users to skip human-computer interaction and turn to manual services, thereby increasing the processing pressure of manual services. For this reason, it is necessary to provide a human-computer interaction solution that can improve the efficiency of dialogue communication in human-computer interaction and reduce the probability of users turning to manual services. The embodiments of this specification provide an implementable technical solution, which may specifically include the following contents:

[0027] A pre-trained dialogue model may be provided in the computer device. The dialogue model may use a dialogue model commonly used in human-computer interaction (implemented in the form of pure text) (which may be constructed by a variety of different algorithms, for example, by a neural network algorithm, etc.) as an initial model. Then, dialogue data including expression images may be obtained (which may include expression images and text, or may only include expression images, etc.), and the obtained dialogue data may be used as training samples. The training samples may be used to train the above initial model to obtain a trained dialogue model. After being trained in the above manner, the dialogue model may at least be able to recognize expression images. In some cases, it may also be able to send appropriate expression images to users, etc., which may be specifically set according to actual conditions.

[0028] When a user (i.e., a target user) needs to perform human-computer interaction with a computer device, the target user can activate the computer device and start a human-computer interaction application installed in the computer device, such as Figure 2As shown, a dialog box and a user input box may be provided in the application. The target user may input the information to be interacted with in the user input box. After the input is completed, the send button in the application may be clicked. The computer device may obtain the input information in the user input box and may present the input information in the dialog box. At the same time, the input information also needs to be input into the above-mentioned trained dialogue model. The input information is analyzed by the dialogue model, that is, the semantics expressed by the text information therein may be analyzed. At the same time, the meaning expressed by the emoticon image therein may also be analyzed, so as to obtain the complete semantics of the input information to judge the real intention or demand of the target user. Based on the analysis results obtained by the above analysis, the dialogue model may output reply information matching the input information. For example, the input information of the target user is "After receiving the product with the order number of xxxxx, it was found that the product had a bump and damage. ", and you can reply with something like "Don't worry, I'm here for you. We can exchange the product for you unconditionally. Of course, you can also choose a refund. So do you want an exchange or a refund?"

[0029] In step S104, if there is a first expression theme information matching the above-mentioned reply information in the pre-established correspondence between the expression image and the expression theme information, then the expression image corresponding to the first expression theme information is obtained, and the expression subject information in the above-mentioned correspondence is obtained by recognizing the expression image and the character information contained in the expression image through a pre-trained multimodal pre-training model, and the multimodal pre-training model is obtained by model training of a training image containing the expression image and the character information contained in the training image.

[0030] The expression theme information may be the information to be expressed by the expression image (which may be the user's emotional information, etc.). For example, the expression image in the input information in the example of step S102 above may be The theme message of the expression may be "broken heart".

[0031] In implementation, a correspondence between an expression image and expression theme information can be established in advance, and the correspondence can be stored in a computer device. In actual applications, there will be more and more expression images. According to the actual situation, the corresponding expression image can be selected, and the expression theme information corresponding to each expression image can be determined. Each expression image and its corresponding expression theme information can be stored in the above correspondence. Among them, there are multiple ways to determine the expression theme information corresponding to the expression image. For example, it can be determined by a corresponding model. Specifically, multiple expression images can be obtained, and then the characters contained in each expression image can be extracted by a character extraction algorithm to obtain the character information contained in each expression image. The model architecture of the multimodal pre-training model can be constructed based on a pre-set algorithm (specifically, a neural network algorithm, etc.), and the model architecture can include one or more different model parameters to be determined. In addition, the loss function of the multimodal pre-training model can be set according to the actual situation. After the above processing is completed, the multimodal pre-training model can be trained using the multiple expression images obtained above and the character information contained in each expression image, and based on the loss function set above, until the multimodal pre-training model converges, thereby obtaining a trained multimodal pre-training model. The multimodal pre-trained model obtained in the above manner can analyze the expression image in combination with the character information contained in the expression image, determine the semantic information to be expressed by the expression image, and generate the expression theme information of the expression image based on the semantic information to be expressed by the expression image.

[0032] Whenever an expression image is obtained, the character information contained in the expression image can be extracted, and the expression image and the character information contained in the expression image can be input into the above-mentioned trained multimodal pre-trained model to generate expression theme information of the expression image. Through the above method, corresponding expression theme information can be generated for multiple different expression images. Based on the expression information and the generated corresponding expression theme information, the correspondence between the expression information and the expression theme information can be constructed, and the above correspondence can be stored.

[0033] After obtaining the reply information of the target user's input information through the processing of the above step S102, the reply information can be matched with the expression theme information in the above correspondence. If the expression theme information in the above correspondence includes the reply information, or if one or more expression theme information is matched in the reply information, the expression theme information corresponding to the reply information can be obtained (it can be one or more, and the expression theme information obtained in this case is the first expression theme information), and then we can search for the expression image corresponding to the obtained first expression theme information in the above correspondence.

[0034] In step S106, the expression image corresponding to the first expression theme information is provided to the target user as a reply to the above input information.

[0035] In implementation, for example, the reply message may be such as "Don't worry, I'm here", if the pre-established correspondence between the emoticon image and the emoticon theme information includes the emoticon theme information "Don't worry, I'm here" and the corresponding emoticon image (such as Figure 3A As shown), Figure 3B As shown, it can be Figure 3A The emoticon image is sent to the target user as a reply to the above input information, and the target user can view the above emoticon image. In addition, a text message can also be sent to the target user. For example, after sending the above emoticon image, a text message such as "Don't worry, we can exchange the product for you unconditionally. Of course, you can also choose a refund. So do you want an exchange or a refund?" can be sent to the target user. The specific setting can be based on actual conditions, and the embodiments of this specification do not limit this.

[0036] It should be noted that, in actual applications, in addition to providing the corresponding expression image as a reply to the above input information to the target user, the above reply information can also be provided to the target user, that is, the above reply information and the expression image corresponding to the first expression theme information are provided to the target user as a reply to the above input information. In addition, the above reply information and the expression image corresponding to the first expression theme information can also be fused in some way to obtain fused information, such as Figure 3C As shown, the fused information can be provided to the target user as a reply to the above input information. The specific setting can be based on actual conditions, and the embodiments of this specification do not limit this.

[0037] In addition, if there is no expression theme information matching the reply information in the pre-established correspondence between expression images and expression theme information, the reply information can be directly output to the target user.

[0038] The embodiments of the present specification provide a human-computer interaction method, which obtains reply information output by a pre-trained dialogue model in response to input information of a target user. If a first expression theme information matching the reply information exists in the pre-established correspondence between expression images and expression theme information, then an expression image corresponding to the first expression theme information is obtained. The expression subject information in the correspondence is obtained by recognizing the expression image and the character information contained in the expression image through a pre-trained multimodal pre-training model. The multimodal pre-training model is obtained by model training using a training image containing the expression image and the character information contained in the training image. The expression image corresponding to the first expression theme information can be provided to the target user as a reply to the input information. In this way, in human-computer interaction, the computer device can understand the expression image sent by the user or the computer device can send a suitable expression image to the user, thereby improving the user satisfaction rate, improving the efficiency of dialogue communication in human-computer interaction, and reducing the probability of users turning to manual services.

[0039] Embodiment 2

[0040] like Figure 4 As shown, the embodiment of this specification provides a human-computer interaction method, the execution subject of the method can be a computer device, wherein the computer device can be a terminal device, a server or a corresponding machine, etc. The terminal device can be a mobile terminal device such as a mobile phone, a tablet computer, or a computer device such as a laptop or a desktop computer, or it can also be an IoT device (specifically such as a smart watch, a car device, etc.), etc. The machine can be an intelligent customer service machine of a certain business, etc., which can be set according to the actual situation. The method can specifically include the following steps:

[0041] In step S402, a pre-stored interaction log is obtained, where the interaction log includes text information, a second expression image, and expression theme information of the second expression image.

[0042] The interaction log may be a log recording information and other contents during the information interaction process between the two parties. The interaction log may include information sent by the parties, the time when the information is sent, and the information may include text information and / or emoticon images, etc. For emoticon images, the emoticon theme information of the emoticon images may also be included. The information recorded in the interaction log may be information on the interaction between any two or more parties (including human-computer interaction and interaction between people, etc.) within a certain period of time (such as within the last year or the last month, etc.).

[0043] In implementation, the interaction logs may be obtained in a variety of ways. For example, whenever a specified period is reached, the computer device may obtain the interaction logs from a certain database or multiple different databases, and the obtained interaction logs may be stored locally. Alternatively, when the dialogue model needs to be trained or updated, the computer device may obtain the interaction logs from a certain database or multiple different databases, and the obtained interaction logs may be stored locally, etc. The specific settings may be based on actual conditions. When the dialogue model needs to be trained or updated, the computer device may obtain the above-mentioned pre-stored interaction logs.

[0044] In step S404, the dialogue model is trained based on the text information in the interaction log and the expression theme information of the second expression image to obtain a trained dialogue model.

[0045] In implementation, the dialogue model can be constructed by a variety of algorithms, such as convolutional neural network algorithms, deep neural network algorithms, etc. The dialogue model can include multiple parts, such as natural language understanding, dialogue strategies, etc. The corresponding functions of the dialogue model can be constructed according to different parts, and finally the model architecture of the dialogue model can be obtained. In addition, in order to improve the efficiency of model construction, the currently commonly used dialogue model can also be directly used, such as the dialogue model based on plain text as described above.

[0046] After obtaining the model architecture of the dialogue model, we can determine the loss function corresponding to this model training, and then, Figure 5 As shown (where Tij (including T11, T12, Tt1, Tt2, etc.) represents the text information sent by the i-th user in the j-th round, Pij (including P22, P42, etc.) represents the expression theme information of the second expression image sent by the i-th user in the j-th round, and Rsp represents the predicted result, which may include text information and / or expression theme information, etc.), the text information in the above interaction log (including Figure 5 The text message input by one party and / or the text message replied by the other party) and the second emoticon image (including Figure 5 The expression theme information of the expression image input by one party and / or the expression image replied by the other party) and the above loss function are used to train the dialogue model until the dialogue model converges, and finally a trained dialogue model can be obtained.

[0047] In step S406, a training image including an expression image is obtained.

[0048] In implementation, training images containing expression images can be obtained in a variety of different ways. For example, training images containing expression images can be obtained from one or more different databases, or expression images can be crawled from the Internet using a web crawler, and the crawled expression images can be used as training images, etc. The specific settings can be based on actual conditions, and the embodiments of this specification do not limit this.

[0049] In step S408, the characters contained in the training image are recognized to obtain the character information contained in the training image.

[0050] In implementation, the characters contained in the training image can be recognized and extracted by a character extraction algorithm. Specifically, for example, the characters contained in the training image can be recognized and extracted by an OCR (Optical Character Recognition) algorithm to obtain the character information contained in the training image. Alternatively, a character extraction model can be constructed by a neural network model. After a certain model training of the character extraction model, the character extraction model can be used to recognize and extract the characters contained in the training image to obtain the character information contained in the training image, etc. The specific setting can be based on actual conditions, and the embodiments of this specification are not limited to this.

[0051] In step S410, model training is performed on the multimodal pre-trained model based on the training image containing the expression image and the character information contained in the training image to obtain a trained multimodal pre-trained model.

[0052] In implementation, the multimodal pre-training model can be constructed by a variety of algorithms, for example, it can be constructed by a convolutional neural network algorithm, a deep neural network algorithm, etc., so as to obtain the model architecture of the multimodal pre-training model. Then, the loss function of the multimodal pre-training model can be determined according to the actual situation. Then, the multimodal pre-training model can be trained using the above-mentioned training images containing expression images and the character information contained in the training images, as well as the above-mentioned loss function until the multimodal pre-training model converges. Finally, the trained multimodal pre-training model can be obtained. The multimodal pre-training model can be used to generate corresponding expression theme information for expression images.

[0053] In order to facilitate the computer device to send emoticon images to the user, the correspondence between emoticon images and emoticon theme information can be pre-constructed. Subsequently, the computer device can select a suitable emoticon image for the user to interact with the user through the correspondence, which can specifically include the processing of the following steps S412 to S418.

[0054] In step S412, a third expression image to be processed is obtained.

[0055] The third expression image may be any expression image, and the third expression image may be one or more, which may be set according to actual conditions.

[0056] In step S414, the characters included in the third expression image are recognized to obtain character information included in the third expression image.

[0057] The specific processing process of step S414 can refer to the aforementioned related content and will not be repeated here.

[0058] In step S416, the third expression image and the character information contained in the third expression image are input into the multimodal pre-training model to obtain the second expression theme information corresponding to the third expression image.

[0059] In step S418, the second expression theme information and the third expression image are stored in a correspondence relationship between the expression image and the expression theme information.

[0060] For the correspondence relationship constructed above, new expression images and corresponding expression theme information can also be added to the above correspondence relationship, and the expression images or expression theme information in the above correspondence relationship can also be modified. The following provides a method for modifying expression images. For the method for modifying expression theme information, please refer to the following content processing, which will not be repeated here. Specifically, it can include the processing of the following steps A2 to A6.

[0061] In step A2, a modification request for the above correspondence is received, where the modification request includes the image identifier of the third expression image and the fourth expression image.

[0062] In step SA4, based on the image identifier, a third expression image corresponding to the image identifier is obtained from the correspondence between the expression image and the expression theme information.

[0063] In step SA6, the third expression image is deleted from the above correspondence, and the fourth expression image and the second expression theme information are stored in the above correspondence in correspondence.

[0064] Through the above processing, a dialogue model capable of recognizing expression images, a multimodal pre-trained model for generating expression theme information for expression images, and a correspondence between expression images and expression theme information are obtained respectively. The subsequent computer device can interact with the target user based on the above models and correspondence. For details, please refer to the processing of the following steps S420 to S430.

[0065] In step S420, the response information output by the pre-trained dialogue model in response to the input information of the target user is obtained.

[0066] In step S422, the reply information is segmented to obtain one or more segmented words contained in the reply information.

[0067] In implementation, the above-mentioned reply information can be segmented in a variety of different ways. For example, the above-mentioned reply information can be segmented by a pre-set word segmentation algorithm, wherein the word segmentation algorithm can be based on a dictionary, a word library matching word segmentation algorithm, a word segmentation algorithm based on word frequency statistics, a word segmentation algorithm based on preset rules, etc. In addition, the above-mentioned reply information can also be segmented by a pre-trained word segmentation model, wherein the word segmentation model can be a hidden Markov model, a conditional random field (CRF) model, etc., which can be specifically set according to actual conditions, and the embodiments of this specification do not limit this.

[0068] In step S424, each segmented word is matched with the expression theme information in the pre-established correspondence between the expression image and the expression theme information, and the segmented words existing in the correspondence between the expression image and the expression theme information are determined.

[0069] In step S426, the expression theme information corresponding to the determined word segment is determined as the first expression theme information matching the reply information.

[0070] In step S428, an expression image corresponding to the first expression theme information is obtained.

[0071] In step S430, corresponding participles in the reply information are replaced with emoticon images corresponding to the first emoticon theme information to obtain replaced reply information, and the replaced reply information is provided to the target user as a reply to the input information.

[0072] The human-computer interaction system of computer equipment constructed in the above way was tested in Track 1 of the Dialogue System Technology Challenge (DSTC-10), and achieved good results in all 6 test sets of three subtasks.

[0073] The embodiments of the present specification provide a human-computer interaction method, which obtains reply information output by a pre-trained dialogue model in response to input information of a target user. If a first expression theme information matching the reply information exists in the pre-established correspondence between expression images and expression theme information, then an expression image corresponding to the first expression theme information is obtained. The expression subject information in the correspondence is obtained by recognizing the expression image and the character information contained in the expression image through a pre-trained multimodal pre-training model. The multimodal pre-training model is obtained by model training using a training image containing the expression image and the character information contained in the training image. The expression image corresponding to the first expression theme information can be provided to the target user as a reply to the input information. In this way, in human-computer interaction, the computer device can understand the expression image sent by the user or the computer device can send a suitable expression image to the user, thereby improving the user satisfaction rate, improving the efficiency of dialogue communication in human-computer interaction, and reducing the probability of users turning to manual services.

[0074] Embodiment 3

[0075] The above is a human-computer interaction method provided in the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide a human-computer interaction device, such as Figure 6 shown.

[0076] The human-computer interaction device includes: a reply information acquisition module 601, an expression determination module 602 and an expression reply module 603, wherein:

[0077] A reply information acquisition module 601 is used to acquire reply information output by a pre-trained dialogue model in response to input information of a target user;

[0078] The expression determination module 602 obtains an expression image corresponding to the first expression theme information if there is a first expression theme information matching the reply information in the pre-established correspondence relationship between the expression image and the expression theme information, wherein the expression subject information in the correspondence relationship is obtained by recognizing the expression image and the character information contained in the expression image through a pre-trained multimodal pre-training model, wherein the multimodal pre-training model is obtained by model training using a training image containing the expression image and the character information contained in the training image;

[0079] The expression reply module 603 provides the expression image corresponding to the first expression theme information to the target user as a reply to the input information.

[0080] In the embodiment of this specification, the device further includes:

[0081] A log acquisition module, which acquires a pre-stored interaction log, wherein the interaction log includes text information, a second expression image, and expression theme information of the second expression image;

[0082] The first model training module performs model training on the dialogue model based on the text information in the interaction log and the expression theme information of the second expression image to obtain a trained dialogue model.

[0083] In the embodiment of this specification, the device further includes:

[0084] A training image acquisition module, which acquires a training image including an expression image;

[0085] A first character recognition module recognizes characters contained in the training image to obtain character information contained in the training image;

[0086] The second model training module performs model training on the multimodal pre-training model based on the training image containing the expression image and the character information contained in the training image to obtain a trained multimodal pre-training model.

[0087] In the embodiment of this specification, the device further includes:

[0088] An expression image acquisition module is used to acquire a third expression image to be processed;

[0089] A second character recognition module recognizes characters contained in the third expression image to obtain character information contained in the third expression image;

[0090] An expression theme determination module, inputting the third expression image and character information contained in the third expression image into the multimodal pre-training model to obtain second expression theme information corresponding to the third expression image;

[0091] The information storage module stores the second expression theme information and the third expression image in correspondence with each other in the correspondence relationship between the expression image and the expression theme information.

[0092] In the embodiment of this specification, the device further includes:

[0093] A modification request module receives a modification request for the corresponding relationship, wherein the modification request includes an image identifier of the third expression image and a fourth expression image;

[0094] An image acquisition module, based on the image identifier, acquires the third expression image corresponding to the image identifier from the correspondence between the expression image and the expression theme information;

[0095] The adjustment module deletes the third expression image from the corresponding relationship, and stores the fourth expression image and the second expression theme information in the corresponding relationship in correspondence.

[0096] In the embodiment of the present specification, the expression reply module 603 provides the reply information and the expression image corresponding to the first expression theme information to the target user as a reply to the input information.

[0097] In the embodiment of this specification, the device further includes:

[0098] A word segmentation module performs word segmentation processing on the reply information to obtain one or more word segments contained in the reply information;

[0099] A matching module matches each of the segmented words with the expression theme information in the pre-established correspondence between the expression image and the expression theme information, and determines the segmented words existing in the correspondence between the expression image and the expression theme information;

[0100] A first expression theme determination module determines the expression theme information corresponding to the determined word segment as the first expression theme information matching the reply information;

[0101] The expression reply module 603 uses the expression images corresponding to the first expression theme information to replace the corresponding participles in the reply information to obtain replaced reply information, and provides the replaced reply information to the target user as a reply to the input information.

[0102] The embodiment of the present specification provides a human-computer interaction device, which obtains reply information output by a pre-trained dialogue model in response to input information of a target user. If a first expression theme information matching the reply information exists in the pre-established correspondence between expression images and expression theme information, then an expression image corresponding to the first expression theme information is obtained. The expression subject information in the correspondence is obtained by recognizing the expression image and the character information contained in the expression image through a pre-trained multimodal pre-training model. The multimodal pre-training model is obtained by model training using a training image containing the expression image and the character information contained in the training image. The expression image corresponding to the first expression theme information can be provided to the target user as a reply to the input information. In this way, in human-computer interaction, the computer device can understand the expression image sent by the user or the computer device can send a suitable expression image to the user, thereby improving the user satisfaction rate, improving the efficiency of dialogue communication in human-computer interaction, and reducing the probability of users turning to manual services.

[0103] Embodiment 4

[0104] The above is a human-computer interaction device provided in the embodiment of this specification. Based on the same idea, the embodiment of this specification also provides a human-computer interaction device, such as Figure 7 shown.

[0105] The human-computer interaction device may be the computer device provided in the above embodiment, etc.

[0106] The human-computer interaction device may have relatively large differences due to different configurations or performances, and may include one or more processors 701 and memory 702, and the memory 702 may store one or more storage applications or data. Among them, the memory 702 may be a short-term storage or a persistent storage. The application stored in the memory 702 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the human-computer interaction device. Furthermore, the processor 701 may be configured to communicate with the memory 702 to execute a series of computer executable instructions in the memory 702 on the human-computer interaction device. The human-computer interaction device may also include one or more power supplies 703, one or more wired or wireless network interfaces 704, one or more input and output interfaces 705, and one or more keyboards 706.

[0107] Specifically in this embodiment, the human-computer interaction device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions in the human-computer interaction device, and the one or more programs are configured to be executed by one or more processors, including the following computer executable instructions:

[0108] Obtain the response information output by the pre-trained dialogue model in response to the input information of the target user;

[0109] If there is first expression theme information matching the reply information in the pre-established correspondence between expression images and expression theme information, then obtaining an expression image corresponding to the first expression theme information, wherein the expression subject information in the correspondence is obtained by recognizing the expression image and the character information contained in the expression image through a pre-trained multimodal pre-training model, wherein the multimodal pre-training model is obtained by model training using a training image containing the expression image and the character information contained in the training image;

[0110] The expression image corresponding to the first expression theme information is provided to the target user as a reply to the input information.

[0111] In the embodiment of this specification, it also includes:

[0112] Acquire a pre-stored interaction log, wherein the interaction log includes text information, a second expression image, and expression theme information of the second expression image;

[0113] Model training is performed on the dialogue model based on the text information in the interaction log and the expression theme information of the second expression image to obtain a trained dialogue model.

[0114] In the embodiment of this specification, it also includes:

[0115] Obtain training images containing expression images;

[0116] Recognizing characters contained in the training image to obtain character information contained in the training image;

[0117] Based on the training image containing the expression image and the character information contained in the training image, the multimodal pre-training model is trained to obtain a trained multimodal pre-training model.

[0118] In the embodiment of this specification, it also includes:

[0119] Obtaining a third expression image to be processed;

[0120] Recognize characters contained in the third expression image to obtain character information contained in the third expression image;

[0121] Inputting the third expression image and character information contained in the third expression image into the multimodal pre-training model to obtain second expression theme information corresponding to the third expression image;

[0122] The second expression theme information and the third expression image are stored in correspondence with each other in the correspondence relationship between the expression image and the expression theme information.

[0123] In the embodiment of this specification, it also includes:

[0124] receiving a modification request for the corresponding relationship, wherein the modification request includes an image identifier of the third expression image and a fourth expression image;

[0125] Based on the image identifier, acquiring the third expression image corresponding to the image identifier from the correspondence between the expression image and the expression theme information;

[0126] The third expression image is deleted from the corresponding relationship, and the fourth expression image and the second expression theme information are stored in the corresponding relationship in correspondence.

[0127] In the embodiment of the specification, providing the emoticon image corresponding to the first emoticon theme information as a reply to the input information to the target user includes:

[0128] The reply information and the expression image corresponding to the first expression theme information are provided to the target user as a reply to the input information.

[0129] In the embodiment of this specification, it also includes:

[0130] Performing word segmentation processing on the reply information to obtain one or more word segments contained in the reply information;

[0131] Respectively matching each of the segmented words with the expression theme information in the pre-established correspondence relationship between the expression image and the expression theme information, and determining the segmented words existing in the correspondence relationship between the expression image and the expression theme information;

[0132] Determining the expression theme information corresponding to the determined word segment as the first expression theme information matching the reply information;

[0133] The step of providing the target user with the expression image corresponding to the first expression theme information as a reply to the input information includes:

[0134] The corresponding word segments in the reply information are replaced respectively with the emoticon images corresponding to the first emoticon theme information to obtain replaced reply information, and the replaced reply information is provided to the target user as a reply to the input information.

[0135] The embodiment of the present specification provides a human-computer interaction device, which obtains reply information output by a pre-trained dialogue model in response to input information of a target user. If a first expression theme information matching the reply information exists in the pre-established correspondence between expression images and expression theme information, then an expression image corresponding to the first expression theme information is obtained. The expression subject information in the correspondence is obtained by recognizing the expression image and the character information contained in the expression image through a pre-trained multimodal pre-training model. The multimodal pre-training model is obtained by model training using a training image containing the expression image and the character information contained in the training image. The expression image corresponding to the first expression theme information can be provided to the target user as a reply to the input information. In this way, in human-computer interaction, the computer device can understand the expression image sent by the user or the computer device can send a suitable expression image to the user, thereby improving the user satisfaction rate, improving the efficiency of dialogue communication in human-computer interaction, and reducing the probability of users turning to manual services.

[0136] Embodiment 5

[0137] Furthermore, based on the above Figure 1A to Figure 5 In one embodiment, the present specification further provides a storage medium for storing computer executable instruction information. In a specific embodiment, the storage medium may be a USB flash drive, an optical disk, a hard disk, etc. When the computer executable instruction information stored in the storage medium is executed by the processor, the following process can be implemented:

[0138] Obtain the response information output by the pre-trained dialogue model in response to the input information of the target user;

[0139] If there is first expression theme information matching the reply information in the pre-established correspondence between expression images and expression theme information, then obtaining an expression image corresponding to the first expression theme information, wherein the expression subject information in the correspondence is obtained by recognizing the expression image and the character information contained in the expression image through a pre-trained multimodal pre-training model, wherein the multimodal pre-training model is obtained by model training using a training image containing the expression image and the character information contained in the training image;

[0140] The expression image corresponding to the first expression theme information is provided to the target user as a reply to the input information.

[0141] In the embodiment of this specification, it also includes:

[0142] Acquire a pre-stored interaction log, wherein the interaction log includes text information, a second expression image, and expression theme information of the second expression image;

[0143] Model training is performed on the dialogue model based on the text information in the interaction log and the expression theme information of the second expression image to obtain a trained dialogue model.

[0144] In the embodiment of this specification, it also includes:

[0145] Obtain training images containing expression images;

[0146] Recognizing characters contained in the training image to obtain character information contained in the training image;

[0147] Based on the training image containing the expression image and the character information contained in the training image, the multimodal pre-training model is trained to obtain a trained multimodal pre-training model.

[0148] In the embodiment of this specification, it also includes:

[0149] Obtaining a third expression image to be processed;

[0150] Recognize characters contained in the third expression image to obtain character information contained in the third expression image;

[0151] Inputting the third expression image and character information contained in the third expression image into the multimodal pre-training model to obtain second expression theme information corresponding to the third expression image;

[0152] The second expression theme information and the third expression image are stored in correspondence with each other in the correspondence relationship between the expression image and the expression theme information.

[0153] In the embodiment of this specification, it also includes:

[0154] receiving a modification request for the corresponding relationship, wherein the modification request includes an image identifier of the third expression image and a fourth expression image;

[0155] Based on the image identifier, acquiring the third expression image corresponding to the image identifier from the correspondence between the expression image and the expression theme information;

[0156] The third expression image is deleted from the corresponding relationship, and the fourth expression image and the second expression theme information are stored in the corresponding relationship in correspondence.

[0157] In the embodiment of the specification, providing the emoticon image corresponding to the first emoticon theme information as a reply to the input information to the target user includes:

[0158] The reply information and the expression image corresponding to the first expression theme information are provided to the target user as a reply to the input information.

[0159] In the embodiment of this specification, it also includes:

[0160] Performing word segmentation processing on the reply information to obtain one or more word segments contained in the reply information;

[0161] Respectively matching each of the segmented words with the expression theme information in the pre-established correspondence relationship between the expression image and the expression theme information, and determining the segmented words existing in the correspondence relationship between the expression image and the expression theme information;

[0162] Determining the expression theme information corresponding to the determined word segment as the first expression theme information matching the reply information;

[0163] The step of providing the target user with the expression image corresponding to the first expression theme information as a reply to the input information includes:

[0164] The corresponding word segments in the reply information are replaced respectively with the emoticon images corresponding to the first emoticon theme information to obtain replaced reply information, and the replaced reply information is provided to the target user as a reply to the input information.

[0165] The embodiment of the present specification provides a storage medium, which obtains reply information output by a pre-trained dialogue model in response to input information of a target user. If there is a first expression theme information matching the reply information in the pre-established correspondence between expression images and expression theme information, then the expression image corresponding to the first expression theme information is obtained. The expression subject information in the correspondence is obtained by recognizing the expression image and the character information contained in the expression image through a pre-trained multimodal pre-training model. The multimodal pre-training model is obtained by model training using a training image containing the expression image and the character information contained in the training image. The expression image corresponding to the first expression theme information can be provided to the target user as a reply to the input information. In this way, in human-computer interaction, the computer device can understand the expression image sent by the user or the computer device can send a suitable expression image to the user, thereby improving the user satisfaction rate, improving the efficiency of dialogue communication in human-computer interaction, and reducing the probability of users turning to manual services.

[0166] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0167] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0168] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0169] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0170] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0171] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0172] The embodiments of this specification are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable fraud case serial and parallel device to produce a machine, so that the instructions executed by the processor of the computer or other programmable fraud case serial and parallel device produce a device for implementing the function specified in one process or multiple processes of the flowchart and / or one box or multiple boxes of the block diagram.

[0173] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable fraud case serial and parallel device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0174] These computer program instructions may also be loaded onto a computer or other programmable fraud case serial and parallel device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows of the flowchart and / or one or more blocks of the block diagram.

[0175] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0176] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0177] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0178] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0179] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0180] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0181] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0182] The above description is only an embodiment of this specification and is not intended to limit the present application. For those skilled in the art, this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification should be included in the scope of the claims of this specification.

Claims

1. A human-computer interaction method, the method comprising: Obtain the response information output by the pre-trained dialogue model in response to the input information of the target user; If there is first expression theme information matching the reply information in the pre-established correspondence between expression images and expression theme information, then obtaining an expression image corresponding to the first expression theme information, wherein the expression subject information in the correspondence is obtained by recognizing the expression image and the character information contained in the expression image through a pre-trained multimodal pre-training model, wherein the multimodal pre-training model is obtained by training a training image containing the expression image and the character information contained in the training image, and the expression theme information is information to be expressed by the expression image, and the expression theme information contains the user's emotional information; providing the target user with an expression image corresponding to the first expression theme information as a reply to the input information; The method comprises: Performing word segmentation processing on the reply information to obtain one or more word segments contained in the reply information; Respectively matching each of the segmented words with the expression theme information in the pre-established correspondence relationship between the expression image and the expression theme information, and determining the segmented words existing in the correspondence relationship between the expression image and the expression theme information; The expression theme information corresponding to the determined word segment is determined as the first expression theme information matching the reply information.

2. The method according to claim 1, further comprising: Acquire a pre-stored interaction log, wherein the interaction log includes text information, a second expression image, and expression theme information of the second expression image; Model training is performed on the dialogue model based on the text information in the interaction log and the expression theme information of the second expression image to obtain a trained dialogue model.

3. The method according to claim 1, further comprising: Obtain training images containing expression images; Recognizing characters contained in the training image to obtain character information contained in the training image; Based on the training image containing the expression image and the character information contained in the training image, the multimodal pre-training model is trained to obtain a trained multimodal pre-training model.

4. The method according to claim 3, further comprising: Obtaining a third expression image to be processed; Recognize characters contained in the third expression image to obtain character information contained in the third expression image; Inputting the third expression image and character information contained in the third expression image into the multimodal pre-training model to obtain second expression theme information corresponding to the third expression image; The second expression theme information and the third expression image are stored in correspondence with each other in the correspondence relationship between the expression image and the expression theme information.

5. The method according to claim 4, further comprising: receiving a modification request for the corresponding relationship, wherein the modification request includes an image identifier of the third expression image and a fourth expression image; Based on the image identifier, acquiring the third expression image corresponding to the image identifier from the correspondence between the expression image and the expression theme information; The third expression image is deleted from the corresponding relationship, and the fourth expression image and the second expression theme information are stored in the corresponding relationship in correspondence.

6. The method according to claim 4, wherein providing the emoticon image corresponding to the first emoticon theme information as a reply to the input information to the target user comprises: The reply information and the expression image corresponding to the first expression theme information are provided to the target user as a reply to the input information.

7. The method according to claim 4, wherein providing the emoticon image corresponding to the first emoticon theme information as a reply to the input information to the target user comprises: The corresponding word segments in the reply information are replaced respectively with the emoticon images corresponding to the first emoticon theme information to obtain replaced reply information, and the replaced reply information is provided to the target user as a reply to the input information.

8. A human-computer interaction device, comprising: A reply information acquisition module obtains the reply information output by the pre-trained dialogue model in response to the input information of the target user; The expression determination module obtains an expression image corresponding to the first expression theme information if there is first expression theme information matching the reply information in the pre-established correspondence between the expression image and the expression theme information, wherein the expression subject information in the correspondence is obtained by recognizing the expression image and the character information contained in the expression image through a pre-trained multimodal pre-training model, wherein the multimodal pre-training model is obtained by model training using a training image containing the expression image and the character information contained in the training image, and the expression theme information is information to be expressed by the expression image, and the expression theme information contains the user's emotional information; An expression reply module, providing the expression image corresponding to the first expression theme information as a reply to the input information to the target user; A word segmentation module performs word segmentation processing on the reply information to obtain one or more word segments contained in the reply information; A matching module matches each of the segmented words with the expression theme information in the pre-established correspondence between the expression image and the expression theme information, and determines the segmented words existing in the correspondence between the expression image and the expression theme information; The first expression theme determination module determines the expression theme information corresponding to the determined word segment as the first expression theme information matching the reply information.

9. A human-computer interaction device, comprising: processor; as well as a memory arranged to store computer executable instructions which, when executed, cause the processor to: Obtain the response information output by the pre-trained dialogue model in response to the input information of the target user; If there is first expression theme information matching the reply information in the pre-established correspondence between expression images and expression theme information, then obtaining an expression image corresponding to the first expression theme information, wherein the expression subject information in the correspondence is obtained by recognizing the expression image and the character information contained in the expression image through a pre-trained multimodal pre-training model, wherein the multimodal pre-training model is obtained by training a training image containing the expression image and the character information contained in the training image, and the expression theme information is information to be expressed by the expression image, and the expression theme information contains the user's emotional information; providing the target user with an expression image corresponding to the first expression theme information as a reply to the input information; Performing word segmentation processing on the reply information to obtain one or more word segments contained in the reply information; Respectively matching each of the segmented words with the expression theme information in the pre-established correspondence relationship between the expression image and the expression theme information, and determining the segmented words existing in the correspondence relationship between the expression image and the expression theme information; The expression theme information corresponding to the determined word segment is determined as the first expression theme information matching the reply information.

10. A storage medium for storing computer executable instructions, wherein the executable instructions implement the following process when executed by a processor: Obtain the response information output by the pre-trained dialogue model in response to the input information of the target user; If there is first expression theme information matching the reply information in the pre-established correspondence between expression images and expression theme information, then obtaining an expression image corresponding to the first expression theme information, wherein the expression subject information in the correspondence is obtained by recognizing the expression image and the character information contained in the expression image through a pre-trained multimodal pre-training model, wherein the multimodal pre-training model is obtained by training a training image containing the expression image and the character information contained in the training image, and the expression theme information is information to be expressed by the expression image, and the expression theme information contains the user's emotional information; providing the target user with an expression image corresponding to the first expression theme information as a reply to the input information; Performing word segmentation processing on the reply information to obtain one or more word segments contained in the reply information; Respectively matching each of the segmented words with the expression theme information in the pre-established correspondence relationship between the expression image and the expression theme information, and determining the segmented words existing in the correspondence relationship between the expression image and the expression theme information; The expression theme information corresponding to the determined word segment is determined as the first expression theme information matching the reply information.