Dialogue method and device

Abstract

The embodiments of the present specification provide a conversation method and device, wherein the conversation method includes: obtaining a language feature vector of a user's conversation information; obtaining an image feature vector of an image corresponding to the conversation information, wherein the image feature vector is a merged vector of feature vectors of several proposed regions of the image in a spatial dimension; performing feature fusion on the language feature vector and the image feature vector based on an attention mechanism to obtain a fused feature vector; and calculating candidate replies based on the fused feature vector to obtain candidate replies that meet the requirements.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of intelligent technology, and more particularly to a dialogue method. One or more embodiments of this specification also relate to a dialogue device, a computing device, and a computer-readable storage medium. Background Art

[0002] In some conversational products, given the user's conversation history and current question, selecting the correct answer from a set of answer candidates is a common conversational task. Currently, this is done by simply performing classification calculations on the language model and selecting a reply.

[0003] However, since people focus on different things when asking questions, the responses from the classification choices may not be accurate. Summary of the invention

[0004] In view of this, the embodiments of this specification provide a conversation method. One or more embodiments of this specification also relate to a conversation device, a computing device, and a computer-readable storage medium to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of the present specification, a conversation method is provided, including: obtaining a language feature vector of a user's conversation information; obtaining an image feature vector of an image corresponding to the conversation information, wherein the image feature vector is a merged vector of feature vectors of several proposed regions of the image in a spatial dimension; performing feature fusion on the language feature vector and the image feature vector based on an attention mechanism to obtain a fused feature vector; and calculating candidate replies based on the fused feature vector to obtain candidate replies that meet the requirements.

[0006] Optionally, the calculating of candidate replies based on the fused feature vector to obtain candidate replies that meet the requirements includes: acquiring a language feature vector of the user's historical conversation information; and calculating candidate replies based on the language feature vector of the historical conversation information and the fused feature vector to obtain candidate replies that meet the requirements.

[0007] Optionally, the calculation of the candidate replies includes: subjecting the feature vectors involved in the calculation of the candidate replies to a full connection operation to obtain joint features of various dimensions; inputting the joint features of various dimensions into a decoder based on likelihood value scoring to obtain candidate replies with scores that meet the requirements.

[0008] Optionally, the feature fusion of the language feature vector and the image feature vector based on the attention mechanism includes: performing dot product of the language feature vector and the image feature vector to obtain a similarity vector; fully connecting the similarity vector to obtain weight coefficients of each dimension; performing weighted summation with the language feature vector using the weight coefficient to obtain the language feature vector in the fused feature vector; performing weighted summation with the image feature vector using the weight coefficient to obtain the image feature vector in the fused feature vector.

[0009] Optionally, the method further includes: receiving current round dialogue information input by a user through a dialogue interface; and determining an image corresponding to the current round dialogue information through an image displayed on the dialogue interface.

[0010] Optionally, the method further includes: if the images corresponding to the current round of dialogue information are multiple images, splicing the multiple images into one image.

[0011] According to the second aspect of the embodiments of this specification, there is provided a conversation device, comprising: a language feature acquisition module, configured to acquire a language feature vector of a user's conversation information. An image feature acquisition module, configured to acquire an image feature vector of an image corresponding to the conversation information, wherein the image feature vector is a merged vector of feature vectors of several proposed regions of the image in a spatial dimension. An attention module, configured to perform feature fusion on the language feature vector and the image feature vector based on an attention mechanism to obtain a fused feature vector. A reply calculation module, configured to calculate candidate replies based on the fused feature vector to obtain candidate replies that meet the requirements.

[0012] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions: obtaining a language feature vector of a user's conversation information; obtaining an image feature vector of an image corresponding to the conversation information, wherein the image feature vector is a merged vector of feature vectors of several proposed regions of the image in a spatial dimension; performing feature fusion on the language feature vector and the image feature vector based on an attention mechanism to obtain a fused feature vector; and calculating candidate replies based on the fused feature vector to obtain candidate replies that meet the requirements.

[0013] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the dialogue method described in any embodiment of this specification are implemented.

[0014] An embodiment of the present specification provides a conversation method. Since the method obtains the language feature vector of the user's conversation information, obtains the image feature vector of the image corresponding to the conversation information, and the image feature vector is a merged vector of the feature vectors of several proposed regions of the image in the spatial dimension, the language feature vector and the image feature vector are feature-fused based on the attention mechanism to obtain a fused feature vector. Therefore, the user's language features can be fused with the features of different regions of the image to obtain a visual fusion feature related to the language without being disturbed by background information or other information irrelevant to the question. Further, the candidate replies are calculated based on the fused feature vector to obtain candidate replies that meet the requirements. It can be seen that the method provided in the embodiment of the present specification selects the visual area through the multimodal attention mechanism of the conversation information perception, so that the conversation information can be fused with the features of different regions of the image, capturing the focus of the user's conversation, and selecting the candidate replies based on the fused features, which can provide a more accurate visual conversation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of a conversation method provided by an embodiment of this specification;

[0016] Figure 2 It is a schematic diagram of a model structure provided by an embodiment of this specification;

[0017] Figure 3 is a schematic diagram of a model structure provided by another embodiment of this specification;

[0018] Figure 4 This is a schematic diagram of effect comparison provided by an embodiment of this specification;

[0019] Figure 5 is a module structure diagram of an attention mechanism provided by an embodiment of this specification;

[0020] Figure 6 This is a diagram of a dialogue interface provided by an embodiment of this specification;

[0021] Figure 7 is a schematic diagram of the structure of a dialogue device provided by an embodiment of this specification;

[0022] Figure 8 is a structural diagram of a dialogue device provided by another embodiment of the present specification;

[0023] Fig. 9 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0024] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0025] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0026] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0027] In this specification, a conversation method is provided. This specification also relates to a conversation device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0028] Figure 1 A flowchart of a dialogue method provided according to an embodiment of the present specification is shown, including steps 102 to 108.

[0029] Step 102: Obtain the language feature vector of the user's conversation information.

[0030] For example, the LSTM language model can be used to model the data to obtain the language feature vector of the conversation information. The user's current round of conversation information is input into the language feature extraction model to obtain the language feature vector of the current round of conversation information;

[0031] Step 104: Obtain an image feature vector of the image corresponding to the dialogue information, where the image feature vector is a merged vector of feature vectors of several proposed regions of the image in a spatial dimension.

[0032] For example, the image corresponding to the current round of dialogue information can be input into an image feature extraction model based on a region proposal network to obtain an image feature vector. For example, Faster-RCNN can be used to extract 36 proposals (proposed regions) from a picture, and the 36 proposed region features are added together in the spatial dimension to form a proposed region feature to obtain an image feature vector.

[0033] Step 106: Fusing the language feature vector and the image feature vector based on an attention mechanism to obtain a fused feature vector.

[0034] Step 108: Calculate candidate responses based on the fused feature vector to obtain candidate responses that meet the requirements.

[0035] Since the method obtains the language feature vector of the user's dialogue information, obtains the image feature vector of the image corresponding to the dialogue information, and the image feature vector is a merged vector of the feature vectors of several proposed regions of the image in the spatial dimension, the language feature vector and the image feature vector are feature-fused based on the attention mechanism to obtain a fused feature vector. Therefore, the user's language features can be fused with the features of different regions of the image to obtain visual fusion features related to the language without being disturbed by background information or other information irrelevant to the question. Further, the candidate replies are calculated based on the fused feature vector to obtain candidate replies that meet the requirements. It can be seen that the method provided in the embodiment of this specification selects the visual area by using the dialogue information perception multimodal attention mechanism, so that the dialogue information can be fused with the features of different regions of the image, capturing the focus of the user's dialogue, and selecting candidate replies based on the fused features, which can provide a more accurate visual dialogue.

[0036] Human beings perceive the world in a "multimodal" rather than a "unimodal" way, and learning knowledge is also carried out under multimodal conditions. Among all the perceptual modalities, vision and text (language) are two important categories. Converting visually perceived content into text information is an important way for people to understand the world. In the multimodal visual dialogue task, the method provided in the embodiment of this specification can answer questions for users based on the content of the picture for images (such as visual content such as videos and static images) and questions input by users. For example, the system conducts the next round of dialogue based on the picture and the current or even historical dialogue. In this process, the system finds the answer to the question for the user. The user can ask several questions in succession, and the system has an answer for each question to achieve the goal of visual dialogue. According to the method provided in the embodiment of this specification, the selection of image areas can be modeled based on the question, so that when the modal is fused, different questions can fuse the features of different regions of the image, thereby achieving better dialogue effects.

[0037] The method provided in the embodiments of this specification does not limit the calculation method of candidate replies. For example, in one or more embodiments, the calculation of candidate replies based on the fused feature vector to obtain candidate replies that meet the requirements may include: obtaining the language feature vector of the user's historical conversation information; calculating the candidate replies based on the language feature vector of the historical conversation information and the fused feature vector to obtain candidate replies that meet the requirements.

[0038] When calculating candidate replies, the specific implementation of the decoder used by the model can be set according to the needs of the implementation scenario. For example, in one or more embodiments, the calculation of the candidate replies may include: subjecting the feature vectors involved in the calculation of the candidate replies to a full connection operation to obtain joint features of each dimension; inputting the joint features of each dimension into a decoder based on likelihood value scoring to obtain candidate replies with scores that meet the requirements. Among them, each dimension refers to each dimension corresponding to each candidate reply. For example, when training the model, the maximum likelihood value of the true answer can be maximized. During testing, each candidate reply is sorted based on log likelihood.

[0039] For example, when the feature vector involved in the candidate reply calculation is the fused feature vector, the fused feature vector can be subjected to a full connection operation to obtain joint features of each dimension. Figure 2 The structural diagram of a model is shown in FIG. 1 , which uses the joint representation of QI for encoding and performs feature fusion through the attention mechanism. Where Q is the user's conversation information, for example, the conversation information can be a question raised by the user; I is an image. The Q language feature vector and I image feature vector are input Figure 2 The attention module "attention" shown in the figure obtains a fused feature vector. The fused feature vector passes through the fully connected layer before the "Decoder" to obtain the joint features of each dimension. The joint features are input into the Decoder decoder to obtain the candidate responses that meet the requirements.

[0040] For another example, when the feature vectors involved in the candidate reply calculation include the fused feature vector and the language feature vector of the historical conversation information, the fused feature vector and the language feature vector of the historical conversation information may be subjected to a full connection operation to obtain joint features of each dimension. Figure 3As shown in the structural diagram of another model shown in the figure, the joint representation of QIH is used for encoding. Among them, H is the historical dialogue information, and H can be regarded as a sentence for encoding. The language feature vector of H is obtained through LSTM. The Q language feature vector and the I image feature vector enter the attention module "attention" to obtain the fused feature vector. The language feature vector and the fused feature vector of H pass through the fully connected layer before "Decoder" to obtain the joint features of each dimension, and the joint features are input into the Decoder decoder to obtain the candidate response that meets the requirements.

[0041] Next, the attention mechanism of the method provided in the embodiment of this specification is described in detail. For example, the feature fusion of the language feature vector and the image feature vector based on the attention mechanism may include: performing a dot product of the language feature vector and the image feature vector to obtain a similarity vector; passing the similarity vector through a fully connected layer to obtain weight coefficients of each dimension; performing a weighted summation with the language feature vector using the weight coefficient to obtain a language feature vector in the fused feature vector; performing a weighted summation with the image feature vector using the weight coefficient to obtain an image feature vector in the fused feature vector. Through the attention mechanism of this embodiment, the language features of the dialogue information are used to select the visual area in the image, and the questions raised by the user can be fused with the features of different regions of the image without being disturbed by background information or other information irrelevant to the question.

[0042] By comparing with other modeling methods, QI+attention mechanism and QIH+attention mechanism achieve better dialogue effect. For example, other methods include: Q, encoding method using only Q information; QH: encoding method using joint representation of QH; QI: encoding method using joint representation of QI; QIH: encoding method using joint representation of QIH. Figure 4 The effect comparison diagram shown in the analysis shows that the effect of late fusion (QI plus attention mechanism) has significantly improved various indicators compared to late fusion (QI) (such as NDCG: +4.5%; MRR: +3.6%), and the result of (QIH plus attention mechanism) has significantly improved various indicators compared to QIH (NDCG: +4%; MRR: +4.1%).

[0043] For ease of calculation, for example, Figure 5 The module structure diagram of the attention mechanism is shown in the figure. In the attention mechanism, the similarity vector can be passed through two fully connected layers respectively to obtain two sets of weight systems of each dimension. One set of weight coefficients is weighted summed with the language feature vector to obtain the weighted language feature vector in the fused feature vector, and the other set of weight coefficients is weighted summed with the image feature vector to obtain the weighted image feature vector in the fused feature vector.

[0044] As Figure 6 Taking the dialogue interface shown in the figure as an example, the method may further include: receiving the current round of dialogue information input by the user through the dialogue interface; and determining the image corresponding to the current round of dialogue information through the image displayed on the dialogue interface. Figure 6 As shown, a question area for users to ask questions in the current round can be specially provided in the dialogue interface, and the corresponding image can be displayed at a preset position in the question area. For another example, an image selection operation method can also be provided to allow users to select images in other applications or other interfaces to establish a corresponding relationship between dialogue information and images. The method provided in the embodiments of this specification is not limited to this. In addition, if the image corresponding to the current round of dialogue information is multiple images, the multiple images can be spliced ​​into one image, and the spliced ​​image can be used in the operation of visual dialogue.

[0045] Corresponding to the above method embodiment, this specification also provides a dialogue device embodiment, Figure 7 FIG. 1 shows a schematic diagram of the structure of a dialogue device provided by an embodiment of the present specification. Figure 7 As shown, the device includes: a language feature extraction module 702, an image feature extraction module 704, an attention module 706 and a reply calculation module 708.

[0046] The language feature extraction module 702 may be configured to obtain a language feature vector of the user's conversation information.

[0047] The image feature extraction module 704 may be configured to obtain an image feature vector of the image corresponding to the dialog information, where the image feature vector is a merged vector of feature vectors of several proposed regions of the image in a spatial dimension.

[0048] The attention module 706 may be configured to perform feature fusion on the language feature vector and the image feature vector based on an attention mechanism to obtain a fused feature vector.

[0049] The reply calculation module 708 may be configured to calculate candidate replies based on the fused feature vector to obtain candidate replies that meet the requirements.

[0050] Since the device obtains the language feature vector of the user's dialogue information and the image feature vector of the image corresponding to the dialogue information, the image feature vector is a merged vector of the feature vectors of several proposed regions of the image in the spatial dimension, and the language feature vector and the image feature vector are feature fused based on the attention mechanism to obtain a fused feature vector, the user's language features can be fused with the features of different regions of the image to obtain a visual fusion feature related to the language without being disturbed by background information or other information irrelevant to the question. Further, the candidate replies are calculated based on the fused feature vector to obtain candidate replies that meet the requirements. It can be seen that the device provided in the embodiment of this specification selects feature regions by using the dialogue information perception multimodal attention mechanism, so that the dialogue information can be fused with different feature regions of the image, capturing the focus of the user's dialogue, and selecting candidate replies based on the fused features, which can provide a more accurate visual dialogue.

[0051] Figure 8 FIG. 2 shows a schematic diagram of the structure of a dialogue device provided by another embodiment of the present specification. Figure 8 As shown, the device can calculate candidate replies in combination with the user's historical conversations. Specifically, for example, the reply calculation module 708 may include:

[0052] The historical feature acquisition submodule 7082 may be configured to acquire language feature vectors of the user's historical conversation information.

[0053] The reply calculation submodule 7084 may be configured to calculate candidate replies based on the language feature vector of the historical conversation information and the fusion feature vector to obtain candidate replies that meet the requirements.

[0054] When calculating candidate responses, the specific implementation of the decoder used by the model can be set according to the needs of the implementation scenario. Figure 8 As shown, the reply calculation submodule 7084 may include:

[0055] The joint feature submodule 7084a may be configured to perform a full-connection operation on the feature vectors involved in the candidate reply calculation to obtain joint features of various dimensions.

[0056] The scoring submodule 7084b may be configured to input the joint features of the dimensions into a decoder based on likelihood scoring to obtain candidate responses with scores that meet the requirements.

[0057] like Figure 8 As shown, in one or more embodiments of this specification, the attention module 706 may include:

[0058] The dot multiplication submodule 7062 may be configured to perform a dot multiplication on the language feature vector and the image feature vector to obtain a similarity vector.

[0059] The coefficient calculation submodule 7064 may be configured to fully connect the similarity vectors to obtain weight coefficients of various dimensions.

[0060] The first fusion submodule 7066 may be configured to perform weighted summation on the language feature vector using the weight coefficient to obtain the language feature vector in the fused feature vector.

[0061] The second fusion submodule 7068 may be configured to perform weighted summation on the image feature vector using the weight coefficient to obtain the image feature vector in the fused feature vector.

[0062] Combined with Figure 6 The dialogue interface shown is as follows: Figure 8 As shown, the device may also include:

[0063] The dialogue receiving module 710 may be configured to receive the current round dialogue information input by the user through the dialogue interface.

[0064] The image determination module 712 may be configured to determine the image corresponding to the current round of dialogue information through the image displayed on the dialogue interface.

[0065] In some embodiments, the image determination module 712 may also be configured to stitch the multiple images into one image if the images corresponding to the current round of dialogue information are multiple images.

[0066] The above is a schematic scheme of a dialogue device of this embodiment. It should be noted that the technical scheme of the dialogue device and the technical scheme of the dialogue method described above are of the same concept, and the details not described in detail in the technical scheme of the dialogue device can be found in the description of the technical scheme of the dialogue method described above.

[0067] Fig. 9 The structure block diagram of a computing device 900 provided according to an embodiment of the present specification is shown. The components of the computing device 900 include but are not limited to a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0068] The computing device 900 also includes an access device 940 that enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 940 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0069] In one embodiment of the present specification, the above components of the computing device 900 and Fig. 9 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig. 9 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0070] The computing device 900 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 900 may also be a mobile or stationary server.

[0071] The processor 920 is used to execute the following computer executable instructions:

[0072] Obtain the language feature vector of the user's conversation information;

[0073] Acquire an image feature vector of the image corresponding to the dialogue information, where the image feature vector is a merged vector of feature vectors of several proposed regions of the image in a spatial dimension;

[0074] Performing feature fusion on the language feature vector and the image feature vector based on an attention mechanism to obtain a fused feature vector;

[0075] Candidate replies are calculated based on the fused feature vector to obtain candidate replies that meet the requirements.

[0076] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned dialogue method belong to the same concept, and the details of the technical scheme of the computing device that are not described in detail can all be referred to the description of the technical scheme of the above-mentioned dialogue method.

[0077] An embodiment of the present specification further provides a computer-readable storage medium storing computer instructions, which are used when executed by a processor to:

[0078] Obtain the language feature vector of the user's conversation information;

[0079] Acquire an image feature vector of the image corresponding to the dialogue information, where the image feature vector is a merged vector of feature vectors of several proposed regions of the image in a spatial dimension;

[0080] Performing feature fusion on the language feature vector and the image feature vector based on an attention mechanism to obtain a fused feature vector;

[0081] Candidate replies are calculated based on the fused feature vector to obtain candidate replies that meet the requirements.

[0082] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned dialogue method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned dialogue method.

[0083] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0084] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0085] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0086] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0087] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A conversation method, comprising: Obtain the language feature vector of the user's conversation information; Acquire an image feature vector of an image corresponding to the dialogue information, wherein the image feature vector is a merged vector of feature vectors of a plurality of proposed regions of the image in a spatial dimension, wherein the dialogue information is displayed in a question area in a dialogue interface, and the image corresponding to the dialogue information is displayed at a preset position in the question area; The language feature vector and the image feature vector are subjected to feature fusion based on an attention mechanism to obtain a fused feature vector, wherein the feature fusion of the language feature vector and the image feature vector based on the attention mechanism includes: obtaining a similarity vector according to the language feature vector and the image feature vector; inputting the similarity vector into a fully connected layer respectively, processing the similarity vector with the language feature vector to obtain a language feature vector in the fused feature vector, and processing the similarity vector with the image feature vector to obtain an image feature vector in the fused feature vector; Candidate replies are calculated based on the fused feature vector to obtain candidate replies that meet the requirements.

2. According to the method of claim 1, the calculating of candidate replies based on the fused feature vector to obtain candidate replies that meet the requirements comprises: Obtain the language feature vector of the user's historical conversation information; Candidate replies are calculated based on the language feature vector of the historical conversation information and the fused feature vector to obtain candidate replies that meet the requirements.

3. The method according to claim 1 or 2, wherein the calculation of the candidate responses comprises: The feature vectors involved in the candidate response calculation are fully connected to obtain joint features of each dimension; The joint features of the various dimensions are input into a decoder based on likelihood value scoring to obtain candidate responses with scores that meet the requirements.

4. According to the method of claim 1, the step of fusing the language feature vector with the image feature vector based on an attention mechanism comprises: Performing a dot product of the language feature vector and the image feature vector to obtain a similarity vector; Fully connect the similarity vectors to obtain weight coefficients of each dimension; Performing weighted summation on the language feature vector using the weight coefficient to obtain the language feature vector in the fused feature vector; The image feature vector in the fused feature vector is obtained by performing weighted summation with the weight coefficient and the image feature vector.

5. The method according to claim 1, further comprising: Receiving the current round of dialogue information input by the user through the dialogue interface; The image corresponding to the current round of dialogue information is determined through the image displayed on the dialogue interface.

6. The method according to claim 5, further comprising: If the images corresponding to the current round of dialogue information are multiple images, the multiple images are spliced ​​into one image.

7. A conversation device, comprising: A language feature acquisition module, configured to acquire a language feature vector of the user's conversation information; an image feature acquisition module, configured to acquire an image feature vector of an image corresponding to the dialogue information, wherein the image feature vector is a merged vector of feature vectors of a plurality of proposed regions of the image in a spatial dimension, wherein the dialogue information is displayed in a question area in a dialogue interface, and the image corresponding to the dialogue information is displayed in a preset position in the question area; The attention module is configured to perform feature fusion on the language feature vector and the image feature vector based on an attention mechanism to obtain a fused feature vector, wherein the feature fusion on the language feature vector and the image feature vector based on the attention mechanism includes: obtaining a similarity vector according to the language feature vector and the image feature vector; inputting the similarity vector into a fully connected layer respectively, processing it with the language feature vector to obtain a language feature vector in the fused feature vector, and processing it with the image feature vector to obtain an image feature vector in the fused feature vector; The reply calculation module is configured to calculate candidate replies based on the fused feature vector to obtain candidate replies that meet the requirements.

8. A computing device comprising: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions: Obtain the language feature vector of the user's conversation information; Acquire an image feature vector of an image corresponding to the dialogue information, wherein the image feature vector is a merged vector of feature vectors of a plurality of proposed regions of the image in a spatial dimension, wherein the dialogue information is displayed in a question area in a dialogue interface, and the image corresponding to the dialogue information is displayed at a preset position in the question area; The language feature vector and the image feature vector are subjected to feature fusion based on an attention mechanism to obtain a fused feature vector, wherein the feature fusion of the language feature vector and the image feature vector based on the attention mechanism includes: obtaining a similarity vector according to the language feature vector and the image feature vector; inputting the similarity vector into a fully connected layer respectively, processing the similarity vector with the language feature vector to obtain a language feature vector in the fused feature vector, and processing the similarity vector with the image feature vector to obtain an image feature vector in the fused feature vector; Candidate replies are calculated based on the fused feature vector to obtain candidate replies that meet the requirements.

9. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the dialogue method described in any one of claims 1 to 6.

10. A computer program product, characterized in that It comprises computer instructions, which, when executed by a processor, implement the steps of the dialogue method described in any one of claims 1 to 6.

Images