Online man-machine conversation method, customer service system, equipment and storage medium

Abstract

The embodiment of the invention provides an online man-machine conversation method, a customer service system, equipment and a storage medium. The method comprises the following steps: obtaining dialogue information of a user; based on multi-mode information provided by a service party, determining at least two modes of dialogue voice information capable of responding to the dialogue information; and processing the dialogue speech information of the at least two modes according to the dialogue information by using a trained first preset model, and outputting a multi-mode dialogue copywriting. In the scheme provided by the embodiments of the invention, the multi-modal dialogue language information of the dialogue copywriting is determined based on the multi-modal information provided by the service party, the multi-modal information in the shop of the merchant is fully combined to serve as the dialogue material, the advantages of the artificial intelligence technology are utilized, the accurate dialogue information with multiple modal styles is output, and the accuracy of the dialogue copywriting is improved. And the requirements of users are met.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an online human-computer dialogue method, a customer service system, a device and a storage medium. Background Art

[0002] Intelligent customer service, also known as customer service robots, can interact with users. Its mission is to help users solve problems or chat, so as to reduce the workload of manual customer service and provide fast service response. The dialogue between users and customer service robots is called human-machine dialogue. Most of the existing customer service robots use knowledge graphs or knowledge bases to retrieve knowledge with high semantic similarity to the questions raised by users, and finally return answers to users based on the retrieved knowledge.

[0003] However, with the development of technology, people have higher and higher requirements for the intelligence of customer service robots, and single-modal text answers can no longer meet the needs of multiple parties. Summary of the invention

[0004] The embodiments of the present application provide an online human-computer dialogue method, a customer service system, an electronic device, a storage medium, and a computer program product, which combine multimodal information provided by the service provider as reply material, making the reply material of the online customer service richer and more diverse.

[0005] In the first embodiment of the present application, an online human-computer dialogue method is provided. The method comprises:

[0006] Get the user's conversation information;

[0007] Determining, based on the multimodal information provided by the service provider, reply utterance information of at least two modes that can respond to the dialogue information;

[0008] The trained first preset model is used to process the reply corpus information of the at least two modalities according to the dialogue information, and a multimodal reply copy is output.

[0009] In the second embodiment of the present application, an online human-computer dialogue method is provided. The method comprises:

[0010] In response to the interface operation, the dialog information input by the user through at least one operation is displayed;

[0011] Displaying or displaying and playing the multimodal reply text generated for the dialogue information;

[0012] The multimodal reply text is obtained by processing reply utterance information of at least two modalities for the dialogue information using a trained first preset model; and the reply utterance information of at least two modalities is determined based on the multimodal information provided by the service provider.

[0013] In the third embodiment of the present application, an online human-computer dialogue method is provided. The method comprises:

[0014] Obtaining the dialog information input by the user through at least one operation;

[0015] Determine a query Query according to the conversation information;

[0016] According to the commodity identifier ItemID included in the Query, obtain multimodal information related to the commodity corresponding to the ItemID;

[0017] Based on the multimodal information related to the commodity corresponding to the ItemID, determining at least two modal reply corpus information matching the Query;

[0018] The trained first preset model is used to process the reply corpus information of the at least two modalities according to the dialogue information, and a multimodal reply copy is output.

[0019] In a fourth embodiment of the present application, a customer service system is provided. The system includes:

[0020] The client is used to respond to the interface operation and display the dialogue information input by the user through at least one operation;

[0021] The server is configured to obtain the conversation information; determine at least two modal reply utterances that can respond to the conversation information based on the multimodal information provided by the service provider; and process the at least two modal reply utterances according to the conversation information using the trained first preset model to output a multimodal reply text;

[0022] The client is used to display or display and play the multimodal reply text generated for the dialogue information.

[0023] In a fifth embodiment of the present application, an electronic device is provided. The electronic device includes a memory and a processor. The memory is used to store executable instructions; the processor implements the steps in the above-mentioned method embodiments by running the executable instructions.

[0024] In a sixth embodiment of the present application, a computer-readable storage medium is provided, wherein the storage medium stores computer instructions, and when the instructions are executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0025] The seventh embodiment of the present application further provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, the processor executes the steps in the above-mentioned method embodiments.

[0026] The technical solution provided by the embodiment of the present application combines the advantages of artificial intelligence (i.e., the trained first preset model) with the multimodal information on the e-commerce platform to construct an intelligent customer service system with multimodal reply copy that can respond to user dialogue information. In the solutions provided by the embodiments of the present application, the multimodal reply text information of the reply copy is determined based on the multimodal information provided by the service provider (such as the multimodal information in the merchant store on the e-commerce platform, etc.), and the multimodal information provided by the service provider is fully combined as the reply material, and the advantages of artificial intelligence technology are utilized to output accurate reply information with multiple modal styles to meet the needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 A schematic diagram of the technical architecture corresponding to the technical solution provided in the embodiment of the present application;

[0029] Figure 2 A schematic diagram of a customer service system provided in an embodiment of the present application;

[0030] Figure 3 A flowchart of an online human-computer dialogue method provided by an embodiment of the present application;

[0031] Figure 4 A schematic diagram of a human-computer dialogue interface shown in an embodiment of the present application;

[0032] Figure 5a , 5b 5c shows a schematic diagram of a scenario of three multimodal reply texts outputted for user's dialogue information by the technical solution provided in an embodiment of the present application;

[0033] Figure 6 A flowchart of an online human-computer dialogue method provided by another embodiment of the present application;

[0034] Figure 7 A schematic diagram showing a scenario in which a user inputs dialogue information through multiple rounds of interaction;

[0035] Figure 8 A schematic diagram showing a flow chart of an online human-computer dialogue method provided by another embodiment of the present application is shown;

[0036] Fig. 9A structural block diagram of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0038] In some processes described in the specification, claims and the above-mentioned figures of the present application, multiple operations appearing in a specific order are included, and these operations may not be executed or executed in parallel in the order in which they appear in this article. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit "first" and "second" to different types. In addition, the embodiments described herein are only a part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0039] Before introducing various embodiments, some technical terms appearing in this document are briefly introduced.

[0040] The concept of multimodality originated from the study of information representation in the field of computer human-computer interaction. The term "modality" is defined as the expression and interaction of information on a specific physical medium. Pictures are a modality, videos are a modality, audio is a modality, text is a modality, animated images are a modality, and so on.

[0041] Query: Query means query. Query is a query parameter class, which is a message sent by a search engine or database to search for files, pictures, websites, records or a series of records in the database. Query is a query parameter class that encapsulates query conditions, paging, sorting and other functions.

[0042] Pre-trained model: refers to a machine learning model with a large number of parameters generated by training with massive data using self-supervised methods or unsupervised methods. It provides excellent distributed feature representation and model generalization capabilities for downstream tasks. The pre-trained model can be called a generalized large model. The first preset model, the second preset model, etc. mentioned below can all be called pre-trained models, but the model structure may be different.

[0043] An e-commerce service provider, or e-commerce platform, or simply e-commerce platform, is a platform that provides online transactions and negotiations for enterprises or individuals. An e-commerce service provider is a place that coordinates and integrates the orderly association and flow of information flow, goods flow, and capital flow, that is, by establishing an e-commerce portal, it provides online shopping services to ordinary consumers, or provides business applications to enterprises, in order to seek a better combination between the Internet and traditional commercial and industrial systems. Enterprises, merchants and individuals can make full use of the shared resources such as network infrastructure and payment platforms provided by e-commerce platforms to carry out their own business activities. In this article, e-commerce service providers can be store bazaar e-commerce platforms, shopping mall e-commerce platforms or social e-commerce platforms. Store bazaar e-commerce platforms and shopping mall e-commerce platforms can be called traditional e-commerce. Social e-commerce is a new derivative model of e-commerce. It uses social networking sites, social apps and other communication channels to assist the purchase and sale of goods through social interaction, user-generated content and other means, and applies social elements such as attention, communication, discussion and interaction to the e-commerce transaction process. Social e-commerce integrates e-commerce and social media.

[0044] Merchants are a general term for individuals and organizations that engage in commercial activities (production and operation of related items) through e-commerce platforms.

[0045] With the rapid development of big model technology, its application prospects in the field of intelligent customer service are becoming more and more broad, especially in improving conversation understanding and natural response, showing great potential and becoming an optional solution for optimizing customer experience. However, a key limitation of current big models is that they are not yet able to process multimodal information.

[0046] At the same time, when listing products, merchants often provide rich and detailed multimodal information, including but not limited to high-definition pictures, detailed descriptions, demonstration videos, etc. This information greatly enhances the explainability and attractiveness of the product. However, these information resources have not been fully utilized in traditional intelligent customer systems, resulting in a gap between customer service responses and user expectations, especially for complex queries that require cross-modal understanding and response.

[0047] Therefore, how to closely combine the advantages of large language models with the multimodal product information provided by merchants to build a customer service system that can accurately generate multimodal responses has become an innovative issue that needs to be solved. This can not only greatly improve customer service efficiency and quality, but also create a more immersive and personalized interactive experience for users, and further promote the field of intelligent customer service to a higher level of intelligence and humanity.

[0048] For this purpose, this application is proposed. The technical solutions of each embodiment of this application combine artificial intelligence technology with multimodal information on the e-commerce platform. In order to facilitate the understanding of the technical solution, the underlying technical architecture on which each embodiment of this application is based is briefly described. The technical solutions provided in the following embodiments are all implemented based on this technical architecture. Figure 1 As shown in the figure, the technical architecture diagram of the technical solution provided by this application. Figure 1 As shown, the technical architecture of the technical solution provided by this application includes:

[0049] 1. Build an information database Figure 1 Winning bid number 101)

[0050] 1.1 Collect multimodal information, such as multimodal information provided by e-commerce service providers and multimodal information in merchant stores. Among them, the multimodal information of e-commerce service providers may include but is not limited to: activity promotion pictures provided by e-commerce platforms, activity promotion videos provided by e-commerce platforms, activity promotion animated pictures provided by e-commerce platforms, and activity rules provided by e-commerce platforms. The activity rules provided by e-commerce platforms can be text, or text plus pictures, or text plus videos, etc., and this application does not limit this. The multimodal information in the merchant stores of e-commerce platforms may include but is not limited to: multimodal information uploaded by merchants to the platform, multimodal information uploaded by users (such as user evaluation texts, pictures, videos, etc. for goods in merchant stores), etc. The multimodal information uploaded by merchants to the platform may include but is not limited to: pictures of goods in stores, product display videos, advertising pictures, advertising videos, product description texts, product details pictures, product instructions videos, store activity promotion pictures and store activity promotion videos, etc.

[0051] 1.2. Processing information items of different modalities

[0052] The text does not need to be processed. If it is not processed, it does not need to be stored in the information database.

[0053] For pictures, the second preset model (such as the picture processing model) can be used to identify and classify text information from the picture, and then the text information can be sorted and stored in the information library. For pictures containing text, the image OCR (Optical Character Recognition) technology and the picture cutting model can also be used to locate the text and cut the picture, and the cut sub-pictures containing text can be sorted and stored in the information library.

[0054] Video or animated picture (also known as dynamic picture) is two categories, video is an image, and animation is Figure 1The first is an image. Video can be converted into dynamic pictures. Dynamic pictures cannot be restored to videos. Video quality is relatively high and clear, while the dynamic pictures commonly seen on the Internet are generally of low quality. The most common dynamic picture format is gif. Therefore, in this article, videos and dynamic pictures are distinguished, but the same technology can be used for both processing.

[0055] For videos and animated images, image recognition and semantic understanding technologies can be used to analyze the content of videos and animated images, and then generate content summaries or keyword tags corresponding to the videos or animated images, etc., for subsequent retrieval. The content summaries or keyword tags corresponding to the videos or animated images are organized and stored in the information library.

[0056] It should be noted here that when organizing and storing the text information corresponding to the picture, the sub-picture of the picture, the content summary or keyword tags corresponding to the video or animated picture in the information database, the text information can be stored in association with the picture identifier of the picture corresponding to the text information, the sub-picture can be stored in association with the identifier of the original picture (or parent picture) of the sub-picture, and the content summary or keyword tags can be stored in association with the identifier of the corresponding video or animated picture.

[0057] It should also be added that: the above examples include text, pictures, videos, and animated images, but not audio modal information items. There are very few information items in a single audio modality on e-commerce platforms. Usually, audio is included with video, so the above video can be understood as audio and video. If there is an information item in a single audio modality, audio and video recognition technology can also be used to identify the audio content to generate text information corresponding to the audio, and then the text information and the identifier of the audio corresponding to the text information are associated and stored in the information library.

[0058] 2. Model Deployment Figure 1 Winning bid number 102)

[0059] Directly select a trained large language model (LLM). Alternatively, build training samples suitable for the scenarios of intelligent customer service and adjust and train the basic model, so that the reply text output by the adjusted and trained model is more in line with the specific scenario. Among them, the basic model is also trained using a large amount of data using self-supervision or unsupervised methods.

[0060] For example, using the constructed training samples, a fine-tuning method (such as the LoRA method) is used to train the Fine Tuning basic model to enhance the logic and accuracy of the pre-trained model output in the customer service scenario of the e-commerce platform. LoRA: A model Fine Tuning method that freezes the model parameters of the basic model and injects the trainable low-rank matrix into the attention layer of the Transformer (a model that uses the attention mechanism to increase the model training speed) architecture, greatly reducing the number of trainable parameters of downstream tasks and improving the training speed.

[0061] The base model can be a large language model (LLM) or other machine learning models. The base model has learned a rich feature representation and pattern recognition capability. Through fine-tuning, these learned feature representations can be used to quickly and effectively adapt to tasks in specific scenarios (such as tasks in e-commerce customer service scenarios). Fine-tuning is faster than training the model from scratch, which accelerates the training process.

[0062] 3. Online Link Construction ( Figure 1 Winning bid number 103)

[0063] In the specific implementation, online deployment can be completed based on Rlab (a high-level programming language for engineering application calculation) tools. Of course, online deployment can also be completed based on other tools. This application does not make specific restrictions on this. Here, only the use of Rlab tools is exemplified. Online link construction also requires the following but not limited to: EAS, Igraph, pre-trained model development tools (Tars Studio), PAI and Itag tools and other supports. Among them, EAS (Elastic Algorithm Service, model online service) is a machine learning platform product to achieve one-stop model development and deployment applications, and provides model online services for online reasoning scenarios. It supports the deployment of model services in public resource groups or exclusive resource groups, and realizes real-time response to model loading and data requests based on heterogeneous hardware (such as CPU or GPU). Igraph is an online graph storage and query system that provides storage, query, update and calculation services for large-scale graph data. The pre-trained model development tool (Tars Studio) provides a visual development tool. PAI is a machine learning platform that allows developers to use artificial intelligence AI technology more efficiently, concisely and standardly. Itag provides data annotation services to help users complete data annotation tasks quickly and efficiently.

[0064] The links built online may include but are not limited to:

[0065] S11. Determine the target Query.

[0066] S12. Rewrite the target Query.

[0067] The target query rewriting may include but is not limited to:

[0068] Query normalization: rewrite the target query through simple string processing, such as removing spaces, converting uppercase and lowercase letters, processing special characters, etc.

[0069] Query error correction: Actively correct target queries with spelling errors;

[0070] Query expansion: expand the query according to the semantics or intent of the target query, find a set of queries that are consistent with the target query, and search together with the target query;

[0071] etc.

[0072] S13. Obtain data based on Igraph.

[0073] For example, the data obtained may be: data retrieved from the database corresponding to the e-commerce platform, more specifically, multimodal information matching the target query retrieved from the above-mentioned "information library", including but not limited to: text, pictures, videos, animated pictures, etc.

[0074] It should be noted here that the retrieval method for texts matching the target Query may be the same as the currently commonly used retrieval method, and this embodiment does not limit this.

[0075] S14: pre-process the acquired data.

[0076] This application does not specifically limit "preprocessing", which can be, but is not limited to, operations such as data cleaning, standardization, and format conversion. For example, useful data is filtered out from the acquired data through the data cleaning process. For example, unstructured data (such as text) may need to be structured through steps such as word segmentation and stem extraction. According to the input requirements of the pre-trained model, the data needs to be converted into an appropriate format, such as JSON format or text format.

[0077] S15, Prompt splicing

[0078] The target Query and the pre-processed data (text, pictures, videos and / or animated images, etc.) can be spliced ​​into the Prompt. The quality of the output (or response) of the first preset model depends on the prompt word (i.e., Prompt). Simply put, the Prompt is the true intention of the user's dialogue information, which is sent as input to the first preset model. The first preset model will try to understand this input and then output a suitable answer or response. Splicing the Prompt here is to make the first preset model output more accurate and useful copy.

[0079] Prompt is an important part. Prompt is used to guide the behavior of the model. By adjusting Prompt, the quality of the model output copy can be improved. In the specific implementation, the designer can manually design the prompt splicing rules so that the splicing can be performed according to the preset splicing rules when executing prompt splicing.

[0080] 4. Front-end Function Deployment

[0081] With the technical architecture support of the above parts one, two and three, the corresponding functional interfaces can be deployed on the front-end application (APP) or browser web page, so that users can call the corresponding services of the back-end (such as the server with the above technical architecture) through the functional interface, such as intelligent customer service, intelligent question and answer functions, etc.

[0082] The functional interface can be in the form of a functional control on the APP page or browser webpage, and the user can call the corresponding service in the backend by clicking the functional control. Alternatively, a specific logo is set on the APP page or browser webpage, such as an intelligent customer service logo or an intelligent question and answer logo. The logo can be any pattern or animation, which is not limited here. The user can wake up the intelligent customer service function or the intelligent question and answer function through voice or gesture. When the intelligent customer service or question and answer dialogue interface is opened, the user can sense that the intelligent customer service or question and answer function has been successfully awakened. The user can call the corresponding service in the backend by interacting with the intelligent customer service. That is, the product form presented by the above technical architecture on the client side can be an intelligent customer service logo.

[0083] The technical solutions corresponding to the technical architecture mentioned above (such as Figure 1 The solutions corresponding to parts one, two, three and four in the above) can be deployed on the server side. The server side can be a server, a service cluster, a virtual server or a cloud, etc., which is not specifically limited in this embodiment. The server side provides corresponding functional services to the client, such as intelligent customer service, intelligent question and answer services, etc. Users can enter the dialogue interface through the browser, client application (APP), web application H5 (HyperText MarkupLanguage 5, the fifth generation of HTML, Hypertext Markup Language), light application (also called applet, a lightweight application) or cloud application on the client device. The client device can be but is not limited to: a smart phone, a smart wearable device, a tablet computer, a laptop computer, a desktop computer, etc. As Figure 2 As shown, an embodiment of the present application provides a customer service system, which includes a server and a client.

[0084] The client 202 is used to respond to the interface operation and display the dialogue information input by the user through at least one operation;

[0085] The server 201 is used to obtain the dialogue information; determine at least two modal reply utterances that can respond to the dialogue information based on the multimodal information provided by the service provider; use the trained first preset model to process the at least two modal reply utterances according to the dialogue information, and output a multimodal reply copy;

[0086] The client 202 is used to display or display and play the multimodal reply text generated for the dialogue information.

[0087] Alternatively, in another embodiment, the client 202 is used to respond to the interface operation and display the dialogue information input by the user through at least one operation. The server 201 is used to obtain the dialogue information input by the user through at least one operation; determine the query Query and the product identifier ItemID according to the dialogue information; obtain the multimodal information related to the product corresponding to the ItemID according to the ItemID; determine the reply corpus information of at least two modes matching the Query based on the multimodal information related to the product corresponding to the ItemID; use the trained first preset model to process the reply corpus information of at least two modes according to the dialogue information, and output the multimodal reply copy;

[0088] The client 202 is used to display or display and play the multimodal reply text generated for the dialogue information.

[0089] The technical solution provided by this embodiment combines the advantages of artificial intelligence (i.e., the trained first preset model) with the multimodal information on the e-commerce platform to construct an intelligent customer service system with multimodal reply copy that can respond to user dialogue information. In the solutions provided by the embodiments of this application, the multimodal reply text information of the reply copy is determined based on the multimodal information in the merchant store on the e-commerce platform, and the multimodal information in the merchant store is fully combined as the reply material, and the advantages of artificial intelligence technology are utilized to output accurate reply information with multiple modal styles to meet the needs of users.

[0090] It is important to note that the above is an introduction to the technical solution of this application using the e-commerce platform scenario as an example. In fact, the technical architecture provided by this application can also be applied to other specific scenarios, such as the intelligent question-and-answer scenario on the corporate website, etc.

[0091] The technical solution provided in this application will be described below in the form of a method embodiment.

[0092] Figure 3 The flowchart of the information processing method provided by an embodiment of the present application is shown. The execution subject of the method provided by this embodiment can be the server in the above customer service system. Figure 3 , the method may include:

[0093] 301. Obtaining user's conversation information;

[0094] 302. Determine, based on the multimodal information provided by the service provider, reply utterance information of at least two modes that can respond to the dialogue information;

[0095] 303. Using the trained first preset model, process the reply corpus information of the at least two modalities according to the dialogue information, and output a multimodal reply copy.

[0096] In the above 301, the user's conversation information may include the conversation content input by the user at least once, and the mode of the conversation content input by each operation may be different. For example, Figure 4 In the example shown, the user inputs a picture of a top in the first operation, and inputs the text "What color clothes are worn under this top?" in the second operation. Based on the two operations of the user, the acquired dialogue information includes the dialogue contents (picture and text) inputted twice.

[0097] In the above 302, the service provider may include but is not limited to: an e-commerce service provider, a merchant, etc. The multimodal information provided by the service provider includes at least one of the following: multimodal information in the merchant's store and multimodal information of the e-commerce service provider. The multimodal information in the merchant's store includes multiple items of the following: pictures of goods in the store, product display videos, advertising pictures, advertising videos, user comment texts, user comment pictures, user comment videos, product description texts, product details pictures, store activity promotion pictures, store activity promotion videos, activity promotion pictures provided by the e-commerce service provider, activity promotion videos provided by the e-commerce service provider, activity promotion animated pictures provided by the e-commerce service provider, and activity rules provided by the e-commerce service provider.

[0098] In the above 303, the first preset model may be a pre-trained large language model.

[0099] In the solution provided in this embodiment, the multimodal reply text information of the reply copy is determined based on the multimodal information provided by the service provider (such as the multimodal information in the merchant store on the e-commerce platform, etc.), and the multimodal information in the merchant store is fully combined as the reply material, and the advantages of artificial intelligence technology are utilized to output accurate reply information with multiple modal styles to meet the needs of users.

[0100] In the above 302, when determining the reply utterance information of at least two modes in response to the dialogue information, it can be retrieved from the information library mentioned above. That is, in a specific implementation, the multimodal information includes information items of at least two modes; accordingly, the above step 302 "determining the reply utterance information of at least two modes that can respond to the dialogue information based on the multimodal information provided by the service provider" may include:

[0101] 3021. Perform content mining on the information item to obtain a mining result corresponding to the information item.

[0102] Among them, the mining results corresponding to the information items can be stored in the information database (see Figure 1 The information library stores mining results corresponding to each modal information item in the multimodal information provided by the service provider.

[0103] Accordingly, determining the reply utterance information of at least two modes capable of responding to the dialogue information may include:

[0104] 3022. Retrieve at least one target mining result related to the conversation information from the information database.

[0105] 3023. Determine the reply corpus information of the at least two modalities according to the at least one target mining result and the information item of the modality corresponding to the target mining result.

[0106] In some specific examples, if a target mining result retrieved is text information in a picture, the text information and the picture corresponding to the text information are used as reply text information. If a target mining result retrieved is a sub-picture of a picture, the original picture to which the sub-picture belongs is used as reply text information. If a target mining result retrieved is a content summary or keyword label corresponding to a video or animated picture, the video or animated picture corresponding to the content summary or keyword label is used as reply text information.

[0107] The following will explain how to conduct content mining for information items of different modalities.

[0108] As mentioned above, multimodal information may include pictures, videos, audio, text, animated pictures, etc. For pictures in multimodal information, the multimodal information includes a first modal information item, which is a picture. Accordingly, the above step 3021 "perform content mining on the information item to obtain the mining result corresponding to the information item" may be specifically: perform content mining on the picture to obtain the mining result corresponding to the picture.

[0109] Furthermore, “performing content mining on the image to obtain mining results corresponding to the image” may include:

[0110] The image is processed using the trained second preset model to obtain text information reflecting the content of the image; the text information is the mining result corresponding to the image.

[0111] The second preset model may be a machine learning model (such as a deep neural network model, a convolutional neural network model, etc.) that processes the image to output text information reflecting the content of the image. Figure 4 The top picture input by the user shown in the figure may be a women's top with ** color and ** collar type after content mining.

[0112] For information items in text mode, content mining may not be performed.

[0113] Based on this, step 3022 of this embodiment, “retrieve at least one target mining result related to the user conversation information in the information database” may include:

[0114] S21, using at least one retrieval algorithm, searching the information database for a plurality of candidate text information that are relevant to the conversation information;

[0115] S22: Select at least one target text information from the multiple candidate text information as a target mining result.

[0116] In an achievable technical solution, two retrieval algorithms can be used, such as vector retrieval algorithm and BM25 algorithm. Among them, vector retrieval is an information retrieval method based on the vector space model, which has been widely used in text retrieval, image retrieval, audio retrieval and other fields. Vector retrieval is based on the vector space model, represents the query and document as vectors in the vector space, and uses the similarity between vectors for matching and sorting. Specifically, the text, image, audio and other information in the query and document are converted into vectors. This vector is composed of a set of feature items, each of which represents a type of information, such as words in text, pixels in images, frequencies in audio, etc. The similarity between vectors can be calculated using algorithms such as cosine similarity and Euclidean distance. BM25 is an algorithm for information retrieval that measures document relevance and is based on OkapiTF-IDF improvement. It considers document length, query term frequency and document frequency. The algorithm involves IDF (Inverse Document Frequency), the weight of query term frequency and document length adjustment, and calculates the degree of matching through a scoring function. The results of vector retrieval and BM25 recall are merged to remove duplicate content. The results are sorted according to the relevance score of the content, and the results with a high relevance to the user's conversation information are retained. The retained results are multiple candidate text information that are relevant to the user's conversation information.

[0117] Furthermore, for the picture, this embodiment also provides the following processing steps. That is, the above step of "mining the content of the picture to obtain the mining result corresponding to the picture" may also include:

[0118] S31, locating the text area in the image;

[0119] S32, cutting the image to obtain a sub-image containing the text area;

[0120] That is, the sub-image is the mining result corresponding to the image.

[0121] The above method of using the trained second preset model to process the picture to obtain text information reflecting the content of the picture is applicable to pictures without text. Of course, pictures with text can also obtain text information of the picture through this method. The above method of step S21 and step S22 is applicable to pictures with text.

[0122] In specific implementation, the above S31 and S32 can use image OCR (Optical Character Recognition) technology and cut picture model to accurately locate text and cut product pictures, and then organize and store this information in the information library. Correspondingly, for the user's conversation information, step 3022 "retrieve at least one target mining result related to the conversation information in the information library" in this embodiment can also include:

[0123] S23, identifying text content in the sub-graph in the information library;

[0124] S24, evaluating the relevance between the text content of each sub-image in the information database and the conversation information;

[0125] S25. At least one subgraph whose correlation meets the requirements is the at least one target mining result.

[0126] Correspondingly, if the at least one target mining result contains a sub-graph, when determining the response corpus information of the at least two modalities based on the at least one target mining result and the information item of the modality corresponding to the target mining result, the original image to which the sub-graph belongs can be determined as the response corpus information.

[0127] For videos and animated images, the multimodal information includes a second modal information item or a third modal information item; the second modal information item is a video, and the third modal information item is a animated image. Accordingly, in this embodiment, step 3021 "performing content mining on the information item to obtain a mining result corresponding to the information item" can be specifically: performing content mining on the video or animated image to obtain a mining result corresponding to the video or animated image.

[0128] In an achievable technical solution, the step of "mining the content of the video or animated image to obtain the mining result corresponding to the video or animated image" may include:

[0129] S41, identifying key elements in the video or animated image.

[0130] In a specific embodiment, the video frames in the video or the frame images in the animated image may be extracted first. For example, some frame extraction tools may be used for frame extraction. Models such as convolutional neural networks (CNN) are used to extract features of images in video frames or frame images in animated images. Features may include underlying features such as color, texture, shape, and may also include more advanced semantic features. Trained target retrieval models (such as YOLO, SSD, etc.) may also be used to identify objects in video frames or frame images, such as key elements such as commodities, people, scenes, and the like.

[0131] S42: Perform semantic analysis on the video or animated image to obtain a content description of the video or animated image.

[0132] For example, natural language processing (NLP) and semantic analysis models can be used to describe objects, backgrounds, etc. in video frames or animated frames. Content description can generate content summaries or keyword tags for the video or animated image to facilitate subsequent retrieval.

[0133] S43: Based on the key elements and the content description, determine the mining result corresponding to the video or animated image.

[0134] Furthermore, the above S43 “determining the mining result corresponding to the video or animated image based on the key elements and the content description” may include:

[0135] S431. Determine the category to which the video or animated image belongs based on the key elements and the content description.

[0136] For example, by analyzing the objects, background, colors and other information in the video frames of the video or the frames of the animated image, the category to which the video or animated image belongs can be identified (for example, in a model display video or animated image, the category to which it belongs may be the type of clothes displayed in the modality, such as clothing, women's clothing, or further coats, etc.).

[0137] S432, generating a content summary or keyword tag of the video or animated image according to the category to which the video or animated image belongs, the key elements and the content description;

[0138] That is, the content summary or keyword tag is the mining result corresponding to the video or animated image.

[0139] In order to improve the search speed, the video or animated image can be classified to store the mining results corresponding to the video or animated image in groups according to the classification. In this way, when searching, you can search within the group where the target classification is located. That is, the embodiment of the present application can also include the following steps:

[0140] According to the category to which the video or animated image belongs, the mining results corresponding to the video or animated image are grouped;

[0141] The mining results corresponding to the videos or animated images in the information library are stored in groups for classification retrieval.

[0142] Accordingly, in this embodiment, step 3023 of “determining the reply corpus information of the at least two modalities according to the at least one target mining result and the information item of the modality corresponding to the target mining result” may also include:

[0143] If the at least one target mining result contains a content summary or keyword tag of a target video or target animated image related to the conversation information, then

[0144] Obtaining the target video or target animated image;

[0145] From the target video or target animated picture, extract a video clip or an animated picture clip that matches the conversation information;

[0146] The video clip or the animated picture clip is used as the reply text information.

[0147] For videos and animated images, in specific implementations, image recognition and semantic understanding technologies (such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs)) can be used to locate and classify videos or animated images, extract elements, and perform content mining. For user conversation information, by analyzing the correlation between the mining results of videos or animated images in the information library and the user conversation information, target videos or target animated images that are highly correlated with the user conversation information can be retrieved. Furthermore, video clips or animation clips that match the conversation information can be captured from the target video or target animated image.

[0148] Furthermore, the above step 3023 may also include the following steps:

[0149] If the at least one target mining result contains text information of a target image related to the dialogue information, then adding the text information of the target image to the reply corpus information;

[0150] If the at least one target mining result contains a sub-image of a target image related to the dialogue information, obtaining an original image of the sub-image, and adding the original image to the reply corpus information;

[0151] If the at least one target mining result contains target text information related to the dialogue information, the target text information is added to the reply corpus information.

[0152] It should be added that the above examples include text, pictures, videos, and animated images, but not audio information items. There are very few audio information items on e-commerce platforms. Usually, audio is included with video, so the above video can be understood as audio and video.

[0153] Most intelligent customer service services do not support users sending pictures, videos, and animated pictures. That is, if a user uploads pictures, videos, animated pictures, etc. through the interactive interface, the reply information returned by the intelligent customer service is mostly irrelevant information. The solution provided in the embodiment of the present application supports multimodal dialogue information input by users. For example, the dialogue information includes at least one modal information item; when the dialogue information includes at least one of pictures, videos, audio, and animated pictures, the method provided in the embodiment of the present application may also include:

[0154] Performing image and / or audio / video recognition on at least one of the pictures, videos, audios, and animated images included in the conversation information;

[0155] Based on the recognition result, a query Query corresponding to the dialogue information is determined, so as to determine reply corpus information of at least two modes matching the Query based on the multimodal information provided by the service provider.

[0156] It should be noted here that Query not only plays a role in determining reply text information of at least two modes, but also plays a role when the first preset model outputs multi-modal reply text.

[0157] Specifically, the above “determining the query Query corresponding to the conversation information based on the recognition result” may include the following steps:

[0158] S51, when the recognition result is text recognized from the picture, video, audio and / or animation, the text recognized from the picture, video, audio and / or animation is combined with the text in the user conversation information to analyze the user's real intention and obtain the user's intention;

[0159] S52: Determine Query according to the user's intention.

[0160] Furthermore, when executing the above S51, information related to the user, such as historical information that is allowed to be obtained by the user, may also be combined. Combining the historical information can more accurately analyze the user's intention.

[0161] for example, Figure 4In the example shown, the user's conversation information includes a picture and the text "What color clothes are worn under this top?". By recognizing the picture, the recognition result is a women's top of ** color, ** style, and *** material. Combining the recognition result with the text in the user's conversation information, it can be analyzed that the user's real intention is: outfit recommendation. Based on the user's intention, it can be determined that the query is to retrieve a top that matches the color, style, and material of the women's top.

[0162] Furthermore, in order to retrieve results that better meet the user's requirements, if the user's historical purchase and browsing information can be combined, that is, the user's preferences can be combined, for example, whether the user prefers stable colors (such as black, gray, white, etc.) or bright colors (such as yellow-green, red, etc.). User preferences can be integrated into the query.

[0163] After obtaining the reply utterance information of at least two modalities, step 303 of this embodiment, "using the trained first preset model to process the reply utterance information of at least two modalities according to the dialogue information, and outputting a multimodal reply copy" may specifically include:

[0164] 3031. Preprocess the response corpus information of the at least two modes to convert it into a format suitable for input into a first preset model.

[0165] The present application does not specifically limit "preprocessing", which may include but is not limited to: data standardization, format conversion and other operations. For example, unstructured data (such as text) may need to be structured through steps such as word segmentation and stem extraction. According to the input requirements of the first preset model, the data needs to be converted into an appropriate format, such as JSON format or text format.

[0166] 3032. Construct corresponding scenarios and contexts based on the user's conversation information and the reply text information of at least two modes after conversion, so that the first preset model can better understand and generate accurate and natural multimodal reply text.

[0167] In specific implementation, the target Query mentioned above and the reply text information of at least two modes after the above preprocessing can be spliced ​​into the Prompt. The quality of the output (or response) of the first preset model depends on the prompt word (i.e., Prompt). Simply put, the Prompt is the question the user wants to ask, which is sent as input to the first preset model. The first preset model will try to understand this input and then output a suitable answer or response. Splicing the Prompt here is to make the first preset model output more accurate and useful copy.

[0168] Prompt is an important part. Prompt is used to guide the behavior of the model. By adjusting Prompt, the quality of the model output copy can be improved. In the specific implementation, the designer can manually design the prompt splicing rules so that the splicing can be performed according to the preset splicing rules when executing prompt splicing.

[0169] See also Figure 5a , 5b In the example shown in FIG5c, the multimodal reply copy output by the first preset model according to the user's dialogue information and at least two modal reply corpus information can be Figure 5a The text shown in the multimodal reply text includes: a shirt image, text, and controls corresponding to a webpage link. A user can click the "More shirts of the same style >" control in the multimodal reply text to enter a product browsing page, which displays at least one product (i.e., a shirt) sold by a merchant on the e-commerce platform.

[0170] Furthermore, the first preset model may also have a drawing function, that is, the first picture 1 in the user's dialogue information and the second picture in the reply text information are combined into a matching picture, see Figure 5b shown. Figure 5b The multimodal reply copy shown includes: a diagram of wearing a top in the first picture 1 with a bottoming shirt in the second picture, text, and controls corresponding to a web page link.

[0171] Furthermore, if the reply corpus information includes a model display video of a bottoming shirt that can be matched with the top in the first picture 1 in a certain merchant's store, the multimodal reply copy output by the first preset model is as follows: Figure 5c As shown, it may include: controls corresponding to a model display video, text, and a web page link of a base shirt suitable for wearing under the top in the first picture 1.

[0172] If the user's conversation information is not about dressing, but instructions for use of a certain commodity, product introduction, etc., the solution provided in the embodiment of the present application can also output a multimodal reply to the user through the various method steps mentioned above, including: detailed description text, pictures and videos of the commodity or product.

[0173] The present application also provides an information processing method in an embodiment. The execution subject of the method provided in this embodiment may be a client in the above service system. Figure 6 As shown, the method includes:

[0174] 401. Respond to the interface operation and display the dialog information input by the user through at least one operation;

[0175] 402. Displaying or displaying and playing a multimodal reply text generated for the dialogue information;

[0176] The multimodal reply text is obtained by processing the reply utterance information of at least two modalities according to the dialogue information using the trained first preset model; the reply utterance information of at least two modalities is determined based on the multimodal information provided by the service provider. For more relevant content, please refer to the above, which will not be elaborated here.

[0177] The user's conversation information may also include information in at least two modes. Figure 4 As shown, the interface operation may include: dialogue information input by the user in multiple consecutive operations. For example, the user imports the first picture 1 in the first operation, and inputs the text "What color clothes are worn under this top?" in the second operation. Or, in multiple rounds of interaction, the dialogue information input by the user in each round. One human-computer dialogue is one round of interaction. For example Figure 7 As shown, in the first round of interaction, the user first inputs the first picture 1, and the intelligent customer service responds, "What questions do you want to ask about the above picture?" In the second round of interaction, the user inputs, "What color clothes are worn under this top?"

[0178] The present application also provides an information processing method in one embodiment. The execution subject of the method provided in this embodiment may be the client in the above customer service system. Alternatively, the execution subject of some steps in the method provided in this embodiment is the client, and the execution subject of other steps is the server. Figure 8 As shown, the method includes:

[0179] 501. Acquire the dialog information input by the user through at least one operation;

[0180] 502. Determine a query Query according to the conversation information;

[0181] 503. According to the commodity identifier ItemID included in the Query, obtain multimodal information related to the commodity corresponding to the ItemID;

[0182] 504. Determine, based on the multimodal information related to the commodity corresponding to the ItemID, at least two modal reply corpus information matching the Query;

[0183] 505. Using the trained first preset model, process the reply corpus information of the at least two modalities according to the dialogue information, and output a multimodal reply copy.

[0184] It should be noted that the solution provided in this embodiment is applicable to the user's dialogue information for a certain product. That is to say, the content that the user wants to consult or dialogue is targeted at the product. Therefore, the Query determined based on the dialogue information contains the product identifier ItemID.

[0185] Furthermore, the conversation information includes at least one modal information item; when the conversation information includes at least one item of a picture, a video, an audio, or a moving picture,

[0186] Determining a query Query according to the conversation information includes:

[0187] Performing image and / or audio / video recognition on at least one of the pictures, videos, audios, and animated images included in the conversation information;

[0188] Based on the recognition result, the Query corresponding to the conversation information is determined.

[0189] For the specific implementation of the above method steps, please refer to the relevant content above and will not be repeated here.

[0190] See also Figure 8 As shown, the solution provided by this embodiment may also include: asynchronously querying the newly added multimodal information on the e-commerce platform. For example, a fixed time duration is triggered to query whether there is newly added multimodal information on the e-commerce platform, or query whether there is newly added multimodal information on the e-commerce platform when idle.

[0191] Furthermore, the method provided in this embodiment may further include the following steps:

[0192] 506. If there is newly added multimodal information, compare the newly added multimodal information with the multimodal information already in the database.

[0193] 507. If there is no information identical to the newly added multimodal information in the database, content mining is performed on the newly added multimodal information.

[0194] 508. Storing mining results mined from the newly added multimodal information into an information database.

[0195] In specific implementation, the execution body of the above-mentioned steps of querying the newly added multimodal information and steps 507 to 508 can be the server, and can be further subdivided into multiple execution bodies on the server. For example, on the server side, there is a first execution body for discovering the newly added multimodal information on the Dianping e-commerce platform (the execution body can be an independent server on the server side, a virtual server, or a functional module running on the server or the cloud, etc.), and there is also a second execution body for content mining of multimodal information using a variety of tools (such as openCV cutting tools, video frame extraction tools, image recognition tools, semantic recognition tools, etc.) (the execution body can also be an independent server on the server side, a virtual server, or a functional module running on the server or the cloud, etc.). Accordingly, the above-mentioned step 506 "If there is newly added multimodal information, compare the newly added multimodal information with the multimodal information already in the database" may include:

[0196] If there is newly added multimodal information, the first executor initiates a mining request for the newly added multimodal information;

[0197] The second executor uses at least one tool to perform content mining on the newly added multimodal information to obtain mining results;

[0198] The first execution body monitors the mining result, and stores the mining result corresponding to the newly added multimodal information mined by the second execution body into the information database to update the information database.

[0199] The mining request may carry a URL (website) corresponding to the multimodal information, so that the second execution body can obtain the multimodal information according to the URL.

[0200] An embodiment of the present application also provides an information processing device. The information processing device includes: an acquisition module, a determination module and an output module. The acquisition module is used to acquire the user's conversation information; the determination module is used to determine at least two modal reply utterance information that can respond to the conversation information based on the multimodal information provided by the service provider; the output module is used to use the trained first preset model to process the at least two modal reply utterance information for the conversation information and output the multimodal reply text.

[0201] The multimodal information on the e-commerce platform includes at least one of the following: multimodal information in the merchant's store and multimodal information provided by the e-commerce service provider; the multimodal information includes information items of at least two modes. Accordingly, when the determination module determines the reply corpus information of at least two modes that can respond to the dialogue information based on the multimodal information provided by the service provider, it is specifically used to: perform content mining on the information item to obtain the mining results corresponding to the information item; retrieve at least one target mining result related to the dialogue information in the information library; wherein the information library stores the mining results corresponding to each modal information item in the multimodal information provided by the service provider; and determine the reply corpus information of at least two modes based on the at least one target mining result and the information item of the modality corresponding to the target mining result.

[0202] The multimodal information includes a first modal information item, and the first modal information item is a picture. Accordingly, when the determination module performs content mining on the picture and obtains the mining result corresponding to the picture, it is specifically used to: use the trained second preset model to process the picture to obtain text information reflecting the content of the picture; the text information is the mining result corresponding to the picture.

[0203] Correspondingly, when the determination module retrieves at least one target mining result related to the user conversation information in the information database, it is specifically used to:

[0204] Using at least one retrieval algorithm, multiple candidate text information related to the dialogue information is retrieved from the information database; from the multiple candidate text information, at least one target text information is selected as the target mining result.

[0205] The multimodal information includes a first modal information item, which is a picture. Accordingly, when the determination module performs content mining on the picture and obtains the mining result corresponding to the picture, it is specifically used to: locate the text area in the picture; cut the picture to obtain a sub-picture containing the text area; the sub-picture is the mining result corresponding to the picture.

[0206] Correspondingly, when the determination module retrieves at least one target mining result related to the conversation information in the information base, it is specifically used to: identify the text content in the sub-graph in the information base; evaluate the correlation between the text content of each sub-graph in the information base and the conversation information; at least one sub-graph whose correlation meets the requirements is the at least one target mining result.

[0207] When the determination module determines the reply corpus information of the at least two modalities based on the at least one target mining result and the information item of the modality corresponding to the target mining result, it is specifically used to: if the at least one target mining result contains a sub-image, the original image to which the sub-image belongs is determined as the reply corpus information.

[0208] The multimodal information includes a second modal information item or a third modal information item; the second modal information item is a video, and the third modal information item is a moving picture. Accordingly, when the determination module performs content mining on the video or moving picture and obtains the mining result corresponding to the video or moving picture, it is specifically used to: identify the key elements in the video or moving picture; perform semantic analysis on the video or moving picture to obtain the content description of the video or moving picture; and determine the mining result corresponding to the video or moving picture based on the key elements and the content description.

[0209] Among them, when the determination module determines the mining results corresponding to the video or animated image based on the key elements and the content description, it is specifically used to: determine the category to which the video or animated image belongs based on the key elements and the content description; generate a content summary or keyword label of the video or animated image according to the category to which the video or animated image belongs, the key elements and the content description; the content summary or keyword label is the mining result corresponding to the video or animated image.

[0210] Furthermore, the device provided in the embodiment of the present application may also include a storage module. The storage module is used to group the mining results corresponding to the video or animation according to the category to which the video or animation belongs; the mining results corresponding to the video or animation in the information library are stored in groups for classification retrieval.

[0211] Further, when the determination module determines the reply corpus information of the at least two modalities according to the at least one target mining result and the information item of the modality corresponding to the target mining result, it is specifically used to:

[0212] If at least one of the target mining results contains a content summary or keyword tag of a target video or target animated picture related to the dialogue information, the target video or target animated picture is obtained; a video clip or animated picture segment matching the dialogue information is captured from the target video or target animated picture; and the video clip or animated picture segment is used as the reply data information.

[0213] Furthermore, the determination module in this embodiment is also used for: when the conversation information includes information items of at least one modality; when the conversation information includes at least one of pictures, videos, audios, and animated pictures, performing image and / or audio and video recognition on at least one of the pictures, videos, audios, and animated pictures contained in the conversation information; based on the recognition result, determining the query Query corresponding to the conversation information, so as to determine the reply corpus information of at least two modalities matching the Query based on the multimodal information provided by the service provider.

[0214] The multimodal information in the merchant store includes multiple items of the following: product pictures in the store, product display videos, advertising pictures, advertising videos, user review texts, user review pictures, user review videos, product description texts, product detail pictures, store activity promotion pictures, store activity promotion videos, activity promotion pictures provided by the e-commerce service provider, activity promotion videos provided by the e-commerce service provider, activity promotion animated pictures provided by the e-commerce service provider, and activity rules provided by the e-commerce service provider.

[0215] The present application also provides an information processing device, which includes a display module, wherein the display module is used to respond to interface operations, display dialogue information input by the user through at least one operation, and display or display and play multimodal reply text generated for the dialogue information.

[0216] The multimodal reply text is obtained by processing reply utterance information of at least two modalities for the dialogue information using a trained first preset model; and the reply utterance information of at least two modalities is determined based on the multimodal information provided by the service provider.

[0217] The present application also provides another information processing device, which includes an acquisition module, a determination module and an output module. The acquisition module is used to acquire dialogue information input by the user through at least one operation. The determination module is used to determine the query Query based on the dialogue information. The acquisition module is also used to acquire multimodal information related to the commodity corresponding to the ItemID according to the commodity identifier ItemID included in the Query. The determination module is also used to determine the reply corpus information of at least two modalities matching the Query based on the multimodal information related to the commodity corresponding to the ItemID. The output module is also used to process the reply corpus information of at least two modalities according to the dialogue information using the trained first preset model, and output the multimodal reply text.

[0218] Furthermore, the conversation information includes information items of at least one modality; when the conversation information includes at least one of pictures, videos, audio, and animated pictures, the determination module, when determining the query Query based on the conversation information, is specifically used to: perform image and / or audio and video recognition on at least one of the pictures, videos, audio, and animated pictures contained in the conversation information; and determine the Query corresponding to the conversation information based on the recognition result.

[0219] It should be noted here that: the various devices provided in the above embodiments can implement the technical solutions described in the corresponding method embodiments above. The specific implementation principles of the above modules or units can be found in the corresponding contents of the above method embodiments, which will not be repeated here.

[0220] An embodiment of the present application also provides an electronic device. Fig. 9 As shown, the electronic device includes a processor 42 and a memory 41. The memory 41 is used to store one or more computer programs; the processor 42 is coupled to the memory 41 and is used for the at least one or more computer programs to implement the steps in the method provided in each embodiment of the present application.

[0221] The above-mentioned memory 41 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0222] Furthermore, the electronic device also includes other components such as a communication component 43, a display 44, a power component 45, and an audio component 46. Only some components are schematically shown here, which does not mean that the electronic device only has these components.

[0223] Accordingly, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, the method steps or functions provided in the above-mentioned embodiments of the present application can be implemented.

[0224] The method in the present application can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. Thus, the present application also provides a computer program product. The computer program product includes a computer program / instruction, and when the computer program / instruction is executed by an electronic component such as a processor, the steps or functions in the method provided in each embodiment of the present application can be executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, a core network device, an OAM or other programmable device.

[0225] Through the description of the above implementation modes, those skilled in the art can clearly understand that each implementation mode can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0226] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An online human-computer dialogue method, characterized in that: include: Get the user's conversation information; Determining, based on the multimodal information provided by the service provider, reply utterance information of at least two modes that can respond to the dialogue information; The trained first preset model is used to process the reply corpus information of the at least two modalities according to the dialogue information, and a multimodal reply copy is output.

2. The method according to claim 1, characterized in that The multimodal information provided by the service provider includes at least one of the following: multimodal information in the merchant's store and multimodal information of the e-commerce service provider; the multimodal information includes information items of at least two modes; as well as Determining, based on the multimodal information provided by the service provider, reply utterance information of at least two modes that can respond to the dialogue information, including: Performing content mining on the information item to obtain mining results corresponding to the information item; Retrieving at least one target mining result related to the conversation information in an information database; wherein the information database stores mining results corresponding to each modal information item in the multimodal information provided by the service provider; According to the at least one target mining result and the information item of the modality corresponding to the target mining result, the reply corpus information of the at least two modalities is determined.

3. The method according to claim 2, characterized in that The multimodal information includes a first modal information item, where the first modal information item is a picture; as well as Perform content mining on the image to obtain mining results corresponding to the image, including: Using the trained second preset model, the image is processed to obtain text information reflecting the content of the image; The text information is the mining result corresponding to the image.

4. The method according to claim 3, characterized in that In the information database, at least one target mining result related to the user conversation information is retrieved, including: Retrieving a plurality of candidate text information related to the conversation information from the information database using at least one retrieval algorithm; At least one target text information is selected from the multiple candidate text information as a target mining result.

5. The method according to any one of claims 2 to 4, characterized in that The multimodal information includes a first modal information item, the first modal information item is a picture, and Perform content mining on the image to obtain mining results corresponding to the image, including: Locating a text area in the image; Cut the image to obtain a sub-image containing the text area; The sub-image is the mining result corresponding to the image.

6. The method according to claim 5, characterized in that In the information database, at least one target mining result related to the conversation information is retrieved, including: Identifying text content in a sub-image in the information library; Evaluating the relevance of the text content of each sub-image in the information base to the conversation information; At least one subgraph whose correlation meets the requirements is the at least one target mining result.

7. The method according to claim 6, characterized in that Determining the reply corpus information of the at least two modalities according to the at least one target mining result and the information item of the modality corresponding to the target mining result includes: If the at least one target mining result contains a sub-graph, the original picture to which the sub-graph belongs is determined as the reply corpus information.

8. The method according to claim 2, characterized in that: The multimodal information includes a second modal information item or a third modal information item; the second modal information item is a video, and the third modal information item is a moving picture; as well as Perform content mining on the video or animated image to obtain mining results corresponding to the video or animated image, including: Identify key elements of the video or animated graphic; Performing semantic analysis on the video or animated image to obtain a content description of the video or animated image; Based on the key elements and the content description, a mining result corresponding to the video or animated image is determined.

9. The method according to claim 8, characterized in that Based on the key elements and the content description, determining a mining result corresponding to the video or animated image includes: Based on the key elements and the content description, determine the category to which the video or animated image belongs; Generate a content summary or keyword tag of the video or animated image according to the category to which the video or animated image belongs, the key elements and the content description; The content summary or keyword tag is the mining result corresponding to the video or animated image.

10. The method according to claim 9, characterized in that Also includes: According to the category to which the video or animated image belongs, the mining results corresponding to the video or animated image are grouped; The mining results corresponding to the videos or animated images in the information library are stored in groups for classification retrieval.

11. The method according to claim 9, characterized in that Determining the reply corpus information of the at least two modalities according to the at least one target mining result and the information item of the modality corresponding to the target mining result includes: If the at least one target mining result contains a content summary or keyword tag of a target video or target animated image related to the conversation information, then Obtaining the target video or target animated image; From the target video or target animated picture, extract a video clip or an animated picture clip that matches the conversation information; The video clip or the animated picture clip is used as the reply text information.

12. The method according to claim 1, characterized in that The conversation information includes at least one modal information item; when the conversation information includes at least one item of a picture, a video, an audio, and a moving picture, the method further includes: Performing image and / or audio / video recognition on at least one of the pictures, videos, audios, and animated images included in the conversation information; Based on the recognition result, a query Query corresponding to the dialogue information is determined, so as to determine reply corpus information of at least two modes matching the Query based on the multimodal information provided by the service provider.

13. The method according to claim 1, characterized in that The multimodal information provided by the service provider includes multiple items of the following: product pictures in the store, product display videos, advertising pictures, advertising videos, user review texts, user review pictures, user review videos, product description texts, product detail pictures, product instruction videos, store activity promotion pictures, store activity promotion videos, activity promotion pictures provided by the e-commerce service provider, activity promotion videos provided by the e-commerce service provider, activity promotion animated pictures provided by the e-commerce service provider, and activity rules provided by the e-commerce service provider.

14. An online human-computer dialogue method, characterized in that: include: In response to the interface operation, the dialog information input by the user through at least one operation is displayed; Displaying or displaying and playing the multimodal reply text generated for the dialogue information; The multimodal reply text is obtained by processing reply utterance information of at least two modalities for the dialogue information using a trained first preset model; and the reply utterance information of at least two modalities is determined based on the multimodal information provided by the service provider.

15. An online human-computer dialogue method, characterized in that: include: Obtaining the dialog information input by the user through at least one operation; Determine a query Query according to the conversation information; According to the commodity identifier ItemID included in the Query, obtain multimodal information related to the commodity corresponding to the ItemID; Based on the multimodal information related to the commodity corresponding to the ItemID, determining at least two modal reply corpus information matching the Query; The trained first preset model is used to process the reply corpus information of the at least two modalities according to the dialogue information, and a multimodal reply copy is output.

16. The method according to claim 15, characterized in that The conversation information includes at least one modal information item; when the conversation information includes at least one item of a picture, a video, an audio, or a moving picture, Determining a query Query according to the conversation information includes: Performing image and / or audio / video recognition on at least one of the pictures, videos, audios, and animated images included in the conversation information; Based on the recognition result, the Query corresponding to the conversation information is determined.

17. A customer service system, characterized in that: include: The client is used to respond to the interface operation and display the dialogue information input by the user through at least one operation; A server, used to obtain the conversation information; Based on the multimodal information provided by the service provider, determine at least two modal reply utterances that can respond to the dialogue information; use the trained first preset model to process the at least two modal reply utterances according to the dialogue information, and output a multimodal reply copy; The client is used to display or display and play the multimodal reply text generated for the dialogue information.

18. An electronic device, characterized in that: comprising a memory and a processor; wherein, The memory is used to store executable instructions; The processor implements the steps of the information processing method described in any one of claims 1 to 13, or the steps of the information processing method described in claim 14, or the steps of the information processing method described in claim 15 or 16 by running the executable instructions.

19. A computer-readable storage medium, characterized in that: The storage medium stores computer instructions, which, when executed by a processor, can implement the steps of the information processing method described in any one of claims 1 to 13, or the steps of the information processing method described in claim 14, or the steps of the information processing method described in claim 15 or 16.

Images