A method and system for semantically configurable multimodal intelligent customer service dialogue

Abstract

This application discloses a method and system for semantically configurable multimodal intelligent customer service dialogue. The method includes: preparing text and image corpora; using a text semantic model to optimize the text corpora and output semantic vectors; using an image semantic model to extract features from the image corpora and perform image-to-text semantic conversion, outputting semantic vectors; granularizing the processing flow into thinking units and configuring guidance scenarios using these thinking units; converting the intent statements in all guidance scenarios into semantic vectors and organizing them into a semantic query tree; converting multimodal input information into semantic vectors, searching the semantic query tree to determine if an intent is triggered; if triggered, entering a thinking unit in the guidance scenario; starting from the current thinking unit, calculating each thinking unit sequentially and searching for the next thinking unit until the final answer is returned. This application implements an intelligent customer service dialogue that supports multimodal user interaction, including text and images.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for semantically configurable multimodal intelligent customer service dialogue. Background Technology

[0002] AI-powered customer service systems are increasingly being applied across various industries and business scenarios. However, the ever-changing nature of these scenarios presents challenges for the practical implementation of AI models. Two common solutions exist: one is to train a small number of general-purpose models applicable to all business scenarios, but this reduces the model's accuracy in specific business situations; the other common approach is to train models specifically for each business scenario, which improves accuracy but adds significant tasks related to data preparation, model training, and model validation. Furthermore, in the actual operation of intelligent customer service, increasingly more business scenarios are not limited to text; multimedia elements such as images are also being used. Integrating these elements with text to serve intelligent customer service is another challenge for practical implementation.

[0003] How to implement intelligent customer service conversations that support multimodal user interaction, such as text and images, in flexible, dynamic, and configurable intelligent guidance scenarios is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for semantically configurable multimodal intelligent customer service dialogue to solve the problems mentioned in the above-mentioned technical background.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] The first aspect of this application provides a method for semantically configurable multimodal intelligent customer service dialogue, including:

[0007] S1. Corpus Preparation: Prepare text corpus for text semantic model tuning, including question pairs and similarity labels; prepare image corpus for image semantic model training, including images and corresponding text descriptions.

[0008] S2. Semantic Representation Model Training: The text corpus is optimized using a pre-set text semantic model. After training according to the classification task, the vector representing the semantics of the whole sentence is set as the output and the model is exported. The image corpus is feature extracted using a pre-set image semantic model. Then, a Decoder network is added to perform semantic conversion from image to text, and the semantic vector representation with the same shape as the text semantic model is output.

[0009] S3. Configure the guided scenario: The possible processing flow of intelligent customer service is granularized into thinking units. Each thinking unit represents a processing process, including intent analysis, option matching or condition judgment. Select M repeatable thinking units, configure the parameters of each thinking unit, and use unidirectional edges to organize all thinking units into a directed acyclic graph to form a guided scenario.

[0010] S4. Multimodal Intent Analysis: Convert the textual messages in all configured guidance scenarios into semantic vectors using preset text semantic models and image semantic models, and organize them into a semantic query tree. Convert the user's multimodal input information into semantic vectors using text semantic models and image semantic models, and use the semantic query tree to find out whether the intent is triggered.

[0011] S5. Guided Path Calculation: When a user triggers an intent, it means entering a thinking unit in the guided scenario. Starting from the current thinking unit, each thinking unit is calculated one by one, and the next thinking unit is searched until the final answer is returned.

[0012] Preferably, step S1 specifically includes the following steps:

[0013] S11. Prepare questions for text semantic model tuning, and collect similar questions and questions that are literally similar but semantically dissimilar as text corpus;

[0014] S12. Annotate the collected text corpus. The annotation format is: Question A + Question B + Tag. The tags are divided into similar and dissimilar.

[0015] S13. Prepare image corpus for training the image semantic model, including images and corresponding text descriptions;

[0016] S14. Convert the text description corresponding to the image into a semantic vector using a text semantic model, and use it as the label for the image during the next step of model training.

[0017] Preferably, step S2 specifically includes the following steps:

[0018] S21. Take the text corpus, which includes question pairs and labels, and perform model optimization training for the classification task based on the BERT pre-trained model. After training, output an N-dimensional semantic vector.

[0019] S22. The image semantic model consists of a VGG classification pre-trained model and a Decoder network, which outputs the same N-dimensional semantic vector as the BERT pre-trained model. Multi-class cross-entropy is used as the loss function.

[0020] Preferably, in step S3, the intent analysis and option matching includes K statements, where K ≥ 1, and the statements include text or images.

[0021] Preferably, step S4 specifically includes the following steps:

[0022] S41. Convert the T dialogues included in the L guidance scenarios into T corresponding semantic vectors V using a preset text semantic model and image semantic model. Each guidance scenario includes O intents, and each intent includes P dialogues, where O≥1, L≥1, and P≥1.

[0023] S42. Calculate the center point vector S of the P semantic vectors in each intent.

[0024]

[0025] S43. Organize the T semantic vectors V into a two-level semantic query tree according to the central point vector S of the intent and the semantic vectors V;

[0026] S44. Convert the text or image input by the user into a query vector Q using the trained text semantic model or image semantic model. Use the query vector Q to find the top H center vectors with the highest similarity, where H>=1. Then find the most similar verbal vector V. If the similarity between the query vector Q and the verbal vector V is greater than the preset threshold R, it means that the intent is triggered; otherwise, it means that no intent is triggered.

[0027] Preferably, step S5 specifically includes the following steps:

[0028] S51. After semantic lookup, when a user triggers an intent, it indicates that they have entered a thinking unit E in a guided scenario;

[0029] S52. Starting from thinking unit E, calculate each thinking unit sequentially and find the next thinking unit. If there is a cached thinking unit, start the search for the next thinking unit from the cached thinking unit.

[0030] S53. If the calculation result of the current thinking unit is the user option, then return the user option and cache the current thinking unit;

[0031] S54. If the calculation result of the current thinking unit is the answer, then return the answer;

[0032] S55. Repeat steps S52 to S54 until the answer is returned.

[0033] The second aspect of this application provides a system for semantically configurable multimodal intelligent customer service dialogue, including:

[0034] The corpus preparation module is used to prepare the text corpus for text semantic model optimization. The text corpus includes question pairs and similarity labels. The image corpus is used to prepare the image semantic model for training. The image corpus includes images and their corresponding text descriptions.

[0035] The semantic representation model training module is used to fine-tune the model for text corpora using a preset text semantic model. After training by classification task, the vector representing the semantics of the whole sentence is set as the output and the model is exported. The preset image semantic model is used to extract features from image corpora, and then a Decoder network is added to perform semantic conversion from image to text, outputting a semantic vector representation with the same shape as the text semantic model.

[0036] The guided scenario configuration module is used to granulate the possible processing flow of intelligent customer service into thinking units. Each thinking unit represents a processing procedure, including intent analysis, option matching or condition judgment. After selecting M repeatable thinking units and configuring the parameters of each thinking unit, a unidirectional edge is used to organize all thinking units into a directed acyclic graph to form a guided scenario.

[0037] The multimodal intent analysis module is used to convert the verbal messages on the intents in all configured guidance scenarios into semantic vectors using preset text semantic models and image semantic models, and organize them into a semantic query tree. It converts the user's multimodal input information into semantic vectors through text semantic models and image semantic models, and uses the semantic query tree to find out whether the intent is triggered.

[0038] The guided path calculation module is used to guide users to a thinking unit in the guided scenario when they trigger an intent. Starting from the current thinking unit, it calculates each thinking unit one by one and searches for the next thinking unit until the final answer is returned.

[0039] A third aspect of this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0040] The memory is used to store computer programs;

[0041] When the processor executes a program stored in the memory, it implements the steps of any of the above methods.

[0042] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0043] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0044] Based on a small number of general models, this application realizes intelligent customer service sessions that support multimodal user interaction, including text and images, in a flexible, dynamic, and configurable intelligent guidance scenario. Attached Figure Description

[0045] The accompanying drawings, which constitute a part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0046] Figure 1 This is a flowchart of a preferred embodiment of the present invention of a method for semantically configurable multimodal intelligent customer service dialogue;

[0047] Figure 2 This is an example diagram of a directed acyclic graph of a guiding scenario in step S32 of the present invention;

[0048] Figure 3 This is an example diagram of a two-level semantic query tree formed in step S43 of the present invention;

[0049] Figure 4 This is a logical schematic diagram of a method for semantically configurable multimodal intelligent customer service dialogue according to the present invention;

[0050] Figure 5 Here is a specific example diagram of a guided scenario;

[0051] Figure 6 A schematic diagram of the structure of a semantically configurable multimodal intelligent customer service dialogue system provided in this application embodiment;

[0052] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0054] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0055] Example

[0056] Figure 1 This is a flowchart of a method for semantically configurable multimodal intelligent customer service dialogue.

[0057] like Figure 1 As shown, a method for semantically configurable multimodal intelligent customer service dialogue includes the following steps:

[0058] Step S1, Corpus Preparation:

[0059] S11, prepare questions for text semantic model tuning, and collect similar questions and questions that are literally similar but semantically dissimilar as text corpus.

[0060] S12, annotate the text corpus from the previous step. The annotation format is: Question A + Question B + Label, and the labels are divided into similar and dissimilar.

[0061] S13, Prepare the image corpus for training the image semantic model. The image corpus includes images and their corresponding text descriptions.

[0062] S14: Convert the text description corresponding to the image into a semantic vector using a text semantic model, which will be used as the label for the image during the next step of model training.

[0063] Step S2, Semantic Representation Model Training:

[0064] S21. Using the annotated text corpus from the previous step, perform model tuning and training for the classification task based on the BERT pre-trained model. After training, the model outputs an N-dimensional semantic vector.

[0065] BERT (Bidirectional Encoder Representations from Transformers) is a large-scale pre-trained language model based on bidirectional Transformers, released by Google. This pre-trained model can capture word and sentence-level representations separately, efficiently extracting text information and applying it to various NLP tasks. BERT pre-trained models typically require powerful computing capabilities and large datasets, taking considerable time to train. Therefore, they are usually trained by large companies like Google, and other users only need to download and fine-tune the pre-trained model. When using BERT for sequence labeling, a pre-trained model is typically loaded and fine-tuned. BERT then uses Softmax to predict the label for each token, achieving high accuracy. Because of BERT's high classification accuracy and the fact that it only requires fine-tuning, the BERT pre-trained model has become the preferred choice for sequence labeling.

[0066] S22, the image semantic model consists of a VGG classification pre-trained model + a Decoder network. The image semantic model outputs the same N-dimensional vector as the text semantic model. Multi-class cross-entropy is used as the loss function, as detailed below:

[0067]

[0068] Where L represents the loss;

[0069] i represents the i-th sample;

[0070] predict(i) represents the predicted value for the i-th sample;

[0071] label(i) represents the true value of the i-th sample;

[0072] C represents the dimension of the N-dimensional vector;

[0073] n represents the total number of samples in a single batch.

[0074] Step S3: Configure the boot scene:

[0075] S31 breaks down the possible processing steps of intelligent customer service into thinking units. Each thinking unit represents a processing procedure, such as intent analysis, option matching, and condition judgment. The thinking units for intent analysis and option matching can include K (K≥) dialogues, which can be text or images.

[0076] S32, select M repeatable thinking units, configure the parameters of each thinking unit, and then use unidirectional edges to organize all thinking units into a directed acyclic graph, such as... Figure 2As shown.

[0077] Step S4, Multimodal Intent Analysis:

[0078] S41, take L (L>=1) intelligent guidance scenarios, each scenario has O (O>=1) intents, and each intent has P (P>=1) dialogues, for a total of T dialogues, and use the text semantic model and image semantic model trained in the previous step to convert them into corresponding T semantic vectors V, where,

[0079]

[0080] S42, calculate the center point vector S of the P semantic vectors in each intent.

[0081]

[0082] S43, organize the T semantic vectors V into a two-level semantic query tree according to the central vector S of the intent and the semantic vectors V, such as Figure 3 As shown.

[0083] S44. The text or image input by the user is converted into a query vector Q using a trained text semantic model or image semantic model. The query vector Q is used to find the top H (H>=1) center vectors with the highest similarity. Then, the most similar verbal vector V is found. If the similarity between the query vector Q and the verbal vector V is greater than the preset threshold R, it means that the intent is triggered. Otherwise, it means that no intent is triggered.

[0084] Step S5, Guide Path Calculation:

[0085] S51, after the semantic lookup in the previous step, when a user triggers an intent, it means entering a thinking unit E in the guided scenario Li.

[0086] S52, starting from the current thinking unit E, calculate each thinking unit one by one and find the next thinking unit. If there is a cached thinking unit, start the search for the next unit from the cached thinking unit.

[0087] S53, if the result calculated by the current think cell is the user option, then return the user option and cache the current think cell.

[0088] S54: If the calculation result of the current thinking unit is the answer, then return the answer.

[0089] S55. Repeat the above three steps until the answer is returned.

[0090] The logical diagram of the above-described method for semantically configurable multimodal intelligent customer service dialogue is shown below. Figure 4 As shown.

[0091] Figure 5 A specific example diagram of a guidance scenario is provided, concerning the damage and repair of a manhole cover. This guidance scenario includes multiple thought units that the intelligent customer service may need to process, including multiple intent analysis units (such as API location acquisition, location information clarification, management department analysis, etc.) and option matching units (such as determining whether it is a repair process or a repair report). Each thought unit represents a processing procedure, and all thought units are organized into a directed acyclic graph using unidirectional edges. When the user inputs a relevant image or text, after image semantic recognition or text semantic recognition, if an intent in the guidance scenario is triggered, then starting from the current thought unit, each thought unit is calculated one by one, and the next thought unit is searched until the final answer is returned.

[0092] On the other hand, this application also discloses a system based on semantically configurable multimodal intelligent customer service dialogue. See also... Figure 6 As shown, the system includes a corpus preparation module 100, a semantic representation model training module 200, a guidance scenario configuration module 300, a multimodal intent analysis module 400, and a guidance path calculation module 500.

[0093] The corpus preparation module 100 is used to prepare text corpus for text semantic model optimization, the text corpus includes question pairs and similarity labels; and to prepare image corpus for image semantic model training, the image corpus includes images and corresponding text descriptions.

[0094] The semantic representation model training module 200 is used to fine-tune the text corpus using a preset text semantic model, and after training according to the classification task, set the vector representing the semantics of the whole sentence as the output and export the model; after extracting features from the image corpus using a preset image semantic model, a Decoder network is added to perform semantic conversion from image to text, and the output semantic vector representation with the same shape as the text semantic model is output.

[0095] The guidance scenario configuration module 300 is used to granulate the processing flow that the intelligent customer service may need into thinking units. Each thinking unit represents a processing procedure, including intent analysis, option matching or condition judgment. After selecting M repeatable thinking units and configuring the parameters of each thinking unit, a unidirectional edge is used to organize all thinking units into a directed acyclic graph to form a guidance scenario.

[0096] The multimodal intent analysis module 400 is used to convert the verbal messages on the intents in all configured guidance scenarios into semantic vectors using preset text semantic models and image semantic models, and organize them into a semantic query tree. It converts the user's multimodal input information into semantic vectors through text semantic models and image semantic models, and uses the semantic query tree to find out whether the intent is triggered.

[0097] The guidance path calculation module 500 is used to enter a thinking unit in the guidance scenario when a user triggers an intention, starting from the current thinking unit, calculating each thinking unit one by one, and searching for the next thinking unit until the final answer is returned.

[0098] On the other hand, embodiments of this application also provide an electronic device 600, such as... Figure 7 As shown, it includes a processor 601, a communication interface 602, a memory 603, and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0099] The memory 603 is used to store computer programs.

[0100] When the processor 601 executes the program stored in the memory 603, it implements the steps of the above-described method for semantically configurable multimodal intelligent customer service dialogue.

[0101] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0102] The communication interface 602 is used for communication between the aforementioned electronic device 600 and other devices.

[0103] The memory 603 may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory 603 may also be at least one storage device located remotely from the aforementioned processor 601.

[0104] The processor 601 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0105] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the methods described above for semantically configurable multimodal intelligent customer service dialogue.

[0106] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the steps of any of the semantically configurable multimodal intelligent customer service dialogue methods described in the above embodiments.

[0107] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0108] The specific embodiments of the present invention have been described in detail above, but they are merely examples, and the present invention is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications and substitutions to the present invention are also within the scope of the present invention. Therefore, all equivalent transformations and modifications made without departing from the spirit and scope of the present invention should be covered within the scope of the present invention.

Claims

1. A method for semantically configurable multimodal intelligent customer service dialogue, characterized in that, include: S1. Corpus Preparation: Prepare text corpus for text semantic model tuning. The text corpus includes question pairs and similarity labels. Prepare image corpora for training the image semantic model. The image corpora include images and their corresponding text descriptions. S2. Semantic Representation Model Training: The text corpus is optimized using a pre-set text semantic model. After training according to the classification task, the vector representing the semantics of the whole sentence is set as the output and the model is exported. The image corpus is feature extracted using a pre-set image semantic model. Then, a Decoder network is added to perform semantic conversion from image to text, and the semantic vector representation with the same shape as the text semantic model is output. S3. Configure the guided scenario: The possible processing flow of intelligent customer service is granularized into thinking units. Each thinking unit represents a processing process, including intent analysis, option matching or condition judgment. Select M repeatable thinking units, configure the parameters of each thinking unit, and use unidirectional edges to organize all thinking units into a directed acyclic graph to form a guided scenario. S4. Multimodal Intent Analysis: Convert the textual messages in all configured guidance scenarios into semantic vectors using preset text semantic models and image semantic models, and organize them into a semantic query tree. Convert the user's multimodal input information into semantic vectors using text semantic models and image semantic models, and use the semantic query tree to find out whether the intent is triggered. S5. Guided Path Calculation: When a user triggers an intent, it means entering a thinking unit in the guided scenario. Starting from the current thinking unit, each thinking unit is calculated one by one, and the next thinking unit is searched until the final answer is returned. Step S4 specifically includes the following steps: S41. Convert the T dialogues included in the L guidance scenarios into T corresponding semantic vectors V using a preset text semantic model and image semantic model. Each guidance scenario includes O intents, and each intent includes P dialogues, where O≥1, L≥1, and P≥1. ; S42. Calculate the center point vector S of the P semantic vectors in each intent. ； S43. Organize the T semantic vectors V into a two-level semantic query tree according to the central point vector S of the intent and the semantic vectors V; S44. Convert the text or image input by the user into a query vector Q using the trained text semantic model or image semantic model. Use the query vector Q to find the top H center vectors with the highest similarity, where H>=1. Then find the most similar verbal vector V. If the similarity between the query vector Q and the verbal vector V is greater than the preset threshold R, it means that the intent is triggered; otherwise, it means that no intent is triggered.

2. The method for semantically configurable multimodal intelligent customer service dialogue according to claim 1, characterized in that, Step S1 specifically includes the following steps: S11. Prepare questions for text semantic model tuning, and collect similar questions and questions that are literally similar but semantically dissimilar as text corpus; S12. Annotate the collected text corpus. The annotation format is: Question A + Question B + Tag. The tags are divided into similar and dissimilar. S13. Prepare image corpus for training the image semantic model, including images and corresponding text descriptions; S14. Convert the text description corresponding to the image into a semantic vector using a text semantic model, and use it as the label for the image during the next step of model training.

3. The method for semantically configurable multimodal intelligent customer service dialogue according to claim 1, characterized in that, Step S2 specifically includes the following steps: S21. Take the text corpus, which includes question pairs and labels, and perform model optimization training for the classification task based on the BERT pre-trained model. After training, output an N-dimensional semantic vector. S22. The image semantic model consists of a VGG classification pre-trained model and a Decoder network, which outputs the same N-dimensional semantic vector as the BERT pre-trained model. Multi-class cross-entropy is used as the loss function.

4. The method for semantically configurable multimodal intelligent customer service dialogue according to claim 1, characterized in that, In step S3, the intent analysis and option matching include K statements, where K ≥ 1, and the statements include text or images.

5. The method for semantically configurable multimodal intelligent customer service dialogue according to claim 1, characterized in that, Step S5 specifically includes the following steps: S51. After semantic lookup, when a user triggers an intent, it indicates that they have entered a thinking unit E in a guided scenario; S52. Starting from thinking unit E, calculate each thinking unit sequentially and find the next thinking unit. If there is a cached thinking unit, start the search for the next thinking unit from the cached thinking unit. S53. If the calculation result of the current thinking unit is the user option, then return the user option and cache the current thinking unit; S54. If the calculation result of the current thinking unit is the answer, then return the answer; S55. Repeat steps S52 to S54 until the answer is returned.

6. A system for semantically configurable multimodal intelligent customer service dialogue, characterized in that, include: The corpus preparation module is used to prepare the text corpus for text semantic model optimization. The text corpus includes question pairs and similarity labels. Prepare the image corpus for training the image semantic model. The image corpus includes images and their corresponding text descriptions. The semantic representation model training module is used to fine-tune the model for text corpora using a preset text semantic model. After training by classification task, the vector representing the semantics of the whole sentence is set as the output and the model is exported. The preset image semantic model is used to extract features from image corpora, and then a Decoder network is added to perform semantic conversion from image to text, outputting a semantic vector representation with the same shape as the text semantic model. The guided scenario configuration module is used to granulate the possible processing flow of intelligent customer service into thinking units. Each thinking unit represents a processing procedure, including intent analysis, option matching or condition judgment. After selecting M repeatable thinking units and configuring the parameters of each thinking unit, a unidirectional edge is used to organize all thinking units into a directed acyclic graph to form a guided scenario. The multimodal intent analysis module is used to convert the verbal messages on the intents in all configured guidance scenarios into semantic vectors using preset text semantic models and image semantic models, and organize them into a semantic query tree. It converts the user's multimodal input information into semantic vectors through text semantic models and image semantic models, and uses the semantic query tree to find out whether the intent is triggered. Specifically include: The T dialogues included in the L guidance scenarios are transformed into T corresponding semantic vectors V using a pre-defined text semantic model and image semantic model. Each guidance scenario includes O intents, and each intent includes P dialogues, where O≥1, L≥1, and P≥1. ; Calculate the center point vector S of the P semantic vectors in each intent. ； Organize the T semantic vectors V into a two-level semantic query tree according to the central point vector S of the intent and the semantic vectors V; The text or image input by the user is converted into a query vector Q using a trained text semantic model or image semantic model. The query vector Q is used to find the top H center vectors with the highest similarity, where H>=1. Then, the most similar verbal vector V is found. If the similarity between the query vector Q and the verbal vector V is greater than a preset threshold R, it means that the intent is triggered; otherwise, it means that no intent is triggered. The guided path calculation module is used to guide users to a thinking unit in the guided scenario when they trigger an intent. Starting from the current thinking unit, it calculates each thinking unit one by one and searches for the next thinking unit until the final answer is returned.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the steps of the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 5.

Images