Multi-mode RAG system rapid construction method in intelligent customer service field

Abstract

The invention discloses a multi-mode RAG system rapid construction method in the field of intelligent customer service, relates to the technical field of intelligent customer service, and solves the problems that an existing RAG system based on a pure large language model can only generate plain texts, cannot understand and generate pictures, cannot well complete output interaction with the pictures, and cannot generate the pictures. Therefore, the technical problem of great limitation in many scenes requiring both pictures and texts is solved. The method comprises the following steps: dividing an original context corpus into a positive / negative sample corpus; mixing the divided corpora with a general data set of a large language model according to a preset proportion, performing training optimization on the large language model, constructing a multi-modal database, extracting corresponding document slices from the multi-modal database according to questions of a user, inputting the document slices into the large language model, generating answers, and pushing the answers to the user; according to the method and the device, richer and more complete output is provided, and the user experience and the universality of application scenes are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent customer service, and specifically is a method for quickly constructing a multimodal RAG system in the field of intelligent customer service. Background Art

[0002] With the continuous development and implementation of big language model technology, more and more industries are using big language model technology to build their own intelligent knowledge question answering system (RAG) to help answer various professional questions of internal employees and customers, especially in professional knowledge retrieval, product instructions and common troubleshooting, which has great use value.

[0003] The RAG system enhances the generation capability of the large language model by retrieving an external knowledge base. It inputs the retrieved knowledge into the large language model to optimize the model's generation results, making it more accurate, more contextual, and reducing the generation of misleading information. This technology installs a "knowledge plug-in" for the large language model, allowing it to answer the latest and most professional relevant knowledge in the vertical field without the need for training, thereby improving the accuracy, real-time and relevance of the answers. However, the current RAG system based on a pure large language model can only generate plain text, cannot understand and generate pictures, and cannot complete output interactions with pictures well, resulting in great limitations in many scenarios that require both text and pictures. Therefore, the present invention provides a method for quickly building a multimodal RAG system in the field of intelligent customer service. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a method for quickly building a multimodal RAG system in the field of intelligent customer service, which is used to solve the technical problem that the current RAG system based on a pure large language model can only generate plain text, cannot understand and generate pictures, and cannot complete output interaction with pictures well, resulting in great limitations in many scenarios that require both text and pictures.

[0005] To achieve the above objectives, a first aspect of the present invention provides a method for quickly building a multimodal RAG system in the field of intelligent customer service, comprising the following steps:

[0006] Step 1: Collect several reading comprehension datasets from open source data and add image placeholders to the locations with image identifiers to generate original context corpus;

[0007] Step 2: Send the original context corpus with the image placeholder and the corresponding question into the big model to generate the corresponding answer, and use the big model to review the question and answer in combination with the context, and divide the context corpus into positive sample corpus and negative sample corpus;

[0008] Step 3: Mix the positive sample corpus and the negative sample corpus with the general data set of the large language model in proportion, and train and optimize the large language model;

[0009] Step 4: construct a multimodal database based on the knowledge documents submitted by the user; wherein the multimodal database includes a text database and an image database;

[0010] Step 5: Based on the user's question, the multimodal RAG system extracts the corresponding document slices from the multimodal database, inputs them into the large language model, generates answers and pushes them to the user.

[0011] Preferably, the positive sample corpus is a corpus in which both the context corpus and the answer have picture placeholders; the negative sample corpus is a corpus in which the context corpus has picture placeholders but the answer does not have picture placeholders.

[0012] Preferably, the step of constructing a multimodal database based on the knowledge documents submitted by the user comprises the following steps:

[0013] Step S1: Parse the entire document;

[0014] Step S2: Cut the parsed document into several document slices and store them in a database for future use, thereby obtaining a text database;

[0015] Step S3: Number the images in the document slices in sequence with the document slices, and rename them in association. Use image placeholders to replace the document slice image positions, store the image names and images in a database for future use, and obtain an image database.

[0016] Preferably, the document slices with the image placeholders added are sent to the large language model to generate question-answer pairs, an index is created with the question vector, and the document slices are used as the corresponding answers, which are respectively inserted into the multimodal database for standby use.

[0017] Preferably, according to the user's question, the multimodal RAG system extracts the corresponding document slice from the text vector library, inputs it into the large language model, outputs the answer, and pushes it to the user, including:

[0018] Determine whether there is an image placeholder in the output answer; if so, use the name of the image placeholder to reverse search the image database, replace the image placeholder with the original image, and then push it to the customer; if not, push the answer directly to the user.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] 1. The present invention uses image placeholders and large models to replace the conventional way of editing specific image-text pairs to construct data sets. This method does not require a large amount of collection and production of real and specific image-text data sets, making data construction more flexible and reducing the difficulty and cost of data set production;

[0021] 2. The present invention uses the context understanding and reasoning capabilities of a pure large language model to achieve image-text reasoning and interaction, so that the RAG system can also achieve multi-modal interaction of images and text without training a large multi-modal model, reducing the system's computing power requirements and the threshold for model optimization, saving time for application implementation;

[0022] 3. The present invention combines the advantages of a large language model, and is not only used as a recall model, but also has the function of a generation model, effectively reducing the accumulated errors in the traditional RAG system due to the multi-stage recall and sorting process, significantly improving the accuracy of document knowledge questions and answers, and optimizing the aesthetics of the answer format;

[0023] 4. The present invention collects several reading comprehension data sets, generates original context corpus, and divides positive sample corpus and negative sample corpus; mixes the divided corpus with the general data set of the large language model according to a preset ratio, and trains and optimizes the large language model; constructs a multimodal database based on the knowledge documents submitted by the user; the multimodal RAG system extracts corresponding document slices from the multimodal database according to the user's question, and inputs them into the large language model to generate answers and push them to the user; by constructing a multimodal RAG system, the limitation that the pure large language model RAG system can only generate plain text is overcome, and a richer and more complete output is provided, which significantly improves the user experience and the breadth of application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 It is a schematic diagram of the RAG system flow of the present invention;

[0026] Figure 2 The figure is a schematic diagram of the corpus generation process of the present invention. DETAILED DESCRIPTION

[0027] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] See also Figure 1-Figure 2 The first aspect of the present invention provides a method for quickly building a multi-modal RAG system in the field of intelligent customer service, comprising the following steps:

[0029] Step 1: Collect several reading comprehension datasets from open source data and add image placeholders to the locations with image identifiers to generate original context corpus;

[0030] Specifically, several Chinese and English reading comprehension datasets are collected from open source data. First, the question and answer corpus in the field of intelligent customer service is preprocessed. Then, image placeholders are added in sequence in the context of words such as "as shown in the figure", "as shown in the figure below", and "picture" to construct the original context corpus.

[0031] Step 2: Send the original context corpus with image placeholders and the corresponding questions into the big model to generate the corresponding answers, and use the big model to review the rationality of the questions and answers in combination with the context to obtain positive sample corpus and negative sample corpus;

[0032] Specifically, the original context corpus with image placeholders and the corresponding questions are sent to the large model (such as the Spark 75B large model) to generate the corresponding answers. The large model is used to review the rationality of the questions and answers in combination with the context. That is, the large language model's context association ability and language understanding ability are mainly used for automatic judgment. The judgment logic is mainly achieved by adjusting the prompt.

[0033] The corpus with image placeholders in both the context corpus and the answer is retained as the positive sample, and the corpus with image placeholders in the context corpus but no image placeholders in the answer is retained as the negative sample.

[0034] For example: Positive sample corpus example:

[0035] Context: 1. If the camera cannot load images or is offline, please follow the steps below: ① Check the network status of the camera location. Use the website http: / / www.speedtest.cn / to test the upload bandwidth. It should be above 15m / s. Alternatively, you can test the camera and router by powering off and restarting. ② Check whether the memory card is plugged in or unplugged with power or the card recording is abnormal. Power off and unplug the memory card to test the machine. If it is normal, you need to format the memory card or replace it. ③ Make sure the APP and device firmware versions are the latest; the specific steps are shown in the figure below: [IMAGE].

[0036] 2. What problem do you want to ask about the water dispenser? You can describe in detail: ① Operation type, you can reply: How to set the child lock function of the water dispenser ② Product failure type, you can reply: The water dispenser does not produce water.

[0037] User asked: "What should I do if the camera cannot connect to the network?"

[0038] Answer: If the camera cannot connect to the Internet, you can troubleshoot the problem by following the steps below:

[0039] 1. Check whether the network status of the camera is good. Use the website http: / / www.speedtest.cn / to test the upload bandwidth. It should be above 15m / s. Alternatively, you can test the camera and router by powering off and restarting them.

[0040] 2. Whether the memory card is plugged or unplugged with power on or the card recording is abnormal. Turn off the power and unplug the memory card to test the machine. If it is normal, you need to format the memory card or replace it.

[0041] 3. Make sure the APP and device firmware versions are up to date; the specific steps are as shown in the figure below: [IMAGE].

[0042] Intelligent customer service will help you find solutions~

[0043] In this example, both the context and the answer contain the image placeholder [IMAGE], so this is a positive sample corpus.

[0044] Negative sample corpus example:

[0045] Context: Same as above.

[0046] User question: "Water smells strange"

[0047] Answer: Hello, are you asking about the problem of odor coming out of the water dispenser?

[0048] You can describe in detail:

[0049] ① Operation type: you can reply: water dispenser child lock function;

[0050] ② Product failure type, you can reply: What should I do if the water dispenser does not produce water?

[0051] Intelligent customer service will help you find solutions~

[0052] In this example, the context only contains the image placeholder [IMAGE], but the answer does not mention images, so this is a negative sample.

[0053] Step 3: Mix the positive sample corpus and the negative sample corpus with the general data set of the large language model in a preset ratio; train and optimize the large language model based on the mixed general data set;

[0054] Specifically, the Lora training method is used to optimize its own large language model (the large language model used in vertical fields has a smaller parameter scale, generally 3 to 8B, and the token length is at least 32k), so that it has the ability to generate text and understand the relationship between images and text.

[0055] It should be noted that the mixing of data sets is determined by algorithm engineers based on experience such as task difficulty and model convergence. The ratio is not fixed and is generally 5:5.

[0056] Step 4: construct a multimodal database based on the knowledge documents submitted by the user; wherein the multimodal database includes a text database and an image database;

[0057] It should be noted that the formats of knowledge documents include word, pdf, etc.

[0058] Specifically, the multimodal database construction process is as follows:

[0059] Step S1: Parse the entire document;

[0060] Step S2: Cut the parsed document into several document slices and store them in a database for future use, thereby obtaining a text database;

[0061] The present invention unevenly segments the document, and the text slicing methods include by paragraph, based on fixed characters, by text semantics, by fixed size, by custom rules, and the like.

[0062] Step S3: Number the images in the document slices in sequence with the document slices, and rename them in association. Use image placeholders to replace the document slice image positions, store the image names and images in a database for future use, and obtain an image database.

[0063] Among them, the document slices with image placeholders added are sent to the large language model to generate question-answer pairs, the question vector is used to create an index, and the document slices are used as the corresponding answers, which are inserted into the multimodal database for standby use.

[0064] For example: Suppose we have a travel guide document, which contains not only text information, but also photos, maps and some travel tips of various attractions.

[0065] First, parse the document into a processable format, such as HTML or XML, so that the text and image information in the document can be extracted more easily.

[0066] Secondly, the parsed document is cut into multiple document slices according to certain rules. For example, it can be divided according to the information of each scenic spot, ensuring that each document slice contains all the information of a scenic spot. In this way, if a scenic spot has more information, the document slice may be larger; otherwise, it will be smaller. Each document slice will be stored as an independent entry in the text database;

[0067] Then, for each document slice, we number and rename the images in the order they appear in the document. For example, the first image of the first spot is named "Spot1_Img1", the second image is named "Spot1_Img2", and so on. These images and their names are stored in the image database;

[0068] Finally, in the original document slice, the position of the image will be replaced by the corresponding placeholder, such as using a label like "[Spot1_Img1]". These document slices with placeholders will be fed into the large language model to generate question-answer pairs. For example, for "Spot1_Img1", the question that may be generated is: "Which famous scenic spot in which country was this photo taken?" And the answer is the text description of the scenic spot in the document slice.

[0069] The questions in the question-answer pair are generated by a large language model, converted into vector form, and indexed for fast retrieval. Each question vector points to its corresponding document slice, that is, the answer to the question. This information will be integrated into the multimodal database, allowing users to quickly obtain relevant text and image information by asking questions.

[0070] For example, suppose there is a travel guide document about Paris, France, and one of the document slices contains relevant information about the Eiffel Tower, including a description, a photo of the Eiffel Tower, and a traffic guide map. In the process of building a multimodal database:

[0071] Document slice: "The Eiffel Tower is one of the most famous landmarks in Paris, France, attracting millions of tourists every year. [Spot1_Img1][Spot1_Img2]"

[0072] Image database:

[0073] "Spot1_Img1": Photo of the Eiffel Tower

[0074] "Spot1_Img2": How to get to the Eiffel Tower

[0075] Question and answer:

[0076] Question: "Where was this photo taken?"

[0077] Answer: "The Eiffel Tower is one of the most famous landmarks in Paris, France, attracting millions of tourists every year."

[0078] Step 5: Based on the user's question, the multimodal RAG system extracts the corresponding document slices from the multimodal database, inputs them into the large language model, generates answers and pushes them to the user.

[0079] Determine whether there is an image placeholder in the output answer; if so, use the name of the image placeholder to reverse search the image database, replace the image placeholder with the original image, and then push it to the customer; if not, push the answer directly to the user.

[0080] Specifically, when a customer asks a question to the RAG system, the customer's question is used to recall the corresponding document slices in the text vector library, and the document slices are combined with the corresponding prompt and sent to the large language model. The final answer is generated by using the large model's context understanding, reasoning, and text polishing capabilities.

[0081] For example, a user asks: "Where is the Eiffel Tower?" The multimodal processing flow is as follows:

[0082] Step 1: Extract document slices from the multimodal database;

[0083] Based on the user's question, the system retrieves relevant document slices from the multimodal database. Assume that the document slices found are as follows:

[0084] Document Slice: The Eiffel Tower is one of the most famous landmarks in Paris, France, attracting millions of tourists every year. [Spot1_Img1][Spot1_Img2]

[0085] Step 2: Input document slices into the large language model;

[0086] The system inputs the extracted document slices into a large language model to generate answers.

[0087] Answers generated by the large language model:

[0088] The Eiffel Tower is located in Paris, France, and is one of the most famous landmarks in the world. It attracts millions of tourists every year. [Spot1_Img1](This photo shows the spectacular view of the Eiffel Tower); [Spot1_Img2](This is a map showing how to get to the Eiffel Tower).

[0089] Step 3: Determine whether there is an image placeholder in the answer

[0090] The system checks if there are image placeholders in the generated answer. In this example, the answer does contain image placeholders [Spot1_Img1] and [Spot1_Img2].

[0091] Step 4: Reverse search the image database and replace the image placeholder

[0092] The system uses the names of the image placeholders Spot1_Img1 and Spot1_Img2 to look up the image database, find the corresponding images, and replace the image placeholders with the original images.

[0093] Image database:

[0094] Spot1_Img1: Photo of the Eiffel Tower

[0095] Spot1_Img2: How to get to the Eiffel Tower

[0096] Step 5: The system pushes the final answer and picture to the user.

[0097] The final answer is: "The Eiffel Tower is located in Paris, France, and is one of the most famous landmarks in the world. It attracts millions of tourists every year.

[0098] [Eiffel Tower][Spot1_Img1]

[0099] [How to get to the Eiffel Tower][Spot1_Img2] ”

[0100] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A method for rapidly constructing a multimodal RAG system in the field of intelligent customer service, characterized in that: The following steps are involved: Step 1: Collect several reading comprehension datasets from open source data and add image placeholders to the locations with image identifiers to generate original context corpus; Step 2: Send the original context corpus with the image placeholder and the corresponding question into the big model to generate the corresponding answer, and use the big model to review the question and answer in combination with the context, and divide the context corpus into positive sample corpus and negative sample corpus; Step 3: Mix the positive sample corpus and the negative sample corpus with the general data set of the large language model in proportion, and train and optimize the large language model; Step 4: construct a multimodal database based on the knowledge documents submitted by the user; wherein the multimodal database includes a text database and an image database; Step 5: Based on the user's question, the multimodal RAG system extracts the corresponding document slices from the multimodal database, inputs them into the large language model, generates answers and pushes them to the user.

2. According to claim 1, a method for rapidly constructing a multimodal RAG system in the field of intelligent customer service, characterized in that: The positive sample corpus is a corpus with picture placeholders in both the context corpus and the answer; the negative sample corpus is a corpus with picture placeholders in the context corpus but no picture placeholders in the answer.

3. The method for rapidly constructing a multimodal RAG system in the field of intelligent customer service according to claim 1, characterized in that: The multimodal database is constructed according to the knowledge documents submitted by the user, including the following steps: Step S1: Parse the entire document; Step S2: Cut the parsed document into several document slices and store them in a database for future use, thereby obtaining a text database; Step S3: Number the images in the document slices in sequence with the document slices, and rename them in association. Use image placeholders to replace the document slice image positions, store the image names and images in a database for future use, and obtain an image database.

4. The method for rapidly constructing a multimodal RAG system in the field of intelligent customer service according to claim 3, characterized in that: The document slices with image placeholders added are sent to the large language model to generate question-answer pairs. An index is created with the question vector, and the document slices are used as the corresponding answers, which are inserted into the multimodal database for backup.

5. The method for rapidly constructing a multimodal RAG system in the field of intelligent customer service according to claim 1, characterized in that: According to the user's question, the multimodal RAG system extracts the corresponding document slice from the text vector library and inputs it into the large language model to generate the answer and push it to the user, including: Determine whether there is an image placeholder in the output answer; if so, use the name of the image placeholder to reverse search the image database, replace the image placeholder with the original image, and then push it to the customer; if not, push the answer directly to the user.

Images