An artificial agent generation method and system based on RAG technology

Abstract

The application belongs to the technical field of file data processing, and discloses an artificial intelligence agent generation method and system based on RAG technology. The method creates a knowledge base, uses a bound vector model and a document processing model to retrieve relevant documents and generate answers or texts, the document processing model is responsible for preprocessing and vectorization of the document library, the vector model processes user input queries, and the queries are converted into vector representation; import data into the knowledge base, queue the data import time sequence into vectorized numerical values; create a robot application, associate the knowledge base; publish the robot application and chat. The application solves the problems of personnel implementation of RAG technology, content missing, missing top-ranked documents and difficulty in processing unstructured documents.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of file data processing, and particularly relates to an artificial intelligence agent generation method and system based on RAG technology. BACKGROUND

[0002] Retrieval-augmented Generation, referred to as RAG, is one of the current popular large model front technologies. In 2020, Facebook AI Research (FAIR) proposed the paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks". This paper first proposed the concept of RAG and made a detailed introduction and explanation of the concept. The technical principle of the concept is as follows:

[0003] The retrieval-augmented generation model combines language models and information retrieval techniques. Specifically, when the model needs to generate text or answer questions, it will first retrieve relevant information from a large document set, and then use the retrieved information to guide the generation of text, thereby improving the quality and accuracy of prediction.

[0004] The current problem of this large model enhancement technology is that a large number of professional technical personnel are needed to participate in the data training, fine-tuning and data vectorization calculation process. In addition, when there is no actual answer in the knowledge base, the RAG system may provide seemingly reasonable but incorrect answers; sometimes important documents may not appear in the top results returned by the system retrieval component, resulting in the correct answer being ignored. When extracting information in non-text format, there are difficulties in processing unstructured documents. SUMMARY

[0005] In order to overcome the problems in the related art, the application discloses an artificial intelligence agent generation method and system based on RAG technology.

[0006] The technical solution is as follows: the artificial intelligence agent generation method based on RAG technology comprises:

[0007] S1, creating a knowledge base, retrieving relevant documents and generating answers or text by using a bound vector model and a document processing model, the document processing model being responsible for preprocessing and vectorizing the document library, the vector model processing user input query and converting the query into a vector representation;

[0008] S2, importing data into the knowledge base, queuing the data according to the time sequence of data import into a vectorized numerical value;

[0009] S3, create a robot application, associate a knowledge base;

[0010] S4, publish the robot application, chat;

[0011] S5, data correction of the robot question and answer data, so that the robot can provide more accurate answers.

[0012] In step S1, the knowledge base is created, including:

[0013] Query preprocessing: cleaning and tokenizing the document;

[0014] Vector representation: convert the query into a vector using the same language model as the document, so that the query and the document are in the same vector space for similarity calculation;

[0015] Using the query vector and document vector generated by the vector model, the system calculates the similarity between the query and each document; through similarity calculation, the system retrieves the N most relevant documents from the document library;

[0016] The retrieved relevant documents are input into the generation model based on the Transformer architecture together with the original query, and the generation model generates the final answer or text using these inputs.

[0017] Further, the generation model generates the final answer or text using these inputs, including:

[0018] Input construction: concatenate the query and the retrieved relevant documents into an input sequence and provide it to the generation model;

[0019] Context enhancement: the generation model effectively utilizes the information of the relevant documents through the attention mechanism to generate informative answers.

[0020] In step S2, the data is queued in the order of the time of data import into a vectorized numerical value, including:

[0021] Text cleaning: remove noise from the document;

[0022] Tokenization and normalization: tokenize the document text and perform normalization processing;

[0023] Vector representation: use a pre-trained language model to convert the processed document into a fixed-dimensional vector;

[0024] In the training phase, the document processing model and the vector model are jointly optimized to make the entire RAG system consistent;

[0025] Joint training: during training, optimize the document vector and the query vector so that the relevant documents and the query are close in the vector space;

[0026] Contrastive learning: further optimize the vector model using contrastive learning methods to make similar document and query vectors close to each other, and unrelated document and query vectors far from each other.

[0027] Further, in tokenization and normalization, normalization processing includes stem extraction and morphological reduction.

[0028] In step S3, a robot application is created, and a knowledge base is associated, including:

[0029] A robot ID, name, icon, introduction, and role setting are created, and the knowledge base created in step S1 is selected for binding, multiple knowledge bases are bound, options such as whether to display the dialogue context in the foreground, whether to display the quoted content in the foreground, whether to make the robot public, etc. are set, AI interface, similarity, number of single searches, empty search reply, welcome speech, bottom identification, menu items are set, digital human image is enabled, and digital human video is bound for personification broadcasting.

[0030] In step S4, the robot application is published, and chat is performed, including:

[0031] For content missing, when there is no actual answer in the knowledge base, reasonable but incorrect answers provided by the RAG system are processed by cleaning up data sources and correcting problem answers, and a prompt is designed to encourage the model to admit limitations and transparently convey uncertainty; cleaning up data sources means deleting data lists in the knowledge base; correcting problem answers means modifying the content of question and answer pairs in the knowledge base;

[0032] For missing top-ranked documents, by adjusting the chunk_size and similarity_top_k hyperparameters, the retrieval results are reordered before being sent to the LLM, solving the problem that important documents do not appear in the top results returned by the system retrieval component, resulting in correct answers being ignored.

[0033] Further, by adjusting the chunk_size and similarity_top_k hyperparameters, the retrieval results are reordered before being sent to the LLM, including:

[0034] Use a more advanced similarity measure;

[0035] Add context information, including the relevance between chunks;

[0036] Use a regression model or classification model for reordering;

[0037] For the difficulty of processing unstructured documents, by using optical character recognition (OCR) technology, effective information in unstructured documents is recognized, extracted, and imported into the RAG system knowledge base for data training.

[0038] Further, through optical character recognition (OCR) technology, effective information in unstructured documents is recognized, extracted and imported into the RAG system knowledge base for data training, including:

[0039] First, image preprocessing is performed: grayscale conversion, converting the image to a grayscale image; binarization, converting the grayscale image to a black and white image; noise reduction, removing noise points in the image; correcting the tilt, correcting the text tilt in the image;

[0040] Then, using the Tesseract recognition engine for recognition, post-processing the recognized text to extract effective information;

[0041] Post-processing: remove noise, delete irrelevant or incorrect text; segmentation, segment the text by paragraph, sentence or other structure; use natural language processing (NLP) techniques to extract structured information from the text, including named entity recognition (NER) and keyword extraction.

[0042] Another object of the present application is to provide a RAG-based artificial intelligence agent generation system, which implements the RAG-based artificial intelligence agent generation method, and the system comprises:

[0043] A knowledge base module is used to retrieve relevant documents and generate answers or text using a bound vector model and a document processing model, the document processing model is responsible for preprocessing and vectorizing the document library, the vector model processes user input queries and converts the queries into vector representations;

[0044] A vectorization numerical conversion module is used to import data into the knowledge base, and the data is queued in the order of data import time and converted into vectorized numerical values;

[0045] A robot application creation module is used to create a robot application and associate the knowledge base;

[0046] A publishing module is used to publish the robot application and chat;

[0047] A correction module is used to correct the robot question and answer data, so that the robot can provide more accurate answers.

[0048] In combination with all the above technical solutions, the present application has the following beneficial effects: the present application solves the problems of personnel implementation of RAG technology, content missing, missing top-ranked documents and difficulty in processing unstructured documents.

[0049] The RAG technology implementation scheme provided by the present application has the following advantages compared with the prior art through a friendly UI interface and process-based processing: improved user experience: through intuitive UI design, non-technical personnel can also easily create and manage the knowledge base, reducing the technical threshold and improving accessibility. Improved data quality: by cleaning up data sources and correcting problem answers, the accuracy of the knowledge base is improved, thereby improving the answer quality of the RAG system and the trust of users. Enhanced retrieval efficiency: adjusting hyperparameters such as chunk_size and similarity_top_k, and reordering the retrieval results can more effectively capture important documents ranked at the top, improving the accuracy of retrieval. Unstructured data processing capability: using OCR technology, unstructured documents can be converted into structured data, expanding the coverage of the knowledge base and enhancing the application scenarios of the model. Real-time update and maintenance: through the process of data import and release of robot application, the update of the knowledge base can be quickly responded to, ensuring the timeliness of information. Scalability and flexibility: the method of the present application is not only applicable to text data, but also can be extended to multi-modal data such as images and videos, providing more extensive application possibilities.

[0050] The present application provides an integrated artificial intelligence agent generation method and system based on RAG technology, which significantly improves the performance of the generation model in answering and text generation tasks by combining language models and information retrieval technology. The expected benefits and commercial value of this technical solution are: improving the accuracy and quality of the generation model, effectively solving the content missing and ranking problems in the prior art through retrieval-enhanced generation model, improving user experience and satisfaction. Reducing the technical threshold, so that non-technical personnel can easily create and manage the knowledge base, thereby expanding the potential user group and market coverage. Expanding the application scenarios, not limited to text data, but also capable of processing multi-modal data such as images and videos, further improving the flexibility and applicability of the technology. Realizing the processing of unstructured documents, converting them into structured data through OCR technology, expanding the coverage of the knowledge base, and further expanding the application field.

[0051] Internationally, especially in the fields of language generation and information retrieval, the application of RAG technology is relatively new. The present application fills the gap in the combination of language generation and information retrieval in the prior art by providing a systematic method. In China, especially in the fields of enterprises and education, there is an increasing demand for quickly generating high-quality answers and texts. The present application fills the gap in the prior art that cannot effectively solve the problems in these fields, providing a new solution for users.

[0052] People have long hoped to effectively combine language models and information retrieval through technical means to improve the efficiency and accuracy of the generation model. Traditional large models are powerful, but have limitations in processing large-scale data and multi-modal information. The present application effectively overcomes the technical difficulties of existing technologies in processing large-scale data and unstructured documents by optimizing the joint training of vector models and document processing models, using contrastive learning and advanced similarity measurement methods.

[0053] In large-scale data processing and document retrieval, traditional technologies often have biases, and even with advanced language models, it is difficult to avoid biases in the ranking and information retrieval process. The present application effectively reduces the bias in the system by reordering the search results, adjusting the hyperparameters, and combining OCR technology to process unstructured documents, improving the objectivity and accuracy of information retrieval and generation models. BRIEF DESCRIPTION OF DRAWINGS

[0054] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure, together with the description;

[0055] Figure 1 is a flow chart of the artificial intelligence agent generation method based on RAG technology provided by the embodiment of the present application;

[0056] Figure 2 is a schematic diagram of the artificial intelligence agent generation system based on RAG technology provided by the embodiment of the present application;

[0057] In the figure: 1, create a knowledge base module; 2, vector numerical conversion module; 3, robot application creation module; 4, release module; 5, correction module. DETAILED DESCRIPTION

[0058] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below. In the following description, a large number of specific details are set forth in order to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the scope of the present application, therefore the present application is not limited to the specific implementation disclosed below.

[0059] The innovation of the present application is that the present application combines language models and information retrieval technology, optimizes the joint training of vector models and document processing models, effectively improves the accuracy of text generation and the efficiency of information retrieval, and at the same time solves the problem of processing unstructured documents and data ranking, significantly improves the practicability and user experience of the system.

[0060] Embodiment 1, in order to solve the problem of RAG technology implementation difficulty for personnel, the application provides a friendly UI (user interface design) interface to process the RAG technology implementation in a process.

[0061] As Figure 1 indicated, the artificial intelligence agent generation method based on RAG technology provided by the embodiment of the application comprises:

[0062] S1, creating a knowledge base, using a bound vector model and a document processing model to retrieve relevant documents and generate answers or texts, the document processing model is responsible for preprocessing and vectorizing the document library, the vector model processes the user input query and converts the query into a vector representation;

[0063] Query preprocessing: similar to document preprocessing, cleaning and word segmentation are performed. In the query preprocessing stage, in addition to simple cleaning and word segmentation, more complex context-aware techniques are introduced. For example, combined with entity recognition (NER) or keyword extraction techniques to capture important information in the query and enhance its semantic understanding ability, so as to more accurately match relevant documents. Considering that documents in different fields may have specific language characteristics and content structures, the document processing model can be developed or adjusted to better adapt to text data in specific fields. This includes domain-specific preprocessing steps, optimization of specific vocabulary, and enhancement of sensitivity to domain-specific information.

[0064] Vector representation: use the same language model as the document to convert the query into a vector, ensuring that the query and the document are in the same vector space for similarity calculation. By further optimizing the pooling strategy. In addition to the conventional average pooling or CLS token pooling, dynamic pooling methods can also be considered, which adaptively select the optimal pooling strategy according to the length and complexity of the text to improve the quality of the vector representation.

[0065] Considering that the knowledge base may need to be continuously updated and optimized, the ability of incremental learning or online learning is introduced. In this way, the system can accept new data at runtime and continuously adjust the document processing model and vector model to reflect the latest language use and semantic changes, maintaining the real-time and accuracy of the knowledge base.

[0066] The specific steps are as follows:

[0067] 1. Select a language model: first, select a pre-trained language model, such as BERT (Bidirectional Encoder Representations from Transformers) or its variants, as the basis for vectorization. These models have been trained on large-scale text corpora and can encode text into dense vector representations, capturing semantic and contextual information.

[0068] 2. Text Preprocessing: The input query and documents are preprocessed, including cleaning the text, tokenization, removing stop words, etc. These steps aim to prepare the text for model processing and ensure input consistency.

[0069] 3. Vectorization Conversion: The preprocessed text is encoded using a selected language model. The specific steps are as follows:

[0070] Tokenization: The text is divided into tokens or subwords, forming a token sequence.

[0071] Embedding: Each token is converted into a corresponding vector representation. These vectors are word embeddings learned by pre-trained models, reflecting the position of words in the semantic space.

[0072] Pooling or Sequence Representation: The embeddings of tokens are aggregated to generate a single vector representation of the entire query or document. Common pooling strategies include average pooling or sequence embedding using special tokens (such as the [CLS] token).

[0073] 4. Similarity Calculation: Once the query and document are converted into vector representations, cosine similarity or other similarity measurement methods in the Vector Space Model can be used to measure their similarity in the semantic space. Cosine similarity is a commonly used measure that measures the angle between two vectors, with closer to 1 indicating greater similarity and closer to 0 indicating less similarity.

[0074] Using the query vector and document vector generated by the vector model, the system calculates the similarity between the query and each document. Through similarity calculation, the system retrieves the N most relevant documents from the document library.

[0075] Technical improvements:

[0076] Use more complex similarity measurement methods: In addition to traditional cosine similarity, use similarity calculation methods based on attention mechanisms, such as using self-attention mechanisms in Transformer models. This method can better capture the semantic relationship between the query and the document, especially in handling long texts or complex contexts.

[0077] Introducing Semantic Matching Models: Incorporate pre-trained semantic matching models, such as BERT-based models for Sentence Embeddings, to match query vectors with document vectors for more accurate similarity measures. These models can learn higher-level semantics

[0078] Considering Local and Global Information: Combine local and global information when calculating similarity. Use Convolutional Neural Networks (CNN) and Long Short-Term Memory Networks (LSTM) to capture local features, then integrate with global Transformer models for a more comprehensive semantic representation and similarity evaluation.

[0079] Integrating Knowledge Enhancement Techniques: Incorporate external knowledge graphs or domain-specific knowledge into the similarity calculation process. These knowledge sources can provide additional semantic context and relationship information for documents and queries, enhancing the accuracy and depth of similarity calculations.

[0080] Optimizing Computational Efficiency: For large-scale document libraries, optimize the efficiency of similarity calculations. Use Approximate Nearest Neighbor (ANN) search algorithms to speed up similarity calculations between query vectors and document vectors, reducing computational complexity and response time.

[0081] The retrieved relevant documents are input into the Transformer-based generative model along with the original query. The generative model generates the final answer or text using these inputs.

[0082] The Transformer-based generative model, when processing tasks based on RAG technology, can be briefly described as follows:

[0083] Transformer Model Structure: Transformer is a deep neural network architecture based on self-attention mechanisms, particularly suitable for processing sequential data such as text. It consists of multiple attention layers, each of which can consider the information of all positions in the input sequence simultaneously, capturing global semantic relationships.

[0084] Input Processing: First, for each query and its retrieved relevant documents, they are represented as input sequences respectively. These sequences are usually marked with special tokens (such as [CLS] tokens) to indicate the beginning and end, and also include the encoding of the text.

[0085] Self-attention mechanism: The core of the Transformer model is the self-attention mechanism. When processing input sequences, the representation of each position can depend on the representations of other positions in the sequence simultaneously. This allows the model to consider all interactions within the input sequence when generating text, rather than being limited to local context.

[0086] Encoder-decoder architecture: The generation model usually adopts an encoder-decoder structure. The encoder part is used to encode the input sequence into a context-aware hidden representation, while the decoder part predicts the next word or generates a complete text sequence based on the output of the encoder and the previously generated part of the text.

[0087] Context enhancement and text generation: When processing RAG tasks, the generation model effectively utilizes the information of the input sequence (i.e., the query and the relevant documents) through the self-attention mechanism. It can extract important context information from these inputs to generate high-quality text related to the query or answer. This approach not only improves the amount of information generated, but also ensures the accuracy and relevance of the generated results.

[0088] Overall, the generation model based on the Transformer architecture can efficiently integrate the semantic information of the query and the relevant documents into the generated text through the powerful self-attention mechanism and encoder-decoder structure, thereby achieving effective application in RAG tasks and improving the ability and effectiveness of the system in text generation and answering questions.

[0089] This process includes: input construction: concatenate the query and the retrieved relevant documents into an input sequence and provide it to the generation model. The specific implementation process is as follows:

[0090] 1、Query and document representation: First, encode or vectorize the query and each retrieved relevant document. Usually, a pre-trained language model is used to generate their word embeddings or sentence representations.

[0091] 2、Special token addition: Before merging the query and the document, special start tokens ([CLS]) and end tokens ([SEP]) need to be added, which is the input format required by the Transformer model. These tokens help the model distinguish different sentences and paragraphs.

[0092] 3、Text length limit processing: If the length of the query and / or the document exceeds the maximum input length limit of the model, truncation or segmentation processing is needed, using truncation or batch processing techniques.

[0093] 4. Concatenation: The query with markers and each relevant document are concatenated in order to form a single input sequence. For example, the [CLS] marker of the query can be concatenated with the content of the first document, then the [SEP] marker between documents, and finally the content of the last document and the [SEP] marker at the end.

[0094] 5. Input sequence preparation: The final input sequence should be a complete, tokenized text sequence containing the query and the content of all relevant documents. This sequence can be directly input into the encoder part of the Transformer model for processing.

[0095] 6. Encoder processing: The encoder part of the Transformer model processes the concatenated input sequence to generate context-aware hidden representations. These representations contain the semantic information of the entire input sequence, allowing the decoder part to generate appropriate outputs based on them.

[0096] Context enhancement: The generative model effectively utilizes the information from relevant documents through attention mechanisms to generate more accurate and informative answers.

[0097] Improved algorithm description

[0098] 1. Input representation and encoding:

[0099] Input sequence construction: The query and retrieved relevant documents are concatenated into an input sequence, ensuring that each document and the query are identified by special markers ([CLS] and [SEP]).

[0100] Word embeddings and position encoding: Each word or clause in the input sequence is converted into a word embedding vector, and position encoding is added to preserve the position information of the words.

[0101] 2. Multi-head self-attention mechanism:

[0102] Encoder stacking: Use multiple layers of Transformer encoders, each containing a multi-head self-attention mechanism and a feed-forward neural network.

[0103] Self-attention calculation: For each position, determine its relevance to other parts of the sequence by calculating attention scores. The improved algorithm can use more complex attention distributions, such as considering local and global information of the text, and interactions between different heads.

[0104] Attention weight calculation: Normalize the attention scores by softmax to calculate the weight of each position to other positions, to obtain a more comprehensive understanding of the context.

[0105] 3. Context enhancement:

[0106] Feature Integration: The context information computed through self-attention mechanisms is integrated into the representation of each position. These representations are not just simple combinations of the original word embeddings, but contain semantic information of the entire input sequence.

[0107] Information Passing: Through the stacking of multiple encoders, each layer can further pass and integrate context information. Each level enhances the richness of the representation through self-attention mechanisms, enabling the model to enhance its higher-level abstraction and text understanding capabilities.

[0108] 4. Generation Model Output:

[0109] Decoder Processing: After the encoder stack, the output of the encoder is processed using the Transformer decoder part to generate appropriate answers or text.

[0110] Context Utilization: The decoder further utilizes the context of the encoder output through self-attention mechanisms, ensuring that the generated text is consistent and relevant to the query and document content in the input sequence.

[0111] Technical improvements and advantages:

[0112] 1. Complex Attention Mechanism: Using improved self-attention mechanisms, asymmetric attention, or attention distribution based on multi-modal input, it can better capture the complex relationships and semantic connections of the input sequence.

[0113] 2. Multi-layer Encoder Stacking: Increase the number of layers and complexity of the encoder to handle deeper semantic understanding and expression, improve the expression ability of the generation model and the quality of text generation.

[0114] 3. Information Passing and Integration: Through the information passing and integration of multiple layers of encoders, it ensures that the representation of each position contains global context information, making the generated answers more comprehensive and accurate.

[0115] S2, import data into the knowledge base, and queue the vectorized numerical values in the order of data import time;

[0116] The system supports manual entry, document import, question and answer pair import, and other data import methods. After data import, the data is queued and converted into vectorized numerical values in the order of data import time. This process is data preprocessing and vectorization. This process includes the following steps:

[0117] Text cleaning: Remove noise in the document, such as HTML tags, special characters, etc.

[0118] Tokenization and normalization: Tokenize the document text and perform normalization processing (such as stem extraction, word form restoration).

[0119] In processing multilingual text, combining subword segmentation techniques and Transformer-based language models, the text is segmented into more fine-grained subword representations, and a Transformer encoder is used to learn context-aware word embeddings. Then, by introducing appropriate normalization processing such as stemming and lemmatization, the efficiency and accuracy of text processing can be further improved. Improvements:

[0120] Subword segmentation: For compound words or long words in specific languages, using subword segmentation techniques (Byte-Pair Encoding, BPE) can better handle out-of-vocabulary (OOV) and out-of-dictionary (OOD) words.

[0121] Multilingual support: For multilingual text, using a sentence piece (SentencePiece) that supports multiple languages can better adapt to the needs of different languages, improving the universality and efficiency of processing.

[0122] Vector representation: Use pre-trained language models to convert processed documents into fixed-dimensional vectors. These vector representations represent the semantic information of the document, facilitating subsequent similarity calculations. Detailed process:

[0123] 1、Preprocessing: Text cleaning: First, preprocess the document to remove unnecessary special characters, punctuation marks, HTML tags, and other noise. Tokenization and tagging: Segment the cleaned text into words or subwords and perform tokenization. This step uses advanced tokenization tools and vocabulary (SentencePiece, WordPiece) to adapt to different languages and text types.

[0124] 2、Word embedding: Pre-trained language model: Use pre-trained deep learning language models BERT (Bidirectional Encoder Representations from Transformers), RoBERTa (Robustly Optimized BERT Pretraining Approach). Through large-scale unsupervised training, learn rich text semantic representations.

[0125] 3. Tokenization: The cleaned and tokenized text is inputted into a pre-trained language model for tokenization, converting the text into an input representation that the model can understand. Encoder outputs: The last hidden state or a specific pooling layer output of the pre-trained language model is obtained for the text. These outputs are generally considered as the semantic representation of the text, capturing the semantic information of the document in context.

[0126] 4. Vectorization: Fixed-dimensional vectors: The output of the pre-trained language model is used as the vector representation of the document. These vectors usually have a fixed dimension, such as 768 or 1024, depending on the configuration of the chosen pre-trained language model. Semantic information extraction: These vectors are designed to effectively capture the semantic information of the document, so that similar documents have close vector representations in the vector space, facilitating subsequent similarity calculation and information retrieval tasks.

[0127] 5. Application scenarios: Similarity calculation: Using the vector representation of the document, the cosine similarity or other distance measures between vectors are calculated to measure the semantic similarity between documents. Information retrieval: Comparing the query vector with the document vector, retrieving the most relevant documents to the query.

[0128] In the training phase, the document processing model and the vector model can be jointly optimized, making the entire RAG system more coordinated and efficient:

[0129] Joint training: During training, both document vectors and query vectors are optimized simultaneously, making relevant documents and queries closer in the vector space.

[0130] Document processing model optimization:

[0131] Objective: The document processing model is responsible for preprocessing and vectorizing the document library, generating semantic vector representations of the documents.

[0132] Optimization method: Using supervised learning and self-supervised learning methods, minimize the loss between the document semantic vector and its corresponding label or automatically generated query vector to optimize the document processing model. Using a Transformer-based model structure, adjust the model parameters through the backpropagation algorithm, so that the document semantic vector better captures the semantic information of the document.

[0133] Query vector model optimization:

[0134] Objective: The query vector model is responsible for converting the user input query into a vector representation for comparison with the document semantic vector.

[0135] Optimization method: Use supervised learning and self-supervised learning methods to optimize the query vector model by minimizing the loss between the query vector and its relevant document semantic vector. This ensures that the query vector effectively reflects the semantic meaning of the query and is close to the semantic vector of the relevant document in the vector space.

[0136] Contrastive learning: Use contrastive learning methods to further optimize the vector model, so that similar document and query vectors are close to each other, and unrelated document and query vectors are far from each other.

[0137] Batch contrastive loss: Take the query vector and the semantic vector of its relevant document as positive samples, and take the query vector and the semantic vector of unrelated documents as negative samples. By adjusting the model parameters, the similarity between positive samples increases, and the similarity between negative samples decreases.

[0138] Self-supervised contrastive learning: Use automatically generated contrastive tasks to learn vector representations in the semantic space. For example, use the same document generator to generate two different versions of the document, then use the document processing model and query vector model to compare the two versions of the document, and optimize the model parameters in the process.

[0139] S3, create a robot application, associate a knowledge base;

[0140] Specifically, create a robot ID, name, icon, introduction, and role setting (guide the chat direction of the application, this content will be fixed at the beginning of the context), and select the knowledge base created in step S1 to bind, multiple knowledge bases can be bound; Set whether to display the conversation context in the foreground, whether to display the quoted content in the foreground, whether to make the robot public, etc.; Set AI interface, similarity, number of single searches, empty search response, welcome message, bottom identifier, menu items, etc.

[0141] User-friendliness and ease of use: The set functions are simple and clear, consistent with the user's intuition and usage habits. For example, the welcome message is friendly and can guide the user to start interaction; menu items are clearly classified, making it easy for users to quickly find the desired function.

[0142] Flexibility and customizability: The system has a certain degree of flexibility, allowing administrators to adjust and customize according to specific application scenarios and user needs. For example, different empty search responses can be set to guide or prompt accordingly according to different situations.

[0143] Practicality and efficiency: The set functions can improve the practicality and efficiency of the system. For example, by setting the number of single searches and the similarity threshold, the number and quality of returned results can be controlled, reducing the amount of information users need to process and improving search efficiency.

[0144] Consistency and Brand Image: The bottom logo and digital human image broadcasting functions should be consistent with the overall brand image and user experience. The design of these functions should conform to the brand style and user expectations, enhancing the overall impression and recognition of the system.

[0145] Enable digital human image, bind digital human video for anthropomorphic broadcasting.

[0146] Digital human image design: Pre-design a suitable digital human image, which can be a virtual character or a robot image, with good visual appeal and user affinity. This image should reflect the intelligent and modern features of the system, while also conforming to the aesthetic and cultural background of the target user group.

[0147] Video anthropomorphic broadcasting: Combine the designed digital human image with speech synthesis technology to generate natural and smooth voice broadcasting content. These contents can include system welcome words, voice explanations for common questions, and user guidance and prompt information. Through video display, information can be conveyed more vividly, enhancing user immersion and interactive experience.

[0148] S4, publish robot application, chat. Can chat in the foreground or publish as a chat window.

[0149] For content missing, that is, when there is no actual answer in the knowledge base, the RAG system may provide seemingly reasonable but incorrect answers. To solve this problem, the invention improves data quality by cleaning up data sources and correcting question answers, and designs better prompts to encourage the model to recognize its limitations and transparently convey uncertainty.

[0150] Among them, cleaning up data sources means deleting data lists in the knowledge base; correcting question answers means modifying the content of question and answer pairs in the knowledge base.

[0151] Principles for cleaning up data sources (deleting data lists in the knowledge base):

[0152] Data expiration and irrelevance: First, identify and delete data that has expired or is no longer relevant. These data may be outdated or no longer meet the latest needs and standards of the system.

[0153] Data quality and accuracy: Delete data with low quality, inaccurate information, or errors. These data may mislead the system to generate incorrect answers or text, reducing the credibility and user experience of the system.

[0154] Legal and Privacy Compliance: Ensure that the deleted data complies with legal regulations and privacy policies. In particular, when dealing with user data or sensitive information, strictly follow relevant data processing laws and regulations to protect users' privacy rights.

[0155] System Optimization and Performance: Delete redundant and unnecessary data to optimize system performance and response speed. These data may occupy storage space or increase system processing burden, affecting system efficiency and stability.

[0156] Modify the detailed description of the question and answer pair:

[0157] Question analysis and identification: First, conduct a comprehensive analysis and evaluation of the existing question and answer pair. Identify possible problems, such as inaccurate answers, outdated or incomplete information, etc.

[0158] Correct the answer content: For the problems found, make specific corrections. This includes updating the answer content to reflect the latest information or knowledge; correcting incorrect information to ensure the accuracy and completeness of the answer.

[0159] Proofreading and verification: The modified question and answer pair needs to go through strict proofreading and verification process. Ensure that the modified answer is consistent with the facts, meets the system's standards and criteria.

[0160] Version control and record: During the modification process, establish a good version control and record mechanism. Record the specific content of each modification, modification time and responsible person, so as to trace back and audit.

[0161] User feedback and iteration: Monitor user feedback and continuously optimize and improve the question and answer pair according to actual use. This includes identifying possible problems or needs from user feedback and making timely corrections and updates.

[0162] For the problem of missing top-ranked documents, important documents may not appear in the top results returned by the system retrieval component, resulting in correct answers being ignored. The solution to this problem includes adjusting the chunk_size and similarity_top_k hyperparameters and reordering the retrieval results before sending them to the LLM.

[0163] The following are the specific implementation steps and detailed description:

[0164] 1. Determine the hyperparameters chunk_size and similarity_top_k

[0165] Determination of chunk_size:

[0166] chunk_size refers to the number or size of documents processed at a time during the retrieval process. Larger chunk_size can increase computational cost but can improve retrieval efficiency as more documents can be processed simultaneously.

[0167] A suitable chunk_size is determined based on the hardware resources and performance requirements of the system. Typically, the optimal value is found through experimentation and performance testing.

[0168] Determination of similarity_top_k:

[0169] similarity_top_k defines the number of most relevant documents returned for each query. This parameter affects the quality of the returned results and the breadth of the search.

[0170] A suitable similarity_top_k value is determined based on the system's requirements and the expected number of search results desired by the user.

[0171] 2. Retrieval and re-ranking process

[0172] 2.1 Data preprocessing and vectorization

[0173] Document preprocessing: For each document, perform cleaning, tokenization, and normalization to ensure uniform formatting and meet analysis requirements.

[0174] Vector representation: Use a pre-trained language model to convert the processed documents into fixed-dimensional vectors. These vectors represent the semantic information of the documents, facilitating subsequent similarity calculations.

[0175] 2.2 Query processing and similarity calculation

[0176] Query preprocessing: Clean, tokenize, and normalize the user's input query to ensure consistent formatting with document processing.

[0177] Vectorized query: Use the same pre-trained language model to convert the processed query into a vector representation, allowing the query and documents to be in the same vector space for similarity calculations.

[0178] Similarity calculation: Calculate the similarity between each query vector and all document vectors. This can be done by calculating the cosine similarity between vectors or other appropriate similarity measures.

[0179] 2.3 Re-ranking process

[0180] Filtering based on similarity_top_k: For each query, select the top similarity_top_k documents from all documents that are most similar to the query vector as the initial candidate set.

[0181] Chunking: Divide the initial candidate document set into several chunks of size chunk_size. The number of documents in each chunk is determined by chunk_size.

[0182] Re-ranking: Perform local ranking within each chunk to further optimize the order of documents. This can be done based on a more refined similarity metric or other ranking criteria, ensuring that the most relevant documents are at the front.

[0183] Merging overall results: Consolidate the reordered documents within all chunks into the final ranking results. These top-ranked documents will become the input for the final LLM generation model processing.

[0184] (1) Experiment and verification:

[0185] Determine the optimal chunk_size and similarity_top_k through experiments:

[0186] Set a set of different chunk_size values (e.g.: 128, 256, 512).

[0187] For each chunk_size value, set a set of different similarity_top_k values (e.g.: 5, 10, 20).

[0188] Record the retrieval and ranking effect under each combination.

[0189] (2) Adjustment and evaluation:

[0190] Quality evaluation of retrieval results for different combinations:

[0191] Relevance of retrieval results: The relevance of the result block to the query.

[0192] This can be measured by calculating the proportion of query-related information contained in each retrieval result block.

[0193] Recall rate: Whether all relevant information blocks are included.

[0194] Recall rate evaluates whether the system can retrieve all information blocks related to the query. It is usually calculated using the following formula:

[0195] Recall = TP / (TP+FN) where TP (1 Positive) is the number of relevant information chunks retrieved, FN (0 Negative) is the number of relevant information chunks not retrieved.

[0196] Precision: How many of the retrieved information chunks are relevant.

[0197] Precision evaluates how many of the retrieved information chunks are relevant to the query. It is usually calculated using the following formula: Precision = TP / (TP+FP) where TP (1 Positive) is the number of relevant information chunks retrieved, FP (0 Positive) is the number of irrelevant information chunks falsely retrieved.

[0198] (3) Re-ranking logic, after adjusting chunk_size and similarity_top_k, further optimize the re-ranking logic:

[0199] Use more advanced similarity measures (such as cosine similarity, BERT embedding similarity).

[0200] Cosine Similarity: Cosine similarity measures the degree of similarity between two vectors in direction. The formula for calculating the cosine similarity between a document and query vector is:

[0201] ;

[0202] where, is the document vector, is the query vector, is the dot product of the vectors, is the norm of the vector.

[0203] BERT Embedding Similarity: Use a pre-trained BERT model to generate embedding vectors for documents and queries, then calculate their similarity. BERT model usually outputs a vector representation, which can be used to calculate the similarity between documents and queries using cosine similarity or other measurement methods.

[0204] Consider adding context information, such as the relevance between chunks.

[0205] Use regression models or classification models for re-ranking.

[0206] To deal with the difficulty of processing unstructured documents, through OCR (Optical Character Recognition) technology, identify the effective information in the unstructured document, extract and import into the RAG system knowledge base for data training.

[0207] Specifically as follows:

[0208] First, image preprocessing:

[0209] Grayscale: Convert the image to grayscale.

[0210] Binarization: Convert the grayscale image to a black and white image.

[0211] Noise reduction: Remove noise points in the image.

[0212] Correct skew: Correct the text skew in the image.

[0213] Then use the Tesseract recognition engine for recognition, and post-process the recognized text to extract valid information. Post-processing steps include:

[0214] Remove noise: Delete irrelevant or incorrect text.

[0215] Segmentation: Divide the text by paragraph, sentence or other structure.

[0216] Use natural language processing (NLP) techniques to extract structured information from text. Including named entity recognition (NER), keyword extraction, etc.

[0217] Specific details steps:

[0218] ### Image preprocessing

[0219] 1. Grayscale

[0220] Formula: G = (R + G + B) / 3

[0221] Where R, G, B represent the red, green and blue channel values of the color image respectively, and G is the converted grayscale value.

[0222] 2. Binarization

[0223] The purpose is to divide the pixels in the image into black and white, making it easier for subsequent text recognition.

[0224] Formula: $B(x, y) = \begin{cases}

[0225] 1&\text{if} G(x, y) \geq T \\

[0226] 0&\text{otherwise}

[0227] \end{cases}$

[0228] Where $B(x,y)$ is the binary pixel value after binarization, $G(x,y)$ is the grayscale value, and $T$ is the threshold value.

[0229] 3. Noise reduction

[0230] Reduce noise in images using filters such as Gaussian filter, median filter, etc.

[0231] 4. Correcting skew

[0232] Detect the orientation of text in images using algorithms like Hough transform and perform rotation correction.

[0233] ### OCR recognition and post-processing

[0234] 1. Tesseract recognition engine

[0235] Use the Tesseract engine to perform text recognition on pre-processed images.

[0236] 2. Remove noise

[0237] Remove irrelevant or incorrect text from the recognition results.

[0238] 3. Segmentation

[0239] Segment text into paragraphs or sentences based on blank lines, punctuation, etc.

[0240] 4. Natural Language Processing (NLP)

[0241] - Named Entity Recognition (NER)

[0242] Identify specific entities in text such as names, locations, organizations, etc.

[0243] - Keyword extraction

[0244] Extract important keywords from text.

[0245] Through the above steps, unstructured image documents can be converted into structured data, which can then be imported into the RAG system knowledge base for data training. This process requires continuous optimization of algorithms and models to improve the accuracy and efficiency of information extraction.

[0246] The above description is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any modification, equivalent replacement, and improvement within the technical range disclosed by the present application, which is within the spirit and principle of the present application, should be covered within the protection scope of the present application.

Claims

1. A method for generating an artificial agent based on RAG technology, characterized in that, The method combines language models and information retrieval techniques, effectively improves the accuracy of text generation and the efficiency of information retrieval by optimizing the joint training of vector models and document processing models, solves the problems of processing unstructured documents and data ranking, and significantly improves the practicability and user experience of the system, including: S1, creating a knowledge base, using the bound vector model and document processing model to retrieve relevant documents and generate answers or text, the document processing model is responsible for preprocessing and vectorizing the document library, the vector model processes the user input query and converts the query into a vector representation; S2, import data into the knowledge base, queue according to the time sequence of data import into the vectorized numerical value; S3, create a robot application, associate the knowledge base; S4, publish the robot application and chat; S5, correct the robot question and answer data, so that the robot can provide more accurate answers; In step S1, creating a knowledge base, including: (1) Select a language model, first, select a pre-trained language model, including BERT or its variants, as the basis for vectorization model; The vectorization basis model is trained by large-scale text corpus to encode text into dense vector representation, capturing semantic and contextual information of text; Match the query vector with the document vector to get a more accurate similarity measure, which can learn more advanced semantics; (2) Text preprocessing, preprocessing the input query and document, including text cleaning, word segmentation, and stop word removal operations; (3) Vector conversion, use the selected language model to encode the preprocessed text; The specific steps are as follows: Tokenization, divide the text into words or subwords to form a token sequence; Embedding, convert each token into a corresponding vector representation; These vectors are word embeddings learned by pre-trained models, reflecting the position of words in the semantic space; Pooling or sequence representation, aggregate the embeddings of tokens to generate a single vector representation of the entire query or document; Pooling strategies include average pooling or sequence embedding using [CLS] token; (4) Similarity calculation, once the query and document are converted into vector representation, use the similarity calculation method based on attention mechanism in vector space model to measure their similarity in semantic space; Using the query vector and document vector generated by the vector space model, the system calculates the similarity between the query and each document; Through similarity calculation, the system retrieves the N most relevant documents from the document library; In step (4) similarity calculation, the similarity calculation method based on attention mechanism uses the self-attention mechanism in the Transformer model to capture the semantic relationship between the query and the document; In step (4), similarity calculation, local and global information of the text are considered, and in the calculation of similarity, local and global information are combined, and convolutional neural network CNN and long short-term memory network LSTM are used to capture local features of the text, and then integrated with the global Transformer model to obtain more comprehensive semantic expression and similarity evaluation; In step (4), similarity calculation, by integrating knowledge enhancement technology, external knowledge graph or professional domain knowledge is introduced into the similarity calculation process; these knowledge sources provide additional semantic background and relationship information for documents and queries; In step (4), similarity calculation, for large-scale document library, the efficiency of similarity calculation is optimized, and approximate nearest neighbor ANN search algorithm is used to accelerate the similarity calculation between query vector and document vector, reduce the calculation complexity and response time; (5) The relevant documents retrieved are input into the generation model based on the Transformer architecture together with the original query, and the generation model generates the final answer or text using these inputs; The generation model based on the Transformer architecture includes the following when processing tasks based on RAG technology: 1) Input representation and encoding; Input sequence construction: concatenate the query and the retrieved relevant documents into an input sequence, and ensure that each document and the query are marked at the beginning and end with [CLS] and [SEP] special tokens; Word embedding and position encoding: convert each word or clause in the input sequence into a word embedding vector and add position encoding to preserve the position information of the word; 2) Multi-head self-attention mechanism; Encoder stacking: use a multi-layer Transformer encoder stack, each layer containing a multi-head self-attention mechanism and a feed-forward neural network; Self-attention calculation: for each position, determine its relevance to other parts of the sequence by calculating attention scores; self-attention calculation takes into account both local and global information of the text, as well as interactions between different heads; Attention weight calculation: calculate the weight of each position corresponding to other positions by softmax normalizing the attention scores to obtain a more comprehensive understanding of the context; 3) Context enhancement; Feature integration: integrate the context information obtained through the self-attention mechanism into the representation of each position; these representations contain a simple combination of the original word embeddings and also contain semantic information of the entire input sequence; Information transmission: through the stacking of multiple layers of encoders, each layer can further transmit and integrate context information; each level enhances the richness of the representation through the self-attention mechanism; 4) Generation model output; Decoder processing: after the encoder stack, use the Transformer decoder part to process the output of the encoder to generate answers or text; Context utilization: the decoder further utilizes the context of the encoder output through the self-attention mechanism to ensure that the generated text is consistent and related to the query and document content in the input sequence; In step S2, the data imported is queued in chronological order and converted into vectorized numerical values, including: Text cleaning: remove noise from the document; Tokenization and normalization: Tokenize the document text and perform normalization processing; The tokenization includes: for compound words or long words in the language, use the sub-word segmentation technique BPE to process the out-of-vocabulary OOV and out-of-dictionary OOD words; Vector representation: use a pre-trained language model to convert the processed document into a fixed-dimensional vector; In the training phase, the document processing model and the vector model are jointly optimized to make the entire RAG system consistent; Joint training: During training, optimize document vectors and query vectors so that relevant documents and queries are close in vector space; The document processing model optimization includes: Using supervised learning and self-supervised learning methods, minimize the loss between document semantic vectors and their corresponding labels or automatically generated query vectors to optimize the document processing model; using a Transformer-based model structure, adjust the model parameters through the backpropagation algorithm to make the document semantic vector better capture the semantic information of the document; The vector model optimization includes: Using supervised learning and self-supervised learning methods, minimize the loss between query vectors and their relevant document semantic vectors to optimize the vector model; ensure that the query vector can effectively reflect the semantic meaning of the query and be close to the semantic vector of the relevant document in the vector space; Contrastive learning: Use contrastive learning methods to further optimize the vector model so that similar document and query vectors are close to each other, and unrelated document and query vectors are far from each other; Same batch contrastive loss: Take the query vector and the semantic vector of its relevant document as positive samples, and take the query vector and the semantic vector of the non-relevant document as negative samples; by adjusting the model parameters, the similarity between positive samples is increased, and the similarity between negative samples is decreased; Self-supervised contrastive learning: Use automatically generated contrastive tasks to learn vector representations in the semantic space, including using the same document generator to generate two different versions of the document, and then using the document processing model and the vector model to compare the two versions of the document. In this process, the model parameters are optimized; In step S3, create a robot application and associate a knowledge base, including: Create a robot ID, name, icon, introduction, and role setting, and select the knowledge base created in step S1 to bind, bind multiple knowledge bases; set whether to display the conversation context in the foreground, whether to display the reference content in the foreground, whether to make the robot public; Set AI interface, similarity, number of single searches, empty search response, welcome speech, bottom identifier, menu item; enable digital human image, bind digital human video for personification broadcast; In step S4, publish the robot application and chat, including: For content missing, when there is no actual answer in the knowledge base, process the reasonable but incorrect answer provided by the RAG system by cleaning up the data source and correcting the question answer, and design a prompt to encourage the model to admit limitations and transparently convey uncertainty; cleaning up the data source means deleting the data list in the knowledge base; correcting the question answer means modifying the content of the question and answer pair in the knowledge base; For missing top-ranked documents, by adjusting chunk_size and similarity_top_k hyperparameters, re-ranking before sending the retrieval results to the LLM, addressing the issue of important documents not appearing in the top results returned by the system's retrieval component, leading to the correct answer being overlooked; By adjusting chunk_size and similarity_top_k hyperparameters, re-ranking before sending the retrieval results to the LLM, including: Using more advanced similarity metrics; Including contextual information, including the relevance between chunks; Using regression or classification models for re-ranking; For the difficulty of processing unstructured documents, by using optical character recognition (OCR) technology, recognizing the effective information in unstructured documents, extracting and importing into the RAG system knowledge base for data training; By using optical character recognition (OCR) technology, recognizing the effective information in unstructured documents, extracting and importing into the RAG system knowledge base for data training, including: First, image preprocessing: grayscale, converting the image to a grayscale image; binarization, converting the grayscale image to a black and white image; noise reduction, removing noise points in the image; correcting the tilt, correcting the text tilt in the image; Then use the Tesseract recognition engine to recognize, post-process the recognized text, and extract effective information; Post-processing: remove noise, delete irrelevant or incorrect text; segmentation, divide the text into paragraphs, sentences, or other structures; use natural language processing (NLP) techniques to extract structured information from the text, including named entity recognition (NER), keyword extraction.

2. The RAG-based artificial agent generation method of claim 1, wherein, In word segmentation and normalization, normalization processing includes stem extraction and lemma reduction.

3. An artificial agent generation system based on RAG technology, characterized by, The system implements the method for generating an artificial intelligence agent based on the RAG technology according to any one of claims 1-2, the system comprising: A knowledge base module (1) for retrieving relevant documents and generating answers or text using a bound vector model and a document processing model, the document processing model being responsible for preprocessing and vectorizing the document library, the vector model processing user input queries and converting the queries into vector representations; A vectorization numerical conversion module (2) for importing data into the knowledge base, queuing the data import time sequence into vectorized numerical values; A robot application creation module (3) for creating a robot application and associating the knowledge base; A publishing module (4) for publishing the robot application and conducting a chat; A correction module (5) for correcting the robot question and answer data to enable the robot to provide more accurate answers.

Images