Method for training large-scale language models using machine learning, and program for utilizing the same

Abstract

To provide a method and program for searching document files that enables high-speed searching of massive text files with high response accuracy.SOLUTION: A method for storing data in a large-scale language model database involves fragmenting text within document files into chunks. Each of the chunks includes a specified number or more of characters from an end of an immediately preceding chunk of continuous text. Each of the chunks is associated with a vectorized numerical value and stored, as unit data to which data including a filename, category, product name, and text are given. A questioning method involves comparing vectorized data of user-input text with vectorized data of text extracted from answer information sources by a large language model and sending n higher-order relevant text data and a prompt to the large language model. When the text is stored in the large language model, the questioning method trains the model through machine learning.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a method for creating a large-scale language model, and also to a program using the same. [Background technology]

[0002] A large language model (LLM) is a language model that has a very large dataset and is constructed using deep learning techniques to speed up search. In recent years, much research has been done on large-scale language models, which have made it possible to process natural language accurately and quickly and output answers to questions. [Prior art documents] [Patent documents]

[0003] Patent Document 1: JP 2024-27070

[0004] Patent Document 1 is a document on the generation of canonical forms for task-oriented dialogue in conversational AI systems and applications, and describes large-scale data processing for processing natural language. However, the document does not mention that the text data used in the large-scale data processing should be limited to documents created and used by humans or organizations in society as a network formed around humans, or that it should be designed to be suitable for such things, thereby making responses in dialogue more accurate or improving the accuracy of those responses. Summary of the Invention [Problem to be solved by the invention]

[0005] Given the above situation, in order to provide language models that can provide answers appropriate to various situations in humans or organizations in a society that is a network formed around humans, there is a challenge of improving the accuracy of answers that appear at the top of the results in searches of the vast amount of documents created and used in those organizations. To process huge amounts of data, it is necessary to create a data structure that is appropriate for that data, and to create an associated structure. In any case, the amount of calculation required for processing using large-scale language models is extremely large, and a high-performance processing device is required to achieve sufficient processing speed. When searching for unpublished data or materials, if a search is performed by incorporating text data from areas where machine learning has not yet been implemented using large-scale language models commonly known as generative AI, there is a problem in that it takes time for the accuracy of answers to be sufficiently high in areas where machine learning has not yet been implemented until machine learning has progressed. [Means for solving the problem]

[0006] In order to solve the above problems, the present invention uses the following means, but is not limited to these. It is necessary to have a data structure and accompanying structure that are suitable for documents created and used by people or organizations in a society that is a network formed around people. The present invention is a method for searching for document files containing text related to products within organizations in society, in which data is saved in a large-scale language model database by dividing the text in the document files into chunks every predetermined number of characters, each chunk containing at least a predetermined number of characters from the end of the previous chunk in the continuous text, each chunk being associated with a vectorized numerical value, and each chunk being saved as unit data containing the file name, category, product name, and text; the search method is performed by comparing a search string together with a prompt as vectorized data with vectors generated from text data in the chunks in the database; and when saving unit data containing chunked text in the large-scale language model database in association with vectorized numerical values, an operation is performed to associate the chunked text data itself with vectorized numerical values ​​and then save the data as unit data. [Effects of the Invention]

[0007] In the conventional method, when a question is posed to a large-scale language model and an answer is obtained, if a question is posed together with a wide range of unique information, the communication volume becomes too large, exceeding the general setting value, and it takes a long time to obtain an answer. However, by using the method of the present invention, the accuracy of the answer is improved, the communication volume is relatively reduced, and the answer time is shortened. This also brings economical usefulness when using a pay-as-you-go large-scale language model. While current searches using general large-scale language models rely on machine learning of only information on the Internet, the large-scale language model used in the method of the present invention is used to learn not only information on the Internet but also information stored within organizations in society that is not generally publicly available, primarily for use within the organization. In this case, by using the method of the present invention, machine learning can significantly improve search accuracy in unlearned areas, thereby improving search speed and efficiency. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing a flow in which a large-scale language model (LLM) outputs an answer from a question based on an answer information source in the present invention. [Figure 2] This is a conceptual diagram of text extraction / data formatting. [Figure 3] FIG. 1 is a conceptual diagram of chunking. [Figure 4] 1 is a diagram showing overlapping portions of the beginning and end of chunked text; [Figure 5] This is a conceptual diagram of text embedding (vectorization). [Figure 6] This is a conceptual diagram comparing the vector data extracted from the question with the vector data stored in the LLM. [Figure 7] This is a conceptual diagram showing that questions and answers with the same meaning can be asked and answered in a foreign language using corresponding vector data. [Figure 8] This is a conceptual diagram showing that if the answer information source is not public, an answer may not be obtained. DETAILED DESCRIPTION OF THE INVENTION

[0009] The following describes exemplary embodiments of the present invention, but the embodiments of the present invention are not limited to the following embodiments as long as they do not deviate from the gist of the present invention.

[0010] The first embodiment of the present invention is shown in the flow chart of FIG. 1 and is exemplified by the following configurations (1) to (3). (1) A method for searching document files containing text related to a product within an organization in a society, Data storage in a large-scale language model database The text in the document file is fragmented into chunks, each of which contains at least a certain number of the tail parts of the previous chunk in the continuous text, Each chunk is associated with a vectorized numerical value, and each chunk is stored as a unit of data with data including file name, category, product name, and text. The questioning method compares vectorized data of user-entered text with vectorized data of text extracted from answer sources in a large-scale language model; The method involves acquiring the top n most relevant text data, and transmitting the text entered as the user's question, the acquired top n most relevant text data, and a prompt to a large-scale language model; When storing fragmented text as chunks in large language models and associating it with other data, This is a method of machine learning a large-scale language model, characterized by first associating vector data that is vectorized from the fragmented text data itself as chunks, and then adding other associated data and saving it as unit data.

[0011] (2) The embodiment of the present invention further includes a program using a large-scale language model that has been machine-learned using the method described in the first embodiment. (3) An embodiment of the present invention further includes a program for improving the accuracy of answers of a large-scale language model when a question from a set of questions is asked to a population of any answer information source by performing machine learning of the method described in the first embodiment on the large-scale language model.

[0012] The large-scale language model 11 in the present invention searches for "data and materials that are not publicly available" that have not been learned by machine learning or have been learned insufficiently, so that the results are clearly far from being saturated, or it can conduct a chat in which instructions and questions are entered, and outputs the answers as text. The unlearned region in machine learning may be set by input operation by the operator, or an automated method such as a program that recognizes some kind of unlearned state may be used. To determine whether a model is unlearned using machine learning, a judgment should be made based on the data and answer results accumulated in a large-scale language model. However, it is thought that the answers and data to be accumulated will change as question trends and question populations change, and it is therefore not always meaningful to clearly define the unlearned state. Regarding the state in which the learning effect saturates and answers become constant, a meaningful definition should depend on the purpose at the time, considering factors such as the degree to which saturation should occur and whether the impact of changes in the question population should be taken into account.

[0013] Narrowing the search range will help improve processing speed, but if this is an area that has not yet been studied by machine learning, it will take time for the accuracy of answers to improve until machine learning advances.

[0014] In each procedure in the first embodiment of the present invention, processing is performed as follows. <Step 1> Pre-processing of response information sources (such as internal data not publicly available) involves text extraction / data shaping 1 and chunking 2. Although not explicitly shown in Figure 1, after chunking 2, metadata is added (in addition to the chunked text, "product name," "category," and "file name"), vectorization 3 (embedding processing) and the vectorized data is saved in a vector database. <Step 2> After the user inputs a question, the vector retrieval process involves vectorizing the user's question (text information) (5), comparing it with the vector database (vector search 4), and retrieving the chunked text from step 1 and the text with added metadata for the top n most relevant vector data (in the case of topk30, the top 30 vector data are retrieved). <Step 3> An API call to the large-scale language model (LLM) is made by adding a prompt to the "chunked text + metadata" from step 1 before vectorizing the top n most relevant vector data obtained in step 2, plus the user's question (raw text information before vectorization), and sending it to the LLM to obtain an answer.

[0015] Here, chunking using an appropriate method or rule may be effective in improving the accuracy of answers from a large-scale language model. In this case, searching using vectorized data of chunked text will result in more searches using vectorized data of small units of text than using vectorized data of the entire contents of the text file. For this reason, it is believed that partial compatibility or response accuracy may be improved compared to vectorizing the text data of the entire file and structuring the vector data. Furthermore, when a user inputs a question to perform a search, the text of the question is converted into a vector and compared with structured / unstructured text data in a large-scale language model. When a search is performed, the component values ​​of the chunked text data vectors are first compared by calculating the dot product of the vectors and calculating the norm, and then classification is performed by product / product information, category of product, document, etc., and finally by file name.

[0016] The technology related to the present invention is included in Retrieval-Augmented Generation (RAG). There are several types of RAG technology, but the RAG technology of the present invention is unique in the following respects.

[0017] 1. Text extraction / data formatting In the present invention, text extraction and the accompanying data shaping are performed as follows: [Figure 2] First, it handles structured and unstructured data sources in different formats. It is possible to process structured and unstructured data sources in different formats such as PDF, Excel, and HTML. For example, PDF files, XLS files created using spreadsheet software, text files recorded in a narrow text format only, document files (DOC files, DOCX files) that record text files with additional information such as formatting information, CSV files that record data in a certain format along with delimiters, HTML files used on websites, and PPT files for presentations can be used.

[0018] 2. Chunking In chunking in the method of machine learning using a large-scale language model of the present invention, text is fragmented and divided in a way that does not lose meaning or context. The length of one chunk may be, for example, 100 to 10,000 characters, 100 to 2,000 characters, 100 to 1,000 characters, or 100 to 200 characters in Japanese text. The typical number of characters per page of manuscript paper used in Japan is 400 characters, and in word processing software, a document with 40 characters x 50 lines per page is approximately 2000 characters, which is less than five pages of manuscript paper.However, when considering extracting a block from a text sentence created by humans in real life, this number of characters can sometimes be appropriate for indicating the content of the sentence being created when determining the length of the chunk. In addition, in this case, each chunk contains the end portion of the previous chunk in the continuous text, for example, several characters or more in duplication in the case of Japanese text. The overlapping portion is, for example, enough to form one sentence, and in Japanese, for example, half a manuscript paper is enough to complete one sentence, so it may be 200 characters or less. For management purposes, by including at least a few characters in the overlapping portion, it becomes easier to compare it with the end of the previous chunk and use it as a criterion for determining whether or not the chunks are consecutive.

[0019] The chunking of the present invention is shown in Figures 3 and 4, but this is for illustrative purposes only and is simplified, and does not directly represent the actual chunking process.

[0020] 3. Embedding (Vectorization) This is the point that contributes to the method of the present invention being able to handle large amounts of data. [Figure 5] Embedding (vectorization) contributes to improving source display and answer accuracy. The vectors in the following figure are for illustrative purposes only; actual vectors will have many more values. For example, one embodiment of the present invention uses vectors in a 3072-dimensional semantic space.

[0021] 4. Retrieval This is what contributes to the method of the present invention being able to handle large amounts of data. [Figure 6] By performing retrieval (vector search) using the method of the present invention, it is expected that the accuracy and speed of answers in large-volume data sources will be improved.

[0022] 5. Large-scale Language Model (LLM) Query Prompts can be embedded and vectorized to search other natural languages ​​(English, Chinese, French, etc.) either in addition to the questions or instructions entered by the operator, or the entered text alone can be used to provide questions or instructions to a large-scale language model. Because the embedding is used for vectorization, by comparing it with a large-scale language model that has been vectorized using a corresponding embedding method in a different language, it is possible to answer the question in the same natural language, regardless of the language of the answer source. While it is not denied that this method can output an answer in a different natural language (e.g., English) to a question input in a natural language (e.g., Japanese), it is not yet confirmed whether the accuracy of answers can be improved by using the method of machine learning the large-scale language model of the present invention.

[0023] In Figure 7, the answer source is Japanese, whereas question 30a is in English, question 30b is in Chinese, and question 30c is in French. In response to English question 50a, answer 50a is in English, question 30b is in Chinese, and question 30c is in French. [Figure 7]

[0024] Text data from non-public answer sources is not machine-learned by general large-scale language models. When searching by adding the content of non-public answer sources to the scope of the search, the accuracy of the answer decreases. [Figure 8]

[0025] As used herein, the following terms are defined as follows: [chunk] A chunk refers to one of the texts obtained by dividing the text in a document file into multiple texts. Generally, when a long text is divided into chunks, the beginning of the portion to be fragmented may be a piece of text that overlaps with the end of the previous portion of the text and has a certain length or more. [Chunking] Chunking is a technique for fragmenting and dividing text in a document file in a way that preserves meaning and context. The points at which text is divided into fragments can be determined based on the number of characters, the presence of punctuation marks, the division of words, phrases, and sentences, specific meanings, characters, etc. In one embodiment of the present invention, a method of dividing by the number of characters and words is adopted to increase the processing speed. The chunking method can be selected according to the type and characteristics of the text data to be handled and the specifications of the hardware that performs the calculations. By using a chunked data structure, a large-scale language model can store multiple vector data sets for a single file, instead of just one if chunking is not performed, and can be used as a search target. This allows a single file to be searched from multiple perspectives. By embedding chunked text data to obtain vector data, the individual vector data are more likely to have values ​​corresponding to their respective parts than when embedding is performed without chunking, resulting in clearer differences in the vector data. This has the effect of clarifying semantic differences because, considering the spread of coordinates in semantic space, the data are scattered over a wider range of space, and the processing of individual position coordinates takes longer. This method can be applied to many currently existing large-scale language models. It is possible to add multiple metadata to a single file without chunking it, and use it for searching as a data structure that essentially has vector data of multiple files attached, but the method of the present invention adopts a data structure that attaches metadata to chunked text data.

[0026] [Large-scale language model] Large-scale language models include OPEN-AI's Chat-GPT, GPT2.0, 3.0, 3.5, 4.0, etc., and Alphabet's Gemini. There are also Japanese domestic large-scale language models such as NTT Group's "tsuzumi" and the University of Tokyo's Matsuo Laboratory's "Weblab-10B." There are currently many large-scale language models for Japanese alone, some with over 100 billion parameters. In recent large-scale language models, parameters are automatically calculated in relation to the dimensions of orthogonal vectors, and the existence of potential new dimensions can be discovered through calculations, such as calculating the eigenvectors of transformations using functions, or by estimating certain regularities or calculating them using functions.

[0027] [Vector Search] In natural language processing, a vector is a numerical representation of text in semantic space, where words and combinations of words are broken down into meanings and expressed as a combination of numerical values ​​on each semantic axis. Fragmented and chunked text can also be vectorized. Using vectorized numerical values, the norm of closeness in the semantic space is measured using methods such as the dot product of vectors, and this is used to measure search and response accuracy.

[0028] [Embedding] Embedding is a technique for converting words, text, and images into numerical vectors that are easy for AI and language models to handle. In the embodiment of the present invention, a 3072-dimensional vector is used, but an n-dimensional vector can also be used, with any natural number n set to suit the computing power of the processing device used. However, if n is too small, it will be impossible to simultaneously determine subtle differences in the accuracy of determining the meaning of language, and if n is too high, the processing device will require advanced processing power and the calculation will tend to take a long time.

[0029] Embedding can use vector coordinates as coordinate axes that are defined from the beginning, but it can also determine coordinate axes by calculating or estimating eigenvectors in vector data as big data. In this case, it is possible to calculate or estimate coordinate axes that are useful for classifying big data and set the necessary coordinates, even if coordinate axes are not defined from the beginning. In this case, as a procedure for adding vector data to the text data that constitutes a large-scale language model, by first classifying the text data using the vector coordinate axes obtained from the chunked text, it is possible to search for similar parts using moderately fragmented text.

[0030] [prompt] A prompt is an instruction or question entered by a user in an interactive system, such as a conversation with an AI or a command line interface (CLI). Instructions and questions may be based on prerequisite conditions, such as a subordinate being asked by a superior in a company, or a customer asking a question to a store clerk. Clear and specific prompts allow AI to generate appropriate responses and results in response to the user's request or question. Prompts can be broadly categorized into two types: instruction prompts, which are used to give specific instructions to AI, and question prompts, which are used to ask AI questions. When a user enters a question, instruction prompts can also be used to add prerequisites for the answer that the AI ​​will provide. [Pre-Linguistic Processing] From files that contain not only text but also images and audio and are saved in multiple file formats, handleable data is extracted and then converted into text data using a specific method, resulting in data and a file format that can be handled by a large-scale language model, and unit data that can be handled as a dataset associated with file names and category information. In the present invention, chunking is performed in principle first by extracting text in the pre-linguistic processing stage.

[0031] [Generation of large-scale language models automatically trained by machine learning] If you limit the search to internal or company documents, the search scope will be narrowed, which will speed up processing, but on the other hand, there is a risk that internal documents may not be accessible under normal circumstances due to security reasons, and internal documents may have unique characteristics or tendencies that make normal search methods inappropriate. If limiting the processing to internal documents is advantageous in terms of processing speed, it can be advantageous to chunk the text database and generate text documents or their vector data to be used for comparison during searches, etc. In this case, the processing capacity and speed gained by limiting the processing to internal documents can be used to optimize the processing of the chunked text database, in accordance with the specifications of hardware designed for the expected large-scale language model. To achieve this, for example, the design or data structure may be such that the number of chunked text data items of the internal documents to be searched is approximately the same as the number of text data items expected in a large-scale language model determined by the hardware specifications. That is, Retrieval-Augmented Generation (RAG) may be performed according to the hardware specifications, and the chunked data may be stored or structured as redundant text data to improve the accuracy of the answers.

[0032] [Answer accuracy] However, in general, the method of measuring answer accuracy is not limited to the above, as long as it does not deviate from the purpose of measuring answer accuracy in a large-scale language model. The measurement of answer accuracy can be performed by a process separate from the search. In this case, the search process and the answer accuracy determination process may each be a plurality of processes, or may be performed as a combination of these processes. By performing multiple search processes in parallel using multiple processing methods and algorithms, and then performing independent measurement processing of the response accuracy on the results, the processing device becomes more complex and larger in scale, but the processing speed can be improved if appropriate processing is included in the parallel data processing without significantly slowing down the processing speed compared to the basic processing. Additionally, the accuracy of the answer may be determined based on whether a large-scale language model can return an answer that provides meaningful information within a certain period of time. In other words, if the large-scale language model responds to a question by providing meaningful information in response to the question, rather than by saying that it cannot answer, it is considered to have answered the question.The percentage of questions in the population that have been answered can be measured, and if this percentage increases, it can be determined that the accuracy of the answers has improved.

[0033] By preparing multiple methods for chunking and redundancy of text data, it is possible to approximate a data structure that matches the characteristics of documents within an organization or company. In this case, tuning of such a redundant text database can be performed during times when questions or instructions are entered but searches are not being performed and specific operations are not being performed.

[0034] Whether a large-scale language model returns a substantive answer to an input question, rather than an answer that conveys that it cannot answer, depends on the answer discrimination method set in the large-scale language model and the large amount of vector data that has been machine-learned and stored, but generally, for a vectorized question, the vector data in the large-scale language model is compared using the set discrimination method, and the answer is determined based on the distance or norm when considered as a position vector in semantic space. Here, if multiple processes are performed in parallel and the results with the best answer accuracy are selected, the answer accuracy can be improved without significantly slowing down the processing speed.

[0035] In large-scale language models for natural language processing, it is common to actually perform processing using a computer to obtain results. In this invention, a processing method is designed and parameters are processed for documents created and used by people or organizations in a society that is a network formed around people, by utilizing characteristics that can be said to be natural laws that the documents possess as their properties.

[0036] The first embodiment of the present invention has already been described in paragraph

[0010] . In contrast to this, in a second embodiment of the present invention, the questions are asked in another country's language, which is a language different from the natural language in which the embedding was performed. The question is entered by the operator, and a prompt appropriate to the situation is added to the question and embedded. By embedding, the vectorized numerical value can be used to search in a natural language other than the natural language in which the question was asked (for example, if the question is asked in Japanese, it can be searched in English, Chinese, French, etc.). Because the embedding is used for vectorization, by comparing it with a large-scale language model that has been vectorized using a corresponding embedding method in a different language, it is possible to answer the question in the same natural language, regardless of the language of the answer source. While it is not denied that this method can output an answer in a different natural language (e.g., English) to a question input in a natural language (e.g., Japanese), it is not yet confirmed whether the accuracy of answers can be improved by using the method of machine learning the large-scale language model of the present invention.

[0037] 7, answer information source 200 is in Japanese, while question 30a is in English, question 30b is in Chinese, and question 30c is in French. In response to English question 50a, answer 50a is in English, answer 50b is in Chinese to question 30b, and question 50c is in French to question 30c.

[0038] In this specification, society refers to human society or society as a network formed around humans. In this specification, an organization refers to an organization in society, such as a government, a local public body, or a company. It is thought that documents created or used by these organizations may have certain characteristics due to considerations of privacy, particularly sensitive information that has become a problem recently, trade secrets and know-how of companies and other organizations, readability, universal design, barrier-free access, etc. Aside from a detailed consideration of how these characteristics affect the text, it is thought that the length of the sentences used in the text, the types of words and characters used, and, for example, the degree to which simple language is preferred depending on the situation, will change, and these results may affect the relationship between data structure and response accuracy. Therefore, by appropriately performing chunking in the present invention, a favorable effect can be exerted on the accuracy of answers. When product-related documents and electronic files are created by organizations in society and text is extracted from them, the text is likely to have the characteristics of text that takes into consideration the trade secrets and know-how of the organization, such as companies, and readability.Furthermore, in text that will be provided to consumers or customers, the text is likely to be influenced by considerations of privacy, etc. Furthermore, for a large-scale language model to perform machine learning, it is necessary to either make information publicly available on the Internet or to access information without restrictions for security reasons, import data, and perform machine learning. However, the large-scale language model of the present invention is intended to handle text information that is not publicly available on the Internet, such as answer information sources used exclusively within an organization. For this reason, it is necessary to analyze a small amount of information from multiple perspectives and output answers with high accuracy. For this reason, the data structure that uses chunked text data in the method of the present application is suitable and useful in many cases.

[0039] Furthermore, files created in an environment that uses WEB 3.0 technology as a distributed Internet technology, and the text extracted from them, are expected to contain a lot of text that is privacy-conscious or has a tendency to use terms and sentences only in specific organizations or groups. In such cases, the data structure that uses chunked text data in the method of the present invention is also useful. [Example]

[0040] Examples of the present invention are given below. In large-scale language models for natural language processing, it is common to actually perform processing using a computer to obtain results. In this invention, a processing method is designed and parameters are adjusted for documents created and used by people or organizations in a society that is a network centered around people, utilizing characteristics that can be said to be natural laws that the documents possess. In the embodiment of the present invention, a 3072-dimensional vector is used. In this specification, response accuracy was measured as follows. When the large-scale language model responded to a question with a meaningful answer, i.e., an answer that provided meaningful information in response to the question rather than an answer that it could not answer, it was deemed to have answered, and was marked as Yes in each table. On the other hand, when it answered that it could not answer or a similar answer was given, it was marked as No. The percentage of people who answered "yes" to all questions in the population was measured, and if this increased, it was determined that the accuracy of the responses had improved. In addition, we calculated the change in answer accuracy when comparing Top30K and Top60K, and the percentage of questions that were answered when comparing the conventional embedding method and the new embedding method. (Answer accuracy) = (Number of answered questions) / (Number of questions in the population)

[0041] The table below shows the experimental results on the change in answer accuracy due to differences in embedding methods. With the current embedding method, embedding was performed in the order of file name, file format, source, category, product, and text, and vector data was added, but with the new embedding method, embedding was performed in the order of text, product, category, and file name, and vector data was added. [Table 1]

[0042] Table 1 compares the current embedding method with the new embedding method. The current embedding method adds metadata to unchunked text, while the new embedding method adds metadata to chunked text. The new embedding method showed an improvement in answer accuracy compared to the current embedding method. With the current embedding method, for the entire population of 83 questions, 64 answers (77.1%) were obtained for both TopK(30) and TopK(60), while the new embedding method yielded 67 answers (60.7%) for TopK(30) and 76 answers (91.6%) for TopK(60). This trend was even more pronounced when the population consisted of only questions related to technical terms. For the entire population of 40 questions, both TopK(30) and TopK(60) yielded 25 answers (62.5%), while the new embedding method yielded 28 answers (70.0%) for TopK(30) and 35 answers (87.5%) for TopK(60). A similar trend was observed when official test questions from research institutions were used as the population, but when questions created by research institutions that were not official test questions were used as the population, the opposite result was observed, with answer accuracy decreasing. It is thought that the level of machine learning is relatively low for questions that are not official test questions, but the relationship between this and the new embedding method cannot be determined in general, as it depends on other conditions. It is possible that fluorescence could change if the answering criteria were made easier. [Table 2]

[0043] Change the amount of data extracted and used when answering questions using the current embedding method [Table 3] New embedding method: Change the amount of data extracted and used when answering [Table 4]

[0044] Cross-Comparison Overall [Table 5] Comparison using technical terms only [Table 6]

[0045] Official research institute test questions [Table 7] Other (questions you have created that are not official test questions from your research institution) [Table 8]

[0046] Answer to the question (excerpt from number 2467) about using the current embedding method. From TopK(30) to TopK(60), the answer sentences to the same question were different, but the availability of the answer did not change. [Table 9]

[0047] Answers to questions (excerpts from numbers 2467 and 2475) using the new embedding method. From TopK(30) to TopK(60), the answers to the same questions were different, and for some questions, the availability of answers changed. [Table 10] [Table 11]

[0048] Comparison of the answers obtained using each embedding method (TopK(30)) [Table 12] [Industrial Applicability]

[0049] This invention uses large-scale language models to search and improve the speed and accuracy of responses in unlearned areas through machine learning. This makes it possible to use this technology for document management within organizations such as companies, and is expected to be applied to a variety of purposes in a wide range of industries. [Explanation of symbols]

[0050] 1. Text extraction / data shaping 2 Chunking 3 Vectorization (Embedding) 4 Vector Retrieval 5 Vectorization (Embedding) 10 Large-scale Language Models (LLM) 11 Vector DB 20 Population of respondent sources 20a Chunked answer source text collection 2011~2015 Chunked answer source text 30 questions 30a English Questions 30a Chinese Questions 30a French Questions 31 questions 40 prompts 50 answers 50a English Answers 50a Chinese Answers 50a French Answers 200 Answer Source 201 Answer information source 300 Selected Answer Sources 301 Selected Response Sources

Claims

1. 1. A method for searching document files containing text related to products within an organization in a society, comprising: Data storage in a large-scale language model database The text in the document file is fragmented into chunks, each of which contains at least a certain number of the tail portion of the previous chunk in the continuous text; Each chunk is associated with a vectorized numerical value, and each chunk is stored as a unit of data with data including file name, category, product name, and text. The questioning method compares vectorized data of user-entered text with vectorized data of text extracted from answer sources in a large-scale language model; The method involves acquiring the top n most relevant text data, and transmitting the text entered as the user's question, the acquired top n most relevant text data, and a prompt to a large-scale language model; When storing fragmented text as chunks in large language models and associating it with other data, This method involves machine learning a large-scale language model, characterized in that among the data to be associated, the text data itself fragmented as chunks is first associated with vectorized vector data, and then other associated data is added and saved as unit data.

2. A program for transmitting and receiving data to and from a large-scale language model machine-learned using the method of claim 1.

3. A program for improving the accuracy of answers of a large-scale language model when a question from a set of questions is asked to a population of arbitrary answer information sources by performing the machine learning method of claim 1 on the large-scale language model.

Images