Data model establishment method based on large language model and local knowledge base

Abstract

The invention discloses a data model establishment method based on a large language model and a local knowledge base. The method comprises the following steps: S1, collecting parallel corpora through a distributed crawler technology; s2, converting data in a non-text format into data in a text format; s3, cleaning and preprocessing all the collected data; s4, denoising, segmenting and aligning the content to construct a parallel corpus; s5, storing the processed parallel corpora and terminology codes in a vector database; s6, constructing a retriever, and retrieving related information from the vector database; and S7, combining the retrieved related parallel corpora with the to-be-translated content in combination with the footstone model to realize high-quality translation generation. According to the method, the flexibility and practicability of the system are greatly improved, the method is suitable for various application scenes such as received text processing and manuscript translation, cross-language communication is more convenient and efficient, and the method has extremely high practical value and wide application prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method for establishing a data model based on a large language model and a local knowledge base. Background Art

[0002] In the field of natural language processing, especially in document translation for specific vertical industries such as customs, traditional methods rely on general large language models for text processing and translation. However, due to the lack of specialized training data for specific fields, these general models often have difficulty in accurately understanding and translating industry-specific terminology and expressions, resulting in uneven quality of translation results. In addition, when faced with policy changes or the emergence of new vocabulary, existing systems usually need to retrain or adjust the entire model, which is not only time-consuming and labor-intensive, but also difficult to adapt to changes quickly. At the same time, the processing capacity for non-textual materials (such as images and scanned documents) is limited, and they are usually unable to be directly converted into text formats that can be used for machine learning, limiting the breadth of data sources. Furthermore, traditional translation systems are inefficient in information retrieval and have difficulty in quickly locating relevant parallel corpora, affecting the quality and efficiency of the final translation. The present invention addresses the above-mentioned technical deficiencies and proposes a data model establishment method based on a large language model and a local knowledge base. By constructing a high-quality parallel corpus, flexibly managing a professional terminology library, and an efficient information retrieval mechanism, the quality and response speed of document translation in specific fields are significantly improved, solving many of the shortcomings in the existing technology. Summary of the Invention

[0003] 1. Technical Problems Solved

[0004] The technical problems to be solved by the present invention are the various problems mentioned in the above background technology, and a method for establishing a data model based on a large language model and a local knowledge base is provided.

[0005] 2. Technical Solution

[0006] To solve the above technical problems, the present invention provides a technical solution: a method for establishing a data model based on a large language model and a local knowledge base, comprising the following steps:

[0007] S1. Collect parallel corpora from customs industry standards, terminology manuals, journal articles, news reports, and other sources using distributed crawler technology;

[0008] S2. Use optical character recognition (OCR) technology to convert non-textual materials into text format;

[0009] S3: Clean and preprocess all collected data, convert documents in different formats into TXT format, and further standardize them into JSON format;

[0010] S4. De-noise, segment, and align the content to construct a parallel corpus. Regularization rules, co-occurrence frequency statistics, and machine learning techniques are used to initially extract professional terms. This is followed by standardization processes such as synonym merging and word form unification.

[0011] S5. Storing the processed parallel corpus and professional terminology codes in a vector database;

[0012] S6. Build a retriever that uses a similarity search algorithm to retrieve relevant information from the vector database and reorder the results based on relevance and importance;

[0013] S7. Combine the cornerstone model and combine the retrieved relevant parallel corpus with the content to be translated to achieve high-quality translation generation.

[0014] As an improvement, the parallel corpus and professional terminology database can be flexibly increased, decreased or modified so as to quickly adapt to policy changes and the emergence of new vocabulary.

[0015] As an improvement, the system provides a kind of "middle platform" capability, which can be used alone or embedded in other application systems, and can be called by other systems through API interfaces.

[0016] As an improvement, in the document receiving processing scenario, after the scanned document is converted into electronic text through text recognition, the accompanying foreign document can be translated into Chinese by clicking the automatic translation function, which is convenient for reading or further editing.

[0017] As an improvement, in the document translation scenario, users can upload the document to be translated or obtain the translation results in the large model by copying and pasting.

[0018] As an improvement, the cornerstone model can be adapted to specific domain requirements without change, and the translation quality in specific domains can be improved by updating the local knowledge base.

[0019] 3. Beneficial Effects

[0020] The advantages of the present invention compared with the prior art are:

[0021] This paper proposes a data modeling method based on a large language model and a local knowledge base. Through systematic steps, it significantly improves the translation quality of specific domains, such as customs documents. First, distributed crawler technology and optical character recognition (OCR) techniques are used to efficiently collect and convert parallel corpora, ensuring the breadth and accuracy of the data source. Next, a high-quality parallel corpus is constructed by cleaning, preprocessing, and standardizing the raw data. Advanced techniques such as machine learning are used to extract specialized terminology, enhancing the accuracy and consistency of terminology. Furthermore, this processed data is stored in an efficient vector database and combined with a similarity search algorithm to enable fast and accurate information retrieval. Finally, by combining the cornerstone model with the retrieved relevant parallel corpora, this method not only achieves high-quality translation generation but also supports the flexible addition, deletion, or modification of the local knowledge base to quickly adapt to policy changes and the emergence of new vocabulary. Specifically, this method provides a "middle platform"-like capability that can be used independently or embedded in other application systems, significantly enhancing the system's flexibility and practicality. It is suitable for a variety of application scenarios, such as incoming document processing and manuscript translation, making cross-language communication more convenient and efficient. It has high practical value and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of a method for establishing a data model based on a large language model and a local knowledge base according to the present invention. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0024] Example 1

[0025] like Figure 1 As shown, a method for establishing a data model based on a large language model and a local knowledge base includes the following steps:

[0026] S1. Collect parallel corpora from customs industry standards, terminology manuals, journal articles, news reports, and other sources using distributed crawler technology;

[0027] S2. Use optical character recognition (OCR) technology to convert non-textual materials into text format;

[0028] S3: Clean and preprocess all collected data, convert documents in different formats into TXT format, and further standardize them into JSON format;

[0029] S4. De-noise, segment, and align the content to construct a parallel corpus. Regularization rules, co-occurrence frequency statistics, and machine learning techniques are used to initially extract professional terms. This is followed by standardization processes such as synonym merging and word form unification.

[0030] S5. Storing the processed parallel corpus and professional terminology codes in a vector database;

[0031] S6. Build a retriever that uses a similarity search algorithm to retrieve relevant information from the vector database and reorder the results based on relevance and importance;

[0032] S7. Combine the cornerstone model and combine the retrieved relevant parallel corpus with the content to be translated to achieve high-quality translation generation.

[0033] The parallel corpus and professional terminology database can be flexibly increased, decreased or modified to quickly adapt to policy changes and the emergence of new vocabulary. The system provides a kind of "middle platform" capability, which can be used independently or embedded in other application systems, and can be called by other systems through the API interface. In the document processing scenario, after the scanned document is converted into electronic text through text recognition, the accompanying foreign document can be translated into Chinese by clicking the automatic translation function, which is convenient for reading or further editing. In the document translation scenario, users can upload the document to be translated or obtain the translation results in the large model by copying and pasting. The cornerstone model can adapt to the needs of specific fields without change, and improve the translation quality in specific fields by updating the local knowledge base.

[0034] The specific implementation of the present invention is as follows:

[0035] First, distributed crawler technology (S1) is used to collect parallel corpora from multiple sources, including customs industry standards, terminology manuals, journal articles, and news reports, ensuring the breadth and representativeness of the data. For non-textual materials, such as images or scanned documents, optical character recognition (OCR) technology (S2) is used to convert them into text format for subsequent processing. Next, all collected data is cleaned and preprocessed, and documents in different formats are uniformly converted to TXT format and further standardized into JSON (S3), laying the foundation for building a high-quality parallel corpus.

[0036] Subsequently, during the construction of the parallel corpus (S4), the content is de-noised, segmented, and aligned. Regularization rules, co-occurrence frequency statistics, and machine learning techniques are used to initially extract specialized terms. Terms are standardized through synonym merging and word form unification, improving their consistency and accuracy. The processed parallel corpus and specialized terminology encodings are then stored in a vector database (S5). A search engine is then constructed, using similarity search algorithms (such as cosine similarity) to efficiently retrieve relevant information from the vector database. The results are then rearranged based on relevance and importance (S6), enabling fast and accurate information search.

[0037] Finally, combined with the cornerstone model (such as Qwen2, ChatGLM, etc.) (S7), the retrieved relevant parallel corpus is combined with the content to be translated to generate high-quality translation results. This system can not only flexibly increase, decrease or modify the parallel corpus and professional terminology library to adapt to policy changes and the emergence of new vocabulary, but also provides a kind of "middle platform" capability, which can be used independently or embedded in other application systems, and can be called externally through the API interface. In addition, this method performs well in actual application scenarios. For example, in the document processing scenario, the scanned document can be automatically translated into Chinese with one click after being converted into electronic text through text recognition. In the document translation scenario, users can upload or copy and paste the document to be translated to obtain instant translation results. The entire process does not require changing the cornerstone model itself. Only the local knowledge base needs to be updated to meet the translation needs of specific fields, which greatly improves the quality and efficiency of translation.

[0038] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0039] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

[0040] The present invention and its embodiments are described above. This description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if a person skilled in the art is inspired by this and, without departing from the purpose of the present invention, designs structures and embodiments similar to this technical solution without inventiveness, they shall fall within the scope of protection of the present invention.

Claims

1. A method for establishing a data model based on a large language model and a local knowledge base, characterized in that: The following steps are involved: S1. Collect parallel corpora from customs industry standards, terminology manuals, journal articles, news reports, and other sources using distributed crawler technology; S2. Use optical character recognition (OCR) technology to convert non-textual materials into text format; S3: Clean and preprocess all collected data, convert documents in different formats into TXT format, and further standardize them into JSON format; S4. De-noise, segment, and align the content to construct a parallel corpus. Regularization rules, co-occurrence frequency statistics, and machine learning techniques are used to initially extract professional terms. This is followed by standardization processes such as synonym merging and word form unification. S5. Storing the processed parallel corpus and professional terminology codes in a vector database; S6. Build a retriever that uses a similarity search algorithm to retrieve relevant information from the vector database and reorder the results based on relevance and importance; S7. Combine the cornerstone model and combine the retrieved relevant parallel corpus with the content to be translated to achieve high-quality translation generation.

2. The method for establishing a data model based on a large language model and a local knowledge base according to claim 1, characterized in that: The parallel corpus and professional terminology database can be flexibly increased, decreased or modified to quickly adapt to policy changes and the emergence of new vocabulary.

3. The method for establishing a data model based on a large language model and a local knowledge base according to claim 1, characterized in that: The system provides a kind of "middle platform" capability, which can be used independently or embedded in other application systems, and can be called by other systems through API interfaces.

4. The method for establishing a data model based on a large language model and a local knowledge base according to claim 1, characterized in that: In the document receiving and processing scenario, after the scanned document is converted into electronic text through text recognition, you can click the automatic translation function to translate the attached foreign document into Chinese for easy reading or further editing.

5. The method for establishing a data model based on a large language model and a local knowledge base according to claim 1, characterized in that: In the document translation scenario, users can upload the document to be translated or obtain translation results in the large model by copying and pasting.

6. The method for establishing a data model based on a large language model and a local knowledge base according to claim 1, characterized in that: The cornerstone model can be adapted to specific domain requirements without change, and the translation quality in specific domains can be improved by updating the local knowledge base.

Images