Systems and methods for document search

Abstract

This paper discloses a document retrieval system comprising a chunking module, a vector transformation module, a database module, and a retrieval module. The chunking module performs a chunking process on predefined key sections of multiple documents in a specific domain to obtain multiple text blocks of the key sections. The vector transformation module uses an embedding model to convert the multiple text blocks of each key section into corresponding multiple retrieval embedding vectors, which are stored in the database module. The vector transformation module uses an embedding model to convert the retrieval content into retrieval content embedding vectors. The retrieval module calculates the similarity between the retrieval content embedding vectors and each retrieval embedding vector in the database module, and generates retrieval results accordingly. This paper also discloses the corresponding document retrieval method.

Description

Technical Field

[0001] This invention relates to a document retrieval system and method, and in particular to a document retrieval system and method for documents in a specific field. Prior Technology

[0002] In the field of document retrieval, there are various challenges in retrieving and processing documents in specific or specialized domains. Many technologies focus on semantic analysis of document data and automated processing, but problems still remain to be solved.

[0003] First, previous technologies often relied on strategies such as "categorization" and "keyword extraction" when processing text semantics. However, these methods are rather rigid in extracting text semantics and fail to fully utilize all the text content. Although they can be combined with automated methods to keep the database up-to-date, the technology for retrieving documents will eventually revert to the more traditional "keyword retrieval".

[0004] Furthermore, many retrieval systems are typically limited to feature processing of static text and are often constrained by the length of the model input, thus limiting user expression and failing to provide sufficient context to fully describe the content to be retrieved, which is detrimental to the flexibility of retrieval. Although some technologies attempt to include other features of the document in the retrieval system, the retrieval strategy relies on a pre-defined retrieval process, affecting the flexibility and accuracy of the retrieval. Summary of the Invention

[0005] In one aspect, the present invention provides a document retrieval system for documents in a specific domain, comprising: a segmentation module; a vector conversion module; a database module; and a retrieval module, wherein: for a given plurality of documents in a specific domain, the segmentation module performs a first segmentation process on a predefined key section in each of the plurality of documents in the specific domain to obtain a plurality of text blocks of the key section; the vector conversion module uses an embedding model to convert the plurality of text blocks of each key section into a corresponding plurality of retrieval embedding vectors, the plurality of retrieval embedding vectors being stored in the database module; the vector conversion module uses the embedding model to convert a retrieval content into a retrieval content embedding vector; the retrieval module calculates the similarity between the retrieval content embedding vector and each retrieval embedding vector stored in the database module, and generates a first retrieval result, which indicates at least one document in the plurality of documents in the specific domain that is most relevant to the retrieval content.

[0006] According to certain embodiments of the present invention, the segmentation module further performs a second segmentation process on the full text of each of the plurality of special domain documents to obtain a plurality of text blocks of the full text. The vector conversion module uses an embedding model to convert each plurality of text blocks of the full text into a corresponding plurality of paragraph embedding vectors, and the plurality of paragraph embedding vectors are stored in the database module.

[0007] According to certain embodiments of the present invention, the document retrieval system further includes a similar paragraph comparison module. The similar paragraph comparison module performs a comparison process to generate a second retrieval result, which includes at least one paragraph in the at least one special domain document that is most similar to the retrieval content. The comparison process includes: performing a third block segmentation process to obtain a plurality of text blocks of the retrieval content, and using an embedding model to convert each of the plurality of text blocks of the retrieval content into a plurality of corresponding retrieval paragraph embedding vectors; calculating the similarity between the plurality of retrieval paragraph embedding vectors and the plurality of paragraph embedding vectors corresponding to the at least one special domain document, and generating the second retrieval result accordingly.

[0008] According to certain embodiments of the present invention, the block size set in the first block segmentation process is 1,900 to 2,100 tokens.

[0009] According to some specific embodiments of the present invention, the block size set in the second block process is 600 to 800 tokens.

[0010] According to certain embodiments of the present invention, the embedding model is a BERT-based model.

[0011] According to certain embodiments of the present invention, the document retrieval system further includes an auxiliary fine-tuning module, which uses a large language model (LLM) to translate multiple domain-specific foreign language documents into multiple translations in the same language as the multiple domain-specific documents, and is used for fine-tuning the embedding model.

[0012] On the other hand, the present invention provides a document retrieval method for documents in a specific domain, comprising: for a given plurality of documents in a specific domain, performing a first block segmentation process on a predefined key section in each of the plurality of documents in the specific domain to obtain a plurality of text blocks of the key section; using an embedding model to convert the plurality of text blocks of each key section into a plurality of corresponding retrieval embedding vectors, the plurality of retrieval embedding vectors being stored in a database; using the embedding model to convert a retrieval content into a retrieval content embedding vector; calculating the similarity between the retrieval content embedding vector and each retrieval embedding vector stored in the database, and generating a first retrieval result that indicates at least one document in the plurality of documents in the specific domain that is most relevant to the retrieval content.

[0013] According to certain embodiments of the present invention, the document retrieval method further includes: performing a second block segmentation process on the full text of each of the plurality of special domain documents to obtain a plurality of text blocks of the full text; using the embedding model, converting each plurality of text blocks of the full text into a corresponding plurality of paragraph embedding vectors, the plurality of paragraph embedding vectors being stored in the database.

[0014] According to certain embodiments of the present invention, the document retrieval method further includes: performing a comparison process to generate a second retrieval result, which includes at least one paragraph in the at least one special domain document that is most similar to the retrieval content; the comparison process includes: performing a third block segmentation process to obtain a plurality of text blocks of the retrieval content, and using an embedding model to convert each of the plurality of text blocks of the retrieval content into a plurality of corresponding retrieval paragraph embedding vectors; calculating the similarity between the plurality of retrieval paragraph embedding vectors and the plurality of paragraph embedding vectors corresponding to the at least one special domain document, and generating the second retrieval result accordingly.

[0015] According to certain embodiments of the present invention, the block size set in the first block process is 1,900 to 2,100 tokens.

[0016] According to some specific embodiments of the present invention, the block size set in the second block process is 600 to 800 tokens.

[0017] According to certain embodiments of the present invention, the embedding model is a BERT-based model.

[0018] According to some embodiments of the present invention, the document retrieval method further comprises: using a large language model (LLM) to translate a plurality of foreign documents in a specific domain into a plurality of translated copies in the same language as the plurality of specific domain documents, and using the translated copies for fine-tuning the embedding model.

[0019] Other objects and advantages of the present invention will be partially described in the following description, or may be understood through the embodiments of the present invention. It should be understood that the foregoing Summary of the Invention and the following Detailed Description are merely exemplary and explanatory, and do not limit the present invention as the scope of claims does. Brief Description of the Drawings

[0020] FIG. 1 is an architectural diagram of a document retrieval system for domain-specific documents according to the present invention.

[0021] FIG. 2 illustrates the flow steps of a document retrieval method for domain-specific documents according to the present invention. Detailed Description

[0022] Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0023] As used herein, the term "a", unless otherwise specified, refers to at least one (one or more than one).

[0024] The present invention provides a document retrieval system for domain-specific documents, comprising: a chunking module; a vector conversion module; a database module; and a retrieval module, wherein: for a given plurality of domain-specific documents, the chunking module performs a first chunking process on a predefined key portion of each of the plurality of domain-specific documents to obtain a plurality of text blocks of the key portion; the vector conversion module uses an embedding model to convert the plurality of text blocks of each key portion into corresponding plurality of retrieval embedding vectors, and the plurality of retrieval embedding vectors are stored in the database module; the vector conversion module uses the embedding model to convert a retrieval content into a retrieval content embedding vector; and The retrieval module calculates the similarity between the retrieval content embedding vector and each retrieval embedding vector stored in the database module, and generates a first retrieval result that indicates at least one special domain document among the plurality of special domain documents that is most relevant to the retrieval content.

[0025] This invention also provides a document retrieval method for documents in a specific field, comprising: For a given plurality of special domain documents, a first block segmentation process is performed on a predefined key section in each of the plurality of special domain documents to obtain a plurality of text blocks of the key section; An embedding model is used to convert multiple text blocks of each key part into multiple corresponding retrieval embedding vectors, and the multiple retrieval embedding vectors are stored in a database; Using this embedding model, a search result is converted into a search result embedding vector; and The similarity between the retrieval content embedding vector and each retrieval embedding vector stored in the database is calculated, and a first retrieval result is generated accordingly, which indicates at least one special domain document among the plurality of special domain documents that is most relevant to the retrieval content.

[0026] Each of the modules described above may independently include a storage device, a processor, and / or a communication chip. Examples of storage devices include hard drives, flash drives, memory, memory cards, etc. Examples of processors include integrated circuits such as microcontrollers, microprocessors, digital signal processors, application-specific integrated circuits (ASICs), logic circuits, or other similar components or combinations thereof. The communication chip may be implemented as LTE, WiMAX, Wi-Fi, Bluetooth, GSM, or PHS, etc.

[0027] The modules mentioned above can be implemented using software technology, hardware technology, or a combination thereof.

[0028] Examples of the specific domain documents include, but are not limited to, patent documents and academic papers or journal articles. The key parts are sections or paragraphs in the specific domain documents that are considered to have important information or show its characteristics for retrieval. In patent documents, this can be defined as the "implementation method" section, and in academic papers or journal articles, it can be defined as the "materials and methods" or "methodology" section.

[0029] [1.] [Module Segmentation / Segmentation Process]

[0030] Documents in specialized fields may contain lengthy text. Preserving all text for embedding would be limited by text length constraints, necessitating text chunking. Appropriate content segmentation offers several advantages: it preserves all text for comparison, and it allows for searching by text chunks, leading to more precise identification of relevant text.

[0031] The strategy used in this invention is based on paragraph or sentence boundaries to preserve the complete semantics as much as possible. Specifically, it segments the text recursively, dividing it into chunks layer by layer according to user-defined delimiters (e.g., newline, period, question mark) until a specified chunk size is reached or other conditions are met.

[0032] According to the present invention, for a given plurality of special domain documents, a first block segmentation process is first performed on the predefined key parts of each plurality of special domain documents to obtain a plurality of text blocks of the key parts. Then, an embedding model is used to convert the plurality of text blocks of each key part into a corresponding plurality of "retrieval embedding vectors", which are used for document retrieval.

[0033] The second segmentation process can be further performed on the full text of each of the multiple special domain documents to obtain multiple text blocks of the full text. Then, the embedding model is used to convert each multiple text block of the full text into a corresponding multiple "paragraph embedding vector", which is used for similar paragraph comparison.

[0034] In addition, a third segmentation process can be executed to obtain multiple text blocks of the search content, and an embedding model can be used to convert the multiple text blocks of each search content into a corresponding multiple "search paragraph embedding vectors".

[0035] When performing chunking, an appropriate chunking size and overlap length between adjacent text chunks must be set. The upper limit of the chunking size is the maximum length allowed by the embedding model (max_length). For example, for "jina-embeddings-v2-base-zh", its max_length is 8,192 tokens. The overlap length between adjacent text chunks can be set to at least 40% of the chunking size, preferably at least 50% or more. In some specific embodiments, the overlap length is set to approximately 50%. The token size here refers to the token size in the embedding model described above. In some specific embodiments, one (Chinese) character roughly corresponds to one token.

[0036] For the first block segmentation process, the block size is preferably 1,000 to 3,000 tokens, more preferably 1,800 to 2,200 tokens, even more preferably 1,900 to 2,100 tokens, and best of all approximately 2,000 tokens.

[0037] For the second and third block processes, the block size can generally be set to 500 to 1,000 tokens, preferably 600 to 800 tokens, and even more preferably about 700 tokens.

[0038] In some preferred embodiments of the present invention, the LangChain RecursiveCharacterTextSplitter technique is used to execute the segmentation process.

[0039] RecursiveCharacterTextSplitter is a text segmentation strategy that effectively handles long text documents, breaking them down into smaller paragraphs. When processing text, this method recursively segments the text, progressively splitting chunks based on user-defined delimiters (e.g., newlines, periods, question marks) until a specified chunk size is reached or other conditions are met. Unlike simple fixed-length splitting, this method considers the structure of the content. When splitting a file, RecursiveCharacterTextSplitter doesn't simply cut by word count but rather tries to preserve the semantic meaning as much as possible by using paragraph or sentence boundaries. In one example, RecursiveCharacterTextSplitter is used to execute the chunking process, with the chunk size set to 2,000 and overlap set to 50%.

[0040] [2.] [Vector Conversion Module / Vector Conversion Process]

[0041] The purpose of vector transformation (embedding) is to transform past text and search content into feature vectors for subsequent similarity calculations during retrieval. It uses an embedding model to convert a piece of text into a feature vector.

[0042] In some preferred embodiments, an embedding model based on a BERT (bidirectional encoder representations from transformers) language model is used, for example, a JinaBERT language model. According to a specific embodiment of the invention, jina-embeddings-v2-base-zh (Gina AI, Germany) is used as the embedding model.

[0043] [3.] [Repository Module / Repository]

[0044] The database module / database is used to store transformed feature vector data, including "retrieval embedding vectors" and / or "paragraph embedding vectors" as described above.

[0045] In one example, an OpenSearch database (Amazon, USA) is used.

[0046] [4.] [Search Module / Search Process]

[0047] The document retrieval system or method of the present invention provides two retrieval methods: "direct retrieval of original input content" and "retrieval after summarizing the input content". The original input content or the summarized retrieval content is first converted into a retrieval content embedding vector using an embedding model.

[0048] Search results are generated by calculating the similarity (e.g., cosine similarity) between the search content embedding vector and each search embedding vector stored in the database module / database. The search results include at least one domain-specific document that is most relevant to the search content among a plurality of domain-specific documents.

[0049] According to certain embodiments of the present invention, a BM25 (best matching 25) algorithm can be used for retrieval. In other words, a "hybrid search" can be performed, combining the original retrieved content with the summarized content. For example, for the retrieved content, both BM25 similarity and embedding similarity can be used for retrieval, and then the results retrieved by the two methods are adaptively weighted to rank the final search results.

[0050] In terms of similarity calculation, the similarity between each chunk of each file and the search content can be calculated. Then, the score with the highest similarity is selected as the final similarity score of the file, and the search results are sorted according to the similarity.

[0051] [5.] [Similar Paragraph Comparison Module / Process]

[0052] Similar paragraph comparison involves calculating the similarity of vectors between the searched content and the retrieved documents. By comparing the feature vectors of the text and the searched content paragraphs, the most similar paragraphs can be identified for subsequent application analysis.

[0053] Specifically, a comparison process is performed to determine the most similar paragraph. This comparison process includes: performing a chunking process to obtain multiple text blocks of the search content, and using an embedding model to convert each of the multiple text blocks of the search content into a corresponding multiple search paragraph embedding vector; calculating the similarity between the multiple search paragraph embedding vectors and the multiple paragraph embedding vectors corresponding to the retrieved file to obtain the paragraph with the highest similarity.

[0054] After retrieving relevant documents, it is desirable to identify similar paragraphs between the target document (the source and basis of the search content) and the retrieved documents in order to provide more accurate search results for subsequent applications.

[0055] According to certain embodiments of the present invention, a third segmentation process is first performed on the target file / retrieval content to obtain a plurality of text blocks (paragraphs). An embedding model is then used to convert the text blocks into corresponding multiple retrieval paragraph embedding vectors. Cosine similarity is then calculated to obtain the content most relevant to the target file (the second retrieval result). The third segmentation process can employ methods including: simple fixed-length segmentation, or the aforementioned RecursiveCharacterTextSplitter technique. Fixed-length segmentation here involves cutting the text according to a preset number of characters (e.g., 500 or 1000 characters). However, to preserve the complete semantics as much as possible, the breakpoint can be taken from the end symbol (e.g., period, question mark) near that number of characters. This method is relatively simple and efficient, suitable for scenarios requiring real-time computation.

[0056] [6.] [Auxiliary Summary Module / Search Content Summary]

[0057] As described above, the document retrieval system or method of the present invention can optionally use the "retrieval after summarizing the input content" method. When the user's input content is too long, a summary can be made of the user's input text, making the retrieval scenario more flexible. Here, a large language model (LLM) is mainly used to condense the user's retrieval content, with the aim of reducing the user's reading burden, improving the accuracy of content retrieval, and ensuring that core topic details (e.g., core technical details) are not obscured by lengthy text.

[0058] According to certain specific embodiments of the present invention, the LLMs used include: Llama 3.1 (Meta AI, USA), Hermes-2 series models (Nous Research, USA), Breeze (MediaTek, Taiwan) or TAIDE (National Science Council, Taiwan).

[0059] Experimental evaluations showed that Breeze and Llama 3.1 have a significant advantage in terms of input context length, enabling them to handle longer text content. According to the Elo Rating ranking after the summarization task evaluation, the performance results are: Llama 3.1 > Hermes-2-Theta > Hermes-2-Pro > Breeze > TAIDE, with Llama 3.1 performing the best. Furthermore, Llama 3.1 also exhibited fewer issues with generating Simplified Chinese and LLM hypothetical scenarios. Therefore, in a preferred embodiment of the present invention, Llama 3.1 is used to perform the summarization task.

[0060] In an example of a patent document in a specific / specific field, Llama 3.1 was used to summarize the search results. The principles for the prompts used were as follows: the generated summary should accurately cover the patent's inventive purpose, technical means, and effects, maintain logical coherence, remain neutral, and avoid generating unrealistic content. Furthermore, in patent searches, in addition to providing the LLM with "implementation methods," "request items" can be added to improve the quality of the LLM-generated summary. Experiments showed that adding "request items" resulted in higher-quality summaries and greater search accuracy.

[0061] In addition, in the example mentioned above, when using Llama 3.1 to generate patent abstracts, it was found that the same sentence was generated multiple times. To solve this problem, several adjustment methods were tested (e.g., adjusting the prompt, model quantization, adjusting LLM parameters, etc.). However, it should be noted that a good abstract generation does not necessarily benefit patent retrieval. Therefore, the actual retrieval performance after the abstract was reconfirmed to ensure that the adjustment would not affect the actual retrieval performance. Using 31 patent citation documents, it was observed whether the abstract generation was unstoppable. First, different prompts and model quantization variables were tested, including two prompts: (1) Guideline Prompt: more detailed task description, (2) Easy Prompt: less detailed task description; and two models: (1) Llama 3.1 without quantization, (2) Llama 3.1 quantized to 4 bits. The results are shown in Table 1 below:

[0062] Table 1: The impact of different prompts and model quantization on repeated generation Unquantified Quantization (4 bits) Guideline Prompt 2 / 31 2 / 31 Easy Prompt 7 / 31 11 / 31

[0063] Building on the previous point, we further adjust the generation parameters of the LLM. The LLM can perform generation tasks using different decoding methods. By setting different parameters, the content of the generated text can be adjusted, balancing the diversity of the generated content. In this example, we primarily test two methods: greedy decoding and multinomial sampling decoding. (1) Greedy Decoding If LLM uses greedy decoding, it means that LLM will choose the word with the highest probability as the next word. In other words, it will directly choose the word with the highest probability from the word distribution predicted by the model. The advantage of using greedy decoding is that the generated words are usually the most "certain", so the generation performance is more stable. However, the disadvantage is that it lacks diversity and is prone to repeated generation. (2) Polynomial sampling decoding The multinomial sampling decoding method involves sampling the next word based on the word probability distribution calculated by the model for each word. This means that every word within the probability distribution has a chance to be selected. As a result, the generated text has a high degree of diversity, which can reduce the occurrence of repeated generation.

[0064] Another parameter of LLM is the "repetition penalty". When this parameter is set, the model will reduce the probability of generating a word that has already been generated when generating the next word.

[0065] Experiments have shown that the combination of polynomial sampling decoding, quantization (4 bits), and duplicate penalty in this example can completely avoid the occurrence of duplicate generation.

[0066] [7.] [Assisted Fine-tuning Module / Embedded Model Fine-tuning]

[0067] The auxiliary fine-tuning module is used to assist in fine-tuning the embedded model. Given a number of domain-specific documents in the primary language used for retrieval (e.g., Traditional Chinese), when the number of such documents is small, the LLM can be used to translate the foreign language data (relative to the primary language used for retrieval) for model fine-tuning, thereby improving the model's retrieval accuracy in specific domains.

[0068] The performance of the above translation can be evaluated according to the following principles: (1) Whether the word choice is appropriate: whether the word choice after translation is faithful to the meaning of the original document (taking the patent document as an example, hierarchical cell structure is correctly translated as “cell hierarchical structure”, while some models will translate cell as cell, which is not appropriate for the classification theme of H04W); and (2) Whether the meaning is fluent: after translation, the inverted sentences in English are easy to cause the meaning to be incoherent and difficult to read, so it is necessary to evaluate whether the sentences translated by the model are fluent.

[0069] In a preferred embodiment of the present invention, the LLM used is Gemma-2 (Google, USA), for example, Gemma-2 9B or Gemma-2 27B.

[0070] If the patent documents are in a specific / specialized field, other possible fine-tuning methods are described below. First, positive samples can be organized using IPC classification. For example, positive samples can be organized using the fifth-order classification of the International Patent Classification (IPC). Samples of the same fifth-order classification can be extracted as positive sample pairs. In practice, this has shown improvement in performance on both the e5-large and bge-m3 LLM models after fine-tuning. Furthermore, changes in the graphical embedding space (results not shown here) reveal that after fine-tuning, patents in the same classification become more clustered and closer together, which is helpful for patent retrieval. Second, since subclasses of IPCs may still be highly technically related to their parent classes even with different IPC classification names, a hierarchical relationship diagram of IPCs can be used to design the extraction of negative samples to avoid extracting highly technically related IPC classifications.

[0071] 100: Document retrieval system for specific domains 110: Segmented Module 120: Vector Transformation Module 130: Database Module 140: Retrieval Module S110~S140: Steps

Claims

1. A document retrieval system for documents in a specific domain, comprising: a block-based module; a vector transformation module; a database module; and a retrieval module, wherein: For a given set of multiple domain-specific documents, the chunking module performs a first chunking process on a predefined key section in each of the multiple domain-specific documents to obtain multiple text blocks of the key section. The vector conversion module uses an embedding model to convert the multiple text blocks of each key section into a corresponding multiple retrieval embedding vectors, which are stored in the database module. The vector conversion module uses the embedding model to convert a search content into a search content embedding vector. The retrieval module calculates the similarity between the search content embedding vector and each retrieval embedding vector stored in the database module, and generates a first retrieval result that indicates at least one domain-specific document in the multiple domain-specific documents that is most similar to the search content. The document retrieval system for domain-specific documents further includes an auxiliary fine-tuning module, which uses a large language model (LLM) to translate the multiple domain-specific foreign language documents into multiple translations in the same language as the multiple domain-specific documents, and uses this translation for fine-tuning the embedding model.

2. The document retrieval system for domain-specific documents as described in claim 1, wherein the segmentation module further performs a second segmentation process on the full text of each of the plurality of domain-specific documents to obtain a plurality of text blocks of the full text, and the vector conversion module uses an embedding model to convert each of the plurality of text blocks of the full text into a corresponding plurality of paragraph embedding vectors, wherein the plurality of paragraph embedding vectors are stored in the database module.

3. The document retrieval system for domain-specific documents as described in claim 2 further includes a similar paragraph comparison module. The similar paragraph comparison module performs a comparison process to generate a second retrieval result, which includes at least one paragraph in the at least one domain-specific document that is most similar to the retrieval content. The comparison process includes: performing a third segmentation process to obtain a plurality of text blocks of the retrieval content, and using an embedding model to convert each of the plurality of text blocks of the retrieval content into a plurality of corresponding retrieval paragraph embedding vectors; calculating the similarity between the plurality of retrieval paragraph embedding vectors and the plurality of paragraph embedding vectors corresponding to the at least one domain-specific document, and generating the second retrieval result accordingly.

4. A document retrieval system for domain-specific documents as described in claim 1, wherein the block size set in the first block process is 1,900 to 2,100 tokens.

5. A document retrieval system for domain-specific documents as described in claim 2, wherein the block size set in the second block process is 600 to 800 tokens.

6. A document retrieval system for domain-specific documents as described in any of claims 1-5, wherein the embedding model is a BERT-based model.

7. A document retrieval method for domain-specific documents, comprising: for a given plurality of domain-specific documents, performing a first block segmentation process on a predefined key section in each of the plurality of domain-specific documents to obtain a plurality of text blocks of the key section; using an embedding model to convert the plurality of text blocks of each key section into a corresponding plurality of retrieval embedding vectors, the plurality of retrieval embedding vectors being stored in a database; using the embedding model to convert a retrieval content into a retrieval content embedding vector; and calculating the similarity between the retrieval content embedding vector and each retrieval embedding vector stored in the database, and generating a first retrieval result indicating at least one domain-specific document in the plurality of domain-specific documents that is most similar to the retrieval content; wherein the document retrieval method for domain-specific documents further comprises: using a large language model (LLM) to translate the plurality of domain-specific foreign language documents into a plurality of translated versions in the same language as the plurality of domain-specific documents, and using the translations for fine-tuning the embedding model.

8. The document retrieval method for special domain documents as described in claim 7 further includes: performing a second block segmentation process on the full text of each of the plurality of special domain documents to obtain a plurality of text blocks of the full text; using the embedding model, converting each plurality of text blocks of the full text into a corresponding plurality of paragraph embedding vectors, the plurality of paragraph embedding vectors being stored in the database.

9. The document retrieval method for a specific domain document as described in claim 8, further comprising performing a comparison process to generate a second retrieval result, which includes at least one paragraph in the at least one specific domain document that is most similar to the retrieval content, the comparison process comprising: performing a third chunking process to obtain a plurality of text blocks of the retrieval content, and using an embedding model to convert each of the plurality of text blocks of the retrieval content into a plurality of corresponding retrieval paragraph embedding vectors; calculating the similarity between the plurality of retrieval paragraph embedding vectors and the plurality of paragraph embedding vectors corresponding to the at least one specific domain document, and generating the second retrieval result accordingly.

10. The document retrieval method for domain-specific documents as described in claim 7, wherein the block size set in the first block process is 1,900 to 2,100 tokens.

11. The document retrieval method for domain-specific documents as described in request item 8, wherein the block size set in the second block process is 600 to 800 tokens.

12. A document retrieval method for domain-specific documents as described in any of requests 7-11, wherein the embedding model is a BERT-based model.

Images