Multi-dimensional document intelligent retrieval method in oil-gas exploration field
By converting tables into text, performing text segmentation and vectorization, and integrating large models with knowledge bases, the problem of inaccurate table processing and retrieval in document retrieval in the oil and gas field is solved, achieving high-quality, multi-dimensional intelligent document retrieval.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA NAT PETROLEUM CORP
- Filing Date
- 2024-11-13
- Publication Date
- 2026-05-15
AI Technical Summary
Traditional document retrieval methods cannot effectively handle table content in the oil and gas field. The retrieval quality depends on the quality of user questions, and there are problems with document segmentation in vector databases, resulting in inaccurate and incomplete retrieval.
By converting tables into text descriptions, performing text segmentation and vectorization, and integrating large models with vectorized knowledge bases, natural language answers are generated using semantic parsing and keyword extraction, enabling multi-dimensional intelligent document retrieval.
It significantly improves the accuracy and comprehensiveness of document retrieval in the oil and gas field, enabling better understanding and processing of professional data, and generating concise yet information-rich query results.
Smart Images
Figure CN122045333A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer technology, specifically relating to a multi-dimensional intelligent document retrieval method in the field of oil and gas exploration. Background Technology
[0002] In the oil and gas industry, a vast amount of document information exists, and retrieving necessary data from business documents is a widespread need in the oil and gas sector. Document retrieval is an important research topic in information retrieval, used to find information that meets user needs from a large number of documents. Due to the characteristics of specialized data in the oil and gas field, such as numerous knowledge carriers, high professional thresholds, strong reliance on experience, and complex reasoning processes, traditional document retrieval methods, such as RAG (Retriever-Augmented Generation), while achieving a certain degree of matching between user queries and document content, have many problems, such as the inability to handle table content in documents, the dependence of retrieval quality on the quality of user queries, and document segmentation problems in vector databases. Summary of the Invention
[0003] The purpose of this invention is to provide a multi-dimensional intelligent document retrieval method in the field of oil and gas exploration, so as to improve the quality and accuracy of retrieval.
[0004] The objective of this invention is achieved through the following technical means: a multi-dimensional intelligent document retrieval method in the field of oil and gas exploration, comprising the following steps:
[0005] S1. Collect relevant business documents in the oil and gas exploration field to form a dataset;
[0006] S2. Convert the tables in the data set documents into text descriptions;
[0007] S3. Perform text segmentation on the document dataset processed in S2 to obtain the segmented document dataset. Segment industry-specific terms and concepts from the document dataset for subsequent retrieval.
[0008] S4. Vectorize the segmented document dataset and establish a vectorized database to store the vectorized document data, forming a knowledge base;
[0009] S5. Integrate the large model with the vectorized knowledge base, and use the large model to perform semantic parsing and keyword extraction on the user's input questions;
[0010] S6. Obtain the keywords extracted by the large model, and based on similarity matching, determine the text from the most relevant document in the knowledge base and return it to the large model;
[0011] S7. Based on the large model's generation capabilities, the selected document content is summarized and reorganized to generate natural language answers, and the final content is returned to the user.
[0012] In step S2, converting the tables of documents in the dataset into text descriptions specifically involves:
[0013] Use XML tags to determine if the table has a header; if the table has a header, extract the header information.
[0014] Construct an N-ary tree using the header information. In the tree, the value of each node represents the text in the header, while the edges represent the hierarchical relationship between the information in the table.
[0015] Perform a preorder traversal on the constructed N-ary tree to generate a description template for the table;
[0016] Using the obtained description template, loop through each row of the table and convert the data in each row into a plain text description;
[0017] The plain text data converted from tables is integrated into the original document data to form a new text dataset.
[0018] In S3, the text segmentation specifically involves:
[0019] Retrieve text containing knowledge related to oil and gas exploration;
[0020] The text is semantically segmented and labeled, and the labeled text is divided into training set and test set;
[0021] The BERT text segmentation pre-trained model is used, and the model is retrained using the training set. The performance of the retrained model is then verified using the test set. Once the performance meets the requirements, it is input into the document dataset processed by S2 to obtain the segmented document dataset.
[0022] In step S4, the specific steps of vectorization are as follows: the segmented document dataset is vectorized using a text vectorization model, the text is converted into mathematical vectors, and the converted mathematical vectors are standardized to ensure consistent dimensions.
[0023] In S5, the large model selected is an open-source generative large model.
[0024] Before integrating the large model with the vectorized knowledge base, S5 also adjusts the large model using the LoRA method.
[0025] While keeping the parameters of the backbone model frozen, inject a trainable low-rank decomposition matrix into each layer of the model;
[0026] Freeze the original pre-training weights of the pre-trained model;
[0027] The model was trained using common question-and-answer pairs related to oil and gas exploration as the training set, and the weights in the low-rank decomposition matrix of the large model were updated using the LoRA method.
[0028] In S5, the Langchain framework is used to integrate the large model with the vectorized knowledge base.
[0029] The specific steps of S6 are as follows:
[0030] Obtain the keyword Q extracted from the user-input question in the large model. user ;
[0031] Iterate through all the vectorized data stored in the knowledge base. For each vectorized data v, calculate the relationship between the vectorized data and the question Q based on the following formula. user cosine similarity(v,Q) user ):
[0032]
[0033] The vectorized data is sorted based on similarity.
[0034] Select the first K vectorized data points and return the corresponding original document text to the large model. K is a set value.
[0035] In step S7, the semantic understanding capability of the large model is used to understand the user's needs. Combined with the document set obtained in step S6, the large model's generation capability is used to return the data needed by the user in different forms such as text and tables.
[0036] The beneficial effects of this invention are as follows: by utilizing a large model and an external knowledge base, the model can better understand and process queries in the private data domain, and present the required query results to the user. Compared with traditional methods, this method demonstrates significant novelty in both its technical solution and expected results. First, in processing table content, a method is proposed to convert table content into searchable text, enhancing the completeness of document retrieval. Second, by introducing multi-query retrieval and compressed retrieval techniques, the quality and accuracy of the retrieval are significantly improved. Attached Figure Description
[0037] Figure 1 A flowchart of a multi-dimensional intelligent document retrieval method in the field of oil and gas exploration;
[0038] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Detailed Implementation
[0039]
Example 1
[0040] like Figure 1 As shown, a multi-dimensional intelligent document retrieval method in the field of oil and gas exploration includes the following steps:
[0041] S1. Collect relevant business documents in the oil and gas exploration field to form a dataset;
[0042] Based on business needs, we collect oil and gas exploration-related business documents to form a dataset, which will be used to save the document data to a knowledge base later.
[0043] S2. Convert the tables in the data set documents into text descriptions;
[0044] In step S2, converting the tables of documents in the dataset into text descriptions specifically involves:
[0045] Use XML tags to determine if the table has a header; if the table has a header, extract the header information.
[0046] Construct an N-ary tree using the header information. In the tree, the value of each node represents the text in the header, while the edges represent the hierarchical relationship between the information in the table.
[0047] Perform a preorder traversal on the constructed N-ary tree to generate a description template for the table;
[0048] Using the obtained description template, loop through each row of the table and convert the data in each row into a plain text description;
[0049] The plain text data converted from tables is integrated into the original document data to form a new text dataset.
[0050] The process of converting table content into text descriptions not only improves information accessibility but also allows table data that would otherwise be difficult for text retrieval algorithms to be included in the search scope. This step significantly enhances the comprehensiveness and usability of the document retrieval system.
[0051] S3. Perform text segmentation on the document dataset processed in S2 to obtain the segmented document dataset. Segment industry-specific terms and concepts from the document dataset for subsequent retrieval.
[0052] In S3, the text segmentation specifically involves:
[0053] Retrieve text containing knowledge related to oil and gas exploration;
[0054] The text is semantically segmented and labeled, and the labeled text is divided into training set and test set;
[0055] The BERT text segmentation pre-trained model is used, and the model is retrained using the training set. The performance of the retrained model is then verified using the test set. Once the performance meets the requirements, it is input into the document dataset processed by S2 to obtain the segmented document dataset.
[0056] Improving retrieval accuracy is key by constructing a text semantic segmentation model specifically for oil and gas exploration data. This model can more accurately understand and segment industry-specific terms and concepts, thereby providing more relevant and accurate results in subsequent retrieval processes.
[0057] The text segmentation pre-training model can be BERT (Bidirectional Encoder Representations from Transformers) or other Chinese text segmentation pre-training models.
[0058] S4. Vectorize the segmented document dataset and establish a vectorized database to store the vectorized document data, forming a knowledge base;
[0059] In step S4, the specific steps of vectorization are as follows: the segmented document dataset is vectorized using a text vectorization model, the text is converted into mathematical vectors, and the converted mathematical vectors are standardized to ensure consistent dimensions.
[0060] For text vectorization models, you can choose a text vectorization model such as the BGE model.
[0061] Vectorization of document data not only transforms text information into a machine-processable format, but also allows algorithms to efficiently compare the similarity between texts in a high-dimensional space. This step is a key technical step in achieving fast and accurate retrieval.
[0062] The obtained vectorized document data is stored in a suitable vectorization database, such as Chroma, depending on the amount of data.
[0063] S5. Based on the Langchain framework, the large model is integrated with the vectorized knowledge base, and the large model is used to perform semantic parsing and keyword extraction on the user input questions;
[0064] In S5, the large model selected is an open-source generative large model, such as ChatGLM, LLaMa, etc.
[0065] This technique is based on open-source generative large models, considering their capabilities in text processing and data mining. The performance of these models in processing natural language understanding, particularly data from oil and gas exploration, is analyzed.
[0066] Before integrating the large model with the vectorized knowledge base, the large model was also adjusted using the LoRA method.
[0067] While keeping the parameters of the backbone model frozen, inject a trainable low-rank decomposition matrix into each layer of the model;
[0068] Freeze the original pre-training weights of the pre-trained model;
[0069] The model was trained using common question-and-answer pairs related to oil and gas exploration as the training set. The LoRA method was employed to update only the weights in the low-rank decomposition matrix of the large model, without altering the original pre-trained parameters. This adjustment method allows for fine-tuning with a small number of new parameters, enabling the model to better understand common terminology and scenarios in the oil and gas exploration field.
[0070] Once the training stopping condition is met, model fine-tuning is complete. In the LoRA method, maintaining the backbone model parameters unchanged during fine-tuning of a large model is crucial. This preserves the model's generalization ability across a wide range of data while allowing the fine-tuning to focus on the specific needs of the oil and gas sector. This approach effectively balances the model's generality and customization requirements.
[0071] S6. Obtain the keywords extracted by the large model, and based on similarity matching, determine the text from the most relevant document in the knowledge base and return it to the large model;
[0072] The specific steps of S6 are as follows:
[0073] Obtain the keyword Q extracted from the user-input question in the large model. user ;
[0074] Iterate through all the vectorized data stored in the knowledge base. For each vectorized data v, calculate the relationship between the vectorized data and the question Q based on the following formula. user cosine similarity(v,Q) user ):
[0075]
[0076] The vectorized data is sorted based on similarity.
[0077] Select the first K vectorized data points and return the corresponding original document text to the large model. K is a set value.
[0078] Using similarity matching to retrieve documents not only improves the flexibility and accuracy of retrieval, but also allows the system to understand and interpret user queries from multiple perspectives, further enhancing the user experience and efficiency of the retrieval system.
[0079] S7. Based on the generative capabilities of large models, the selected document content is summarized and reorganized to generate natural language answers, which are then returned to the user. This ensures that the user receives a concise yet informative response.
[0080] In step S7, the semantic understanding capability of the large model is used to understand the user's needs. Combined with the document set obtained in step S6, the large model's generation capability is used to return the data needed by the user in different forms such as text and tables.
[0081] This method can also be widely applied to online knowledge bases, enterprise information retrieval, academic research and other fields. It only requires changing the collected document data according to different fields.
Claims
1. A multi-dimensional intelligent document retrieval method in the field of oil and gas exploration, characterized in that: Includes the following steps, S1. Collect relevant business documents in the oil and gas exploration field to form a dataset; S2. Convert the tables in the data set documents into text descriptions; S3. Perform text segmentation on the document dataset processed in S2 to obtain the segmented document dataset. Segment industry-specific terms and concepts from the document dataset for subsequent retrieval. S4. Vectorize the segmented document dataset and establish a vectorized database to store the vectorized document data, forming a knowledge base; S5. Integrate the large model with the vectorized knowledge base, and use the large model to perform semantic parsing and keyword extraction on the user's input questions; S6. Obtain the keywords extracted by the large model, and based on similarity matching, determine the text from the most relevant document in the knowledge base and return it to the large model; S7. Based on the large model's generation capabilities, the selected document content is summarized and reorganized to generate natural language answers, and the final content is returned to the user.
2. The multi-dimensional intelligent document retrieval method in the field of oil and gas exploration according to claim 1, characterized in that: In step S2, converting the tables of documents in the dataset into text descriptions specifically involves: Use XML tags to determine if the table has a header; if the table has a header, extract the header information. Construct an N-ary tree using the header information. In the tree, the value of each node represents the text in the header, while the edges represent the hierarchical relationship between the information in the table. Perform a preorder traversal on the constructed N-ary tree to generate a description template for the table; Using the obtained description template, loop through each row of the table and convert the data in each row into a plain text description; The plain text data converted from tables is integrated into the original document data to form a new text dataset.
3. The multi-dimensional intelligent document retrieval method in the field of oil and gas exploration according to claim 1, characterized in that: In S3, the text segmentation specifically involves: Retrieve text containing knowledge related to oil and gas exploration; The text is semantically segmented and labeled, and the labeled text is divided into training set and test set; The BERT text segmentation pre-trained model is used, and the model is retrained using the training set. The performance of the retrained model is then verified using the test set. Once the performance meets the requirements, it is input into the document dataset processed by S2 to obtain the segmented document dataset.
4. The multi-dimensional intelligent document retrieval method in the field of oil and gas exploration according to claim 1, characterized in that: In step S4, the specific steps of vectorization are as follows: the segmented document dataset is vectorized using a text vectorization model, the text is converted into mathematical vectors, and the converted mathematical vectors are standardized to ensure consistent dimensions.
5. The multi-dimensional intelligent document retrieval method in the field of oil and gas exploration according to claim 1, characterized in that: In S5, the large model selected is an open-source generative large model.
6. The multi-dimensional intelligent document retrieval method in the field of oil and gas exploration according to claim 1, characterized in that: Before integrating the large model with the vectorized knowledge base, S5 also adjusts the large model using the LoRA method. While keeping the parameters of the backbone model frozen, inject a trainable low-rank decomposition matrix into each layer of the model; Freeze the original pre-training weights of the pre-trained model; The model was trained using common question-and-answer pairs related to oil and gas exploration as the training set, and the weights in the low-rank decomposition matrix of the large model were updated using the LoRA method.
7. The multi-dimensional intelligent document retrieval method in the field of oil and gas exploration according to claim 1, characterized in that: In S5, the Langchain framework is used to integrate the large model with the vectorized knowledge base.
8. The multi-dimensional intelligent document retrieval method in the field of oil and gas exploration according to claim 1, characterized in that: The specific steps of S6 are as follows: Obtaining a large model through a keyword Q extracted by a user input question user ; Iterate through all the vectorized data stored in the knowledge base. For each vectorized data v, calculate the relationship between the vectorized data and the question Q based on the following formula. user cosine similarity (b, Q) user ): The vectorized data is sorted based on similarity. Select the first K vectorized data points and return the corresponding original document text to the large model. K is a set value.
9. The multi-dimensional intelligent document retrieval method in the field of oil and gas exploration according to claim 1, characterized in that: In step S7, the semantic understanding capability of the large model is used to understand the user's needs. Combined with the document set obtained in step S6, the large model's generation capability is used to return the data needed by the user in different forms such as text and tables.