Railway whole-process consultation document digitization and retrieval method based on large model

Abstract

The application discloses a railway whole-process consultation document digitization and retrieval method based on a large model, which comprises the following steps: S1, classifying and grading railway whole-process consultation documents; S2, constructing a virtual environment required by large model deployment; S3, constructing a digital retrieval database; S4, constructing a digital extraction standard for the classified and graded railway whole-process consultation documents; S5, document retrieval; determining which category of documents the retained paragraphs belong to according to the retrieval result of S5, and extracting corresponding content according to the digital extraction standard constructed in S4; S7, fusing the content extracted in S6 with the document paragraphs retrieved in S5, and realizing accurate answering to the user's retrieval demand. The method not only refines the original document content, but also retains the key content, avoids missing of key information in the summarization process of the large model and content fragmentation in the traditional segmentation mode, realizes real-time analysis and overall control of the engineering progress, and improves the accuracy of the whole-process consultation.

Description

Technical Field

[0001] This invention relates to the field of railway engineering management, and in particular to a method for digitizing and retrieving railway full-process consulting documents based on a large model. Background Technology

[0002] Railway full-process engineering consulting runs through all stages of railway construction projects, including project initiation and decision-making, survey and design, engineering implementation, completion acceptance and operation and maintenance. The fragmentation of engineering consulting services can be solved by integrating and innovating consulting work models at each stage.

[0003] Railway full-process engineering consulting provides clients with comprehensive engineering consulting services, encompassing overall project consulting management and specific specialized consulting services. The overall project consulting management, guided by the overall goals of full-process engineering consulting, focuses on schedule management, quality management, and investment management, organically integrating specialized consulting content throughout the entire railway construction project. This is the core and key to realizing the value of full-process engineering consulting.

[0004] Railway full-process engineering consulting, as a comprehensive and integrated intellectual service, involves numerous document types and a large volume of data. The data at each stage is diverse, and data standards are inconsistent. For example, structured data from the surveying and design stage differs significantly in format from unstructured reports from construction / operation and maintenance. Unstructured data, such as technical reports, meeting minutes, and drawing annotations, constitutes a large proportion and has low semantic relevance. Furthermore, there is a lack of effective data analysis methods for both structured design drawings and unstructured text in cross-disciplinary and cross-stage collaborative consulting, making reading and connecting these data difficult.

[0005] Document digitization refers to the digital processing of documents using information technology to improve the efficiency of document storage, retrieval, and use. It can significantly enhance the speed and accuracy of information retrieval, and ensure document security through access control. The core value of document digitization lies in using technological means to process fragmented and inefficient traditional documents, upgrading them into a new model of knowledge interconnection and intelligent driving.

[0006] In response to the challenges of numerous document types and large volumes in railway end-to-end consulting, there is an urgent need for a document digitization method to achieve unified management of multi-source heterogeneous data from various stages, including surveying and design, construction / operation and maintenance, eliminate data barriers between stages, realize cross-stage information and knowledge association, and improve the processing efficiency of railway end-to-end engineering consulting documents. Summary of the Invention

[0007] The purpose of this invention is to provide a method for digitizing railway full-process consultation documents based on a large model, in order to solve the problems of low correlation between multi-source information, inaccurate extraction of core information, and bias in semantic understanding in existing technologies.

[0008] Therefore, the present invention adopts the following technical solution:

[0009] A method for digitizing and retrieving railway full-process consultation documents based on a large model includes the following steps:

[0010] S1, Classification and Grading of Railway Full-Process Consulting Documents:

[0011] The railway whole-process consulting documents are classified into folders according to project and object type. The folder directory is divided into two levels: project level and category level. The category level folders are classified according to different stages of the railway whole-process consulting project.

[0012] S2, the virtual environment required for deploying large models;

[0013] S3, Construct a digital retrieval database: Read the contents of all railway full-process consultation documents in each project-level folder in sequence, and perform semantic segmentation on the read contents; after the segmented text content is encoded into text structure storage, it is stored in a structured database, and after the segmented text content is encoded into vector storage, it is stored in a vector database;

[0014] S4 establishes a digital extraction standard for railway full-process consultation documents after S1 classification and grading.

[0015] S5, Document retrieval: Extract keywords from the user-input search content, obtain the top 1 to 5 vectors with the highest similarity to the search content from the vector database through the search enhancement generation method, and match the vectors to the corresponding paragraphs in the vector database; perform noun retrieval on the structured database, compare the retrieval results of the structured database with the matching results of the vector database, and retain the consistent results;

[0016] S6. Based on the search results retained in S5, determine which document category the retained paragraph belongs to, and then extract the corresponding content according to the digital extraction standard constructed in S4.

[0017] S7 integrates the content extracted by S6 with the document paragraphs retained by S5 to provide an accurate answer to the user's search needs.

[0018] In step S2 above, the langchain and transformer framework modules required for deploying the large language model are configured in the virtual environment, and the Baichuan large language model is deployed locally.

[0019] In step S1 above, the railway whole-process consultation documents include Word, PDF and Excel formats, and the document types include engineering reporting documents, standard specifications, official documents and whole-process consultation weekly and monthly reports.

[0020] Preferably, in step S3 above, the content of the railway full-process consultation document is read by Python programming.

[0021] The segmentation method in step S3 above is as follows: using a large language model, the semantic similarity between two adjacent natural paragraphs is judged by the prompt method. If they are similar, the number 1 is output; if they are not similar, the number 0 is output. Based on the output number, the semantically similar paragraphs are merged into one paragraph.

[0022] In step S3 above, the method of text structured storage encoding is as follows: the text content after segmentation remains unchanged, and the segmented text is tagged by constructing a label field "content"; the method of vector storage encoding is as follows: the text content after segmentation is encoded by a natural language processing model.

[0023] In step S4, the digitization extraction criteria for engineering reporting documents are: nouns, key parameters, and corresponding data; for standard and specification documents, the digitization extraction criteria are: proper nouns, corresponding expressions, and relevant specifications; for official documents, the digitization extraction criteria are: keywords and content summary; and for weekly and monthly consultation reports, the digitization extraction criteria are: weekly / monthly report number, context with words like "key points," "important," or "difficult points," and numbers other than the date.

[0024] Preferably, in step S5, the structured database is searched using the Elasticsearch module.

[0025] This invention establishes a method for digital storage and retrieval of railway construction engineering documents. It adopts a natural paragraph adaptive segmentation method based on a large language model to construct a parallel dual storage method of text storage and vector storage, defines the digital extraction standard for classified document data, and realizes key information extraction and information fusion based on a large language model.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. This invention achieves database querying through parallel character matching and semantic querying, and limits the key content within paragraphs according to digital extraction standards. This not only refines the content of the original document, but also retains the key content of the original document, preventing the large model from omitting key information in the original document during the summarization process.

[0028] 2. This invention adopts a semantic segmentation approach, which uses a large language model to determine the semantic similarity between natural paragraphs, and divides content with consistent semantics into the same small paragraph, thus achieving semantic segmentation; by using a large language model to determine the semantic similarity between two adjacent natural paragraphs, paragraphs with similar semantics are divided into the same paragraph, avoiding the content fragmentation of traditional segmentation methods;

[0029] 3. This invention uses a large language model to digitally retrieve and extract documents for the entire process of railway construction projects, solving the problems of inaccurate key information and incomplete summarization caused by traditional retrieval and extraction methods. It effectively improves the processing efficiency of full-process documents and enhances the accuracy and comprehensiveness of key information acquisition.

[0030] 4. This invention establishes a digital extraction standard for classified railway full-process consultation documents. It classifies documents of different formats according to consultation reports at each stage, such as engineering filing documents, standard specifications, official documents, weekly and monthly full-process consultation reports, and proposes a digital extraction standard for railway full-process consultation documents based on the core content of different categories of documents, and specifies the key content of different categories of documents.

[0031] 5. The method of this invention enables the precise extraction of key and difficult issues in the engineering construction from design documents and provides accurate consultation on these issues, thereby facilitating high-quality project construction. Attached Figure Description

[0032] Figure 1 This is a flowchart of the method for digitizing and retrieving railway full-process consultation documents based on a large model in an embodiment of the present invention;

[0033] Figures 2-4 The results were obtained when using the Deepseek website's large model to summarize important work in the embodiments of the invention. Detailed Implementation

[0034] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings and embodiments. Obviously, the following embodiments are only some embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0035] Example

[0036] See Figure 1 The present invention provides a method for digitizing and retrieving railway full-process consultation documents based on a large model, comprising the following steps:

[0037] S1, Classification and Grading of Railway Full-Process Consulting Documents:

[0038] Railway full-process consultation documents are available in formats such as Word, PDF, and Excel. Document types include engineering reporting documents, standards and specifications, official documents, and weekly and monthly consultation reports. This invention categorizes railway full-process consultation documents according to project and object type, using folders. The folder structure is divided into two levels: project-level and category-level. Project-level folders categorize different railway full-process consultation projects, while category-level folders categorize according to different stages of each project. Files within each category-level folder have different formats and types.

[0039] S2, the virtual environment required for deploying large models:

[0040] In this embodiment, the virtual environment is built using conda. The construction method is as follows: conda create -ndigitization python=3.10 (the python version can be upgraded according to the development of subsequent versions); activate the virtual environment: conda activate digitization.

[0041] In addition, configure the langchain and transformer framework modules required for the deployment of the large language model in the virtual environment, and deploy the Baichuan (2-13B) large language model locally.

[0042] S3, Constructing a digital retrieval database includes the following steps:

[0043] S31 uses Python programming to sequentially read the content of all formats and types of railway full-process consultation documents in each project-level folder, segments the read text content, and splits multiple long documents into multiple text segments.

[0044] Existing segmentation methods segment by natural paragraphs, while this invention uses semantic segmentation. By using a large language model to determine the semantic similarity between adjacent natural paragraphs, content with consistent semantic meaning is merged into one paragraph, ultimately achieving semantic segmentation.

[0045] When using a large language model to determine the semantic similarity between two adjacent paragraphs, a prompt method is used for judgment. The prompt content is: "Please judge the semantic similarity between the two paragraphs. If they are similar, please output the number 1. If they are not similar, please output the number 0." Based on the output number, paragraphs with similar semantics are merged into one paragraph.

[0046] S32 constructs a database for the segmented text content. Details are as follows:

[0047] The text content after S31 segmentation is encoded. Two encoding methods are used: text structured storage (electsearch) encoding and vector storage encoding. These two methods correspond to different retrieval methods. Specifically:

[0048] The text structured storage encoding method is as follows: the text content remains unchanged after segmentation, and the segmented text is tagged by constructing a label field "content" before being stored in a structured database.

[0049] The vector storage encoding method is as follows: the segmented text content is encoded (embedding) using a natural language processing model. In this embodiment, based on the proposed document semantic retrieval requirements, a semantic encoding model is used to encode the segmented results, obtain a fixed-length (e.g., 1024) text vector, automatically assign an ID, and store it in the vector database.

[0050] S4 establishes digital extraction standards for documents classified and graded in S1. Among them:

[0051] For engineering reporting documents, the digital extraction criteria are: nouns, key parameters, and corresponding data;

[0052] For standard and specification categories, the digital extraction standard is: proper nouns, corresponding expressions, and relevant specifications;

[0053] For official documents, the digital extraction criteria are: keywords and content summary;

[0054] For weekly and monthly reports on the entire consultation process, the digital extraction criteria are: the weekly / monthly report number, the context containing words such as "key points," "important," and "key difficulties," and numbers other than the date.

[0055] S5, Document Retrieval. Details are as follows:

[0056] Users describe the content they want to retrieve using natural language. This invention places no restrictions on how the user input is expressed. In this embodiment, the user input is "What are the important tasks for XX Railway in Month X?".

[0057] Keywords were extracted from the user's input, resulting in three keywords: XX Railway, X Month, Important Work.

[0058] By using a retrieval enhancement generation method, the top 3 vectors with the highest similarity to the retrieved content are obtained, and the vectors are matched to the corresponding paragraphs in the vector database.

[0059] To ensure the reliability of the search results, the Elasticsearch (character query algorithm) module is used to perform a noun search on the structured database. The nouns "XX Railway" and "Month X" from the three keywords are used to locate the working documents of XX Railway in Month X.

[0060] Then, the search results from the structured database are compared with the search results from the vector database matched by the search enhancement generation method, and the consistent results are retained.

[0061] S6, standardize the extraction of the results retained in S5: determine the document category (engineering report, standard specification, official document, or weekly / monthly consultation report) of the retained paragraphs based on the search results retained in S5, and then extract the corresponding content according to the digital extraction standard constructed in S4.

[0062] In this embodiment, based on the results obtained in S5, the retrieved document paragraphs are determined to belong to the weekly and monthly reports of the whole-process consultation. The extraction method uses a large language model. Since the extraction criteria for weekly and monthly report documents are: the weekly / monthly report number, context containing words like "key," "important," and "critical points," and numbers other than the date, the obtained content consists of the following three paragraphs:

[0063] "Key task progress. The Railway X Institute completed the preparation of the revised design document for XXX Station (due to adjustments in the overall layout, changes to the station's front and XXXX areas) by XX month XX day. Design and cost consulting opinions have been solicited during the process, and the document is ready for review."

[0064] "Key and difficult engineering projects. XXX Tunnel: Shaft #1 excavated 195 meters this month, totaling 722 meters (53%); Shaft #2 excavated 189.5 meters this month, totaling 919.1 meters (69%); the entrance excavation this month was 26.4 meters, totaling 70.2 meters (4%). This is approximately two months behind the guiding construction schedule, mainly due to construction organization and excavation progress indicators falling short of expectations. XX Tunnel: The entrance excavation this month was 86 meters, totaling 529.5 meters (25%); the exit section at the smaller mileage excavation this month was 71 meters, totaling 172 meters (5%); the larger mileage excavation this month was 4.8 meters, totaling 130.4 meters (58%). - 2" - The overall construction schedule is about 1.5 months behind the guidance plan, mainly due to restrictions on nighttime construction and the late arrival of mechanized equipment. XXXX Bridge: The main pier pile foundation construction for swing bridge piers 219#~220# has been completed; the turntable under pier 220# has been completed; the foundation abutment under pier 220# is 3.5~4.5m high. Concrete pouring completed; no delays in progress. See the table below for the overall project progress. "II. Key Coordination and Concern Matters This Month (I) Important Meetings. On [Date], Deputy General Manager XXXXX visited the XX Railway for research and guidance, inspecting the XXX beam fabrication yard, the entrance to the XX tunnel, and the entrance to the XXX tunnel. After listening to the reports, he affirmed the project's phased achievements and put forward four requirements: First, fully leverage the advantages of all participating parties, focus on key and difficult control projects, and work together to overcome difficulties and bottlenecks to create conditions for project construction; second, adhere to the goals and fully promote the first and second phases of the XX Railway according to plan to achieve the construction schedule targets; third, make overall plans and organize operational intervention and external environment remediation in advance; fourth, strictly adhere to the bottom line and create a safe and high-quality project for the next century."

[0065] "(III) Key and Difficult Points XXXX. 1. XX County XXXXX Mine. On XX Month XX Day, the XX County Railway XX Department organized a meeting to coordinate the handling of XXXXX Mine. The XX City XX Office, XXXX Company, XXXX Command Center, and XX Project Department participated. After discussion among all parties, the meeting reached the following consensus: First, there are still risks in the slope of the mine, and the risks and hidden dangers should be eliminated in time; Second, XX County XX should identify the mining rights of the mine and clarify the ownership of the mine in writing to provide a basis for subsequent XXXXXX; Third, after the safety hazards of the mine are eliminated and the ownership is identified and confirmed, the "one case, one discussion" coordination and handling should be adopted in accordance with the requirements of document XXXX No. [XXXX] XXX. 2. XX County XX Water Plant XX. On XX Month XX Day, XX On [Date], XXXX Company organized a coordination and promotion meeting for the reservoir project of XX Company in XX County. The meeting agreed to the overall XX project for the reservoir of XX Company and the XX project on the south side. It was agreed that the XXXXXX Department would be responsible for the design, implementation, and policy handling of the XX project. The plan is to obtain XX approval in X month and complete the reservoir project in XX month. The XXX Bureau should coordinate the railway construction schedule. The full consultation, design, and cost review will be involved throughout the process to review and control the project and costs, and control investment. 3. XX Village Road Project in XX County. On [Date], XXXX Company organized a site survey to study the XX Village Road Project in XX County. Regarding the issues of one section of XXXX Road not being able to be connected and two sections of XX wall located within the XX boundary, factors such as the safe distance between the road and railway, the setting of railway fences, and road drainage should be taken into account to further optimize the longitudinal and transverse sections of the road. Opinions will be solicited from local, construction, and operation units to clarify the XX project. "Investment completed this month: X.XX". The annual investment target was XX.XXX billion yuan, representing 36.4% of the total annual target of XX billion yuan. It is recommended to further strengthen coordination of external resources and urge relevant local departments to submit the required documentation to ensure the smooth completion of the annual investment.

[0066] S7 integrates the content extracted by S6 with the document paragraphs retained by S5 to provide an accurate answer to the user's search needs.

[0067] In this embodiment, a large model approach is used, with fusion performed via prompt. The prompt in this embodiment is defined as "extracting and integrating the key themes of the <paragraph> content, retaining the weekly / monthly report number, contextual elements marked with 'key points,' 'important,' or 'difficult points,' and numbers other than dates; all of the above content is strictly preserved according to the original text," and the semantic understanding of the large model is used to understand and integrate the acquired content.

[0068] The fusion result of this embodiment, i.e., the query result, is as follows:

[0069] "Progress of key tasks. The Railway X Institute has completed the preparation of the revised design documents for XXX Station (due to changes in the station's front-end and electrical engineering caused by adjustments to the overall layout). Design and cost consulting opinions have been solicited during the process, and the documents are ready for review."

[0070] "Regarding the progress of key control projects: For the XXX Tunnel, the No. 1 inclined shaft has been excavated 195 meters this month, with a cumulative total of 722 meters (53%); the No. 2 inclined shaft has been excavated 189.5 meters this month, with a cumulative total of 919.1 meters (69%); the entrance has been excavated 26.4 meters this month, with a cumulative total of 70.2 meters (4%). This is about two months behind the guiding construction plan, mainly due to the construction organization and excavation progress indicators not meeting expectations. For the XX Tunnel, the entrance has been excavated 86 meters this month, with a cumulative total of 529.5 meters (25%); the exit work area has been excavated 71 meters this month at the small mileage level, with a cumulative total of 172 meters (5%); and the large mileage level has been excavated 4.8 meters this month, with a cumulative total of 130.4 meters (58%). Overall, this is about 1.5 months behind schedule, mainly due to restrictions on nighttime construction and late equipment arrival. For the XXX Bridge, the main pier pile foundations of the 219#~220# swing bridge have all been completed, the turntable under the 220# pier has been completed, and the 3.5~4.5m concrete pouring of the 220# pier's foundation has been completed, with no delays in progress."

[0071] "The Deputy General Manager of XXXXX visited the XXXX project site (including the XXX beam fabrication yard, the entrance to the XX tunnel, and the entrance to the XXX tunnel, etc.) for research. After affirming the project's phased achievements, he put forward four requirements: First, all parties should work together to overcome difficulties and bottlenecks in key control projects; second, the goals should be firmly adhered to and all efforts should be made to ensure that the first and second phases of the project are completed on schedule; third, operational intervention and external environment remediation should be planned in advance; and fourth, the bottom line of safety and quality should be strictly adhered to."

[0072] "Key and difficult points XX involve three work sites: 1. XX County XXXXX Mine, 2. XX County XX Water Plant XX, and 3. XX County XX Village Road XX." "This month's completed investment is X.XX billion yuan, with a cumulative annual investment of XX.XXX billion yuan, accounting for 36.4% of the annual target of XX billion yuan; it is recommended to further strengthen the coordination of external resource guarantees, urge relevant local units to submit XXXX compilation materials, etc., to ensure the smooth completion of the annual investment."

[0073] For comparison, we used the DeepSeek website's large model to summarize key work, asking the same questions as in this embodiment, and obtained the following results: Figures 2-4 As shown (sensitive information in the image has been obscured).

[0074] By comparing the query results of this invention with those of Deepseek, it can be seen that, based on the key points of the full-process consultation document, the method of this invention completely preserves the project progress data, extracts the investment data completely, and accurately extracts the quality-related requirements and statements. Furthermore, the method of this invention filters out non-key points summarized by Deepseek, such as "cleaning up the slag pile in the initial section; strictly monitoring the progress of lagging work points." Therefore, the answer from this invention not only extracts the original document content but also retains the key points, avoiding the problem of large models omitting crucial information during the summarization process.

[0075] The accurate consultation results obtained from this embodiment show that the method of the present invention can accurately extract investment data from multiple dimensions such as the entire line, districts / counties, and sections for investment control in the whole-process consultation, thereby achieving overall control of investment consultation and optimization of investment control. For progress control in the whole-process consultation, by extracting and analyzing the progress information of key projects, real-time analysis and overall control of project progress can be achieved, thereby comprehensively improving the accuracy of whole-process consultation for railway construction projects and effectively promoting the high-quality construction of railway construction projects.

Claims

1. A method for digitizing and retrieving railway full-process consultation documents based on a large model, characterized in that, Includes the following steps: S1, Classification and Grading of Railway Full-Process Consulting Documents: The railway whole-process consulting documents are classified into folders according to project and object type. The folder directory is divided into two levels: project level and category level. The category level folders are classified according to different stages of the railway whole-process consulting project. S2, the virtual environment required for deploying large models; S3, Construct a digital retrieval database: Read the content of all railway full-process consultation documents within each project-level folder sequentially, and perform semantic segmentation on the read content; after semantic segmentation, perform text structured storage encoding on the semantically segmented text content and store it in a structured database; after segmentation, perform vector storage encoding on the segmented text content and store it in a vector database; where: The method for semantic segmentation is as follows: using a large language model, the semantic similarity between two adjacent natural paragraphs is judged by the prompt method. If they are similar, the number 1 is output; if they are not similar, the number 0 is output. Based on the output number, paragraphs with similar semantics are merged into one paragraph. The method for structured text storage encoding is as follows: the content of the segmented text remains unchanged, and the segmented text is tagged by constructing a tag field "content"; The method for vector storage encoding is as follows: the segmented text content is encoded using a natural language processing model; S4 establishes a digital extraction standard for railway full-process consultation documents classified and graded by S1: For engineering reporting documents, the digitization extraction criteria are: nouns, key parameters, and corresponding data; For standard and specification documents, the standard for digitization extraction is: proper nouns, corresponding expressions, and relevant specifications; For official documents, the criteria for digital extraction are: keywords and content summary; For weekly and monthly reports on the entire consulting process, the digitization extraction criteria are: the weekly / monthly report number, the context containing words such as "key points," "important," and "key difficulties," and numbers other than the date; S5, Document retrieval: Extract keywords from the user-input search content, obtain the top 1 to 5 vectors with the highest similarity to the search content from the vector database through the search enhancement generation method, and match the vectors to the corresponding paragraphs in the vector database; perform noun retrieval on the structured database, compare the retrieval results of the structured database with the matching results of the vector database, and retain the consistent results; S6. Based on the search results retained in S5, determine which document category the retained paragraph belongs to, and then extract the corresponding content according to the digital extraction standard constructed in S4. S7 integrates the content extracted by S6 with the document paragraphs retained by S5 to provide an accurate answer to the user's search needs.

2. The method for digitizing and retrieving railway full-process consultation documents based on a large model according to claim 1, characterized in that: In S2, the langchain and transformer framework modules required for the deployment of the large language model are configured in the virtual environment, and the Baichuan large language model is deployed locally.

3. The method for digitizing and retrieving railway full-process consultation documents based on a large model according to claim 1, characterized in that: The railway whole-process consultation documents mentioned in S1 include Word, PDF and Excel formats, and the document types include engineering reporting documents, standard specifications, official documents and whole-process consultation weekly and monthly reports.

4. The method for digitizing and retrieving railway full-process consultation documents based on a large model according to claim 3, characterized in that: In S3, the content of the railway full-process consultation document is read using Python programming.

5. The method for digitizing and retrieving railway full-process consultation documents based on a large model according to claim 1, characterized in that, In S5, the Elasticsearch module is used to perform noun retrieval on the structured database.

Images