Large model-based multi-type text hierarchical directory construction method, device and equipment

Abstract

The application relates to the field of electronic technology and discloses a multi-type text hierarchical directory construction method, device and equipment based on a large model. The method comprises the following steps: acquiring a plurality of first text data, performing format conversion on each document data to obtain second text data corresponding to a target layout format; recognizing position information of a plurality of initial title marks in each second text data, determining corresponding title line information based on the position information of the initial title marks; calling a locally deployed pre-trained large model to perform semantic analysis on each title line information of each second text data, and obtaining hierarchical information corresponding to different title lines in the corresponding second text data; and updating the initial title marks of the plurality of title line information in the corresponding target text data based on the hierarchical information corresponding to the different title lines in each second text data, so as to obtain hierarchical directory information of the corresponding second text data, and provide reliable structured data support for text slicing or RAG question answering and other scenes.

Description

Technical Field

[0001] This invention relates to the field of electronic technology, and more specifically to a method, apparatus, and device for constructing multi-type text hierarchical directories based on a large model. Background Technology

[0002] Identifying the heading levels of text and generating a hierarchical directory clearly presents the text's logical structure, quickly distinguishes primary and secondary content, and avoids disorganized long blocks of text. This allows for more efficient location of key information, understanding of the text's context, and improved reading and information retrieval efficiency. Furthermore, constructing a hierarchical directory can also meet the needs of Retrieval-Augmented Generation (RAG) question-answering systems and other related applications for high-quality structured text.

[0003] In related technologies, a series of rules are set to match text titles. However, due to the extremely complex hierarchical directory numbering types of different document types, there are not only common standard numbers, but also user-defined non-standard numbering forms. This makes it difficult for unified preset rules to cover all possible situations, resulting in rule matching easily failing and failing to accurately identify different levels of titles. Summary of the Invention

[0004] In view of this, the present invention provides a method, apparatus and device for constructing multi-type text hierarchical directories based on a large model, so as to solve the problem in related technologies that it is impossible to accurately identify different levels of document headings.

[0005] In a first aspect, the present invention provides a method for constructing a multi-type text hierarchical directory based on a large model. The method includes: acquiring multiple first text data, the multiple first text data including text data in different document formats; converting the format of each document data to obtain second text data corresponding to the target typesetting format; identifying the position information of multiple initial title markers in each second text data, and determining the corresponding title line information based on the position information of each initial title marker; calling a locally deployed pre-trained large model to perform semantic analysis on each title line information of each second text data to obtain the hierarchical information corresponding to different title lines in the corresponding second text data; updating the initial title markers of multiple title line information in the corresponding target text data based on the hierarchical information corresponding to different title lines in each second text data to obtain the hierarchical directory information of the corresponding second text data.

[0006] The present invention provides a method for constructing a hierarchical directory for multiple types of text based on a large model. By introducing the semantic analysis capabilities of a pre-trained large model, it first converts documents of various formats into a unified target layout format and locates the initial titles. Then, the large model determines the level of each title based on semantic understanding, rather than relying solely on numbering rules for matching. Finally, it updates the title tags to generate a hierarchical directory. This method not only overcomes the limitations of preset rules in adapting to non-standard numbering and accurately identifies the title levels of various documents, but also, through the combination of unified format conversion and semantic analysis by the large model, is compatible with different document formats and reduces the cost of manually maintaining rules. Ultimately, it generates a hierarchical directory with a clear structure, providing reliable structured data support for subsequent scenarios such as efficient reading, text slicing, or RAG question answering.

[0007] In one optional implementation, the hierarchical directory information includes multiple target title line information after updating the initial title tags. The method further includes: determining the target text content data corresponding to each target title line information in the hierarchical directory information of each second text data; associating each target title line information in the hierarchical directory information of each second text data with the corresponding target text content data to obtain the structured text data of the corresponding second text data.

[0008] The method provided in this optional implementation, after completing the title level identification, associates each target title line with the corresponding text content data and generates structured text data. The core beneficial effect is that it breaks the separation between title and content, allowing the originally potentially messy text to form a clear correspondence between title-guided content and content belonging to title. Readers or subsequent processing systems can intuitively know which level of title a certain piece of content belongs to. This not only avoids the problem of low reading efficiency caused by confused content attribution, but also provides a structured and directly callable data foundation for subsequent scenarios such as text slicing and RAG question answering, without the need for additional processing of the correspondence between title and content, greatly reducing the operational cost of subsequent applications.

[0009] In one optional implementation, the pre-trained large model determines the hierarchical information corresponding to different header lines in each second text data through the following steps: redundancy identification is performed on multiple header line information of each second text data based on a preset redundancy identification rule to obtain the invalid header line information and valid header line information of the corresponding second text data; the hierarchical level of the invalid header line information in each second text data is set to 0; the priority of each valid header line information in each second text data is determined based on semantic analysis, and the hierarchical list of multiple valid header line information in the corresponding second text data is determined based on the priority of multiple valid header line information.

[0010] In one optional implementation, the priority of each valid title line information in each second text data is analyzed and determined, and the hierarchical information corresponding to different title lines in the corresponding second text data is determined based on the priority of multiple valid title line information, including: detecting whether multiple title line information of each second text data contains a numeric directory; if multiple title line information of the second text data contains a Chinese numeric directory, the corresponding second text data is determined to be historical art text data; if multiple title line information of the second text data does not contain a Chinese numeric directory, the corresponding second text data is determined to be regular text data; determining the first priority of each valid title line information in the regular text data, and determining the first hierarchical information list of multiple valid title line information in the regular text data based on the first priority of multiple valid title line information; determining the second priority of each valid title line information in the historical art text data, and determining the second hierarchical information list of multiple valid title line information in the historical art text data based on the second priority of multiple valid title line information.

[0011] In one optional implementation, the priority of each valid header line information in each second text data is determined based on semantic analysis, and a hierarchical list of multiple valid header line information in the corresponding second text data is determined based on the priority of multiple valid header line information, including: determining the priority of each valid header line information in each second text data; determining an initial hierarchical list of multiple valid header line information in the corresponding second text data based on the priority of multiple valid header line information; if the initial hierarchical list is not a purely numeric list, redundant information in the initial hierarchical list is deleted to obtain a hierarchical list of multiple valid header line information in the corresponding second text data, wherein the hierarchical list is a numeric list.

[0012] In an optional implementation, the method further includes: determining multiple text slice data corresponding to each second text data based on the structured text data of each second text data, wherein each text slice data includes a target title line information and the corresponding target text content data; generating multiple question-and-answer pairs based on the multiple slices of each second text data, wherein the multiple slices correspond one-to-one with the multiple question-and-answer pairs; and storing the multiple question-and-answer pairs of each second text data in a preset database.

[0013] Secondly, the present invention provides a multi-type text hierarchical directory construction device based on a large model. The device includes: an acquisition module for acquiring multiple first text data, the multiple first text data including text data in different document formats; a conversion module for converting the format of each document data to obtain second text data corresponding to a target layout format; a first determination module for identifying the position information of multiple initial title markers in each second text data and determining the corresponding title line information based on the position information of each initial title marker; an analysis module for calling a locally deployed pre-trained large model to perform semantic analysis on each title line information of each second text data to obtain the hierarchical information corresponding to different title lines in the corresponding second text data; and a second determination module for updating the initial title markers of multiple title lines in the corresponding target text data based on the hierarchical information corresponding to different title lines in each second text data to obtain the hierarchical directory information of the corresponding second text data.

[0014] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the method for constructing a multi-type text hierarchical directory based on a large model as described in the first aspect or any corresponding embodiment.

[0015] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the method for constructing a multi-type text hierarchical directory based on a large model according to the first aspect or any corresponding embodiment described above.

[0016] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the method for constructing a multi-type text hierarchical directory based on a large model, as described in the first aspect above or any corresponding embodiment thereof. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a method for constructing a multi-type text hierarchical directory based on a large model according to an embodiment of the present invention.

[0019] Figure 2This is a flowchart illustrating another method for constructing a multi-type text hierarchical directory based on a large model according to an embodiment of the present invention;

[0020] Figure 3 This is a structural block diagram of a multi-type text hierarchical directory construction device based on a large model according to an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] In related technologies, a series of rules are set to match text titles. However, due to the extremely complex hierarchical directory numbering types of different document types, there are not only common standard numbers, but also user-defined non-standard numbering forms. This makes it difficult for unified preset rules to cover all possible situations, resulting in rule matching easily failing and failing to accurately identify different levels of titles.

[0024] In view of this, the present application provides a method for constructing a hierarchical directory for multiple types of text based on a large model, which can be applied to a server to realize the construction of a hierarchical directory for multiple types of text. The method provided in this application introduces the semantic analysis capability of a pre-trained large model, first converts multiple format documents into a unified target typesetting format and locates the initial title, then allows the large model to determine the level of each title based on semantic understanding, rather than relying solely on numbering rules for matching, and finally updates the title tags to generate a hierarchical directory. This not only overcomes the limitations of preset rules in adapting to non-standard numbering and accurately identifies the title level of various types of documents, but also, through the combination of unified format conversion and semantic analysis by the large model, is compatible with different document formats, reduces the cost of manual rule maintenance, and ultimately generates a well-structured hierarchical directory, providing reliable structured data support for subsequent scenarios such as efficient reading, text slicing, or RAG question answering.

[0025] According to an embodiment of the present invention, a method for constructing a multi-type text hierarchical directory based on a large model is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0026] This embodiment provides a method for constructing multi-type text hierarchical directories based on a large model, which can be used in the aforementioned server. Figure 1 This is a flowchart of a multi-type text hierarchical directory construction method based on a large model according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0027] Step S101: Obtain multiple first text data, which include text data in different document formats.

[0028] For example, the first text data may be text data that needs to be reconstructed from the directory, and the multiple first text data may include, but are not limited to, text data in various formats such as PDF and Word. This application embodiment does not limit the specific content of the multiple first text data, which can be determined by those skilled in the art according to their needs.

[0029] Step S102: Convert the format of each document data to obtain the second text data corresponding to the target layout format.

[0030] For example, the target formatting can include, but is not limited to, a lightweight text markup language format (Markdown, MD). In this embodiment, mature document processing libraries, such as PyPDF2 and python-docx in Python, are used to convert text in PDF, Word, and other formats into Markdown format. During the conversion process, the basic structure and content of the text are preserved, while all headings are uniformly processed into the first-level heading form in Markdown format.

[0031] Step S103: Identify the position information of multiple initial title markers in each second text data, and determine the corresponding title line information based on the position information of each initial title marker.

[0032] For example, the initial title marker refers to the representation in the second text data used to identify the title line, and its position is at the beginning of each line in the text. In this embodiment, the title line is identified by checking the beginning of each line in the text and whether it is "#". The converted Markdown text is scanned to extract all text content that conforms to the title format, and a title index mapping table is established, retaining the original position information to form a set of initial title markers. A "title index mapping table" is established based on the title line information to record the position of each title in the second text data, avoiding the loss of position association in subsequent processing.

[0033] Step S104: Call the locally deployed pre-trained large model to perform semantic analysis on the information of each title line of each second text data to obtain the hierarchical information corresponding to different title lines in the second text data.

[0034] For example, in this embodiment of the application, a hierarchical analysis prompt template is constructed, and the extracted set of titles and prompt words are input into a pre-trained large model, such as the qwen3 series and Wenxin Yiyan. Based on the semantic understanding and contextual analysis capabilities of the large model, each title is judged in terms of hierarchy and classified into the corresponding first-level, second-level, third-level, etc. level directories. Finally, the hierarchical information corresponding to different title lines in the second text data is output. The hierarchical information corresponding to different title lines in the second text data may include, but is not limited to, a numerical list representing the title hierarchy information.

[0035] Step S105: Update the initial title tags of multiple title lines in the corresponding target text data based on the hierarchical information corresponding to different title lines in each second text data, and obtain the hierarchical directory information of the corresponding second text data.

[0036] For example, in this embodiment of the application, the headings are re-marked and formatted in the Markdown file based on the refined heading hierarchy information of the large model to construct a complete text hierarchy directory structure. Specifically, a new heading identifier is created to replace the old heading identifier based on the heading line information of each second text data and the processed heading hierarchy number list.

[0037] This embodiment provides a method for constructing multi-type text hierarchical directories based on a large model. By introducing the semantic analysis capabilities of a pre-trained large model, it first converts documents of various formats into a unified target layout format and locates the initial titles. Then, the large model determines the level of each title based on semantic understanding, rather than relying solely on numbering rules for matching. Finally, it updates the title tags to generate a hierarchical directory. This method not only overcomes the limitations of preset rules in adapting to non-standard numbering and accurately identifies the title levels of various documents, but also, through the combination of unified format conversion and semantic analysis by the large model, it is compatible with different document formats, reduces the cost of manually maintaining rules, and ultimately generates a well-structured hierarchical directory. This provides reliable structured data support for subsequent scenarios such as efficient reading, text slicing, or RAG question answering.

[0038] This embodiment provides a method for constructing multi-type text hierarchical directories based on a large model, which can be used in the aforementioned server. Figure 2 This is a flowchart of a multi-type text hierarchical directory construction method based on a large model according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0039] Step S201: Obtain multiple first text data sets, which include text data in different document formats. For details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0040] Step S202 involves converting the format of each document data to obtain the second text data in the corresponding target layout format. For details, please refer to [link to relevant documentation]. Figure 1 Step S102 of the illustrated embodiment will not be described again here.

[0041] Step S203: Identify the position information of multiple initial title markers in each second text data, and determine the corresponding title line information based on the position information of each initial title marker. For details, please refer to [link to relevant documentation]. Figure 1 Step S102 of the illustrated embodiment will not be described again here.

[0042] Step S204: Call the locally deployed pre-trained large model to perform semantic analysis on the information of each title line of each second text data to obtain the hierarchical information corresponding to different title lines in the second text data.

[0043] In some alternative implementations, the pre-trained large model determines the hierarchical information corresponding to different header lines in each second text data through the following steps:

[0044] Step a1: Based on the preset redundancy recognition rules, perform redundancy recognition on multiple header line information of each second text data to obtain the invalid header line information and valid header line information of the corresponding second text data.

[0045] For example, in this embodiment of the application, it is determined whether the title content is an inserted image or table. If its text content is 'Insert Fig' or 'Insert Table', then the title row information is determined to be invalid title row information.

[0046] Step a2: Set the level of invalid header line information in each second text data to 0.

[0047] For example, in this embodiment of the application, the level of invalid header line information is set to 0.

[0048] Step a3: Determine the priority of each valid header line information in each second text data based on semantic analysis, and determine the hierarchical list of multiple valid header line information in the corresponding second text data based on the priority of multiple valid header line information.

[0049] Specifically, step a3 above includes:

[0050] Step a31: Detect whether the multiple title lines of each second text data contain a numeric directory. If the multiple title lines of the second text data contain a Chinese numeric directory, the corresponding second text data is identified as historical art text data. If the multiple title lines of the second text data do not contain a Chinese numeric directory, the corresponding second text data is identified as regular text data.

[0051] Exemplarily, in the embodiments of the present application, it is determined whether there is a Chinese numeral directory "One, Two, Three... (One), (Two), (Three)..." in the multiple title line information of each second text data in the discrimination layer. This directory usually appears in historical art text data. If the above directory does not exist, it is classified as conventional text data.

[0052] Step a32: Determine the first priority of each valid title line information in the conventional text data, and determine the first-level information list of multiple valid title line information in the conventional text data based on the first priority of the multiple valid title line information.

[0053] Exemplarily, in the embodiments of the present application, it is set that " " represents Arabic numerals 1, 2, 3... The hierarchical relationship is determined according to the following principles, and each title is assigned a value in ascending order, and a hierarchical number list is output. The list form is in the following format:

[0054]

[0055] Step a33: Determine the second priority of each valid title line information in the conventional text data, and determine the second-level information list of multiple valid title line information in the conventional text data based on the second priority of the multiple valid title line information.

[0056] Exemplarily, in the embodiments of the present application, it is set that "#" represents Chinese numerals One, Two, Three... The hierarchical relationship is determined according to the following principles, and each title is assigned a value in ascending order, and a hierarchical number list is output. The form of the hierarchical number list can be represented in the following format: "[1, 2, 2, 3, 4, 1, 2, 1, 2]: Chapter One > Section One > [#] > [(#)] > = .] > ( )] > )]". In the embodiments of the present application, the form of the hierarchical number list is not specifically limited, and those skilled in the art can determine it according to needs.

[0057] In some optional implementation manners, the above step a3 further includes:

[0058] Step a34: Determine the priority of each valid title line information in each second text data, and determine the initial hierarchical list of multiple valid title line information in the corresponding second text data based on the priority of the multiple valid title line information.

[0059] Step a35: If the initial hierarchical list is not a pure number list, delete the redundant information in the initial hierarchical list to obtain the hierarchical list of multiple valid title line information in the corresponding second text data, and the hierarchical list is a number list.

[0060] For example, in this embodiment of the application, it is determined whether the returned content is a pure list of numbers. If not, all irrelevant content (such as analysis steps and process explanations) is deleted, and only the list of numbers is retained, such as [1,2,2,3,4,1,2,1,2].

[0061] Step S205: Update the initial title tags of multiple title lines in the corresponding target text data based on the hierarchical information corresponding to different title lines in each second text data, and obtain the hierarchical directory information of the corresponding second text data.

[0062] In some optional implementations, the above method further includes:

[0063] Step S206: Determine the target text content data corresponding to each target title line information in the hierarchical directory information of each second text data.

[0064] For example, in this embodiment of the application, the target text content data corresponding to each target title line information is determined by analyzing the second text data. This embodiment of the application does not limit the specific analysis process, and those skilled in the art can determine it according to their needs.

[0065] Step S207: Associate the target title line information in the hierarchical directory information of each second text data with the corresponding target text content data to obtain the structured text data of the corresponding second text data.

[0066] For example, in this embodiment of the application, hierarchical directory information is associated with the original text content to form structured text data, which provides support for subsequent text slicing and RAG query-answering applications.

[0067] In some optional implementations, the above method further includes:

[0068] Step b1: Based on the structured text data of each second text data, determine multiple text slice data corresponding to the second text data. Each text slice data includes a target title line information and the corresponding target text content data.

[0069] For example, in this embodiment of the application, the text slicing will fully consider the hierarchical directory structure of the text, and ensure that related content is stored in the same slice as much as possible, avoiding two pieces of content that are too different from each other from appearing in the same slice, and storing the target title line information and the target text content data corresponding to the title line in the same slice.

[0070] Step b2: Generate multiple question-answer pairs based on multiple slices of each second text data, with each slice corresponding to one question-answer pair.

[0071] Step b3: Store multiple question-answer pairs of each second text data into a preset database.

[0072] For example, in this embodiment of the application, the generated text slices will be used to generate question-answer pairs, which will be stored in a preset database, and knowledge-based question answering will be performed based on this database using a RAG architecture. This embodiment of the application does not limit the specific content of the preset database; those skilled in the art can determine it according to their needs.

[0073] The method provided in this application uses document processing libraries such as PyPDF2 and python-docx to quickly convert PDF, Word, and other formatted text into Markdown format. It uniformly processes all headings as Markdown Level 1 headings, eliminating format differences and facilitating subsequent processing. Heading lines are identified using the "#" symbol, and the Markdown text is scanned to extract heading content. An index mapping table is established to record the original position, ensuring the completeness and accuracy of heading information. A prompt template is constructed and combined with a pre-trained large model (such as the qwen3 series and Wenxin Yiyan) to determine the heading hierarchy. By removing redundant information, distinguishing between historical and artistic text types and regular text types, and classifying according to customized rules, a precise list of heading hierarchy numbers is output. Based on the hierarchy number list and heading index generated by the large model, the Markdown file headings are re-marked and formatted, and the table of contents is associated with the original text to form structured text data, providing high-quality support for text slicing and RAG question answering.

[0074] The following specific embodiment illustrates the method for constructing multi-type text hierarchical directories based on a large model provided in this application.

[0075] Example:

[0076] The method for constructing a multi-type text hierarchical directory based on a large model provided in this application includes the following steps:

[0077] Step 1: Text Format Conversion: Using mature document processing libraries, such as PyPDF2 and python-docx in Python, convert text in PDF and Word formats to Markdown format. During the conversion process, preserve the basic structure and content of the text, and uniformly process all headings into Markdown's first-level heading format. A specific example is shown below:

[0078] # Chapter 1 Introduction

[0079] # 1.1 Source of the Topic

[0080] This thesis is supported by the following project: ...

[0081] # 1.2 Significance of the Research

[0082] After the Three Gorges Dam was completed, ... support and guarantee system

[0083] # 1.3 Current Status and Trends of Waterway Traffic Safety Research

[0084] # 1.3.1 Water Traffic Safety

[0085] The concept of water traffic safety involves related concepts such as safety, hazard, accident, danger, and risk... while hidden dangers (hazards) are a necessary condition for the occurrence of accidents.

[0086] # 1.3.2 The Connotation of Waterway Traffic Safety

[0087] Water traffic safety is related to... strengthening water traffic supervision and other aspects to ensure water traffic safety.

[0088] # 1.3.3 Factors affecting water traffic safety

[0089] #1) Crew Factors

[0090] Crew factors are important human factors, ... and are interconnected and mutually restrictive.

[0091] # insert Table 1

[0092] #2) Ship Factors

[0093] Ship-related factors are another important cause of maritime accidents... hull structural strength, ship equipment...

[0094] ……】

[0095] Step 2: Title Extraction: By examining the beginning of each line in the text and identifying the title line based on whether it contains "#", the converted Markdown text is scanned to extract all text content that conforms to the title format. A title index mapping table is then created, preserving the original position information to form an initial title set. For example, the titles after extracting the above text should be:

[0096] # Chapter 1 Introduction

[0097] # 1.1 Source of the Topic

[0098] # 1.2 Significance of the Research

[0099] # 1.3 Current Status and Trends of Waterway Traffic Safety Research

[0100] # 1.3.1 Water Traffic Safety

[0101] # 1.3.2 The Connotation of Waterway Traffic Safety

[0102] # 1.3.3 Influencing Factors of Water Traffic Safety

[0103] # 1) Crew Factors

[0104] # insert Table 1

[0105] # 2) Vessel Factors

[0106] ……].

[0107] The above titles are all shown as first-level titles (marked with # in front), without distinguishing hierarchical relationships.

[0108] Step 3, Refinement of the large model level: Construct a hierarchical analysis prompt template, and input the extracted title set and prompt words into a pre-trained large model, such as the qwen3 series, Wenxin Yiyan, etc. Based on the semantic understanding and context analysis capabilities of the large model, judge the level of each title and classify it into the corresponding first-level, second-level, third-level, etc. hierarchical directories, and finally output an analyzable list of numbers. The specific steps are as follows:

[0109] 3.1. Remove redundant information: Determine whether the title content is an inserted picture or table. If its text content is 'Insert Fig' or 'Insert Table', set the hierarchical title number to 0.

[0110] 3.2. Classification of hierarchical directories: Determine whether there are Chinese numeral directories "one, two, three... (one), (two), (three)..." in the hierarchical directory. This directory usually appears in historical and art texts; if the above directory does not exist, it is classified as a regular type.

[0111] 3.3. Generation of regular text directories: Set to represent Arabic numerals: 1, 2, 3... Determine the hierarchical relationship according to the following principles, and assign values to each title in ascending order, and output the hierarchical number list in the following format:

[0112]

[0113] 3.4. Generation of historical and art text directories: Set # as Chinese numerals: one, two, three... Determine the hierarchical relationship according to the following principles, and assign values to each title in ascending order, and output the hierarchical number list in the following format [1, 2, 2, 3, 4, 1, 2, 1, 2]: Chapter 1 > Section 1 > [#] > [(#)] > = .] > ( )] > )].[[]]

[0114] 3.5. Hierarchical Directory Numeric List Verification: Determine whether the returned content is a pure numeric list. If not, delete all irrelevant content (such as analysis steps and process explanations), and only retain the numeric list, e.g., [1, 2, 2, 3, 4, 1, 2, 1, 2].

[0115] According to the above steps, the large language model prompt words are constructed as follows:

[0116] Role: You are a professional document structure analysis assistant. Please analyze the following Markdown title list.

[0117] Task: Determine their proper hierarchical relationships (starting from level 1), and return the correct hierarchical numeric list corresponding to all the titles in this document, in the following format: [1, 2, 2, 3, 4, 1, 2, 1, 2]. Note that the number of numbers in the numeric list you finally output should be the same as the number of titles in the title list.

[0118] Requirement: Please strictly ensure that each title returns a hierarchical number, and no title is omitted. Also, each number in the returned numeric list should correspond one-to-one with each title in the title list.

[0119] The specific steps include:

[0120] Step1: If there are titles with content such as 'Insert Fig' or 'InsertTable' for inserting pictures or tables, set their hierarchical title numbers to 0.

[0121] Step2: Determine whether there are Chinese numeric directories such as 'One, Two, Three... (One), (Two), (Three)...' in the hierarchical directory. If not, go to Step3. If so, go to Step4. Otherwise, go to Step3.

[0122] Step3: Set to represent Arabic numerals: 1, 2, 3... Determine the hierarchical relationships according to the following principles, and assign values to each title in ascending order, outputting the hierarchical numeric list: [Abstract] = [Introduction] = [Keywords] = [References] = [Contents] = [Figure Contents] = [Table Contents] = [abstract] = [key words] = [menu] = [Acknowledgements] = [References] = [Introduction] = [Methods] = [results] = [discussion] = [Chapter = = .]> . = . .]>[ . . ]=[ . . .]>[ . . . ]>[ . . . . ]>[( )]>[ )]>[①, ②, ③…]>[a, b, c…].

[0123] Step 4: Set # to Chinese numerals: one, two, three... Determine the hierarchical relationship according to the following principles, and assign values ​​to each title in ascending order, outputting a list of hierarchical numbers: Chapter 1 > Section 1 > [#] > [(#)] > [ ]=[ .]>[( )]>[ )).

[0124] Step 5: Determine whether the returned content is a pure list of numbers. If not, delete all irrelevant content (such as analysis steps and process explanations) and keep only the list of numbers, such as: [1,2,2,3,4,1,2,1,2].

[0125] Based on the prompt, the locally deployed large language model (qwen32B) is invoked, and the returned list of numbers is [1, 2, 2, 2, 3, 3, 3, 4, 4……].

[0126] Step 6: Table of Contents Construction and Integration: Based on the refined heading hierarchy information from the large model, the headings in the Markdown file are re-marked and formatted to construct a complete text hierarchical table of contents structure. Specifically, new headings are created to replace the old ones based on the established heading index mapping table and the processed heading hierarchy number list. Simultaneously, the hierarchical table of contents information is associated with the original text content to form structured text data, providing support for subsequent text slicing and RAG question-and-answer applications. The text content after the hierarchical table of contents reconstruction in this case is shown below:

[0127] # Chapter 1 Introduction

[0128] ## 1.1 Source of the Topic

[0129] This thesis is supported by the following project: ...

[0130] ## 1.2 Significance of the Research

[0131] After the Three Gorges Dam was completed, ... support and guarantee system

[0132] ## 1.3 Current Status and Trends of Waterway Traffic Safety Research

[0133] ### 1.3.1 Water Traffic Safety

[0134] The concept of water traffic safety involves related concepts such as safety, hazard, accident, danger, and risk... while hidden dangers (hazards) are a necessary condition for the occurrence of accidents.

[0135] ### 1.3.2 The Connotation of Waterway Traffic Safety

[0136] Water traffic safety is related to... strengthening water traffic supervision and other aspects to ensure water traffic safety.

[0137] ### 1.3.3 Factors Affecting Waterway Traffic Safety

[0138] #### 1) Crew Factors

[0139] Crew factors are important human factors, ... and are interconnected and mutually restrictive.

[0140] insert Table 1

[0141] #### 2) Ship Factors

[0142] Ship-related factors are another important cause of maritime accidents... hull structural strength, ship equipment...

[0143] ……】

[0144] The method provided in this application, by introducing the powerful semantic understanding capabilities of a large model, can deeply analyze text semantics, accurately distinguish different levels of headings, and effectively avoid the problem of hierarchy confusion, thereby constructing a complete and accurate text hierarchical directory structure. Addressing the diversity of document formats and the complexity of directory numbering, this application designs an innovative processing mechanism that is compatible with various text formats such as PDF and Word, while adapting to various numbering rules, significantly improving the accuracy and universality of heading hierarchy recognition. The constructed reliable text hierarchical directory structure provides a solid basis for text slicing based on directory hierarchy, ensuring that slices maintain a short length while possessing complete semantics, greatly improving the performance and accuracy of the RAG question-answering system and providing users with a better question-answering experience. Compared to directly using a large model to read text and extract the directory, this method, through specific optimization strategies, significantly reduces token consumption, effectively controlling usage costs; at the same time, it improves processing efficiency, enabling rapid completion of hierarchical directory recognition and construction for multiple types of text, better meeting practical application needs.

[0145] This embodiment also provides a multi-type text hierarchical directory construction device based on a large model. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0146] This embodiment provides a multi-type text hierarchical directory construction device based on a large model, such as... Figure 3 As shown, it includes:

[0147] The acquisition module 301 is used to acquire multiple first text data, which include text data in different document formats;

[0148] The conversion module 302 is used to convert the format of each document data to obtain the second text data in the corresponding target typesetting format;

[0149] The first determining module 303 is used to identify the position information of multiple initial title markers in each second text data, and determine the corresponding title line information based on the position information of each initial title marker.

[0150] Analysis module 304 is used to call the locally deployed pre-trained large model to perform semantic analysis on the information of each title line of each second text data, and obtain the hierarchical information corresponding to different title lines in the corresponding second text data.

[0151] The second determining module 305 is used to update the initial title tags of multiple title lines in the corresponding target text data based on the hierarchical information corresponding to different title lines in each second text data, so as to obtain the hierarchical directory information of the corresponding second text data.

[0152] In some alternative embodiments, the above-described apparatus further includes:

[0153] The third determination module is used to determine the target text content data corresponding to each target title line information in the hierarchical directory information of each second text data;

[0154] The association module is used to associate the target title line information in the hierarchical directory information of each second text data with the corresponding target text content data to obtain the structured text data of the corresponding second text data.

[0155] In some alternative implementations, the pre-trained large model determines the hierarchical information corresponding to different header lines in each second text data through the following steps:

[0156] Based on the preset redundancy identification rules, the redundancy of multiple header line information of each second text data is identified to obtain the invalid header line information and valid header line information of the corresponding second text data.

[0157] Set the level of invalid header line information in each second text data to 0;

[0158] Based on semantic analysis, the priority of each valid header line information in each second text data is determined, and based on the priority of multiple valid header line information, a hierarchical list of multiple valid header line information in the corresponding second text data is determined.

[0159] In some optional implementations, the priority of each valid header line information in each second text data is determined based on semantic analysis, and the hierarchical information corresponding to different header lines in the corresponding second text data is determined based on the priority of multiple valid header line information, including:

[0160] Detect whether the multiple header lines of each second text data contain a numbered directory. If the multiple header lines of the second text data contain a Chinese numbered directory, the corresponding second text data is identified as historical art text data. If the multiple header lines of the second text data do not contain a Chinese numbered directory, the corresponding second text data is identified as regular text data.

[0161] Determine the first priority of each valid header line information in the regular text data, and determine the first-level information list of multiple valid header line information based on the first priority of multiple valid header line information in the regular text data;

[0162] Determine the second priority of each valid title line information in the historical art text data, and determine the second-level information list of multiple valid title line information in the historical art text data based on the second priority of multiple valid title line information.

[0163] In some optional implementations, determining the priority of each valid header line information in each second text data based on semantic analysis, and determining a hierarchical list of multiple valid header line information in the corresponding second text data based on the priority of multiple valid header line information, further includes:

[0164] Determine the priority of each valid header line information in each second text data, and determine the initial hierarchy list of multiple valid header line information in the corresponding second text data based on the priority of multiple valid header line information;

[0165] If the initial hierarchy list is not a pure numeric list, remove redundant information from the initial hierarchy list to obtain a hierarchy list of multiple valid header line information in the corresponding second text data. The hierarchy list is a numeric list.

[0166] In some alternative embodiments, the above-described apparatus further includes:

[0167] The fourth determination module is used to determine multiple text slice data corresponding to each second text data based on the structured text data of each second text data. Each text slice data includes a target title line information and the corresponding target text content data.

[0168] The generation module is used to generate multiple question-and-answer pairs based on multiple slices of each second text data, with each slice corresponding to a one-to-one question-and-answer pair.

[0169] The storage module is used to store multiple question-and-answer pairs of each second text data into a preset database.

[0170] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0171] In this embodiment, the multi-type text hierarchical directory construction device based on a large model is presented in the form of functional units. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0172] This invention also provides a computer device having the above-described features. Figure 3 The diagram shows a multi-type text hierarchical directory construction device based on a large model.

[0173] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 4 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 4 Take a processor 10 as an example.

[0174] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0175] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0176] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0177] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0178] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0179] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0180] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0181] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for constructing a multi-type text hierarchical directory based on a large model, characterized in that, The method includes: Acquire multiple first text data, wherein the multiple first text data include text data in different document formats; The format of each document data is converted to obtain the second text data corresponding to the target layout format; Identify the position information of multiple initial title markers in each second text data, and determine the corresponding title line information based on the position information of each initial title marker; The pre-trained large model deployed locally is invoked to perform semantic analysis on the information of each title line in each second text data, so as to obtain the hierarchical information corresponding to different title lines in the second text data. Based on the hierarchical information corresponding to different title lines in each second text data, the initial title tags of multiple title lines in the corresponding target text data are updated to obtain the hierarchical directory information of the corresponding second text data. The pre-trained large model determines the hierarchical information corresponding to different header lines in each second text data through the following steps: Based on the preset redundancy identification rules, the redundancy of multiple header line information of each second text data is identified to obtain the invalid header line information and valid header line information of the corresponding second text data. Set the level of invalid header line information in each second text data to 0; Based on semantic analysis, the priority of each valid header line information in each second text data is determined, and based on the priority of multiple valid header line information, a hierarchical list of multiple valid header line information in the corresponding second text data is determined.

2. The method according to claim 1, characterized in that, The hierarchical directory information includes multiple target header line information after updating the initial header tags, and the method further includes: Determine the target text content data corresponding to each target title line information in the hierarchical directory information of each second text data; By associating the target title line information in the hierarchical directory information of each second text data with the corresponding target text content data, the structured text data of the corresponding second text data is obtained.

3. The method according to claim 1, characterized in that, The step of determining the priority of each valid title line information in each second text data based on semantic analysis, and determining the hierarchical information corresponding to different title lines in the corresponding second text data based on the priority of multiple valid title line information, includes: The system detects whether the multiple title lines of each second text data contain a numeric directory. If the multiple title lines of the second text data contain a Chinese numeric directory, the corresponding second text data is determined to be historical art text data. If the multiple title lines of the second text data do not contain a Chinese numeric directory, the corresponding second text data is determined to be regular text data. Determine the first priority of each valid header line information in the regular text data, and determine the first-level information list of multiple valid header line information based on the first priority of multiple valid header line information in the regular text data; Determine the second priority of each valid title line information in the historical art text data, and determine the second-level information list of multiple valid title line information in the historical art text data based on the second priority of multiple valid title line information.

4. The method according to claim 1, characterized in that, The step of determining the priority of each valid header line information in each second text data based on semantic analysis, and determining the hierarchical list of multiple valid header line information in the corresponding second text data based on the priority of multiple valid header line information, further includes: Determine the priority of each valid header line information in each second text data, and determine the initial hierarchy list of multiple valid header line information in the corresponding second text data based on the priority of multiple valid header line information. If the initial hierarchy list is not a purely numeric list, redundant information in the initial hierarchy list is deleted to obtain a hierarchy list of multiple valid header line information in the corresponding second text data, wherein the hierarchy list is a numeric list.

5. The method according to claim 2, characterized in that, The method further includes: Based on the structured text data of each second text data, multiple text slice data corresponding to the second text data are determined. Each text slice data includes a target title line information and the corresponding target text content data. Multiple question-answer pairs are generated based on multiple slices of each second text data, and the multiple slices correspond one-to-one with the multiple question-answer pairs; Multiple question-and-answer pairs of each second text data are stored in a preset database.

6. A multi-type text hierarchical directory construction device based on a large model, characterized in that, The device includes: The acquisition module is used to acquire multiple first text data, which include text data in different document formats; The conversion module is used to convert the format of each document data to obtain the second text data in the corresponding target layout format; The first determining module is used to identify the position information of multiple initial title marks in each second text data, and determine the corresponding title line information based on the position information of each initial title mark; The analysis module is used to call the locally deployed pre-trained large model to perform semantic analysis on the information of each title line of each second text data, and obtain the hierarchical information corresponding to different title lines in the corresponding second text data. The second determining module is used to update the initial title tags of multiple title lines in the corresponding target text data based on the hierarchical information corresponding to different title lines in each second text data, so as to obtain the hierarchical directory information of the corresponding second text data. The pre-trained large model determines the hierarchical information corresponding to different header lines in each second text data through the following steps: Based on the preset redundancy identification rules, the redundancy of multiple header line information of each second text data is identified to obtain the invalid header line information and valid header line information of the corresponding second text data. Set the level of invalid header line information in each second text data to 0; Based on semantic analysis, the priority of each valid header line information in each second text data is determined, and based on the priority of multiple valid header line information, a hierarchical list of multiple valid header line information in the corresponding second text data is determined.

7. A computer device, characterized in that, include: The system includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes the computer instructions to perform the method for constructing a multi-type text hierarchical directory based on a large model, as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the method for constructing a multi-type text hierarchical directory based on a large model, as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, Includes computer instructions for causing a computer to execute the method for constructing a multi-type text hierarchical directory based on a large model, as described in any one of claims 1 to 5.

Images