A method for processing unstructured legal book data based on rag and a system utilizing the same

Abstract

An unstructured legal book data processing system according to one embodiment includes a user terminal providing an input including an unstructured legal book file and a server generating structured data available for search augmented generation (RAG) from the unstructured legal book file, wherein the server (i) obtains the unstructured legal book file from the user terminal, (ii) stores configuration information including the layout and processing conditions of the unstructured legal book file, (iii) separates and extracts an index including body text and footnote-related elements from the unstructured legal book file based on the configuration information, (iv) normalizes the index to insert tokens indicating paragraph boundaries and replace tables, (v) separates the normalized text into search unit chunks, and (vi) generates a chunk data file in JSON format including the chunks.

Description

Technology Field

[0001] The present invention relates to a method for processing RAG-based unstructured legal book data and a system utilizing the same. Background Technology

[0002] Information search and review in the legal field rely on a diverse range of materials, including statutes, precedents, commentaries, and practical guides. In particular, literature published in book form is frequently referenced in practice due to its high level of expertise and reliability. However, since these legal materials are generally distributed in unstructured document formats such as PDFs or often feature complex layouts involving editing, typesetting, footnotes, tables, and tables of contents, it is difficult for computer systems to consistently extract, classify, and search their content. Furthermore, elements such as page-based organization, headers and footers, chapter, section, and article notation, footnotes, and quotations can serve as meaningful clues for users, but from a machine processing perspective, they can act as noise or cause contextual breaks.

[0003] Recently, with the proliferation of question-answering and summarization functions utilizing large language models, Retrieval-Augmented Generation (RAG) architectures, which search for relevant information within documents and generate responses based on it, are gaining attention. However, search performance and response quality in RAG-based services depend heavily on how input documents are broken down, what metadata is assigned, and how the search index is constructed.

[0004] In particular, for unstructured legal texts, inconsistencies in sentence boundaries, paragraph structure, page boundaries, and relationships between citations and footnotes can lead to reduced consistency in the composition of search units, which may result in unstable recall or precision for the same query. Furthermore, given the critical importance of evidence presentation and citation accuracy in the legal field, a data management method capable of reliably referencing and tracking specific parts of literature is required. The problem to be solved

[0005] Legal books provided in unstructured forms (e.g., commentaries, practical guides, casebooks, etc.) often contain complex elements such as typeset structures, page-based organization, and citations, footnotes, and tables, making it difficult to reliably extract and structure their content computationally and convert them into a searchable format. In particular, for RAG-based question-and-answer systems, the quality of the response depends on properly breaking down literature into search units and accurately retrieving sections highly relevant to the query; however, unstructured legal books are prone to compromising consistency in search unit composition due to unclear sentence and paragraph boundaries and complex relationships between citations and footnotes. Furthermore, as the presentation of evidence and the accuracy of citations are critical in legal practice, it is necessary to manage reference information that can reliably track the location of the original texts used in the response.

[0006] Accordingly, the problem that the present invention aims to solve is to provide a method and a system that implement the same, which can process a large volume of documents while maintaining data quality, by stably deriving meaningful content units from documents and systematically assigning metadata necessary for searching and referencing, so that unstructured legal book data can be converted and managed into a data structure suitable for RAG.

[0007] The problems of the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by a person skilled in the art from the description below. means of solving the problem

[0008] An unstructured legal book data processing system according to an embodiment of the present invention for solving the above problem comprises a user terminal providing an input including an unstructured legal book file and a server generating structured data available for search augmented generation (RAG) from the unstructured legal book file, wherein the server (i) obtains the unstructured legal book file from the user terminal, (ii) stores configuration information including the layout and processing conditions of the unstructured legal book file, (iii) separates and extracts an index including body text and footnote-related elements from the unstructured legal book file based on the configuration information, (iv) normalizes the index to insert tokens indicating paragraph boundaries and replace tables, (v) separates the normalized text into search unit chunks, and (vi) generates a chunk data file in JSON format including the chunks.

[0009] The above server can extract text from the above unstructured legal book file using a first library, detect footnote separators from the above unstructured legal book file using a second library, and separate and extract the above index by converting the coordinate systems of the text and the footnote separators to the same reference coordinate system.

[0010] The server extracts the original text and the threshold text with reduced width, respectively, compares the threshold text and the original text to calculate the feature quantity of line break candidates, determines the paragraph boundary without external judgment when a table of contents pattern is detected, determines one of “concatenation / space / line break” for the remaining candidates, batches and transmits judgment requests on a page-by-page basis, applies legacy rules in case of timeout or failure, and can insert a token indicating the paragraph boundary by determining EOP insertion as a safety mechanism based on the balance of terminal endings or parentheses and quotation marks.

[0011] The server can replace the table by detecting the bounding box of the table on the page, removing the body text that overlaps with the bounding box, and creating the table in a Markdown table format and inserting it at a position corresponding to the bounding box.

[0012] To preserve page boundary continuity, the above server determines whether to combine with the first text of the next page if the last text of the current page does not end with EOP, and if combination is necessary, connects the two texts by assigning a token indicating the page boundary between them, but may prohibit the combination if a table of contents or chapter / section title start pattern is detected.

[0013] The above server, in order to separate into the above chunks, a normalized paragraph sequence Regarding continuous intervals chunk Defined as, and the above chunk The goodness-of-fit score Define as the following mathematical formula 1, and [Mathematical Formula 1] , regarding multiple chunks Determine the chunk boundary so that is maximized, wherein in the above mathematical formula 1, the above is a weight, and the above is the average of the cosine similarity between adjacent paragraph embeddings within a chunk, and the above is the degree of inclusion of the footnote reference set within the chunk, and the above is the representativeness between the candidate header embedding and the chunk embedding, and the above is a penalty for the actual number range included in the chunk, and the above is chunk length go It can be defined as a penalty for going out of range.

[0014] Target chunk length for document segment u Calculate as shown in the following mathematical formula 2, and [Mathematical Formula 2] , above Based on the above Determine the chunk boundary by calculating, and in the above mathematical formula 2, the above is the average paragraph length within interval u, and the above is the standard deviation, and the above is a coefficient, and The value It can be a function that cuts.

[0015] The above chunk The boundary continuity of Defined as, and overlap size Calculate as shown in the following mathematical formula 3, and [Mathematical Formula 3] , the above overlap size Include the token corresponding to in an adjacent chunk by overlapping it, and in the above mathematical formula 3, the above is the overlap ratio coefficient, and the above can be the target chunk length of interval u.

[0016] The above chunk The set of footnote reference expressions detected in Define as, and footnote search margin When defined as such, the chunk is finalized by adjusting the chunk boundaries or expanding the footnote search range to satisfy the following mathematical formula 4, and [Mathematical Formula 4] , in the above mathematical formula 4, the above is a paragraph It is a set of footnote identifiers corresponding to, and the above can be a matching function that includes footnote number normalization.

[0017] A method for a server to process RAG-based unstructured legal book data according to an embodiment of the present invention for solving the above problem comprises the steps of: acquiring an unstructured legal book file; storing configuration information including the layout and processing conditions of the unstructured legal book file; separating and extracting an index including body text and footnote-related elements from the unstructured legal book file based on the configuration information; normalizing the index to insert paragraph boundary tokens and replace tables; separating the normalized text into search unit chunks; and generating a chunk data file in JSON format containing the chunks. Effects of the invention

[0018] According to an embodiment of the present invention, processing unstructured legal books into a form suitable for RAG-based services increases the accuracy of retrieving relevant sections during the search phase and improves the consistency and reliability of responses during the generation phase. By mitigating contextual interruptions or noise issues caused by complex layouts, page-based configurations, citations, footnotes, and tables, the deviation of search results for the same query can be reduced, and the balance between recall and precision can be improved.

[0019] Furthermore, by managing specific sections within the literature in a referenceable format, the connectivity between the response and the original text is strengthened, and the presentation of evidence becomes easier, thereby improving the citation accuracy and verifiability required in the legal field. Moreover, through the standardization and automation of the literature processing process, the time and cost associated with manual organization and verification are reduced, and processing efficiency can be maintained even in operational environments involving the continuous accumulation and updating of large volumes of literature.

[0020] Consequently, the present invention provides a technical effect that improves the quality of legal book-based knowledge search and Q&A services while reducing the burden of management and operation.

[0021] The effects according to the embodiments are not limited to those exemplified above, and a wider variety of effects are included in this specification. Brief explanation of the drawing

[0022] FIG. 1 is a schematic diagram showing the components of a legal book data processing system according to one embodiment of the present invention. FIG. 2 is a flowchart schematically illustrating a method for processing legal book data according to one embodiment of the present invention. Figure 3 is a diagram illustrating the normalization step of Figure 2, specifically the preprocessing step. Figure 4 is a diagram illustrating the normalization step of Figure 2, specifically the paragraph boundary (EOP) determination algorithm. Figure 5 is a diagram illustrating the normalization step of Figure 2, specifically the table substitution algorithm. Figure 6 is a diagram illustrating some steps of the normalization step of Figure 2. Figure 7 is a diagram illustrating the search unit chunk separation step of Figure 2. FIG. 8 is a diagram showing the hardware configuration of a server according to one embodiment of the present invention. Specific details for implementing the invention

[0023] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

[0024] Although terms such as "first," "second," etc., are used to describe various components, it goes without saying that these components are not limited by these terms. These terms are used merely to distinguish one component from another. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical scope of the present invention. A singular expression includes a plural expression unless the context clearly indicates otherwise.

[0025] In this specification, "unstructured" refers to a state in which legal book data is not pre-structured in a conventional table schema or field-value form, but is provided in document formats such as PDF, where semantic units of text (sentences, paragraphs, chapters, articles) and visual layouts (pages, margins, headers, footers, multi-column layouts, footnotes, tables, figures, etc.) are mixed, making it difficult to mechanically separate, align, and index in a consistent manner. In other words, even for the same legal book, typesetting, formatting, citation notation, footnote notation, and page layout methods may vary by book or edition; paragraph boundaries may be distorted by line breaks; or footnotes and tables may overlap with the main text; it is defined as unstructured in that "the content exists as text, but its structure is not explicitly declared."

[0026] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. Identical or similar reference numerals are used for identical components in the drawings.

[0027] FIG. 1 is a schematic diagram showing the components of a legal book data processing system according to one embodiment of the present invention.

[0028] Referring to FIG. 1, a legal book data processing system (hereinafter referred to as the system) may include a user terminal (30) and a server (10). The server (10) may include an AI model (11) and a database (20).

[0029] The user terminal (30) provides an interface for performing tasks such as registering legal book files, inputting processing conditions, and viewing and inspecting processing results. For example, the user terminal (30) may be implemented in the form of a web browser or a dedicated application, and the user may upload unstructured documents such as legal book PDFs or select documents stored on the server (10) through the terminal (30).

[0030] Additionally, the user terminal (30) provides a screen that allows input and modification of configuration information (e.g., layout-related criteria, page ranges, text processing options, etc.) required during the document processing process, and can correct errors or request reprocessing while checking the structured data (e.g., page-unit text, search-unit chunks, metadata, quality issue indicators, etc.) generated as a result of processing.

[0031] Furthermore, the user terminal (30) may include a function to receive and review an AI response along with a searched basis section for a specific query, and at this time, may transmit request conditions such as a paragraph / page range selected by the user, whether to include footnotes, and search range restrictions to the server (10).

[0032] The user terminal (30) can be implemented as, for example, a desktop PC, a laptop, a workstation, a smartphone, a tablet, an enterprise business terminal (an MDM-enabled terminal), a virtual desktop / VDI terminal, or a dedicated kiosk terminal.

[0033] The server (10) is an operator that performs a data processing pipeline based on a legal book file received from a user terminal (30) or a stored legal book file, and integrally performs file collection / registration, configuration information management, text and component extraction, normalization, search unit splitting (chunking), output generation and storage, query processing and response generation.

[0034] The server (10) may be implemented as one or more physical servers, virtual servers, cloud instances, etc., and may be configured so that processing nodes are horizontally scaled according to document throughput or the number of concurrent users. Additionally, the server (10) may be configured to manage logs and version information for each document processing stage so that the history of reprocessing, such as configuration information changes, renormalization, and re-chunking, for the same document can be tracked.

[0035] The AI ​​model (11) is an artificial intelligence component provided in the form of internal or external integration with the server (10) and performs judgments necessary during the processing and utilization of unstructured legal book data. The AI ​​model (11) may include, for example, at least one of (i) an embedding generation model for search, (ii) a large language model that generates response text for a query, (iii) a model that assists in judgments related to text normalization such as sentence boundaries, paragraph boundaries, spacing, and line breaks, and (iv) a classification and extraction model that recognizes table of contents / chapter / article notation or citation expressions. The AI ​​model (11) may be configured to receive input text, candidate boundary information, metadata, etc., from the processing module of the server (10) and provide outputs to improve the quality of the normalization results or increase search accuracy.

[0036] Additionally, the AI ​​model (11) can be configured to generate an answer by receiving a chunk of evidence searched from the database (20) as input when processing a query, and by using attributes necessary for citation such as page information of the evidence section, book title, and year of publication together to enable an evidence-centered response.

[0037] The database (20) is a storage facility that stores and manages legal book data processed by the server (10) and its management information, and is configured to include original files, configuration information, intermediate outputs, final outputs, indexes, and metadata. For example, the database (20) may store (i) book-unit metadata (title, edition, year of publication, etc.), (ii) page-unit text and page metadata (actual page / logical page, context, whether headers and footers have been removed, etc.), (iii) footnote data and linkage information with the main text, (iv) structuring results of unstructured elements such as tables / charts, (v) search-unit chunk data and chunk header / reference information, (vi) embedding vectors and keyword indexes, (vii) processing versions, quality check results, user modification history, etc.

[0038] The database (20) can be implemented in a combined form of relational DB, document DB, vector DB, object storage, etc., and in particular, for efficiency in the search stage, it can be configured so that chunk body, embedding and metadata are stored together.

[0039] In one embodiment, in such a configuration, a user terminal (30) transmits a processing request and settings to a server (10), and the server (10) structures and chunks unstructured legal books using the judgment of an AI model (11) and stores the results in a database (20). Subsequently, the user terminal (30) can receive and view / examine the processing results stored in the database (20) through the server (10), or input a query to receive the results in which the server (10) searches for relevant grounds in the database (20) and generates a response using the AI ​​model (11). If necessary, the server (10) is configured to manage access rights, logs, processing history, and citationable reference information together to ensure the reliability and traceability required in the legal domain.

[0040] FIG. 2 is a flowchart schematically illustrating a method for processing legal book data according to one embodiment of the present invention.

[0041] Referring to FIG. 2, a method for processing legal book data according to one embodiment of the present invention includes a legal book file acquisition step (S100), a configuration information storage step (S200), an index separation extraction step (S300), a normalization step (S400), a search unit chunk separation step (S500), and a chunks.json generation step (S600). Each step is performed on a server (10) and constitutes a flow for converting unstructured legal books into a data form suitable for RAG-based utilization, and can be performed in a linked manner so that the output and settings of the previous step are referenced by the subsequent step.

[0042] First, the legal book file acquisition step (S100) is a step of securing the legal book file to be processed.

[0043] Here, the legal book file may be, for example, a PDF, an e-book format, a scanned image-based document, or a file in a combined form. The file may be provided by uploading via a user terminal (30), or by the server (10) calling from a stored repository (database (20)) or collecting it by linking with an external document repository.

[0044] In the legal book file acquisition step (S100), basic metadata such as file identifiers, filenames, book titles, and editions / publication years may be secured to ensure the reliability of subsequent processing, or a hash value may be generated and stored to verify file integrity. Additionally, when processing multiple books in batches, initial conditions such as processing priority, page range specification, and branching by file type (presence or absence of text layers, etc.) may be collected together.

[0045] Next, the setting information saving step (S200) is a step of generating setting information that reflects the layout and processing conditions of the legal book file and saving it.

[0046] Since non-standardized legal books may have varying editing characteristics—such as headers, footers, page numbers, footnotes, tables, and multi-column layouts—applying the same extraction rules uniformly may result in mis-extraction or omission. Accordingly, in the setting information saving step (S200), extraction criteria for each document component, such as the main text area, footnote area, and header / footer area, can be defined as setting information. Additionally, if necessary, settings for each page type can be saved separately to reflect cases where margins or component positions differ between odd and even pages. Furthermore, the setting information may include document processing options (e.g., exclusion of specific page sections, table of contents area processing method, table processing options, citation / footnote expression recognition options, quality check options, etc.). The setting information may be version-controlled to enable reuse on a book-by-book basis, or stored so that it can be modified or expanded to accommodate revised editions of the same book.

[0047] The index separation extraction step (S300) is a step of separating and extracting text and components from a legal book file by referring to the configuration information stored in the configuration information storage step (S200).

[0048] Here, “index” refers to the result of separating a document by component for search and subsequent normalization, and may include elements such as body text, footnote text, header / footer text, page numbers, table areas, and shapes / lines (footnote separators, etc.). In the index separation extraction step (S300), intermediate structures such as the position each component occupies within the page, extracted text lines, paragraph candidates, footnote candidates, and table candidates can be generated together, and these intermediate structures can be utilized as the basis for normalization and chunking in subsequent steps. Additionally, since there may be a discrepancy between the “logical page (PDF index)” and the “actual printed page number” due to the nature of page-based documents, page number information can be extracted separately using header / footer or page notation patterns in the index separation extraction step (S300), and page metadata that can be referenced in subsequent steps can be constructed based on this.

[0049] The result of the index separation extraction step (S300) is stored in the form of structured data in page units or component units, and can subsequently be used as input for context linking, noise removal, and standardization processing during the normalization process.

[0050] The normalization step (S400) is a step for organizing text and metadata for index data extracted in the index separation extraction step (S300) to make it suitable for searching and generation.

[0051] For example, unnecessary header / footer residue can be removed from the main text and footnotes, line breaks, spaces, and punctuation can be standardized, sentence / paragraph boundaries can be defined reliably, and connection information can be assigned by identifying cases where the context continues at page boundaries.

[0052] In addition, if tables or special formatting are included, normalization can be performed to replace them with expressions suitable for searching or to separate them into a separate structure.

[0053] However, since the specific processing method of the normalization step (S400) is to be explained in detail in the drawings described later, Figure 2 briefly explains it with an emphasis on the fact that normalization is a step that performs “standardization of text and structure and alleviation of context disconnection.”

[0054] The search unit chunk separation step (S500) is a step of breaking down the normalized results into RAG search units.

[0055] A chunk can be the minimum unit stored in a search index or retrieved during query processing; if the chunk is too large, the relevant parts may be diluted, and if it is too small, the context may be insufficient, which may degrade the response quality. Therefore, in the search unit chunk separation step (S500), chunks with appropriate size and semantic units can be generated by considering sentence / paragraph boundaries, chapter / section / article notation, page information, and the need for footnote combination.

[0056] In addition, metadata identifying which book / page / component the chunk originated from may be assigned to the chunk. However, since the specific chunking criteria and additional rules of the search unit chunk separation step (S500) will be explained in detail in the drawings described later, Figure 2 briefly explains that the chunk consists of “unit data for search and presentation of evidence.”

[0057] Finally, the chunks.json generation step (S600) is a step of serializing and storing the chunks separated in the search unit chunk separation step (S500) and their metadata into a predetermined output format.

[0058] Here, chunks.json serves as an exemplary output and may include chunk body text, chunk identifiers, book identification information, page information (logical / actual pages), chapter, section, article, or title information, footnote combination information, processing versions, or quality check results. Additionally, chunks.json may be generated as a single file or split into book units, volume units, or page range units, and may be used as input for building search indexes (e.g., vector index, keyword index).

[0059] In this way, as a standardized output is generated in the chunks.json generation step (S600), the server (10) can be utilized to search for documents in chunk units, generate a response based on the search results, and, if necessary, provide the original location and citation information of the chunk.

[0060] Figure 3 is a diagram illustrating the normalization step of Figure 2, specifically the preprocessing step.

[0061] Referring to FIG. 3, the preprocessing step (S401) is intended to extract necessary components from a separated index (301) so that subsequent normalization (paragraph boundary determination, page boundary connection, standardization, etc.) can be performed stably, and to produce a “preprocessed extracted value (304)” that resolves the problem of discrepancies in extraction coordinate systems that frequently occur in book PDFs. That is, the preprocessing step (S401) is not a simple text extraction step, but a step that aligns results obtained from different extraction means with the same coordinate system and the same standard so that subsequent steps have reliable input.

[0062] First, the input to the preprocessing step (S401) is the separated index (301) generated in the index separation extraction step (S300). The separated index (301) may include information (area coordinates, page range, odd / even page distinction, processing options, etc.) that indicates “what elements to extract from where,” such as the main text area, footnote area, header / footer area, table candidate area, etc., for example, by page. In the preprocessing step (S401), by referencing this separated index (301), the process is configured to perform (i) a text-centered text (302) extraction path and (ii) a footnote separator line (303) extraction path to stably separate the footnote area in parallel. The reason for processing the text (302) and the footnote separator line (303) separately in this way is that in legal book PDFs, “footnote boundaries” are often unclear based solely on the text content, and in actual typesetting, there are many cases where the start of a footnote is visually indicated by a separator line (or a thin shape / line). Therefore, if the footnote dividing line (303) is secured as a separate signal, the boundary between the main text and the footnote can be set more consistently in subsequent steps.

[0063] Text (302) is extracted using a first library (21), such as PyMuPDF. Since PyMuPDF can provide line / block-unit text and location information from the text layer of a page relatively efficiently, it is advantageous in environments where the total page processing volume is large or where large volumes of books are processed. At this time, the text (302) may include not only simple strings but also additional information such as line-unit information for constructing paragraph candidates, bounding boxes, and font size / style. On the other hand, footnote separators (303) are extracted using a second library (22), such as pdfplumber. pdfplumber can handle “graphic elements” such as lines, rectangles, and shape objects drawn on the page relatively directly, making it suitable for detecting shape-based boundaries rather than text, such as footnote separators.

[0064] Since the footnote dividing line (303) may be a single line segment or may exist as several short line segments during the scanning / conversion process, the preprocessing step (S401) may be configured to select candidates based on the length, thickness, slope, and location at the bottom of the page of the line segments, and to merge adjacent line segments to derive “footnote boundary candidates.”

[0065] The critical issue here is that the coordinate systems returned by the first library (21) and the second library (22) may differ from each other, or even within the same library, the interpretation of coordinates may vary depending on PDF attributes (e.g., mediabox, cropbox / trimbox, rotation, scale). In practice, there is a frequent problem where the area coordinates specified in the user interface (e.g., body text area, footnote area) and the actual library extraction coordinates are slightly misaligned, resulting in “text being omitted in the desired area or unwanted areas (headers / footnotes) being mixed in.” If such a discrepancy in coordinate systems persists, it causes the paragraph boundaries to be misjudged or footnote merging / separation to fail during the subsequent normalization stage.

[0066] The coordinate transformation module (210) performs the role of converting text location information obtained from the first library (21) and dividing line location information obtained from the second library (22) into a single reference coordinate system and aligning them. For example, if the extraction coordinates specified in the UI are based on the “page display image,” the coordinate transformation module (210) calculates an offset and a scale factor by reflecting whether the image was rendered based on the PDF’s mediabox, whether a cropbox / trimbox was applied, or whether there was page rotation or non-uniform scaling. Subsequently, the coordinates returned by each library are mapped to the same coordinate system through a transformation that includes (i) translation (offset correction), (ii) enlargement / reduction (scale correction), and (iii) rotation correction if necessary. Through this process, the “area coordinates specified in the setting information” correspond accurately to the actual extraction result, and as a result, the text (302) and the footnote dividing line (303) have a consistent positional relationship with each other. In addition, the coordinate transformation module (210) may apply different transformation parameters for each page type to account for cases where the margin structure differs in odd / even pages, or may handle exceptions where the cropbox changes only in specific page sections as separate rules.

[0067] The preprocessed extract value (304) produced through the coordinate transformation module (210) is a “matched intermediate output” used as input for the subsequent normalization step. The preprocessed extract value (304) may include, for example, (i) text lines / blocks per page and their coordinates before normalization, (ii) matched coordinates and confidence (candidate score) of the footnote separator line, (iii) whether it intersects with the main text / footnote area defined by the setting information, and (iv) classification results regarding where the text is located, either above or below the footnote separator line. That is, the preprocessed extract value (304) is configured in a “form that allows text and footnote boundaries to be handled together in the same coordinate system,” and based on this, it can be utilized to stably separate the main text and footnotes in the subsequent normalization step and to reduce errors that may occur in determining line breaks / paragraph boundaries.

[0068] Consequently, the preprocessing step (S401) can be understood as a functional block that combines multiple extraction paths (text / separator) and a coordinate transformation module (210) to mitigate processing instability caused by the complex layout and coordinate system deviation of the legal book PDF, and provides base data that allows subsequent steps to perform normalization with consistent rules.

[0069] Figure 4 is a diagram illustrating the normalization step of Figure 2, specifically the paragraph boundary (EOP) determination algorithm.

[0070] In this embodiment, EOP is a special token or separator indicating the “end of a paragraph” and functions as a reference point to ensure that page boundary linking, footnote combination, and search unit chunk separation are reliably performed in subsequent steps. Referring to FIG. 4, the EOP determination algorithm does not simply rely on line break characters as they are, but performs a multi-layered judgment to distinguish between (i) automatic line breaks caused by layout influences and (ii) line breaks intended to distinguish actual paragraphs.

[0071] The threshold text extraction step (S411) is a step for generating reference data to determine whether line breaks in text extracted from the original text are “actual paragraph breaks” or “line breaks forced by the layout width.” For example, a threshold area is set by reducing the horizontal width of the body area defined by the setting information by a predetermined ratio (e.g., around 90%), and text is extracted again from the same page based on that threshold area. At this time, the threshold area is usually set so that only the right width is reduced while maintaining left-alignment, thereby inducing automatic line breaks to occur at an earlier point even for the same content. Consequently, the threshold text extraction step (S411) generates comparison text to observe “where the line breaks when the width narrows,” and this is used as a feature quantity to determine line end fullness (whether it is filled to the right end) and the possibility of forced line breaks in a subsequent step.

[0072] The original text comparison step (S412) is a step that calculates quantitative and qualitative characteristics of paragraph boundary candidates (line break candidates) by comparing the threshold text obtained in the threshold text extraction step (S411) with the original text extracted. For example, if a specific line in the original text ends abruptly (lines end early), the question arises whether this is due to an actual paragraph ending or if it was shortened accidentally due to layout, alignment, quotation marks, formulas, etc. In this case, if lines in the same section of the threshold text are cut off more frequently and at earlier points due to width reduction, the reliability of the original text's "shortness" as a paragraph end signal may be lowered. Conversely, if lines are repeatedly cut off near the same location despite the width being reduced, or if a break pattern is maintained uniquely only in specific lines, it is highly likely that the break is an actual line break or a structural paragraph rather than a simple wrapping. The original text comparison step (S412) calculates features such as line length ratio, line end character (period, quotation mark, parenthesis, etc.), next line start form (indentation, number / symbol start, etc.), and breakage amount relative to threshold for each candidate boundary through this comparison, and provides these as inputs for subsequent judgment.

[0073] The table of contents pattern break processing step (S413) is designed to prevent structures such as table of contents, chapter / section titles, and item columns, which are frequent in legal books, from confusing EOP judgments. Table of contents features repeated dotted line readers (……), page numbers, and chapter / section / article notations (e.g., “Chapter 1,” “Section 2,” “Article 3”); concatenating them like regular sentences actually degrades search quality. Therefore, when S413 detects a predefined table of contents pattern (e.g., “Chapter n / Section / Article + space + title + dotted line + number,” or a continuous item column following the “Table of Contents” heading), it immediately performs line breaks (EOP insertion) for the corresponding section without separate external judgment (model / API call). This reduces computational costs while preemptively preventing the problem of paragraph boundaries collapsing due to table of contents sections mixing with the main text.

[0074] The remaining pattern “concatenation / spacing / line break” determination step (S414) is a step for determining how to combine text for each candidate boundary for general body sections that were not processed in the table of contents pattern break processing step (S413). Specifically, between the end of one line and the beginning of the next line, it is selected whether to (i) remove the line break and paste as is (concatenation), (ii) insert a space and paste (spacing), or (iii) insert an EOP because the point is a paragraph boundary (line break). Since this step involves linguistic combinations (processing of particles / endings / abbreviations / quotation marks, etc.) that are difficult to process with rules alone, it can be configured to perform classification based on the input context (e.g., the last few characters of the previous line and the first few characters of the next line, or one or two lines before and after). For example, if the previous line ends with a particle or conjunction and the next line is in the form of a sentence continuation, a space or concatenation is likely to be selected; if the previous line ends with a terminal mark and the next line begins with an indentation or item number, a line break is likely to be selected. This determination result is used as a combination rule applied when reconstructing text within a page.

[0075] The page-unit transmission and legacy rule application step (S415) is a step that enables the determination of the remaining pattern “concatenation / space / newline” in the determination step (S414) from the perspective of efficiency and fault tolerance. For example, candidate boundaries can be grouped into page units for batch transmission, and the candidate pairs generated from one page can be configured to be processed as a single request up to a predetermined number (e.g., around 32 cases). At this time, since there may be cases where the determination results are not fully secured due to network delays, timeouts, or missing partial responses, the page-unit transmission and legacy rule application step (S415) applies legacy rules (fallback rules) to supplement the presence of spaces / newlines. Legacy rules may include, for example, whether parentheses and quotation marks are balanced (if there is only an opening and no closing, the paragraph is not broken), the repeating pattern of table of contents and item columns (if it is a repeat, the paragraph is broken), the end of a line / start of the next line character (e.g., if an item starts after “:”, a newline is given priority for the start of a number or symbol (e.g., “1)”, “A.”, “(1)”), and rules that strengthen candidates for an intermediate newline if an excessively long paragraph is generated. That is, the page-by-page transmission and legacy rule application step (S415) is a step designed to “utilize external judgment if possible, but ensure that the pipeline does not stop even if it fails.”

[0076] The safety device-based EOP insertion step (S416) is a step that ensures consistency and safety of paragraph boundaries by finally checking the results of the first combination of steps S414 and S415. Here, the safety device can be understood as a post-processing step that “reverses cases where a break is necessary due to the actual sentence or paragraph structure, even if it is determined to be a concatenation, and inserts an EOP,” or conversely, “restores cases where a break is not necessary due to the context, even if it is determined to be a line break.” For example, if the previous line ends with a terminal ending or a terminal symbol and the next line begins with a new description after a closed bracket or quotation mark, EOP insertion is reinforced. Conversely, if the next line follows a bracket or quotation mark opened in the previous line without being closed, EOP insertion is suppressed to maintain balance within the same paragraph. Additionally, even if a line is broken at the end of a page, it checks whether the next line is a continuation of the same sentence (concatenating expressions, incomplete phrases, continuation of numbers or units, etc.) to reduce contextual interruption caused by unnecessary EOP insertion. As a result of this step, each line combination is finalized, and “body text for normalization” is generated with EOP tokens inserted at the finalized positions.

[0077] In summary, the EOP determination algorithm operates through a flow that establishes comparison criteria via width reduction and re-extraction, characterizes the nature of line breaks by comparing with the original text, preemptively processes table of contents patterns, classifies general sections into concatenation, spacing, and line breaks, ensures operational stability through page-unit layout processing and fallback rules, and finalizes the EOP using safeguards such as sentence-ending and parenthesis balancing. Due to this multi-layered structure, paragraph boundaries are restored relatively stably even in PDFs of legal books with numerous layout-based line breaks, resulting in improved input quality for subsequent stages (page boundary linking, footnote merging, and search-unit chunk separation).

[0078] Figure 5 is a diagram illustrating the normalization step of Figure 2, specifically the table substitution algorithm.

[0079] Legal books frequently contain information in the form of tables, such as requirement summary tables, comparison tables, procedure tables, and form examples. However, when extracting PDFs, tables are often scattered as “line-by-line text without cell boundaries” or mixed with the main text with the column order jumbled. If the table area is maintained as main text in this manner, (i) the column and row structure of the table collapses in the search index, thereby reducing search precision; (ii) identical content is duplicated in the main text or accumulates as noise, making retrieval results unstable; and (iii) the correspondence between table items in the RAG response is broken, potentially distorting the mapping of facts and requirements. Accordingly, this embodiment detects the table, removes it from the main text, and then replaces and inserts the table in a structured form (e.g., a Markdown table), thereby normalizing it into a form suitable for searching and generation.

[0080] The detection step (S421) is a step of detecting table candidates in the page to be processed (or the body area of ​​the page). For example, the system can determine whether a table exists on each page by using a module that provides a function to convert PDF to Markdown (e.g., a pymupdf4llm family function or an equivalent table detection function), and if a table exists, it can extract the table's bounding box (bbox), the number of rows and columns, the presence of a header, and text per cell. Since multiple tables may exist on a single page, the bbox and structural information for each table candidate can be stored in a list format. Additionally, since false detection may occur if the boundary of the table is adjacent to multi-column editing, footnote areas, or figures / shapes, the reliability of the candidate may be calculated based on criteria such as (i) whether the table bbox overlaps with the body area, (ii) whether the text density inside the bbox (number of lines, ratio of numbers / symbols) matches the characteristics of a table, and (iii) whether the table outline or cell boundary line exists, and only candidates that are above the reliability threshold may be confirmed as tables.

[0081] The text removal step (S422) is a step of removing text corresponding to the table area from the original text body based on the table bbox confirmed in the detection step (S421). Specifically, the system manages text within a page in units of lines or blocks and may retain the coordinates (bbox) of each line / block. At this time, lines / blocks that intersect (overlap) with the table bbox are determined, and the corresponding lines / blocks are excluded from the original text body. The removal can be implemented by “removing only lines that are completely contained within the table” or “removing if they overlap with the table bbox by more than a predetermined ratio.” This prevents the table from remaining duplicated in the text body and causing search noise. In addition, if table titles (e.g., “”, “Requirements Summary Table”) or table annotations are adjacent to the table, to prevent these texts from being excessively removed, (i) buffer margins may not be placed at the top / bottom of the table bbox, or (ii) exception rules may be applied to exclude lines with specific patterns (“Table”, “Table”, “< >”, etc.) from removal. Consequently, the body removal step (S422) is a step that establishes a normalization basis for “leaving tables as tables and body text as body text.”

[0082] The Markdown insertion step (S423) is a step of inserting a structured table representation at the location removed in the body removal step (S422) (i.e., the location where the table existed). The system generates a Markdown table (e.g., a '|' separator and a header separator) using the cell structure of the table extracted in the detection step (S421) and inserts it into the body text stream. At this time, the insertion location can be set immediately before or after the body line closest to the y-coordinate where the table bbox starts (relative to the top of the page), and if multiple tables exist, they can be sorted in order of y-coordinates to preserve the reading order of the document as much as possible. Additionally, to clarify that the table is inserted within a single paragraph flow, paragraph boundary tokens can be added before and after the table, or a separator can be assigned to treat the table itself as an independent block. Furthermore, the metadata of the table (e.g., table ID, source page / coordinates, number of rows and columns, table title, detection confidence, etc.) may be stored in a separate repository and linked to the Markdown table inserted in the body via a link (reference key). In this case, during the subsequent search stage, the table is treated as “structured data” rather than a “block of text,” making it possible to search based on specific columns / rows or present evidence at the table level.

[0083] In summary, the table replacement algorithm detects tables on a page to secure bbox and structural information, removes source lines / blocks overlapping with the corresponding bboxes to prevent table text from remaining as noise in the main body, and then generates the table in a structured Markdown format and inserts it at an appropriate location, thereby normalizing the table information of legal books into a form suitable for searching and generation. As a result, the correspondence of table items is maintained, improving the quality of table-based requirements organization, comparative analysis, and procedural guidance in RAG responses, and mitigating search instability and noise caused by duplication and mixing of table areas.

[0084] Figure 6 is a diagram illustrating some steps of the normalization step of Figure 2.

[0085] The normalization step (S400) may further include (i) a page boundary continuity preservation step (S430) to prevent text segmented by page from causing context breakage during the RAG processing process, (ii) a page number restoration and context generation step (S440) to resolve the issue of discrepancies between logical pages in the PDF and actual page numbers of the book and to strengthen page-by-page reference information, and (iii) a footnote organization and text-footnote combination step (S450) to organize footnotes and link them with the text to enhance the reliability and evidence presentation of the text. These steps receive the results of the preceding preprocessing step (S401) and the paragraph boundary (EOP) determination algorithm as input and perform the role of stabilizing the page, citation, and evidence structure before proceeding to “search unit chunk separation (S500).”

[0086] First, the page boundary continuity preservation step (S430) determines whether to restore a sentence broken at a page boundary to the same paragraph, and if necessary, a token indicating the page boundary (e.g.,<PAGE_SPLIT> This is the step of connecting text with ).

[0087] Since many PDFs of legal books are digitized directly from printed typeset, it is common for a paragraph to be interrupted at the end of a page and continue to the first line of the next page. In such cases, if the text is simply separated by page, "a single paragraph is split into two," causing a break in context during searching, summarizing, and presenting evidence. Accordingly, in the page boundary continuity preservation step (S430), it is checked whether the last line or last paragraph of the current page ends with an EOP, and if it does not end with an EOP, it is determined as "possibility of paragraph continuity" and an attempt is made to connect it with the first paragraph of the next page. When connecting, strings are not simply concatenated; instead, the method of combination is determined by considering the ending form of the previous page (e.g., particles / connectors, incomplete parentheses / quotation marks, segmentation of numbers / units) and the starting form of the next page (e.g., continuous narrative, start of item number, start of title / chapter / section).

[0088] For example, if it is determined that a connection is necessary, one of the following is selected: insert a space between the two sections (spacing), combine them without a space (concatenation), or prohibit the connection itself (start of new paragraph). This combination method can be determined using the combination judgment result (concatenation / spacing / line break) used in the paragraph boundary determination stage or by utilizing separate combination judgment logic. Additionally, if the next page begins with a “start of new structure” pattern, such as a table of contents, chapter / section heading, or index, the connection is prohibited to prevent miscombination, even if the previous page does not end with EOP.

[0089] Furthermore, since page boundary connections can be connected in a sequential manner, the maximum number of pages allowed for a continuous connection is limited (e.g., a maximum of 3 consecutive pages) to suppress the occurrence of abnormally long connections. As a result, through the page boundary continuity preservation step (S430), a balance can be achieved in which the problem of sentences being cut off at page boundaries being segmented more than necessary is reduced, while simultaneously preventing misconnections at structural boundaries such as table of contents and titles.

[0090] Next, the page number restoration and context generation step (S440) is a step for generating metadata to resolve the issue of discrepancies between the “PDF page index” and the “actual page numbers of the book” and to enhance the referability of each page. In legal practice, book citations are often made at the actual page number level, and the page number serves as a key identifier even when citing the same document as evidence.

[0091] However, in PDFs, it is common for logical pages and actual printed page numbers to be misaligned due to the insertion of front covers, table of contents, prefaces, appendices, etc. Therefore, in the page number restoration and context creation step (S440), it is first checked whether a fixed offset value (e.g., page_gap_fixed) for a specific book exists in the database or configuration information, and if it exists, it is applied first to finalize the page mapping.

[0092] Conversely, if there is no fixed value, patterns presumed to be page numbers (e.g., numbers alone, "- 123 -" form, numbers combined with chapter / section notations, etc.) are extracted from headers / footers, and a statistically consistent offset (page_gap) is calculated by determining the difference between the "logical page number" of each page and the "extracted page number candidates." In this case, to reduce misrecognition, candidates in which the same pattern is repeated more than a certain frequency may be prioritized, or candidates may be evaluated by page type by reflecting cases where the header position differs between odd and even pages.

[0093] Once the offset is finalized, the actual page number of each page is restored and saved as page metadata.

[0094] Additionally, in the page number restoration and context generation step (S440), the text of the header / footer remaining after removing the page number itself (e.g., book title, chapter / section title, truncated title, etc.) is stored as a “page context,” and an “full_context” can be generated by combining an even page context and an odd page context. This full_context is utilized as information explaining “which chapter / section the current page belongs to” in the chunk header or the evidence presentation area, thereby enhancing the interpretability of search results and citation accuracy. Consequently, the page number restoration and context generation step (S440) strengthens page-level referencing capabilities and lays the foundation for generating “chunks containing book title / page / chapter / section information” in subsequent steps.

[0095] The footnote organization and text-footnote combination step (S450) is a step that implements the basis structure of a legal book in a RAG-friendly form by normalizing footnotes and clarifying the connection relationship with the text.

[0096] In legal books, footnotes often contain key grounds such as precedent sources, legal grounds, and citations of academic theories; however, when extracting PDFs, footnotes may be mixed with the main text, fragmented beyond page boundaries, or footnote numbers may appear in various forms (superscript, brackets, asterisks, etc.), making it difficult to connect them. Accordingly, in the footnote organization and main text-footnote combination step (S450), line break / paragraph boundary processing logic identical or similar to that of the main text is first applied to the footnote text extracted from the footnote area to remove unnecessary forced line breaks within the footnote and insert EOPs at necessary locations to restore the “footnote paragraph.”

[0097] Next, since footnotes may extend beyond page boundaries, the start pattern of the footnote area on the next page is determined as either a “continuation of footnotes” or a “start of a new footnote” to connect or separate the footnotes. For example, if the footnote area on the next page matches a specific start pattern (e.g., whether footnote numbers are continuous, whether there is a consecutive notation of the same number, or whether there is a repetition of the 'Note' form), it is combined as a continuation of the same footnote; otherwise, it is separated into a new footnote block.

[0098] Subsequently, in the footnote organization and body-footnote combination step (S450), footnote reference expressions (e.g., numerical notation, symbolic notation) within the body text are detected and mapped to a normalized footnote numbering system to link the footnote list with the body text location. This linkage information can be used as a basis for determining “which footnotes to attach to a specific paragraph / chunk of the body text,” and depending on the implementation, it can be stored by including the footnote content in the body text or attaching it as a separate field (e.g., footnote_list, evidence_list). The result of this combination is directly utilized in the subsequent chunking step to construct an “evidence-containing chunk,” thereby increasing the possibility of presenting evidence during Q&A and improving the reliability of the answer.

[0099] These steps (S430–S450) contribute to ensuring consistent search performance and traceability of evidence in subsequent search unit chunk separation and RAG-based utilization by stabilizing the structure of “page, citation, and evidence” in the latter part of the normalization step (S400).

[0100] Figure 7 is a diagram illustrating the search unit chunk separation step of Figure 2.

[0101] Referring to FIG. 7, the RAG-friendly billing module (220) of the server (10) can utilize chunk suitability scores for chunk separation by simultaneously optimizing "search recall + possibility of answering with evidence + contextual consistency".

[0102] Document paragraph (or line block) sequence It is viewed as. Each paragraph Token length for , embedding vector , page number (Based on actual pages), footnote reference set , Title / Subtitle Set Defines. Chunk is a continuous section Includes the paragraphs of.

[0103] In one embodiment, a formula can be used to define a "chunk suitability score (320)" and generate an optimal split. Instead of a person determining "where to cut" by rule, a method is used to define chunk quality scores and split the chunk so that their sum is maximized. Chunk The goodness-of-fit score Define as in mathematical formula 1.

[0104]

[0105] Here, each term directly reflects the "RAG affinity" as follows (Equations 2, 3).

[0106]

[0107]

[0108] It is structured so that the score increases for chunks that have better attached footnotes (evidence) (Mathematical Formula 4).

[0109]

[0110] In mathematical formula 4 It is a set of candidate subtitles (e.g., 30 characters or less) to be accumulated in the chunk, and is the chunk embedding (e.g., average). That is, the score is increased "the better the header represents the chunk content" (Equation 5).

[0111]

[0112] Points are deducted if the page range becomes excessively wide (multiple pages are combined) (Equation 6).

[0113]

[0114] In mathematical formula 6 and if it goes out of the minimum / maximum token range, a squared penalty is applied.

[0115] Now, the total partition is defined by the following optimization (Equation 7).

[0116]

[0117] According to the embodiment, a chunk boundary score may also be utilized in the chunk boundary determination (311) that determines whether to cut a boundary candidate i (Equation 8).

[0118]

[0119] In mathematical formula 8 is the "degree of interruption in footnote flow," for example It could be. It is set as a rule-based signal (1 if present, 0 if absent, or a weight) such as sentence termination, table of contents start, or definition start. The truncation is Defines to perform if...

[0120] According to the embodiment, the target chunk length (312) L according to the document / section characteristics * (u) can be automatically adjusted (Equation 9).

[0121]

[0122] Here is the mean / standard deviation of paragraph lengths in the current interval u (e.g., same chapter / section), and is the number of target chunk tokens in the corresponding section. In other words, it can automatically adapt to "short chunks if there are many short units, such as statutes / case summaries, and long chunks if explanatory paragraphs are long."

[0123] According to the embodiment, the overlap (313) width can be defined as "information loss minimization".

[0124]

[0125] It is a method that reduces context loss by increasing overlap when topic conversion is large at the boundary (small cosine similarity), and reduces costs by decreasing overlap when conversion is small.

[0126] According to the embodiment, header accumulation (314) (subtitle accumulation) can be defined as a "damped weighted sum." Instead of simply attaching subtitles, accumulation weights can be assigned by reflecting "recency" (Equation 11).

[0127]

[0128] In mathematical formula 11 is the distance between the paragraph index where the subtitle appears and the end t of the chunk. The chunk header is You can select the subheading most similar to it or define it to combine the top m. This approach can mathematically support the effect that "headers represent chunk content."

[0129] According to the embodiment, the attachment of footnotes (315) (reasons) can be specified as a "condition for satisfying the reasoning." The most beneficial point in the RAG is that "chunks are configured so that a reasoning can be attached to the answer." This can be set as a conditional expression (Equation 12).

[0130]

[0131] Here is a set of footnote reference expressions detected in the chunk body (e.g., "(footnote 12)", "^12"), and is a navigation margin that reflects "footnotes may extend to the next page." match is a matching function that includes number normalization / formation transformation. If this condition is not satisfied, it can be designed to (i) adjust chunk boundaries, (ii) expand the footnote navigation range, or (iii) record a “missing basis” flag in the meta.

[0132] Below is a list summarizing the definitions of the parameters and symbols used in the mathematical formulas presented earlier.

[0133] As a basic symbol for documents, paragraphs, and chunks, is the i-th paragraph (or line block) and is the total number of paragraphs. Is is the token length (or character / subword length) and Is It is the embedding vector of Is is the actual page number (or actual page index) to which it belongs. Is or referenced in It is a set of footnote identifiers corresponding to Is It is a set of titles / subtitles (candidate headers) linked to it. is the k-th chunk and is chunk It is a paragraph index range included in is an arbitrary chunk section representation and is the chunk length (token sum) and is chunk embedding (e.g., )am.

[0134] As symbols related to the chunk fitness objective function, is chunk It is the total goodness-of-fit score. is the semantic consistency (average cosine similarity) between adjacent paragraphs within a chunk, and is the inclusion and linkage degree of the chunk's footnotes (foundations) (a value normalized by the number of reference expressions relative to the size of the footnote set) and is the representativeness between the chunk content and the candidate header (maximum cosine similarity between the header embedding and the chunk embedding) and is a set of candidate headers (e.g., subtitles of 30 characters or less) to be accumulated in the chunk, and is candidate header It is an embedding of is a deduction term for page ranges spanning chunks (e.g., ) and is the page range penalty strength parameter and is a length constraint penalty. is the allowed minimum / maximum chunk length (number of tokens). are each It is the weighting coefficient of the term.

[0135] As a symbol related to boundary score-based clipping judgment, is a paragraph and It is the cutoff (boundary) score between is the weight of each term of the boundary score. is an indicator function that indicates whether it is a page-changing boundary, and is as the degree of interruption in the flow of footnotes between adjacent paragraphs at is the difference in symmetry, is a union and is the number of elements and is a rule-based signal (or its weighted sum) such as sentence termination / table start / definition start, and is a threshold value that determines whether to cut.

[0136] As an adaptive target length related symbol, is the current section (e.g., the same chapter / section, etc.) and is the section It is the target chunk length (number of tokens) in is the section My average paragraph length, is the standard deviation and is the standard deviation reflection strength coefficient. The value It is a function that truncates by a range.

[0137] As an overlap-related symbol, is chunk It is the number of tokens (or paragraphs) to nest into the next chunk. is the overlap default ratio factor. is an indicator of semantic continuity at the boundary. is the end-of-chunk index. is a ceiling function.

[0138] As a symbol related to header accumulation (damped weighted sum), is chunk It is the accumulated header vector of is a candidate header index. is the j-th candidate header text (or item) and is that embedding and is the location where the candidate header appeared and the end of the chunk It is the distance between (paragraph index difference, etc.) is a damping rate (strength reflecting primacy) parameter and is the number of top candidates to be combined into the final header.

[0139] As a symbol related to the condition satisfying the basis of the footnote, is a set of footnote reference expressions detected in the chunk body, and is that element (individual reference representation) and is a footnote identifier or footnote text unit and is the navigation margin (paragraph / page range expansion amount) considering that footnotes may extend to the next page, and is a reference expression and footnotes It is a function that returns whether a match exists (including number normalization, shape transformation, symbol conversion, etc.).

[0140] FIG. 8 is a diagram showing the hardware configuration of a server according to one embodiment of the present invention.

[0141] Referring to FIG. 8, the server (10) may be implemented as a computing device comprising a processor (110), memory (120), a transmitting / receiving device (130), an input interface device (140), an output interface device (150), a storage device (160), and a bus (170). The processor (110) performs operations to execute the legal book data processing pipeline of the present invention, for example, performing calculations required in the process of file acquisition, index separation extraction, normalization, chunk generation, and output generation. The memory (120) includes ROM and RAM and stores program code, intermediate processing data (page text, coordinate information, temporary index, etc.), and execution status information performed by the server (10). The storage device (160) is a non-volatile storage medium and may store original legal book files, configuration information, processing result data (pages.json, chunks.json, etc.), logs, and version information. The transmitting and receiving device (130) performs network communication with a user terminal to receive uploaded files, processing requests, and query inputs, and transmits processing results, search results, or AI responses. The input interface device (140) and output interface device (150) are configured to allow the server (10) to interact with an external device or management console, such as enabling settings and status checks through a keyboard, mouse, display, or remote management interface.

[0142] The AI ​​model (11) can be implemented as a module running inside the server (10) or as an externally connected computational resource, and can be configured to perform functions such as text normalization judgment, embedding generation, and response generation by being called through the processor (110) and bus (170). The bus (170) serves as a data transmission path between each of the above components, enabling efficient exchange of computation results and control signals.

[0143] Although embodiments according to the technical concept of the present invention have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention may be implemented in other specific forms without changing its technical concept or essential features. The embodiments described above should be understood as illustrative in all respects and not restrictive. Explanation of the symbols

[0144] 10: Server 11: AI Model 20: Database 21: 1st Library 22: Second Library 30: Terminal 210: Coordinate Transformation Module 220: RAG Affinity Billing Module 301: Separated index 302: Text 303: Footnote separator 304: Preprocessed extract value 311: Determine Chunk Boundary 312: Target Chunk Length 313: Overlap 314: Header accumulation 315: Footnote 320: Chunk fit score

Claims

Claim 1 A RAG-based unstructured legal book data processing system comprises: a user terminal providing an input including an unstructured legal book file; and a server generating structured data usable for search augmented generation (RAG) from the unstructured legal book file, wherein the server comprises: (i) acquiring the unstructured legal book file from the user terminal; (ii) storing configuration information including the layout and processing conditions of the unstructured legal book file; (iii) separating and extracting an index including body text and footnote-related elements from the unstructured legal book file based on the configuration information; (iv) normalizing the index to insert tokens indicating paragraph boundaries and replacing tables; (v) separating the normalized text into search unit chunks; and (vi) generating a chunk data file in JSON format containing the chunks, wherein the server, for separating into chunks, a normalized paragraph sequence Regarding continuous intervals chunk Defined as, and the above chunk The goodness-of-fit score Define as the following mathematical formula 1, and [Mathematical Formula 1] , regarding multiple chunks Determine the chunk boundary so that is maximized, wherein in the above mathematical formula 1, the above is a weight, and the above is the average of the cosine similarity between adjacent paragraph embeddings within a chunk, and the above is the degree of inclusion of the footnote reference set within the chunk, and the above is the representativeness between the candidate header embedding and the chunk embedding, and the above is a penalty for the actual number range included in the chunk, and the above is chunk length go An unstructured legal book data processing system defined by a penalty for exceeding the scope. Claim 2 An unstructured legal book data processing system according to claim 1, wherein the server extracts text from the unstructured legal book file using a first library, detects footnote separator lines from the unstructured legal book file using a second library, converts the coordinate systems of the text and the footnote separator lines to the same reference coordinate system, and separates and extracts the index. Claim 3 An unstructured legal book data processing system according to claim 1, wherein the server extracts the original text and the width-reduced threshold text respectively, calculates the feature quantity of line break candidates by comparing the threshold text and the original text, determines the paragraph boundary without external judgment when a table of contents pattern is detected, determines one of "concatenation / space / line break" for the remaining candidates, batches and transmits judgment requests on a page-by-page basis, applies legacy rules in case of timeout or failure, determines EOP insertion as a safety device based on the balance of terminal endings or parentheses / quotation marks, and inserts a token indicating the paragraph boundary. Claim 4 An unstructured legal book data processing system according to claim 1, wherein the server detects a bounding box of a table on a page, removes body text overlapping with said bounding box, generates said table in a Markdown table format and inserts it at a position corresponding to said bounding box to replace said table. Claim 5 An unstructured legal book data processing system according to claim 1, wherein the server determines whether to combine with the first text of the next page if the last text of the current page does not end with EOP in order to preserve page boundary continuity, and if combination is necessary, connects the two texts by assigning a token indicating a page boundary between them, but prohibits the combination if a table of contents or chapter / section title start pattern is detected. Claim 6 delete Claim 7 In claim 1, for section u of the document, the target chunk length Calculate as shown in the following mathematical formula 2, [Mathematical Formula 2] , above Based on the above Determine the chunk boundary by calculating, and in the above mathematical formula 2, the above is the average paragraph length within interval u, and the above is the standard deviation, and the above is a coefficient, and The value An unstructured legal book data processing system that is a function for cutting. Claim 8 In claim 1, the chunk The boundary continuity of Defined as, and overlap size Calculate as shown in the following mathematical formula 3, [Mathematical Formula 3] , above overlap size Include the token corresponding to in an adjacent chunk by overlapping it, and in the above mathematical formula 3, the above is the overlap ratio coefficient, and the above An unstructured legal book data processing system where is the target chunk length of interval u. Claim 9 In claim 1, the chunk The set of footnote reference expressions detected in Define as, and footnote search margin When defined as such, the chunk is finalized by adjusting the chunk boundaries or expanding the footnote search range to satisfy the following Equation 4, [Equation 4] , in the above mathematical formula 4, the above is a paragraph It is a set of footnote identifiers corresponding to, and the above An unstructured legal book data processing system that is a matching function including footnote number normalization. Claim 10 A method for a server to process RAG-based unstructured legal book data comprises: a step of acquiring an unstructured legal book file; a step of storing configuration information including the layout and processing conditions of the unstructured legal book file; a step of separating and extracting an index including body text and footnote-related elements from the unstructured legal book file based on the configuration information; a step of normalizing the index to insert paragraph boundary tokens and replace tables; a step of separating the normalized text into search unit chunks; and a step of generating a chunk data file in JSON format containing the chunks, wherein, in the step of separating into chunks, the server [represents] a normalized paragraph sequence Regarding continuous intervals chunk Defined as, and the above chunk The goodness-of-fit score Define as the following mathematical formula 5, and [Mathematical Formula 5] , regarding multiple chunks It includes determining the chunk boundary so that is maximized, and in the above mathematical formula 5, the is a weight, and the above is the average of the cosine similarity between adjacent paragraph embeddings within a chunk, and the above is the degree of inclusion of the footnote reference set within the chunk, and the above is the representativeness between the candidate header embedding and the chunk embedding, and the above is a penalty for the actual number range included in the chunk, and the above is chunk length go A method for processing unstructured legal book data, defined by a penalty for out-of-range cases.

Images