Title-level semantic partitioning method and equipment for multi-document batch processing

Abstract

The invention discloses a title-level semantic partitioning method and device for multi-document batch processing, and the method comprises the steps: obtaining a plurality of to-be-processed input documents, and adding the plurality of input documents into a document processing task queue; allocating quotas for the document visual identification tasks in the document processing task queue to obtain page identification texts and structured contents of the documents; after all the pages of the multiple input documents are recognized, target outline titles are extracted from all the documents on the basis of page recognition texts; determining a corresponding page position of the target outline title in the document according to the page identification text and the structured content of each document; and on the basis of the corresponding page position of the target outline title in the document, executing cross-page segmentation according to the title boundary on the page recognition text of the corresponding document to generate a plurality of title-level semantic blocks capable of keeping the context continuity of the content. Therefore, a processing link from page recognition to structure understanding to semantic partitioning is realized.

Description

Technical Field

[0001] This application relates to the field of document parsing technology, and in particular to a method and device for header-level semantic segmentation for batch processing of multiple documents. Background Technology

[0002] With the continuous growth of digital content, enterprises and organizations need to handle a vast amount of documents in diverse formats, including scanned copies, reports, data entry forms, and business documents with complex layouts. These documents typically require content recognition, structural understanding, and fragment-level content organization during the data entry, archiving, and knowledge extraction processes to support subsequent data analysis and business applications. However, the diverse sources, inconsistent formats, and significant differences in layout make automated processing quite complex in real-world environments.

[0003] Traditional document processing workflows typically rely on fixed rules or single-document processing methods to identify and structurally transform document content. However, these methods often struggle to accommodate the diverse structures present in multi-document batch processing scenarios, failing to effectively address issues such as cross-page layouts, inconsistent heading levels, and blurred chapter boundaries. Furthermore, in large-scale document processing tasks, existing workflows generally lack proper coordination of processing resources, making document processing efficiency susceptible to factors such as task backlog, unbalanced processing order, or uneven resource utilization.

[0004] In the context of structured processing of large volumes of documents, accurately extracting key content that conveys the document's organizational information while maintaining the coherence of paragraphs remains a common challenge in existing technologies. This is especially true when heading levels are irregular, page layouts are unpredictable, and content extends across pages; traditional methods struggle to accurately determine document content boundaries and logical blocks, thus impacting the quality of subsequent document retrieval, classification, and analysis tasks. Summary of the Invention

[0005] This application provides a title-level semantic segmentation method, system, device, storage medium, and program product for multi-document batch processing, which is used to solve at least one of the above-mentioned technical problems.

[0006] In a first aspect, embodiments of this application provide a title-level semantic block method for batch processing of multiple documents, comprising: acquiring multiple input documents to be processed and adding the multiple input documents to a document processing task queue; the document processing task queue maintains multiple document visual recognition tasks, and each document visual recognition task corresponds to at least one input document; allocating quotas to the document visual recognition tasks in the document processing task queue, so that the document visual recognition tasks that have obtained quotas call a visual recognition model to perform page visual recognition processing on the corresponding documents to obtain the page recognition text and structured content of the documents; the structured content includes: page layout structure, the correspondence between page text and page number, and structured tag information of page elements; after all pages of the multiple input documents have completed recognition processing, extracting target outline titles from each document based on the page recognition text; determining the corresponding page position of the target outline title in the document according to the page recognition text and structured content of each document; and performing cross-page segmentation based on the title boundary on the page recognition text of the corresponding document based on the corresponding page position of the target outline title in the document to generate multiple title-level semantic blocks that can maintain the continuity of content context.

[0007] Secondly, embodiments of this application provide a storage medium storing one or more programs including execution instructions, which can be read and executed by electronic devices (including but not limited to computers, servers, or network devices) to perform any of the above-mentioned title-level semantic segmentation methods for multi-document batch processing of this application.

[0008] Thirdly, a computer device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the above-described header-level semantic chunking methods for multi-document batch processing of this application.

[0009] Fourthly, embodiments of this application also provide a computer program product, the computer program product including a computer program stored on a storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to execute any of the above-mentioned title-level semantic block segmentation methods for multi-document batch processing.

[0010] The beneficial effects of the embodiments of this application are as follows: By introducing task queues and quota allocation mechanisms in multi-document batch processing scenarios, the orderly scheduling of document visual recognition tasks is achieved, thereby improving resource utilization efficiency and overall throughput capacity in large-scale document processing. Simultaneously, relying on structured information such as text content, layout structure, and element tags obtained from page visual recognition, the position of target outline titles within the document is accurately determined, enhancing the ability to extract information from documents with diverse formats and significant layout differences. Furthermore, by performing cross-page segmentation based on title boundaries, title-level semantic blocks that maintain semantic continuity and contextual integrity are generated, making the document's structural expression more consistent with natural chapter logic. Thus, a processing chain from page recognition to structural understanding to semantic block segmentation is realized, providing reasonably granular and semantically complete structured foundational data for subsequent document retrieval, classification, and knowledge extraction, significantly improving the overall quality and adaptability of automated multi-document processing. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating an example of a header-level semantic chunking method for multi-document batch processing according to an embodiment of this application is shown. Figure 2 This document illustrates an example of an operation flowchart for allocating quotas to document visual recognition tasks in a document processing task queue according to an embodiment of this application. Figure 3 A flowchart illustrating an example of allocating an initial quota based on the total base quota according to an embodiment of this application is shown. Figure 4 A flowchart illustrating an example of visual recognition processing according to an embodiment of this application is shown. Figure 5 A flowchart illustrating an example of extracting a target outline title based on page recognition text according to an embodiment of this application is shown. Figure 6 A flowchart illustrating an example of determining the corresponding page position of a target outline title in a document according to an embodiment of this application is provided. Figure 7 A flowchart illustrating an example of cross-page header-level semantic segmentation according to an embodiment of this application is shown. Figure 8 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0014] It should also be noted that, in this document, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0015] It should be noted that, with the widespread application of artificial intelligence, document data has become significantly more important in scenarios such as intelligent retrieval, knowledge-based question answering, enterprise knowledge base construction, and intelligent agent reasoning. However, real-world business involves a wide variety of document types, including native electronic documents, PDFs converted from Word or LaTeX, scanned documents, photographs, and documents in mixed formats. These different documents vary in layout, structural identifiers, page distribution, and element arrangement, making document processing extremely complex. Many document processing tools are based on early text stream parsing approaches, failing to fully consider the requirements of modern AI models for semantic consistency, structured expression, and context preservation.

[0016] Currently, related technologies mainly fall into two categories: one is rule-based PDF text parsing tools, such as PyPDF2 and pdfplumber, which rely on the internal object structure or coordinates of the document to extract text, but are completely unable to handle scanned documents, image-based documents, or pages with complex structures; the other is commercial cloud services based on OCR, such as Google DocumentAI, which can recognize scanned documents, but lack semantic segmentation capabilities for AI retrieval and RAG (Retrieval-Augmented Generation) scenarios. Regardless of the method used, documents are generally treated as several independent pages, and content is extracted sequentially without considering the semantic coherence of content across pages, and there is a lack of ability to organize content based on heading hierarchy or document structural logic.

[0017] In practical applications, these methods also have several typical limitations: First, when a chapter, paragraph, or topic in a document is distributed across multiple pages, current splitting strategies often truncate it directly along the page boundaries, resulting in fragmented semantic units and affecting the understanding of downstream AI models; Second, structured elements such as tables and images in the document are often ignored or need to be processed independently, failing to form an overall semantic context with adjacent text; Third, common chunking methods are usually based on simple strategies such as fixed character length, fixed word count, and fixed pages, failing to organize based on semantic units. The generated results are either too large, leading to inaccurate retrieval, or too small, leading to semantic fragmentation, making them unsuitable for scenarios such as RAG and Agent that require high semantic integrity.

[0018] Furthermore, current document processing systems generally lack unified and scalable multi-format processing capabilities. Native PDFs, Office documents, and scanned documents differ in their structural parsing, page number correspondence, and layout understanding, forcing traditional systems to employ multiple processing strategies and hindering the formation of a unified workflow. Moreover, especially in Office documents, the correspondence between text and page numbers depends on formatting elements such as page breaks. Actual documents often do not strictly adhere to these standards, making it difficult to accurately establish page-level mappings and lacking effective methods for title recognition and structural positioning.

[0019] In large-scale document batch processing scenarios, another prominent issue is the difficulty in dynamically matching processing efficiency with hardware computing power. Some systems employ fixed concurrency levels or static task queue models, which cannot automatically adjust the processing flow based on real-time computing resources. This results in slower processing speeds under high loads and an inability to fully utilize available computing power when resources are idle, leading to low overall performance utilization.

[0020] In the context of building enterprise-level RAG systems, commonly used document processing tools in related technologies generally adopt content segmentation methods based on fixed lengths or simple rules. The quality of the generated segments is low, making it difficult to accurately reflect the semantic boundaries and structural logic of the document, thus directly affecting the matching accuracy of vector retrieval and the quality of question answering based on the retrieval results. Through practice, the inventors of this application have discovered that the root of the problem lies in the lack of recognition and preservation of the semantic structure of documents in existing solutions. The "extract first, then segment" processing mode, which converts the document content into plain text before segmenting it, will destroy the original layout relationship. Especially in common scenarios such as content spanning multiple pages, paragraphs spanning multiple blocks, and separation of headings and body text, it is even more difficult to preserve the inherent semantic continuity and hierarchical structure of the document, resulting in the final knowledge blocks failing to accurately carry the meaning of the original text.

[0021] It should be understood that the above description of the relevant technologies is intended only to help the public better understand the inventive spirit and motivation of this application, and is not intended to limit this application. Furthermore, the technical solutions described in the above-mentioned relevant technologies are not prior art, and may also be undisclosed technical solutions, such as those under research or in the laboratory stage.

[0022] The technical solutions in this application, including the collection, storage, use, processing, transmission, provision, and disclosure of users' personal information, comply with relevant laws and regulations and do not violate public order and good morals.

[0023] Figure 1 A flowchart illustrating an example of a header-level semantic chunking method for multi-document batch processing according to an embodiment of this application is shown.

[0024] Regarding the execution subject of the method in the embodiments of this application, it can be any controller or processor with computing or processing capabilities. In some examples, the method in the embodiments of this application can be integrated and configured in an electronic device or terminal through software, hardware or a combination of software and hardware, and the type of terminal or electronic device can be diverse, such as mobile phone, tablet computer, desktop computer or vehicle terminal, etc.

[0025] For example, the execution entity of the method in this application embodiment can be an integrated document semantic segmentation platform. This platform ensures stable output during the multi-document visual recognition stage through task queues and quota control, supports accurate positioning of outline titles with structured content, and drives cross-page semantic segmentation based on title boundaries. Therefore, document indexing, retrieval, clustering, or knowledge extraction modules built upon this platform can directly utilize title-level semantic segments as basic processing units, thereby achieving an overall improvement in the completeness of retrieved relevant content, the rationality of segmentation granularity, and the accuracy of semantic matching. This results in a positive enhancement from the underlying structured processing to the upper-level business application effects.

[0026] Figure 1 A flowchart illustrating an example of a title-level semantic chunking method for multi-document batch processing according to an embodiment of this application is shown.

[0027] like Figure 1 As shown, in step S110, multiple input documents to be processed are obtained and added to the document processing task queue.

[0028] In scenarios involving batch processing of multiple documents, documents from different sources and in different formats are often input into the system at once. By constructing a document processing task queue, the input documents are converted into schedulable document visual recognition tasks, thus achieving a unified scheduling mechanism for managing the processing status of multiple documents. Here, the document processing task queue maintains multiple document visual recognition tasks, and each document visual recognition task corresponds to at least one input document.

[0029] In some implementations, multiple input documents to be processed are obtained from external data sources (such as batch upload directories, file management systems, or business system interfaces), and a corresponding document visual recognition task is created for each document or group of documents. The task queue uses a first-in-first-out (FIFO) or priority queue structure to maintain the status of each task and records metadata such as document identifier, number of pages to be processed, and estimated processing time. The queue can support dynamic expansion to adapt to newly added document tasks in real time.

[0030] In addition, to cover common file formats in the enterprise environment (such as PDF, scanned images, Office documents, etc.), the system pre-converts the documents into a standardized sequence of page images at this stage to ensure that the visual recognition model can directly process them.

[0031] In step S120, quotas are allocated to document visual recognition tasks in the document processing task queue, so that the document visual recognition tasks that have obtained quotas can call the visual recognition model to perform page visual recognition processing on the corresponding documents to obtain the page recognition text and structured content of the documents.

[0032] It should be understood that the visual recognition model used in page recognition tasks can be diverse, such as OCR and layout analysis models, and this should not be restricted here. Furthermore, visual recognition models are computationally intensive; therefore, effectively scheduling computational resources under limited computing power is crucial for improving processing efficiency. In batch processing environments, multiple tasks compete for limited visual recognition resources. Without restrictions, problems such as some tasks monopolizing resources or task blocking can easily occur. By allocating quotas to document visual recognition tasks in the task queue, controllable resource scheduling can be achieved, maintaining a balance between simultaneous processing of multiple documents and system resource load.

[0033] In some implementations, the visual recognition tasks for each document are traversed in a task queue, and currently available processing quotas are allocated to some tasks based on queue strategies (such as polling, priority, document size, etc.). For example, a quota can be defined as the number of pages a task is allowed to submit at once or the number of recognition model sessions it is allowed to occupy. Tasks that receive quotas are then scheduled to perform OCR and visual layout analysis on the corresponding page images, outputting the recognized text and structured content for the page.

[0034] Here, structured content includes page layout structure, the correspondence between page text and page numbers, and structured tag information for page elements. Specifically, page layout structure can include parameters such as region boxes, paragraph blocks, and heading blocks, used to reflect the spatial organization of text on the page; the correspondence between text lines and page numbers supports tracking content position across pages; and structured tags for page elements can include titles, body text, charts, headers, and footers, used to characterize the semantic category of elements. The recognition results are stored at the document-page hierarchy to ensure that cross-page structures can be accurately referenced subsequently.

[0035] Through a quota allocation mechanism, the system balances the model call pressure, maintaining stable recognition throughput in high-concurrency scenarios and avoiding task backlog caused by resource overload. By generating structured content, not only is the text preserved, but also semantically valuable visual cues are retained.

[0036] In step S130, after the recognition processing is completed on all pages of multiple input documents, the target outline title is extracted from each document based on the page recognition text.

[0037] It should be noted that the goal of cross-document title extraction is to identify key content units that represent the document's organizational structure. However, different documents have significant differences in format and title style, so fixed rules cannot be relied upon. By leveraging text and structural tags generated through visual recognition, title identification no longer depends on a single feature but rather on a comprehensive assessment of semantics, format, and structure.

[0038] In some implementations, after acquiring the text for page recognition, a joint semantic and format analysis is performed. On one hand, an NLP (Natural Language Processing) model is used to classify the text lines as potentially headings, including detecting heading-specific short sentence structures, summarizing words, and line spacing patterns. On the other hand, visual structure (such as font size, bolding, top placement, and first-level heading labels) is combined to improve the reliability of the heading recognition results. For documents with hierarchical numbering, the hierarchy can be automatically identified through numbering patterns; for non-standard headings, semantic aggregation is used to identify whether they constitute chapter entry points through paragraph topic abrupt changes.

[0039] In step S140, the corresponding page position of the target outline title in the document is determined based on the page recognition text and structured content of each document.

[0040] Once the semantic location of the title is determined, it still needs to be precisely positioned on a specific page within the document and within the page's internal structure. Here, a page-title mapping relationship is established based on structured content to ensure that pagination boundaries align with logical content boundaries.

[0041] In some implementations, the text location of the target outline title and its corresponding page number are searched throughout the document. Simultaneously, information such as the title level, spatial adjacency between the title and body text, and the global document layout structure is used to analyze the extent of the content following the title within the page, determine the starting page corresponding to the title, and infer that the content preceding the next title belongs to the current title block. Furthermore, a title position index table is generated, containing the page number corresponding to the title, its coordinate range within the page, and the spatial relationship between the title and adjacent content. By jointly locating the title boundary through page layout structure and text semantics, the accurate page position mapped to the title is ensured, avoiding misclassification or incorrect segmentation of chapter content.

[0042] In step S150, based on the corresponding page position of the target outline title in the document, the page recognition text of the corresponding document is split across pages according to the title boundary to generate multiple title-level semantic blocks that can maintain the continuity of content context.

[0043] It should be noted that traditional page-based segmentation methods struggle to maintain the semantic continuity of chapters, while content in complex documents often expands across pages. By performing page-based segmentation according to title boundaries, semantic units consistent with natural chapter boundaries can be generated, making the segmentation results closer to human reading logic. To form semantically complete and hierarchically clear content fragments, the identified text can be segmented according to the sections defined by the titles, maintaining a natural transition of content even when spanning multiple pages.

[0044] In some implementations, titles are sequentially used as semantic boundaries based on a title position index table to determine the content range controlled by each title in the document. For example, in cross-page segmentation, multiple pages belonging to the same semantic segment are identified and merged. Headers, footers, or pagination marks that only serve a formatting function are removed, ensuring logical continuity of semantic paragraphs and achieving cross-page content linking. Furthermore, if titles have a hierarchical relationship, the segmentation granularity can be determined based on the title level, making each semantic block both independent and structurally clear. Additionally, when merging text, internal paragraph and list structures are preserved, allowing the segmented semantic fragments to be directly used as standard input units for subsequent retrieval or knowledge extraction.

[0045] Ultimately, the generated title-level semantic chunks are output as structural units, each with a clearly defined title, page source, and contextual consistency. For example, standard JSONL entries are generated for each title-level semantic chunk. This method of organizing document content into JSONL chunks based on the title dimension is suitable for incorporating this knowledge into a model through post-training methods to create a vertical domain expert model.

[0046] Through the embodiments of this application, the orderly scheduling of visual recognition tasks is realized in multi-document batch processing scenarios, thereby improving resource utilization efficiency; the target outline title is accurately identified and its page position is determined in documents with complex formats and inconsistent layouts; and semantic blocks that maintain contextual continuity are generated by splitting pages according to title boundaries, so that the document structured results are more in line with natural reading logic, significantly improving the quality and applicability of subsequent document processing and knowledge analysis tasks.

[0047] Figure 2 A flowchart illustrating an example of allocating quotas for document visual recognition tasks in a document processing task queue according to an embodiment of this application is shown.

[0048] like Figure 2 As shown, in step S210, the total basic quota is determined based on the server's hardware computing power configuration, and an initial quota is allocated to each document visual recognition task in the document processing task queue according to the total basic quota.

[0049] Here, the total basic quota is calculated based on the server's computing power configuration, and an initial quota is allocated to each task to achieve fair scheduling during the task startup phase. Specifically, the system first reads the currently available computing resource parameters of the server, including the number of GPUs and their memory capacity, the number of CPU cores, the memory size, and the maximum number of concurrent executions of the visual model. Based on these parameters, the system calculates the total basic quota that can be provided, for example, by dividing it into several schedulable quota units proportionally according to the number of concurrent GPUs.

[0050] The base quota refers to the maximum concurrent number of tasks that the system can run simultaneously, calculated based on the server-side hardware computing power (e.g., GPU model, memory capacity, number of cards, etc.). This total quota is used to distribute quotas across different task execution processes. For example, currently only one GPU card is deployed for the parsing service, but the program is integrated and called from multiple places. The computing power of a single GPU service is limited. If there were no restrictions, the concurrent access to the single GPU service could be very high. Under high concurrency, the GPU cannot work efficiently, and there is a risk of crashing, preventing everyone from achieving the desired results. Quotas ensure that the program's access to the GPU remains controlled and optimally efficient regardless of how many places it is used. If there are eight GPU cards, setting the optimal empirical value for the quota for each of the eight GPU cards is sufficient; if using a consumer-grade processor, setting a very low quota ensures broad computing power compatibility for the program.

[0051] In some implementations, the corresponding base quota can be adapted according to the hardware configuration type. For example, the base quota for GPU3090 can be 32, the base quota for a single H20 card can be 82, and the base quota for 8 H20 cards can be 656.

[0052] Subsequently, based on the number of tasks in the document processing task queue and their estimated number of pages, an initial quota is allocated to each document visual recognition task. The initial quota can be allocated using an average allocation strategy or a weighted allocation based on the task size (number of pages to be recognized), so that each task can start page recognition simultaneously in the early stages of processing, reducing queue congestion.

[0053] By implementing a unified quota allocation mechanism during the task initiation phase, it is ensured that each document visual recognition task can fairly occupy computing resources in the initial stage, avoiding long waiting times for tasks due to uneven resource distribution, thereby improving the overall page recognition throughput and scheduling stability of the system. In step S220, during the execution of page visual recognition processing in each document visual recognition task, the number of completed pages and the number of pages to be processed in each document visual recognition task are periodically counted to obtain the task progress of each document visual recognition task.

[0054] In some implementations, during the execution of all document visual recognition tasks, the task status is polled at fixed time intervals (e.g., every 0.5-2 seconds) to read the number of completed pages and the number of pages to be processed for each task. The number of completed pages represents the number of pages for which structured content has been successfully recognized and output, while the number of pages to be processed represents the number of pages for which recognition has not yet been completed. This number can be updated in real time based on the total number of pages in the document. Furthermore, the system calculates the task progress of each task based on the above data (e.g., number of completed pages / total number of pages) and records its rate of change to determine whether a task is in the final stages of execution.

[0055] In step S230, for each document visual recognition task, if the progress of the document visual recognition task is detected to exceed the preset tail progress threshold, the allocation quota for the document visual recognition task is increased to update the initial quota, so that the document visual recognition task occupies the quota resources released by the completed document visual recognition task, thereby accelerating the document visual recognition task to process the page visual recognition of the remaining document pages.

[0056] It should be noted that during multi-task parallel execution, when some document tasks are about to complete recognition but only a few pages remain unprocessed, if execution continues according to the initially set batch size or resource quota, the actual processing efficiency will significantly decrease because the remaining number of pages is insufficient to fully utilize the processing power of the visual recognition model (such as GPU). This unnecessarily prolongs the task's final stage, thus delaying the overall task completion time. To avoid situations where small batches slow down the overall process at the end of a task, it is necessary to detect whether a task has entered the final stage of execution. Once confirmed, the resource quota or processing concurrency is increased to accelerate task completion with limited computing resources and improve the overall system throughput efficiency.

[0057] In some implementations, the progress of each task, which is periodically monitored, is compared with a preset tail progress threshold (e.g., 95% or other values). When the progress of a document visual recognition task falls below this threshold, it indicates that the task has entered the tail stage but there are still unfinished pages; for example, in a task of recognizing 100 documents, only 2 documents remain unprocessed. At this point, the system identifies the quota resources released by completed tasks and reallocates these quota resources to tasks in the tail stage, enabling tail tasks to invoke the visual recognition model at a higher concurrency rate, accelerating the recognition processing of their remaining pages until the task is completely completed.

[0058] By increasing the quota at the end of task execution through the embodiments of this application, the processing time of the last tasks can be significantly shortened, avoiding delays in the completion of the entire batch of documents due to "slow processing of the last few pages". At the same time, this strategy enables the reuse of resources of completed tasks, allowing system resources to continue to operate at high efficiency, thereby improving the throughput and stability of the entire multi-document processing flow.

[0059] Figure 3 A flowchart illustrating an example of allocating an initial quota based on the total base quota, according to an embodiment of this application, is shown.

[0060] like Figure 3 As shown, in step S310, based on the quota manager, a distributed lock is used to maintain the total global available quota, and the total global available quota does not exceed the total basic quota.

[0061] Here, a distributed lock mechanism is used to protect the total global available quota to ensure the atomicity and consistency of the quota allocation process. Specifically, the system sets up a Quota Manager, which maintains a global available quota. Its value does not exceed the base quota and can be set during the initialization phase based on hardware configuration. It is used to control the quota management when multiple processes run and use program functions, ensuring that all processes accessing the same computing power entry point set by the program comply with the quota management.

[0062] The quota manager uses distributed locks (such as Redis distributed locks or file locks) to protect read and write operations on quota variables, ensuring that only one scheduling thread or task can request or modify the quota value at any given time. Redis distributed locks are used to manage scenarios involving distributed applications, while file locks are used to manage scenarios involving offline or network issues. The globally available quota changes dynamically with the start and end of the document visual recognition task. The locking mechanism ensures that the "read-judgment-deduction" process is an indivisible atomic operation, preventing multiple tasks from simultaneously believing they can occupy the same quota unit.

[0063] By using a distributed lock mechanism, the globally available quota is ensured to remain consistent and reliable in a concurrent environment, avoiding resource conflicts and quota allocation disorder, thus guaranteeing the resource scheduling security and stability of the entire multi-document batch processing system from the underlying layer.

[0064] In step S320, in response to the processing request of the document visual recognition task, a lease application is initiated to the quota manager, and the lease is used to occupy at least one quota unit in the total global available quota.

[0065] Here, to ensure that each document visual recognition task has a valid quota before invoking the visual recognition model, a "lease" mechanism is used to manage quota usage. This gives quota usage lifecycle management capabilities, preventing long-term occupation and resource leakage. A lease is a temporary resource authorization mechanism for task execution. When a task needs computational resources, it can request a lease from the quota manager.

[0066] In some implementations, when a document visual recognition task is ready to start, it can submit a lease request to the quota manager. The lease includes information such as the task ID, request time, number of quota units requested, and lease expiration time. The quota manager checks whether the currently available global quotas meet the allocation requirement for at least one quota unit. If the condition is met, the quota is temporarily locked and a corresponding lease is generated; otherwise, no quota is allocated.

[0067] For example, the quota manager checks active leases at fixed intervals (e.g., every 10 or 30 seconds); active tasks will proactively "renew" their leases once (updating their lease timestamps); leases that expire without renewal will be automatically reclaimed. Therefore, the lease mechanism is renewable and revocable, ensuring that tasks have valid authorization during execution, while also guaranteeing that even if a task exits abnormally, the quota can be reclaimed through the lease expiration mechanism. This makes task resource usage traceable and recoverable, avoiding resource waste.

[0068] In step S330, when the lease application is successful, the quota unit corresponding to the lease is deducted from the total global available quota, and the corresponding document visual recognition task is marked as a task that has obtained quota, so as to activate the page visual recognition processing permission of the document visual recognition task.

[0069] Once a lease application is successful, the quota manager, protected by a distributed lock, deducts the quota unit corresponding to the lease from the total available global quota and marks the lease status as "occupied." Subsequently, the document visual recognition task is marked as "quota acquired," officially gaining the permission to call the visual recognition model to perform page recognition. The task executes according to the lease authorization, continuously occupying the quota unit based on the task progress during execution, thereby calling the visual recognition model, and ensuring that the model instance or computing resources are not contested by other tasks.

[0070] This ensures that quota resource allocation and task execution are strictly linked, preventing tasks without quotas from calling the visual recognition model, thereby maintaining the orderly use of resources and improving the overall stability of the system.

[0071] In step S340, after all pages corresponding to the document visual recognition task have completed the recognition process, the lease is released through the quota manager, and the quota unit corresponding to the lease is returned to the total global available quota.

[0072] In some implementations, once all pages of a document visual recognition task have been processed, the system calls the quota manager to release the lease. Protected by a distributed lock, the quota manager updates the lease status to "released" and returns the corresponding quota unit to the total available global quota, making it available for other tasks to request again. Simultaneously, the system can record the lease's execution time for subsequent optimization of resource allocation strategies or assessment of model load, such as allocating it to tasks in their final stages or new tasks. By reclaiming leases, it ensures that each quota unit can quickly return to the global resource pool after the task's lifecycle ends, effectively avoiding long-term resource occupation, improving overall resource reuse efficiency, and accelerating the startup of the next batch of tasks.

[0073] In step S350, when the lease application fails, the start of the document visual recognition task is postponed, and the lease application is resubmitted to the quota manager after waiting for a preset delay, so as to ensure that the total global available quota is not lower than the safe quota threshold.

[0074] Here, a "delay-retry" mechanism is employed, differentiating between quota pools for regular requests and retry requests. This allows tasks to relinquish resources appropriately when quotas are insufficient, awaiting the next quota release. Specifically, when the quota manager determines that the globally available quota is insufficient to satisfy a lease request, the system will report "lease request failed" to the document visual recognition task. In this case, the task will not terminate immediately but will enter a delayed waiting state. Furthermore, the "delay-retry" mechanism may also be triggered in certain special circumstances, such as when the visual recognition task encounters difficult-to-handle special content during execution, leading to processing failure. In such cases, a retry will also be initiated.

[0075] In some implementations, tasks wait for a preset delay (e.g., 100ms to 1000ms) before resubmitting a lease request to the quota manager after the delay. The system can adjust this based on a security quota threshold to ensure that the globally available quota does not fall below the security quota threshold (e.g., 0 or other preset positive numbers) even under high load, thus guaranteeing system stability and ensuring that subsequent new tasks still have a chance to obtain an execution window.

[0076] By using delay and retry mechanisms, flexible scheduling of tasks is achieved in resource-scarce environments, avoiding jitter or deadlock problems caused by resource contention, enabling the system to run smoothly under high concurrency conditions, and ensuring that tasks can be started in a timely manner after resources become available.

[0077] In some examples of embodiments of this application, the document is processed page by page (each page name has page number information), and after processing, they are merged into a whole document. During this process, page number information is inserted for positioning, and title matching and page number positioning are performed based on this whole document.

[0078] In some implementations, the document type of each input document is first identified.

[0079] It should be noted that in multi-document batch processing scenarios, input documents typically come from diverse sources, including native PDFs, Office documents (such as Word and PowerPoint), scanned PDFs, and single-page or multi-page image files. If document types are not differentiated and a uniform page parsing method is directly applied, problems such as parsing failures, missing layout structures, or image quality failing to meet the requirements of visual recognition models can easily occur. For example, document types can be identified by parsing file header information (MagicNumber), MIME type, file extension, and attempting parsing multiple formats.

[0080] If it is a native PDF document, the PDF parsing engine will extract the page image data corresponding to each page of the document.

[0081] Here, the PDF parsing engine can be diverse, such as pdfplumber, PDFium, and Poppler. Native PDFs can contain rich structural information such as vector text, embedded fonts, and path graphics. If parsed directly as images, layout accuracy or font rendering features may be lost. Therefore, a PDF parsing engine needs to accurately render the page, and the visual API should only be called when necessary (e.g., when the parsed content is inadequate) to generate a high-fidelity image that meets the requirements of the visual recognition model. High-fidelity rendering of native PDFs ensures that the structure of text, tables, headings, and graphics remains consistent after conversion, enabling the visual recognition model to accurately identify the layout and content, thus improving the reliability of subsequent structured content output.

[0082] If it is an Office document, convert the Office document into an intermediate PDF document and extract the page image data corresponding to each page of the intermediate PDF document.

[0083] Office document layouts rely on fonts, paragraph rules, object anchors, and template definitions, lacking a stable direct layout rendering method. Directly capturing screenshots or using low-level parsing can lead to layout distortion, content misalignment, or image loss. Therefore, converting Office documents to intermediate PDF documents normalizes the layout structure of different Office formats, providing a unified source of page image input for visual recognition models. After converting to PDF, the system performs the same page rendering process on the intermediate PDF as on the native PDF to generate standardized page image data.

[0084] If the document is a scanned copy or an image document, the individual pages of the document are normalized by image size to generate the corresponding page image data.

[0085] It should be noted that scanned PDFs or original image documents often have issues such as inconsistent resolution, skewed images, noise, and incomplete image cropping. Directly inputting these into a visual recognition model will significantly reduce the accuracy of text recognition and layout extraction. Therefore, image normalization processing is necessary to ensure that scanned documents from different sources meet a uniform quality standard.

[0086] In some implementations, by invoking the collaborative processing of the OCRFlux and Ovis dual models, the system performs normalization processes on each page of scanned image, including size normalization, orientation correction, grayscale or enhancement processing, and aspect ratio correction. It then extracts the embedded image stream page by page and performs the aforementioned image normalization processes on each page. Therefore, the image quality of the scanned document is significantly improved, enabling the visual recognition model to maintain stable recognition accuracy under varying noise conditions and avoiding the loss or incorrect extraction of structured content due to low-quality scanned documents.

[0087] Here, the page image data is standardized image data that matches the input requirements of the visual recognition model.

[0088] It should be noted that although the page sources of different documents vary—whether from PDF parsing, Office conversion, or standardized scanned documents—the visual recognition model requires a stable input format and must adhere to a unified standard. Specifically, all page image data is converted to have a uniform resolution, uniform color channels (such as RGB three channels), and uniform file format (such as PNG or tensor arrays). The system uses these standardized images as input to the visual recognition model to perform text recognition and layout analysis. Thus, image data standardization enables the visual recognition model to maintain stable inference performance across different document types.

[0089] Figure 4 A flowchart illustrating an example of visual recognition processing according to an embodiment of this application is shown.

[0090] In this embodiment, the visual recognition model adopts a pluggable architecture, including a primary visual recognition model and at least one backup visual recognition model. OCR (Optical Character Recognition) and VQA (Visual Question Answering) models are abstracted into a unified interface, supporting models from different vendors (OCRFlux, Ovis, etc.). Each model can be independently configured with parameters and prompts, and automatically downgrades to a backup model upon failure.

[0091] In step S410, based on a unified model calling interface, the main visual recognition model and each backup visual recognition model are abstracted and encapsulated so that the document visual recognition task does not need to care about the specific model type when calling the visual recognition model.

[0092] Here, OCR and VQA models are abstracted into a unified visual recognition model interface, defining a unified input parameter format (such as page image data, optional prompts, scene labels, etc.) and output format (page recognized text and structured content). For models from different vendors (such as OCRFlux, Ovis, etc.), adaptation layers conforming to this interface specification are implemented respectively. The adaptation layer is responsible for handling model-specific parameter configurations, authentication methods, and API addresses.

[0093] Each model instance supports independent configuration of parameters including: recognition language, layout analysis switch, prompt word template, timeout, and maximum number of retries. The primary visual recognition model and at least one backup visual recognition model are registered to the model management module, which maintains the model priority, availability status, and operational metrics (error rate, response time, etc.) to provide a basis for subsequent primary / backup switchover and degradation strategies.

[0094] By encapsulating the above, the document visual recognition task is decoupled from the specific recognition model. This allows the task module to call the visual recognition model only to face a unified interface, which greatly reduces the dependence on the implementation details of the specific model. This makes it easier to replace, extend or upgrade the model without affecting the business logic, thus improving the maintainability and scalability of the system.

[0095] In step S420, when performing page visual recognition processing on page image data, the main visual recognition model is called first to recognize the page image data. If the recognition result of the main visual recognition model triggers a recognition failure event, the backup visual recognition model is called to recognize or compensate for the same page image data.

[0096] It should be noted that in large-scale document recognition tasks, the primary visual recognition model typically performs better in terms of recognition accuracy and layout understanding, but it is also more sensitive to input quality, network conditions, and other factors. If the primary model fails to recognize content in certain scenarios or its output does not conform to structured content specifications, directly halting processing will result in missing content for that page, affecting the integrity of subsequent title-level semantic blocks. In this embodiment, a strategy of prioritizing the primary model and using a backup model for degradation compensation is adopted to improve coverage of complex scenarios while ensuring overall recognition quality.

[0097] More specifically, when performing page visual recognition processing on page image data, the document visual recognition task prioritizes calling the main visual recognition model through a unified model call interface. During the call process, the system monitors recognition failure events in real time. These failure events indicate at least one of the following situations: an operational exception occurs during the process of calling the main visual recognition model to perform page visual recognition processing (such as a network error or abnormal model service return); the main visual recognition model times out, exceeding the preset maximum waiting time; or the page recognition text and structured content output by the main visual recognition model do not conform to structured content specifications, for example, failing the structured content validator test due to missing key fields, severely abnormal page structure tags, or text content confidence below a threshold.

[0098] When any of the above recognition failure events are detected, the system automatically downgrades and calls a backup visual recognition model according to the model management module configuration, and re-performs recognition on the same page image data. The backup model can be a more specialized image model used to compensate for input images that the main model cannot effectively process. Dedicated processing flows are designed for special scenarios such as red stamp interference, table recognition, and image description. For example, when table content is detected to overlap with a red stamp, a faded version of the image is automatically generated and OCR is retried to ensure complete extraction of the table content. Fine-grained optimization significantly improves the processing quality of complex documents. Furthermore, the system uses the page recognition text and structured content output by the backup model as the final recognition result for that page, or merges and aligns it with the main model's result.

[0099] In some examples of embodiments of this application, when the main visual recognition model triggers a recoverable error (such as a brief network anomaly or momentary service unavailability) during a certain call, the system first performs a preset number of retries on the current model (e.g., 1-2 times). During the retry process, a backoff strategy (such as exponential backoff waiting time) can be adopted to improve the success rate and avoid causing momentary impact on the model service. If a recognition result conforming to the structured content specification is still not obtained after retries with the same model, a second-level retry is triggered, that is, a fallback visual recognition model is called. The backup model is mostly a general model that focuses more on robustness and compatibility. Even if it is slightly inferior to the main model in terms of the fineness of understanding some layouts, it can stably output usable results in complex noise, distortion, or abnormal format scenarios. Furthermore, for pages with obvious abnormal patterns in the output results after recognition, or pages that are judged to be special scenarios by content rule detection (e.g., tables covered by a large area of ​​red stamps, repeated noise tails, severe interference from headers and footers, etc.), the system can trigger a third-level dedicated retry process. This process calls specialized preprocessing modules or dedicated models for specific scenarios. For example, it performs image stamp removal preprocessing on tables covered by red stamps, and then calls a dedicated table recognition model; or it calls the "Super Simplified Cleaner" to clean up the repeating patterns at the end of the recognition results, format the text, and generate a merged Markdown file with page number annotations.

[0100] Therefore, when the main model fails to recognize, degradation compensation is automatically triggered, which significantly reduces the impact of a single model failure or insufficient adaptation capability on the overall recognition process. Through a three-level retry mechanism that is hierarchical and differentiated by type, the system can greatly improve the fault tolerance capability for various temporary anomalies, insufficient model adaptation, and special scenarios while ensuring recognition efficiency, thereby improving the robustness of the entire document processing chain.

[0101] In some examples of embodiments of this application, the rendering of document page images and the calling of the visual recognition model API are performed asynchronously. Specifically, asynchronous PNG rendering and model API calls can be implemented based on asyncio, parallelizing the rendering and recognition processes in a pipeline manner. This allows for the control of concurrency limits and rate limiting within the asynchronous framework, preventing momentary overload on the model service and ensuring the stability of the overall recognition service.

[0102] Figure 5 A flowchart illustrating an example of extracting target outline titles based on page recognition text according to an embodiment of this application is shown.

[0103] In step S510, based on the document's structured content, all text lines in the document that have a numbering pattern, line beginning alignment, or carry heading style tags are located to generate a candidate outline heading set.

[0104] In a multi-source document batch processing scenario, the title levels of documents often have obvious layout features, such as line indentation at the beginning of a line, numbering format, font size differences, etc. Therefore, if relying solely on pure text string search, it is very difficult to stably identify titles in scenarios with inconsistent formats, complex layouts, or language diversity. Structured content (such as page layout structure, element tags, position coordinates, etc.) can complement the deficiencies of pure text methods and make title positioning more accurate.

[0105] In some embodiments, the system extracts text lines from the structured content of each page of the document, and simultaneously obtains the layout features and tag information corresponding to each line of text.

[0106] In one example, it can be detected whether the text line conforms to common paragraph numbering formats, such as: numerical numbering (such as "1.", "2.1"), Chinese numbering (such as "一、", "(一)"), bullet points such as dots, dashes, etc. These structures are usually significant signals of chapter titles.

[0107] In another example, the alignment method at the beginning of the line is detected. Titles generally have strong consistency, such as left alignment, flush left display, or a fixed indentation depth. The X coordinate of the text line can be combined with the page boundary position to identify whether it conforms to the title alignment pattern.

[0108] In yet another example, the title style tags are identified. In structured content, some visual recognition models will output tags such as "heading1", "heading2", "title", etc. The text lines with such tags are added to the candidate set.

[0109] In still another example, the cross-page consistency is detected. For titles that may be located in the cross-page area (such as a number appearing at the end of a page and the text beginning at the beginning of the next page), the system combines logical rules to determine whether it is a truncated title to avoid missed detection.

[0110] Through the above detections, the system generates a candidate outline title set in the document that includes all text lines with title features.

[0111] In step S520, based on a preset invalid title filtering rule, the validity of each candidate title in the candidate outline title set is detected, and the invalid candidate titles are screened out to obtain the target outline title set corresponding to the document.

[0112] Here, the invalid title filtering rule includes at least one of the following: removing text lines located in the header, footer area, or page edge, removing text lines with a length exceeding a preset length threshold or containing consecutive punctuation marks, and removing text lines that do not contain any keyword in the preset set of semantic keywords for the main text.

[0113] The candidate outline title set not only includes the actual outline titles, but may also include header and footer text, table column titles, image caption titles, long sentence text, noise recognition products, etc. By using preset invalid title filtering rules, text lines that do not conform to the title semantics or structural specifications are removed from the candidate set.

[0114] In some implementations, the system performs the following validity check on each candidate title based on invalid title filtering rules.

[0115] In one example, text lines located in the header and footer areas are removed. The coordinate information in the structured content is used to determine whether the text line is located in a fixed area at the top or bottom of the page (e.g., less than a certain threshold from the page edge). Header and footer content is usually unrelated to the chapter and should be excluded.

[0116] In another example, text lines located at page edges or in unusual positions are removed. Text lines located at the left / right page edges or deviating too far from the normal text area are considered layout noise, such as page numbers in scanned documents or side notes.

[0117] In another example, remove text lines that exceed a threshold in length or contain consecutive punctuation marks. Excessively long text (e.g., over 40–60 characters) is generally not a heading, and lines containing consecutive punctuation marks such as “…”, “----”, “ / / / ”, “::::”, etc., are often table of contents decorations or noisy content and should not be considered headings.

[0118] In another example, titles that do not contain semantic keywords for the main text are removed. The system can maintain a preset set of semantic keywords for the main text (such as "Overview", "Background", "Function", "Chapter", "Method", "Appendix", etc.). If the candidate title text does not contain any keywords, it is determined to be an invalid title, such as "Image 1" or "Table Description".

[0119] In another example, text that might belong to a chart title is removed. The system can determine whether a text line is located in the layout area surrounding a table or image based on structured content. If it matches the chart title pattern, it is removed from the candidate title set.

[0120] By using validity filtering, the true logical outline headings of a document can be accurately identified, significantly reducing the problem of chapter boundary offset caused by misidentification. This makes the final generated set of target outline headings more stable and reliable, ensuring that the document can be split across pages and semantically segmented according to the true chapter logic.

[0121] Figure 6 A flowchart illustrating an example of determining the corresponding page position of a target outline title in a document according to an embodiment of this application is shown.

[0122] like Figure 6As shown, in step S610, based on the correspondence between page text and page number in the structured content of the document, the page identification text of each page of the document is concatenated in page number order, and page number boundary markers for identifying page number boundaries are inserted between adjacent page identification texts to obtain concatenated document text with page number boundary markers.

[0123] It should be noted that the target outline title is a text fragment appearing throughout the entire document, while the page recognition text is output in units of "pages". Directly locating titles within each page's text makes it difficult to perform unified matching across pages and subsequent sequence verification. By concatenating the page recognition text in page number order and inserting explicit page number boundary markers between pages, the entire document can be treated as a continuous text while preserving page number position information. This facilitates global string matching of the target outline title and allows for quick deduction of the corresponding page number position after matching.

[0124] In some implementations, the page identification text of each page, along with the page number information corresponding to each text segment, is first read from the structured content. Then, the text of each page is concatenated sequentially according to the page numbers in ascending order, with a preset formatted page number boundary marker inserted between every two adjacent pages. For example, a special placeholder format could be used.<PAGE_BREAK_p> " or "[PAGE_p]", where p represents the current page number.

[0125] Therefore, the generated document concatenation text is a simple concatenation of the recognized text of each page of the document, and each page number boundary marker appears only once in the concatenation text, and the order is consistent with the original document page order, so that the page number to which any character position belongs can be determined by tracing back to the nearest page number boundary marker.

[0126] Therefore, by constructing document concatenation text with page number boundary markers, the originally discrete page-level text is transformed into a continuous text stream with position information, making subsequent string matching and position lookup simple and reliable. This avoids the complex logic of cross-page title positioning while retaining accurate page number boundary information.

[0127] In step S620, the target outline title is matched with the document concatenation text to identify one or more candidate starting positions of the target outline title in the document concatenation text.

[0128] It should be noted that each heading in the target outline heading set is a defined text string. To determine its actual location within the document, string matching of that heading needs to be performed within the constructed document concatenation text. Since headings may appear multiple times in the document (e.g., duplicates in the table of contents and body text, or multiple occurrences of the same-named section), it is necessary to identify all possible candidate starting positions.

[0129] In some implementations, for each target outline heading, string matching is performed starting from the beginning of the concatenated document text. This can be done using an exact match or an approximate match that allows for minor character differences (e.g., handling OCR errors or slight punctuation differences), or by using regular expressions to match the heading text. Whenever a match is successful, the starting index position of the matched segment in the concatenated document text is recorded, and this position is designated as a candidate starting position. Thus, one or more candidate starting positions for the target outline heading are identified within the concatenated text.

[0130] In step S630, if there is only one candidate starting position, the page number boundary of the candidate starting position is marked as an interval to determine the page position of the target outline title in the document.

[0131] When a target outline title has only one candidate starting position in the document's concatenated text, it can be assumed that the title has a unique actual location within the document, eliminating the need for a complex disambiguation process. In this case, the page position of the target outline title can be directly determined by marking the page number boundary interval where the candidate starting position is located.

[0132] In step S640, when there are multiple candidate starting positions, multiple candidate page positions are determined according to the page number boundary marker interval where each candidate starting position is located. The candidate page positions are then filtered based on sequence verification and layout verification. The candidate page positions that simultaneously satisfy both sequence verification and layout verification are determined as the page positions of the corresponding target outline title in the document.

[0133] When multiple candidate starting positions for the target outline title appear in the document's concatenated text (e.g., repeated appearance in the table of contents and body text, or the same title appearing in multiple chapters), directly selecting any one may lead to errors in subsequent chapter boundary identification. In this embodiment, by mapping each candidate starting position to its corresponding candidate page position, and combining the order of the titles in the document with the continuity of the page layout, order verification and layout verification are used to filter the candidate page positions, thereby eliminating candidates that do not conform to the overall logic and determining the true title page position.

[0134] Here, the sequence check requires that the page number sequence of the candidate page positions corresponding to the target outline titles arranged in document order is non-decreasing, and the layout check requires that the vertical coordinate change of the candidate page positions of adjacent target outline titles arranged in document order in the page layout structure does not exceed a preset continuity threshold.

[0135] Regarding the details of sequence validation, for target outline headings arranged in the logical order of the document (e.g., T1, T2, T3, etc.), the system combines the candidate page positions corresponding to each heading into several possible page number sequences. The final selected page number sequence must be a non-decreasing sequence, meaning that a later heading should not appear on a page preceding a previous heading. If a candidate page position causes the overall order to be decreasing or disordered, that candidate position is deemed to violate sequence validation and is discarded.

[0136] Regarding the details of layout validation, among the candidate page positions that pass the sequence validation, the system further utilizes the page layout structure information in the structured content to check the vertical coordinate changes of adjacent target outline headings arranged in document order on the corresponding pages. If the difference in the vertical coordinates of two adjacent headings on the page exceeds a preset continuity threshold (for example, indicating an abnormal position where the second heading is outside the top of the page), the corresponding candidate page position is considered to be a table of contents or a non-body heading, and thus the candidate is eliminated. Only candidate page positions that simultaneously satisfy both sequence and layout validation are ultimately determined as the page position of the target outline heading in the document.

[0137] By combining sequence and layout validation, duplicate headings such as those in the table of contents, headers, and footers can be effectively eliminated in multi-candidate scenarios, prioritizing candidate positions that match the natural reading order and page layout continuity. This ensures consistency between heading location and the actual document chapter organization, reduces interference from incorrect page locations on subsequent cross-page segmentation and semantic block division, and improves the accuracy and robustness of the entire document structure understanding process.

[0138] Figure 7 A flowchart illustrating an example of cross-page header-level semantic segmentation according to an embodiment of this application is shown.

[0139] like Figure 7 As shown, in step S710, each target outline title is traversed according to the document order. The starting position of the current target outline title in the document concatenation text is taken as the block starting point, and the starting position of the next target outline title or the ending position of the document concatenation text is taken as the block ending point.

[0140] It should be noted that in complex document layouts, the correspondence between headings and body text is not limited by page boundaries; the body text may extend from the page containing the heading to the next page or even multiple pages. Traditional page-based segmentation methods can lead to the fragmentation of semantic segments, disrupting the continuity of context.

[0141] In this embodiment, the block segmentation is constructed by using the precise starting position of the target outline title in the document concatenation text and the starting position of the next title, rather than relying on the page structure, thus fundamentally solving the problem of continuity of cross-page blocks.

[0142] In some implementations, the target outline titles are processed sequentially from T1, T2, T3, etc., according to the order in which they appear in the document, ensuring that the block order is consistent with the logical order of the document.

[0143] Then, the starting position of the current target outline title in the document concatenation text is marked as the block start point. If there is a next target outline title, the starting position of that title is the block end point of the current block; if the current title is the last title, the ending position of the document concatenation text is taken as the block end point.

[0144] In step S720, for each target outline title, the starting and ending points of the block in the document concatenation text are determined based on their position ranges in the document concatenation text, and the page number boundary markers are used to determine the starting page number, ending page number, and text range within each page of the corresponding block in the document, so as to generate the title-level semantic blocks of the target outline title.

[0145] In some implementations, the starting page number of a block is determined by tracing back the nearest page number boundary marker from the block's start point, and the ending page number is determined by tracing back the page number boundary marker from the block's end point. For pages located between the start and end pages, the text content from the start point to the end of the page is extracted for the start page, the entire page text is extracted for intermediate pages, and the content from the beginning to the end of the page is extracted for the end page.

[0146] Thus, the system generates a complete title-level semantic block that spans multiple pages and strictly corresponds to the semantic content scope of the title. By using a text block concatenation method based on title boundaries, continuous segmentation of cross-page content is achieved, effectively avoiding the problem of text truncation caused by page boundaries, and ensuring that each title-level semantic block can completely and accurately represent the natural semantic structure of the document.

[0147] Through the embodiments of this application, the semantic integrity is maintained by focusing on the title structure. The title is divided into blocks, and the images under the title are summarized into image descriptions, preserving all semantics, so that the title block becomes a basic complete self-describing unit of content.

[0148] In some examples of embodiments of this application, after generating the original semantic blocks, to improve the block quality in diverse document scenarios, this embodiment further provides semantic block optimization operations, making the final output title-level semantic blocks more stable, semantically more complete, and with less noise. The type of semantic block optimization operation can be diverse, and can be one or more optimization operations.

[0149] In one example, excessively short title-level semantic blocks with a text length less than a first text length threshold are filtered out from each title-level semantic block, and these excessively short title-level semantic blocks are merged with adjacent title-level semantic blocks that have the same parent outline title.

[0150] Specifically, the text length of each block can be calculated. If the text length is less than a first text length threshold (a preset minimum content standard), the block is marked as a semantically ambiguous block. If the parent title of this block is the same as its adjacent block, it is semantically merged with the next or previous adjacent semantic block. After merging, the page range and content range of the merged blocks are recalculated to form a more coherent chapter expression. Thus, by automatically merging semantically ambiguous blocks, noisy or meaningless title content can be eliminated, making the block structure more stable and ensuring that each title-level semantic block has a minimum semantic capacity, thereby improving the accuracy of chapter analysis.

[0151] In another example, excessively long title-level semantic blocks with a block text length greater than a second text length threshold are filtered out from each title-level semantic block, and the excessively long title-level semantic blocks are further segmented according to natural paragraph boundaries; the second text length threshold is greater than the first text length threshold.

[0152] Specifically, the text length of each heading-level semantic block is calculated. If the length of a block exceeds a second text length threshold (greater than the first threshold, indicating the maximum acceptable semantic range), it is considered an overly long semantic block. Then, the natural paragraph structure within the block is identified, such as blank lines between paragraphs, first-line indentation, and paragraph punctuation patterns (e.g., periods, semicolons). Based on the natural paragraph boundaries, the overly long block is divided into multiple reasonable semantic segments, which serve as the secondary block results. Thus, through secondary segmentation of natural paragraphs, the problem of giant semantic blocks caused by document formatting issues can be improved, resulting in more granular and clearly structured blocks.

[0153] It should be understood that the first and second text length thresholds can be preset and adjusted to suit the needs of different business scenarios. This allows for the continuous merging of texts with fewer than a certain number of characters (this number can be adjusted as needed, such as 256 or 512) into a larger block, avoiding empty or fragmented blocks.

[0154] On the other hand, some heading matching failures occur when matching outline headings to page numbers, which is most likely to happen in the Office branch. This is because Office documents need to be converted to PDF for page number positioning of outline headings. After extracting the content from the PDF, the outline will differ from the Office branch, causing some outline headings from both Office and PDF to not match one by one, potentially resulting in some large blocks. For excessively long blocks, a secondary segmentation is performed to resolve some difficulties encountered in the initial segmentation. This maintains a reasonable block length and does not alter the semantic meaning of content aggregation centered on headings.

[0155] In another example, it is detected whether the text content of each heading-level semantic block contains only the target outline title text, and the heading-level semantic block that contains only the same text as the corresponding target outline title and does not contain body paragraphs is marked as an empty content heading block. The empty content heading block is used to indicate that the corresponding heading-level semantic block does not participate in the subsequent semantic business analysis processing based on the heading-level semantic block.

[0156] It should be noted that some heading-level semantic blocks, while formatted correctly, contain only text identical to the heading and no body text. These blocks are typically placeholder headings, table of contents entries, or noise; failure to remove them will affect subsequent semantic inference. Specifically, the entity content of each heading-level semantic block can be examined. If the block content only contains the heading text itself and no body paragraphs, it is marked as an empty content heading block. Empty content heading blocks will not participate in subsequent semantic block-based content analysis, summarization, classification, or knowledge extraction operations. Thus, by identifying empty content heading blocks, the system can reduce block noise, increase the analysis task's focus on the true structure of chapters, and avoid generating erroneous or redundant chapter semantic information.

[0157] Based on the business application scenarios of this application, after completing the title-level semantic segmentation, several post-processing mechanisms are further introduced to enhance the usability and robustness of the segmentation results in human reading and agent reasoning scenarios. First, the initially extracted outline titles are format-filtered to generate a concise outline that is human-readable. Specifically, the system automatically removes title texts containing only numbers, complex punctuation combinations, or that are too short, ensuring that the retained titles have clear semantic indicative and chapter organization significance. This operation effectively avoids the interference of format noise on document structure understanding, making the final outline result more suitable for actual reading needs.

[0158] After obtaining a stable outline structure, the system further establishes a solid correspondence between the generated title-level semantic blocks and the original document pages. Since the system has determined the physical page number of each title during the title location phase, the page number of the block can be directly mapped to the physical page of the original document, achieving bidirectional traceability from semantic blocks to the original content. This design not only ensures that the block content can be quickly verified in the original document, but also provides human users with an intuitive and convenient reference when reading or verifying information.

[0159] To address the typical needs of agents in RAG (Retrieval-Augmented Generation) scenarios, a "gent-ready" chunked metadata system was constructed to provide high-quality, referable, structured content for subsequent intelligent question answering. The system injects key metadata such as chunk number, title, character count, corresponding physical page number, and the path to the PNG image obtained from the original page rendering into each semantic chunk. This allows the agent to quickly locate relevant knowledge fragments based on the user's question and load the corresponding content as context for reasoning. Furthermore, because the chunked content maintains semantic integrity through a cross-page aggregation mechanism, it avoids the semantic breaks caused by traditional page-based segmentation methods. Therefore, it avoids issues such as missing content referenced by the agent or incomplete context, significantly improving the accuracy and reliability of the answers.

[0160] The final semantic chunking results are output in a structured format, with each semantic chunk carrying its source filename, chunk title, content length, corresponding page number, and access path to the corresponding page image. This mechanism not only supports the front-end system in quickly locating and reconstructing the original page scene when displaying chunk content, facilitating manual verification, but also provides complete traceable information for subsequent intelligent applications such as full-text search, knowledge extraction, and content comparison. Overall, this application, through cross-page semantic aggregation, title filtering, human-readable outline generation, and agent-ready metadata design, achieves a synergistic improvement in semantic continuity, interpretability, and verifiability in structured document processing, providing a high-quality data foundation for large-scale document understanding and intelligent question-answering systems.

[0161] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0162] In some embodiments, this application also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform any of the above-described header-level semantic chunking methods for multi-document batch processing.

[0163] In some embodiments, this application also provides an electronic device including: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a header-level semantic chunking method for multi-document batch processing.

[0164] The apparatus described in the embodiments of this application can be used to execute the title-level semantic segmentation method for multi-document batch processing according to the embodiments of this application, and accordingly achieve the technical effects achieved by the title-level semantic segmentation method for multi-document batch processing described in the embodiments of this application, which will not be elaborated here. In the embodiments of this application, the relevant functional modules can be implemented using a hardware processor.

[0165] Figure 8 This is a schematic diagram of the hardware structure of an electronic device that performs a title-level semantic chunking method for multi-document batch processing, as provided in another embodiment of this application. Figure 8 As shown, the device includes: One or more processors 810 and memory 820, Figure 8 Take the 810 processor as an example.

[0166] The device for performing a header-level semantic chunking method for multi-document batch processing may further include an input device 830 and an output device 840.

[0167] The processor 810, memory 820, input device 830, and output device 840 can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0168] The memory 820, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the title-level semantic segmentation method for multi-document batch processing in the embodiments of this application. The processor 810 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 820, thereby implementing the title-level semantic segmentation method for multi-document batch processing described in the above-described method embodiments.

[0169] The memory 820 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the device. Furthermore, the memory 820 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 820 may optionally include memory remotely located relative to the processor 810, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0170] Input device 830 can receive input digital or character information and generate signals related to user settings and function control of the device. Output device 840 may include display devices such as a display screen.

[0171] The one or more modules are stored in the memory 820, and when executed by the one or more processors 810, the title-level semantic block segmentation method for multi-document batch processing in any of the above method embodiments is executed.

[0172] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0173] The electronic devices in this application embodiments exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.

[0174] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0175] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players (such as iPods), handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0176] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0177] (5) Other electronic devices with data interaction functions.

[0178] In some embodiments, this application also provides a mobile platform on which the computer device described in any embodiment of this application is installed. The mobile platform includes, but is not limited to, vehicles, tracked robots, bipedal robots, quadrupedal robots, etc., wherein the vehicle can be a passenger car, pickup truck, truck, etc. It should be noted that the above are merely examples, and this application does not limit the specific form of the mobile platform.

[0179] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0180] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A title-level semantic segmentation method for multi-document batch processing, comprising: Obtain multiple input documents to be processed and add them to the document processing task queue; The document processing task queue maintains multiple document visual recognition tasks, and each document visual recognition task corresponds to at least one input document; Allocate quotas to document visual recognition tasks in the document processing task queue so that document visual recognition tasks that obtain quotas can call the visual recognition model to perform page visual recognition processing on the corresponding document to obtain the page recognition text and structured content of the document; the structured content includes the page layout structure, the correspondence between page text and page number, and the structured tag information of page elements; After all pages of the multiple input documents have been identified, the target outline title is extracted from each document based on the identified text on the page. Based on the page recognition text and structured content of each document, determine the corresponding page position of the target outline title in the document; Based on the corresponding page position of the target outline title in the document, the page recognition text of the corresponding document is split across pages according to the title boundary to generate multiple title-level semantic blocks that can maintain the continuity of content context.

2. The method according to claim 1, wherein, The allocation of quotas for document visual recognition tasks in the document processing task queue includes: The total basic quota is determined based on the server's hardware computing power configuration, and an initial quota is allocated to each document visual recognition task in the document processing task queue according to the total basic quota. During the visual recognition process of each document visual recognition task, the number of completed pages and the number of pages to be processed in each document visual recognition task are periodically counted to obtain the task progress of each document visual recognition task. For each document visual recognition task, if the progress of the document visual recognition task is detected to exceed a preset tail progress threshold, the allocation quota for the document visual recognition task is increased to update the initial quota, so that the document visual recognition task occupies the quota resources released by the completed document visual recognition tasks, thereby accelerating the document visual recognition task to process the page visual recognition of the remaining document pages.

3. The method according to claim 2, wherein, The step of allocating an initial quota to each document visual recognition task in the document processing task queue based on the total basic quota includes: Based on the quota manager, a distributed lock is used to maintain the total global available quota, and the total global available quota does not exceed the total basic quota. In response to a document visual recognition task processing request, a lease application is initiated to the quota manager, and the lease is used to occupy at least one quota unit in the total global available quota. When a lease application is successful, the quota unit corresponding to the lease is deducted from the total global available quota, and the corresponding document visual recognition task is marked as a task that has obtained a quota, so as to activate the page visual recognition processing permission of the document visual recognition task. After all pages corresponding to the document visual recognition task have completed the recognition process, the lease is released through the quota manager, and the quota unit corresponding to the lease is returned to the total global available quota. When a lease application fails, the start of the document visual recognition task is postponed, and a new lease application is sent to the quota manager after a preset delay to ensure that the total global available quota is not lower than the security quota threshold.

4. The method according to claim 1, wherein, After obtaining multiple input documents to be processed, the method further includes: Identify the document type of each input document; If it is a native PDF document, the page image data corresponding to each page of the document will be extracted through the PDF parsing engine; If it is an Office document, the Office document is converted into an intermediate PDF document, and the page image data corresponding to each page of the intermediate PDF document is extracted; If it is a scanned document or an image document, the individual pages of the document are normalized by image size to generate the corresponding page image data; The page image data is standardized image data that matches the input requirements of the visual recognition model.

5. The method according to claim 1 or 4, wherein, The visual recognition model adopts a pluggable architecture, including a primary visual recognition model and at least one backup visual recognition model. The method further includes: Based on a unified model calling interface, the main visual recognition model and each of the backup visual recognition models are abstracted and encapsulated so that the document visual recognition task does not need to care about the specific model type when calling the visual recognition model; When performing page visual recognition processing on page image data, the main visual recognition model is called first to recognize the page image data. If the recognition result of the main visual recognition model triggers a recognition failure event, the backup visual recognition model is called to recognize or compensate for the same page image data. The recognition failure event is used to indicate at least one of the following situations: an operational exception occurs during the process of calling the main visual recognition model to perform page visual recognition processing, the main visual recognition model times out, or the page recognition text and structured content output by the main visual recognition model do not conform to the structured content specification.

6. The method according to claim 1, wherein, The extraction of target outline titles from each document based on page-based text recognition includes: Based on the structured content of the document, locate all text lines in the document that have a numbering pattern, first-line alignment, or carry heading style tags to generate a set of candidate outline headings; The validity of each candidate title in the candidate outline title set is checked based on the preset invalid title filtering rules, and invalid candidate titles are filtered out to obtain the target outline title set corresponding to the document. The invalid title filtering rules include at least one of the following: removing text lines located in the header, footer area or page edge; removing text lines whose text length exceeds a preset length threshold or contains consecutive punctuation marks; and removing text lines that do not contain any keyword in the preset set of semantic keywords for the main text.

7. The method according to claim 1, wherein, The step of determining the corresponding page position of the target outline title in the document based on the page recognition text and structured content of each document includes: Based on the correspondence between page text and page number in the structured content of the document, the page identification text of each page of the document is concatenated in page number order, and page number boundary markers are inserted between adjacent page identification texts to identify page number boundaries, so as to obtain the document concatenated text with page number boundary markers. The target outline title is matched with the document concatenation text to identify one or more candidate starting positions of the target outline title in the document concatenation text; If there is only one candidate starting position, mark the page number boundary of the candidate starting position to determine the page position of the target outline title in the document; When there are multiple candidate starting positions, multiple candidate page positions are determined according to the page number boundary mark interval where each candidate starting position is located. The candidate page positions are then filtered based on sequence verification and layout verification. The candidate page positions that simultaneously satisfy both the sequence verification and the layout verification are determined as the page positions of the corresponding target outline title in the document. The sequence verification requires that the page number sequence of the candidate page positions corresponding to the target outline titles arranged in document order is non-decreasing, and the layout verification requires that the vertical coordinate change of the candidate page positions of adjacent target outline titles arranged in document order in the page layout structure does not exceed a preset continuity threshold.

8. The method according to claim 7, wherein, Based on the corresponding page position of the target outline title in the document, the page recognition text of the corresponding document is segmented across pages according to the title boundaries to generate multiple title-level semantic blocks that can maintain the continuity of content context, including: Traverse each target outline title in the document order, take the starting position of the current target outline title in the document concatenation text as the block start point, and take the starting position of the next target outline title or the ending position of the document concatenation text as the block end point. For each target outline title, the starting and ending points of the block in the document's concatenated text are determined based on their position ranges within the document's concatenated text, combined with the page number boundary markers, to generate the title-level semantic blocks of the target outline title.

9. The method according to claim 8, wherein, After generating the heading-level semantic blocks of the target outline headings, the method further includes performing a semantic block optimization operation, which includes at least one of the following operations: Filter out excessively short title-level semantic blocks whose block text length is less than the first text length threshold, and merge the excessively short title-level semantic blocks with adjacent title-level semantic blocks that have the same parent outline title; Filter out excessively long heading-level semantic blocks whose text length exceeds a second text length threshold, and perform secondary segmentation on these excessively long heading-level semantic blocks according to natural paragraph boundaries; the second text length threshold is greater than the first text length threshold; or The system detects whether the text content of each heading-level semantic block contains only the target outline title text, and marks the heading-level semantic block that contains only the same text as the corresponding target outline title and does not contain body paragraphs as an empty content heading block. The empty content heading block is used to indicate that the corresponding heading-level semantic block will not participate in the subsequent semantic business analysis processing based on the heading-level semantic block.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory, wherein, The processor executes the computer program to implement the steps of the method according to any one of claims 1-9.

Images