Multimodal document data processing methods and systems for training large language models

Abstract

This invention discloses a method and system for processing multimodal document data for training large language models. The method includes: receiving several original documents in various formats; extracting structural information from each original document; identifying text and image regions of each original document based on the structural information; employing a parallel OCR recognition engine based on GPU acceleration and heterogeneous computing to perform optical character recognition on the text and image regions, generating corresponding recognized text data for the original documents; performing multi-dimensional quality assessment and cleaning on the recognized text data of each original document, outputting standardized text data; storing the standardized text data in a distributed knowledge base according to a predefined structure, and performing copyright and compliance checks on the standardized text data. By employing a parallel OCR recognition engine based on GPU acceleration and heterogeneous computing, efficient and high-precision batch processing of multimodal documents is achieved, improving processing speed, recognition accuracy, and data quality.

Description

Technical Field

[0001] This invention relates to the field of multimodal data processing, and in particular to a method and system for processing multimodal document data for training large language models. Background Technology

[0002] With the rapid development of generative artificial intelligence technology, natural language processing technologies, represented by large language models, have become a core driving force for promoting the intelligent transformation of industries. These models, pre-trained on massive amounts of high-quality text corpora, have acquired powerful language understanding and generation capabilities, and their performance directly depends on the scale and quality of the training data. Currently, mainstream large language models typically require TB-level or even PB-level structured text data for training. However, most available data in the real world exists in the form of unstructured multimodal documents, including native PDFs, PPT presentations, scanned image files, and other formats. These documents contain rich textual information, but existing technologies face systemic technical challenges in efficiently, accurately, and in batches converting them into high-quality corpora suitable for training large language models.

[0003] In terms of processing efficiency, existing document processing technologies suffer from significant bottlenecks. Most traditional document processing tools employ a single-threaded serial processing mode, typically processing only 0.5 to 1 image per second when handling scanned PDF documents and PPT files with complex layouts. When faced with the need to process terabytes of document data, the entire processing cycle can last for months, completely failing to meet the timeliness requirements of rapid iterative training for large language models. This bottleneck in processing efficiency severely restricts the development cycle and iteration speed of large language models, necessitating breakthrough technical solutions to improve the throughput of the entire data processing workflow.

[0004] Traditional optical character recognition (OCR) technology has significant limitations in terms of recognition accuracy. When processing complex layouts, low-quality scanned documents, documents containing mathematical formulas, and multilingual documents, the accuracy rate of traditional OCR technology is generally below 85%. Especially when processing academic papers, technical documents, and other documents containing a large number of technical terms, special symbols, and complex tables, the error rate can even exceed 20%. These errors directly lead to a decline in the quality of training data, which in turn affects the understanding ability and generation quality of large language models. Therefore, improving text recognition accuracy in complex scenarios has become an urgent technical challenge.

[0005] In terms of computing resource utilization, existing systems suffer from severe underutilization. Most traditional OCR processing tools still rely on serial computation based on the central processing unit (CPU), failing to fully utilize the parallel computing advantages of dedicated hardware such as graphics processing units (GPUs). This architectural design leads to the idle and wasted high-performance computing resources such as GPUs, resulting in low overall system throughput and an inability to fully leverage the performance potential of modern hardware. Furthermore, existing systems lack effective resource scheduling and management mechanisms, hindering dynamic optimization based on task characteristics and system load, further reducing resource utilization efficiency.

[0006] At the data quality control level, existing technical solutions lack a systematic quality assurance mechanism. Traditional processing workflows often focus only on text extraction itself, neglecting the quality assessment and cleaning of the extracted results. This results in a large amount of noisy data (such as headers and footers, watermarks, and meaningless characters), duplicate content, and formatting errors being mixed into the final training set, severely impacting the training performance and generalization ability of large language models. The lack of unified quality standards and a grading system makes it difficult to quantify and assess data quality, and also makes it impossible to precisely control data quality according to different application scenarios.

[0007] In terms of system scalability, existing solutions struggle to meet the demands of large-scale data processing. Traditional architectures lack effective distributed processing capabilities, failing to support parallel processing of petabyte-scale data. Furthermore, these systems exhibit significant shortcomings in load balancing and fault tolerance. This limitation in scalability makes it difficult for existing technologies to meet the stringent data scale requirements of training large language models, becoming another technological bottleneck restricting the development of artificial intelligence. Summary of the Invention

[0008] The purpose of this invention is to provide a method and system for processing multimodal document data for training large language models. By adopting a parallel OCR recognition engine based on GPU acceleration and heterogeneous computing, combined with a multi-dimensional quality assessment and cleaning mechanism, it achieves efficient and high-precision batch processing of multimodal documents, significantly improving data processing speed, recognition accuracy and data quality, and effectively meeting the needs of large language model training for the preparation of high-quality structured corpora.

[0009] To address the aforementioned technical problems, a first aspect of this invention provides a method for processing multimodal document data for training large language models, comprising the following steps:

[0010] Receive several original documents in various formats, extract the structural information of each original document, and identify the text region and image region of each original document based on the structural information;

[0011] A parallel OCR recognition engine based on GPU acceleration and heterogeneous computing is used to perform optical character recognition on the text region and the image region to generate the corresponding recognized text data of the original document;

[0012] The identified text data of each original document is subjected to multi-dimensional quality assessment and cleaning, and standardized text data that meets the quality level is output.

[0013] The standardized text data is stored in a distributed knowledge base according to a predefined structure, and the standardized text data is then checked for copyright and compliance.

[0014] Furthermore, the parallel OCR recognition engine based on GPU acceleration and heterogeneous computing performs optical character recognition on the text region and the image region to generate corresponding recognized text data of the original document, including:

[0015] Each of the text regions and the image regions is preprocessed to obtain a corresponding image to be recognized;

[0016] The text regions and image regions of several original documents are combined into several batches of images to be recognized, and each batch of images to be recognized is decomposed into multiple independent OCR recognition subtasks, with each OCR recognition subtask corresponding to an image to be recognized.

[0017] Based on a heterogeneous computing platform consisting of CPU and GPU, multiple independent OCR recognition subtasks are distributed in parallel to multiple computing threads for processing through a multi-thread scheduler.

[0018] The optical character recognition model based on an improved deep convolutional neural network is invoked to process each OCR recognition subtask. The sequence recognition module based on the Transformer architecture performs text sequence decoding and error correction on the output of the optical character recognition model to generate intermediate text data and its corresponding confidence score.

[0019] The intermediate text data is filtered according to a preset confidence threshold, and the intermediate text data with a confidence score lower than the preset confidence threshold is removed. The intermediate text data with a confidence score greater than or equal to the preset confidence threshold are then aggregated into the recognition text data of the corresponding original document.

[0020] Further, the step of combining all the images to be identified corresponding to the text regions and image regions of several original documents into several batches of images to be identified includes:

[0021] Real-time monitoring of the GPU's current available video memory capacity and computing core utilization;

[0022] Based on the currently available video memory capacity and computing core utilization, the batch size of each batch of images to be identified is dynamically adjusted;

[0023] Based on the batch size, the images to be identified are combined into several batches of images to be identified, and the number of images in each batch of images to be identified is matched with the real-time processing capability of the GPU.

[0024] Furthermore, the invocation of an optical character recognition model based on an improved deep convolutional neural network to process each of the OCR recognition subtasks includes:

[0025] At least two different optical character recognition models are invoked in parallel to recognize the same image to be recognized, and each model outputs a recognition result containing a character sequence and its corresponding character-level confidence.

[0026] The character sequences output by each optical character recognition model are aligned based on the dynamic time warping algorithm to establish the correspondence between character positions;

[0027] For each aligned character position, compare the characters recognized by all optical character recognition models at that character position with their corresponding character-level confidence scores;

[0028] The character with the highest character-level confidence at the specified character position is selected as the character at that position in the final recognition sequence;

[0029] The selected characters at all positions are combined in sequence to obtain the final recognition result of the image to be recognized.

[0030] Furthermore, the filtering of the intermediate text data based on a preset confidence threshold includes:

[0031] Identify the original region type corresponding to the intermediate text data, where the region type includes ordinary text, mathematical formulas, and technical terminology text;

[0032] The corresponding confidence threshold is dynamically selected based on the region type. The first confidence threshold for the mathematical formula and technical term text region is lower than the second confidence threshold for the ordinary text region.

[0033] The intermediate text data generated from the original region type is filtered based on the confidence threshold corresponding to each region type.

[0034] Furthermore, the invocation of an optical character recognition model based on an improved deep convolutional neural network to process each of the OCR recognition subtasks includes:

[0035] The weight parameters of the input layer, output layer and batch normalization layer in the deep convolutional neural network are retained in single-precision floating-point format, while the weight parameters of the intermediate convolutional layers are converted to half-precision floating-point format to obtain a mixed-precision model.

[0036] Optical character recognition inference computation of the mixed-precision model is performed on the GPU, wherein the weight parameters in half-precision floating-point format are accelerated using the dedicated half-precision computing unit of the GPU.

[0037] Furthermore, prior to employing the parallel OCR recognition engine based on GPU acceleration and heterogeneous computing, the following steps are also included:

[0038] A priority score is calculated based on at least one of the following parameters for each original document: document size, format complexity, and user-specified priority.

[0039] Based on the priority scores, the plurality of original documents are sorted into a processing queue;

[0040] Real-time monitoring of CPU utilization and memory usage;

[0041] Based on the CPU utilization and memory usage, the rate and number of concurrent submissions of original documents from the processing queue to the parallel OCR recognition engine are dynamically adjusted.

[0042] Furthermore, the multi-dimensional quality assessment and cleaning of the identified text data for each original document includes:

[0043] The identified text data is subjected to noise content filtering, duplicate content detection, and language model quality assessment in sequence.

[0044] The noise content filtering is based on regular expression rules and machine learning classifiers to identify and clean up noisy text, including headers, footers, and watermarks. The duplicate content detection is based on the Locality Sensitive Hash (LSH) algorithm to calculate the LSH value for each text unit, and identifies and removes duplicate or highly similar text content by comparing the LSH values. The language model quality assessment is achieved by calling a preset language model to calculate the grammatical correctness score and semantic coherence score of the identified text data, and obtains a comprehensive quality score based on the weighted combination of the two.

[0045] Furthermore, the output, which conforms to a quality level, includes normalized text data, including:

[0046] Based on the comprehensive quality score and the results of noise content filtering and duplicate content detection, the identified text data is divided into three quality levels;

[0047] Among them, texts with a comprehensive quality score greater than the first threshold and passing noise filtering and duplicate content detection are rated as Grade A; texts with a comprehensive quality score greater than the second threshold but not greater than the first threshold and passing noise filtering are rated as Grade B; and texts with a comprehensive quality score greater than the third threshold but not greater than the second threshold but failing duplicate content detection are rated as Grade C.

[0048] Only the texts rated A and B are output as the normalized text data.

[0049] Accordingly, a second aspect of the present invention provides a multimodal document data processing system for training large language models, which identifies multimodal documents based on the above-described multimodal document data processing method for training large language models, including:

[0050] A data receiving module is used to receive several original documents in multiple formats, extract the structural information of each original document, and identify the text region and image region of each original document based on the structural information.

[0051] The OCR recognition module is used to perform optical character recognition on the text region and the image region using a parallel OCR recognition engine based on GPU acceleration and heterogeneous computing, and generate corresponding recognition text data of the original document.

[0052] The data processing module is used to perform multi-dimensional quality assessment and cleaning on the identified text data of each original document, and output standardized text data that meets the quality level.

[0053] The data verification module is used to store the standardized text data in a distributed knowledge base according to a predefined structure, and to perform copyright and compliance verification on the standardized text data.

[0054] Accordingly, a third aspect of the present invention provides an electronic device, including: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the above-described multimodal document data processing method for training large language models.

[0055] Accordingly, a fourth aspect of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-described multimodal document data processing method for training large language models.

[0056] The above-described technical solutions of the embodiments of the present invention have the following beneficial technical effects:

[0057] 1. By constructing a parallel processing architecture based on GPU acceleration and heterogeneous computing, a qualitative leap in the efficiency of multimodal document data processing was achieved. A multi-threaded scheduler was used to dynamically decompose OCR recognition tasks into parallel subtasks. Combined with mixed-precision computing and dynamic memory management technology, the advantages of GPU hardware parallel computing were fully utilized, enabling a processing speed of 3-5 images per second, an improvement of over 500% compared to traditional single-threaded processing methods. Simultaneously, system resource utilization was improved by over 75%, completely solving the efficiency bottleneck problem in the preparation of large language model training data.

[0058] 2. By establishing a multi-level intelligent quality control system, the quality and reliability of the output text data are significantly improved. The system integrates a triple guarantee mechanism of noise filtering, duplicate content detection and language model quality assessment. The confidence threshold strategy based on region type adaptation ensures that different content adopts differentiated quality control standards. Combined with a three-level quality rating system, the proportion of A-level quality in the output data is increased from 67% in traditional methods to over 94%, providing high-quality, low-noise structured corpus for training large language models.

[0059] 3. It possesses excellent scalability and stability, capable of meeting the needs of petabyte-scale large-scale data processing; through the intelligent batch processing scheduling module, it dynamically adjusts the task execution strategy based on the system resource status, and combined with the distributed knowledge base storage architecture and containerized deployment scheme, it achieves elastic scaling of processing scale; actual deployment shows that while maintaining high throughput, memory usage efficiency is improved by 37% and manual intervention costs are reduced by 80%, providing reliable technical support for enterprise-level large-scale document data processing. Attached Figure Description

[0060] Figure 1 This is a flowchart of a multimodal document data processing method for training large language models provided in an embodiment of the present invention;

[0061] Figure 2 This is a block diagram of a multimodal document data processing system for training large language models, provided in an embodiment of the present invention.

[0062] Figure label:

[0063] 1. Data receiving module; 2. OCR recognition module; 3. Data processing module; 4. Data verification module. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0065] Please refer to Figure 1 The first aspect of this invention provides a method for processing multimodal document data for training large language models, comprising the following steps:

[0066] Step S100: Receive several original documents in various formats, extract the structural information of each original document, and identify the text region and image region of each original document based on the structural information.

[0067] The multimodal document loading module employs a plug-in architecture, integrating multiple open-source libraries such as PyMuPDF, pdfplumber, and python-pptx, supporting unified processing of PDF, PPT, PPTX, DOC, and DOCX formats. An automatic document type recognition algorithm intelligently selects the optimal parsing strategy, performing deep analysis on the input raw document. This extracts not only document structure information such as page layout and chapter structure, but also metadata such as creation time, author, and title. Based on the extracted structural information, a deep learning-based layout analysis algorithm, combined with an improved EAST text detection algorithm and the U-Net semantic segmentation network, accurately identifies text regions, image regions, and table regions in the document, laying the foundation for subsequent processing.

[0068] Step S200: A parallel OCR recognition engine based on GPU acceleration and heterogeneous computing is used to perform optical character recognition on text regions and image regions to generate recognition text data of the corresponding original document.

[0069] First, each identified text and image region undergoes image normalization preprocessing, including size normalization, contrast enhancement, and noise suppression, generating a standardized image to be recognized. Then, the batch size is dynamically adjusted based on the GPU's available memory capacity and core utilization, combining all images to be recognized from multiple original documents into several batches. Each batch is decomposed into multiple independent OCR recognition subtasks, which are then distributed in parallel across multiple computing threads on a heterogeneous computing platform consisting of CPUs and GPUs using a multi-threaded scheduler.

[0070] During processing, an optical character recognition model based on an improved deep convolutional neural network is invoked. This model employs a mixed-precision calculation strategy, converting model weights from single-precision floating-point format to half-precision floating-point format to accelerate inference. It also utilizes the sequence recognition module of the Transformer architecture for text sequence decoding and error correction, generating intermediate text data and its corresponding confidence scores. Finally, the intermediate text data is filtered according to a preset confidence threshold, and the filtered text content is aggregated into the recognition text data of the corresponding original document.

[0071] Step S300: Perform multi-dimensional quality assessment and cleaning on the recognized text data of each original document, and output standardized text data that meets the quality level.

[0072] First, the identified text data undergoes noise filtering, duplicate content detection, and language model quality assessment sequentially. Noise filtering, based on regular expression rules and a machine learning classifier, identifies and removes noisy text, including headers, footers, watermarks, and meaningless characters. Duplicate content detection uses a Locality Sensitive Hashing (LSH) algorithm to calculate a LSH value for each text unit, identifying and removing duplicate or highly similar text content by comparing hash values. Language model quality assessment is implemented by calling a pre-trained large language model to calculate the grammatical correctness score and semantic coherence score of the text data, and obtains a comprehensive quality score based on a weighted combination of these two scores. Based on the comprehensive quality score and the results of noise filtering and duplicate detection, the text data is divided into three quality levels: A, B, and C. Only text rated A and B is output as normalized text data.

[0073] Step S400: Store the standardized text data in a distributed knowledge base according to a predefined structure, and perform copyright and compliance checks on the standardized text data.

[0074] The system works collaboratively through a knowledge base ingestion module and a legality verification module, supporting multiple database backends including PostgreSQL, MongoDB, Elasticsearch, and Neo4j. It employs knowledge graph technology to organize text data, achieving efficient storage and retrieval of petabyte-scale data. It also integrates a vector database to support semantic retrieval. For legality verification, digital fingerprint technology is used to detect copyrighted content, and a deep learning model is used for content security review to ensure the legality and security of training data, preventing copyright disputes and non-compliant content from entering the training process.

[0075] Through the above steps, this invention achieves comprehensive optimization of the multimodal document data processing workflow. In actual testing, the processing time for 10TB of multi-format document data was reduced from 720 hours using traditional methods to 144 hours, OCR recognition accuracy increased from 85% to 98.2%, the proportion of data quality at grade A reached 94%, system resource utilization increased by 75%, and labor costs decreased by more than 80%. This significantly improved the efficiency and quality of preparing large language model training data, providing a solid data foundation for the development of the artificial intelligence industry.

[0076] Specifically, step S200 employs a parallel OCR recognition engine based on GPU acceleration and heterogeneous computing to perform optical character recognition on text and image regions, generating the corresponding original document's recognized text data, including:

[0077] Step S210: Preprocess each text region and image region to obtain a corresponding image to be recognized.

[0078] Each text and image region undergoes image normalization preprocessing to obtain a standardized image to be recognized. This preprocessing includes size normalization to adjust all input images to a uniform resolution; contrast enhancement using histogram equalization to improve image quality; and noise suppression using a combination of median and Gaussian filtering algorithms to eliminate scanning artifacts and image noise. The preprocessing module supports automatic conversion of various image formats, ensuring that the subsequent OCR recognition engine can process image data from various sources, laying the foundation for high-quality text recognition.

[0079] Step S220: Combine all the images to be recognized corresponding to the text and image regions of several original documents into several batches of images to be recognized, and decompose each batch of images to be recognized into multiple independent OCR recognition subtasks, with each OCR recognition subtask corresponding to one image to be recognized.

[0080] Processing batches are dynamically constructed based on system resource monitoring results, combining all images to be recognized from several original documents into optimized batches. The current available GPU memory capacity and core utilization are monitored in real time, and the number of images in each batch is dynamically adjusted according to resource conditions to ensure precise matching between batch size and GPU real-time processing capabilities. Each batch of images to be recognized is further decomposed into multiple independent OCR recognition subtasks, each processing one image, establishing fine-grained parallel processing units and providing a foundation for load balancing on the subsequent heterogeneous computing platform.

[0081] Step S230: Based on a heterogeneous computing platform consisting of CPU and GPU, multiple independent OCR recognition subtasks are distributed in parallel to multiple computing threads for processing through a multi-thread scheduler.

[0082] Parallel processing scheduling is implemented on a heterogeneous computing platform consisting of CPUs and GPUs. A multi-threaded scheduler, based on task queues and load balancing algorithms, distributes multiple independent OCR recognition subtasks to multiple computing threads for processing. The scheduler dynamically adjusts the task allocation strategy according to the real-time load status of each computing thread, ensuring full utilization of CPU and GPU computing resources while avoiding overload of individual computing nodes, thus achieving optimal distribution of computing tasks across the heterogeneous platform.

[0083] Step S240: The optical character recognition model based on the improved deep convolutional neural network is called to process each OCR recognition subtask. The sequence recognition module based on the Transformer architecture performs text sequence decoding and error correction on the output of the optical character recognition model to generate intermediate text data and its corresponding confidence score.

[0084] Each OCR subtask is processed by an optical character recognition model based on an improved deep convolutional neural network. This model employs an improved ResNet-152 residual network architecture to extract multi-scale feature representations of the image, implements mixed-precision computation within the network, and converts model weights from single-precision floating-point format to half-precision floating-point format to accelerate the inference process. Building upon this, a sequence recognition module based on the Transformer attention mechanism performs text sequence decoding and contextual error correction on the initial recognition results. Through an encoder-decoder architecture, accurate recognition of mixed Chinese and English text is achieved, and a corresponding confidence score is generated for each recognition result, forming intermediate text data containing complete semantic information.

[0085] Step S250: Filter the intermediate text data according to the preset confidence threshold, remove intermediate text data with a confidence score lower than the preset confidence threshold, and aggregate the intermediate text data with a confidence score greater than or equal to the preset confidence threshold into the recognition text data of the corresponding original document.

[0086] The intermediate text data is filtered for quality based on a dynamically adjustable confidence threshold. The confidence threshold is dynamically adjusted according to the type of the original region corresponding to the recognized text. A relatively lower confidence threshold is used for document regions containing mathematical formulas or technical terms to retain more specialized content, while a relatively higher confidence threshold is used for ordinary text regions to ensure text quality. The filtering process removes all intermediate text data with confidence scores lower than the corresponding region type threshold, retaining only high-reliability recognition content. Finally, these filtered text data are aggregated according to their original documents to generate the final recognized text data for the corresponding original documents.

[0087] Through the complete parallel OCR processing workflow described above, significant technical effects have been achieved in practical applications. When processing large-scale, multi-format documents, the processing speed of traditional methods has been increased from 0.8 images per second to 3-5 images per second, and the recognition accuracy has been improved from 85.3% to 98.2%. At the same time, the system throughput has reached 15.6 GB / hour, and the memory utilization efficiency has been improved to 89%, providing a high-quality, large-scale text corpus data foundation for the training of large language models.

[0088] Further, step S220, which involves combining all the images to be recognized corresponding to the text and image regions of several original documents into several batches of images to be recognized, includes:

[0089] Step S221: Monitor the current available video memory capacity and computing core utilization of the GPU in real time.

[0090] The integrated performance monitoring module monitors the GPU's hardware status parameters in real time. This module continuously tracks key metrics such as the GPU's current available memory capacity, compute core utilization, temperature, and power consumption, building a complete device status profile. Monitoring data is updated at millisecond-level intervals, providing real-time decision-making support for subsequent dynamic resource scheduling. In actual operation, when processing academic papers containing numerous high-resolution image scans, it can accurately detect fluctuations in memory usage, ensuring precise resource allocation.

[0091] Step S222: Based on the current available video memory capacity and computing core utilization, dynamically adjust the batch size of each batch of images to be identified.

[0092] Based on real-time monitoring of GPU resource status data, the batch size of each batch of images to be recognized is dynamically adjusted through the collaborative work of a dynamic memory management unit and a parallel computing scheduler. The adjustment algorithm comprehensively considers the balance between available GPU memory capacity and computing core utilization. When available GPU memory is sufficient but computing core utilization is low, the batch size is appropriately increased to improve parallelism; when GPU memory is scarce and computing cores are saturated, the batch size is correspondingly reduced. A predictive model based on historical processing data is also established, which can predict resource requirements based on document type characteristics, achieving proactive batch size optimization.

[0093] Step S223: Based on the batch size, the images to be identified are combined into several batches of images to be identified, with the number of images in each batch matching the real-time processing capability of the GPU.

[0094] Based on the dynamically adjusted batch size, the images to be recognized are intelligently combined into several processing batches. The combination process employs an optimized load balancing algorithm to ensure that the number of images in each batch precisely matches the real-time processing capacity of the GPU. For document pages containing complex tables and mathematical formulas, the number of images in a single batch is automatically reduced to maintain processing quality; while for simple pages primarily consisting of plain text, the batch size is appropriately increased to improve throughput. All batches are arranged into a task queue according to processing priority, awaiting GPU resource scheduling.

[0095] Through the aforementioned refined batch scheduling mechanism, significant technical results were achieved in actual deployment. When processing a large-scale document set containing 500,000 academic papers, this solution maintained GPU resource utilization at a stable level of over 85%, an improvement of more than 40% compared to traditional fixed batch size methods. Simultaneously, it avoided processing interruptions due to insufficient GPU memory, achieving an overall system throughput of 15.6 GB / hour, providing a stable data processing capability guarantee for large-scale language model training.

[0096] Furthermore, the invocation in step S240, based on an improved deep convolutional neural network-based optical character recognition model, processes each OCR recognition subtask, including:

[0097] Step S2411: At least two different optical character recognition models are called in parallel to recognize the same image to be recognized. Each model outputs a recognition result containing the character sequence and its corresponding character-level confidence.

[0098] This parallel processing architecture simultaneously invokes at least two optical character recognition models with different network structures to process the same image. These models, based on their own independent deep convolutional neural network architectures, extract image features from different dimensions and perform character recognition. During processing, each model not only outputs a complete character sequence recognition result but also generates a corresponding character-level confidence score for each character position in the sequence, forming a set of recognition results containing fine-grained reliability indicators. This multi-model parallel processing mechanism effectively utilizes the recognition advantages brought by model diversity, providing rich decision-making basis for subsequent result fusion.

[0099] Step S2412: Align the character sequences output by each optical character recognition model based on the dynamic time warping algorithm to establish the character position correspondence.

[0100] A dynamic time warping algorithm is employed to accurately align character sequences output by multiple optical character recognition models. This algorithm establishes the optimal correspondence between character positions by calculating the minimum cumulative distance between different sequences, effectively solving the alignment problem caused by inconsistent recognition result lengths or character segmentation differences. When processing document images containing mixed Chinese and English content or complex layout formats, this alignment method can accurately match characters recognized by each model at different positions, establishing a unified coordinate framework for character-level confidence comparison.

[0101] Step S2413: For each aligned character position, compare the characters recognized by all optical character recognition models at the character position and the corresponding character-level confidence scores.

[0102] Based on the aligned character position correspondences, a detailed analysis is performed on the recognition results of all optical character recognition models at each character position. For each character position, the specific character recognized by each model at that position and its corresponding character-level confidence score are compared, establishing a complete character-confidence mapping table. This process accurately grasps the candidate character set and its reliability assessment at each character position, providing data support for selecting the optimal recognition result.

[0103] Step S2414: Select the character with the highest character-level confidence at the character position as the character at the final recognition position in the sequence.

[0104] Based on character-level confidence score comparisons, the optimal recognition result is selected for each character position. By comparing the confidence scores of all candidate characters at the same position, the character with the highest confidence score is selected as the final recognized character for that position. This confidence-based selection mechanism ensures that the most reliable recognition result is used for each character position, effectively improving the recognition accuracy of individual characters.

[0105] Step S2415: Combine the selected characters at all character positions in order to obtain the final recognition result of the image to be recognized.

[0106] The optimal characters selected at all character positions are combined according to the original sequence order to construct the complete final recognition result. This combination process preserves the spatial order and contextual relationships of the characters in the original image, ensuring the coherence and readability of the output text. Through this fine-grained character-level fusion strategy, the advantages of multiple OCR models can be fully utilized, significantly improving the overall recognition accuracy while maintaining the integrity of the text structure.

[0107] In actual testing, this solution reduced the character recognition error rate of complex layouts to less than 50% of that of traditional single-model methods. The improvement in recognition accuracy was particularly significant when processing technical documents containing special symbols and mixed fonts, providing a high-quality text data foundation for subsequent text quality assessment and knowledge base construction.

[0108] Further, the filtering of intermediate text data based on a preset confidence threshold in step S250 includes:

[0109] Step S251: Identify the original region type corresponding to the intermediate text data. Region types include ordinary text, mathematical formulas, and technical terminology text.

[0110] By analyzing the contextual features and layout information of intermediate text data, the corresponding original region type is accurately identified. This identification process is based on region annotation information provided by the intelligent segmentation module, combined with semantic features of the text content, to divide text regions into three types: ordinary text, mathematical formulas, and technical terminology text. For mathematical formula regions, identification is performed by detecting special symbols, subscript / superscript structures, and formula formatting features; for technical terminology text regions, determination is made through domain dictionary matching and terminology density analysis; all other text regions are uniformly classified as ordinary text regions. This refined region classification provides accurate type identification for subsequent differentiated quality control.

[0111] Step S252: Dynamically select the corresponding confidence threshold according to the region type. The first confidence threshold for mathematical formulas and technical terms text regions is lower than the second confidence threshold for ordinary text regions.

[0112] Based on the identified region type, a dynamic confidence threshold is selected to establish differentiated quality control standards. A relatively low first confidence threshold is set for mathematical formulas and technical terminology text regions, as this type of content is inherently more difficult to identify, but preserving complete technical content is more important for the quality of training data. A relatively high second confidence threshold is set for ordinary text regions to ensure the accuracy and standardization of the basic text. A threshold configuration table is maintained, containing confidence threshold parameters for each region type optimized based on extensive experimental data. These threshold parameters can be dynamically adjusted according to the needs of specific application scenarios, enabling flexible configuration of the quality control strategy.

[0113] Step S253: Filter the intermediate text data generated from the original region type based on the confidence threshold corresponding to each region type.

[0114] Based on the confidence threshold corresponding to each region type, precise filtering is performed on the intermediate text data generated for the respective region. For mathematical formula regions, a first confidence threshold is used for filtering to retain more text content that may contain important formula information; for technical terminology text regions, the same first confidence threshold is used to ensure the integrity of the technical terms; for ordinary text regions, a higher second confidence threshold is used for strict filtering to eliminate unreliable recognition results. During the filtering process, the confidence score of each piece of intermediate text data is checked, and only data with a score greater than or equal to the corresponding region type threshold is retained, forming high-quality recognized text data output.

[0115] By implementing this region-type-based differential filtering strategy, a balance is effectively struck between text retention and recognition accuracy when processing complex academic documents. This approach significantly improves the completeness of specialized content recognition while ensuring the accuracy of ordinary text. The final output recognized text data maintains the integrity of technical terms and mathematical formulas while achieving a higher level of accuracy for ordinary text, providing a comprehensive and accurate text corpus foundation for training large language models.

[0116] Furthermore, the invocation in step S240, based on an improved deep convolutional neural network-based optical character recognition model, processes each OCR recognition subtask, including:

[0117] Step S2421: The weight parameters of the input layer, output layer and batch normalization layer in the deep convolutional neural network are retained in single-precision floating-point format, while the weight parameters of the intermediate convolutional layers are converted to half-precision floating-point format to obtain a mixed-precision model.

[0118] A model precision conversion operation is performed to construct a hybrid precision recognition model suitable for GPU acceleration. This process first analyzes the sensitivity of each layer in the deep convolutional neural network to numerical precision. It determines that the input layer, output layer, and batch normalization layer, due to their involvement in key calculations such as data standardization and final output, require high numerical precision; therefore, the weight parameters of these layers are retained in single-precision floating-point format. Simultaneously, a large number of intermediate convolutional layers in the recognition network are insensitive to changes in numerical precision, and their weight parameters are converted to half-precision floating-point format. The conversion process is implemented using an automatic precision selector. This selector automatically identifies the layers requiring conversion based on the network structure and computational characteristics. During the conversion process, loss scaling techniques are applied to dynamically adjust the gradient value range to maintain the stability of numerical calculations, ensuring that the model precision does not significantly decrease due to format conversion.

[0119] Step S2422: Perform optical character recognition inference computation of the mixed-precision model on the GPU, wherein the weight parameters in half-precision floating-point format are accelerated using the GPU’s dedicated half-precision computing unit.

[0120] Optical character recognition inference computations using a mixed-precision model are deployed and executed on GPUs. After loading the mixed-precision model, the GPU's dedicated half-precision computing units (such as Tensor Cores) accelerate the parallel computation of weight parameters in half-precision floating-point format, significantly improving the throughput of matrix multiplication and convolution operations. Simultaneously, a dynamic precision coordinator manages data transfer between different precision layers, ensuring correct conversion and concatenation of input and output data between single-precision and half-precision layers, maintaining the integrity of the computation graph. During recognition tasks, the GPU's computing unit utilization is monitored in real time, and the computational task allocation strategy is dynamically adjusted based on the load to avoid performance bottlenecks caused by mixed precision, maximizing the speed advantages of mixed-precision computation.

[0121] By implementing a hybrid precision computing strategy, processing efficiency is significantly improved while maintaining recognition accuracy. In actual tests, this solution increased the inference speed of deep learning models by 3-5 times, achieved GPU core utilization of over 85%, and, due to the reduced memory usage caused by half-precision computing, enabled the processing of larger batches of image data, further improving overall throughput. This optimization allows for significant performance improvements in processing large-scale documents without increasing hardware costs, while also reducing energy consumption and providing a reliable guarantee for long-term stable operation.

[0122] Furthermore, prior to employing a parallel OCR recognition engine based on GPU acceleration and heterogeneous computing in step S200, the following steps are also included:

[0123] Step S201: Calculate a priority score based on at least one of the following parameters for each original document: document size, format complexity, and user-specified priority.

[0124] Each original document is comprehensively evaluated through a task priority assessment module, calculating a priority score based on three dimensions: document size, format complexity, and user-specified priority. The document size parameter is quantified based on the actual storage capacity of the file; the format complexity parameter is derived by analyzing the document's structural hierarchy, image density, and layout complexity; and the user-specified priority is determined based on the urgency set by business requirements. A weighted algorithm combines these three parameters into a unified priority score, with user-specified priority having the highest weight, followed by format complexity, and then document size having the lowest weight, ensuring that task scheduling considers both business needs and processing efficiency.

[0125] Step S202: Sort several original documents into a processing queue according to their priority scores.

[0126] All raw documents to be processed are sorted according to their calculated priority scores to construct a processing queue. The sorting algorithm uses a descending order, placing the document with the highest priority score at the front of the queue to ensure that important documents enter the processing flow first. The processing queue uses a dynamic data structure, allowing new processing tasks to be inserted at any time during system operation without affecting the existing queue order. It also records the metadata and priority score of each document, providing a basis for subsequent resource scheduling decisions.

[0127] Step S203: Real-time detection of CPU utilization and memory usage.

[0128] The performance monitoring module monitors key resource metrics in real time, including CPU utilization and memory usage. It collects system resource data at a rate of one second, establishing a resource usage trend model to accurately reflect the current system load. When CPU utilization consistently exceeds 80% or memory usage exceeds 85%, a resource warning signal is generated, providing a trigger for subsequent processing speed adjustments and ensuring the system operates in a stable state.

[0129] Step S204: Based on CPU utilization and memory usage, dynamically adjust the rate and number of concurrent submissions of original documents from the processing queue to the parallel OCR recognition engine.

[0130] The document submission strategy is dynamically adjusted based on real-time resource monitoring data. When low system resource utilization is detected, the intelligent batch processing scheduling module increases the rate and concurrency of submitting original documents from the processing queue to the parallel OCR recognition engine, making full use of idle computing resources. When resource utilization approaches the warning threshold, the submission rate is reduced accordingly, decreasing the number of concurrent tasks to prevent system overload. The adjustment process adopts a gradual strategy to avoid drastic fluctuations in resource usage and ensure the smooth progress of processing tasks.

[0131] By implementing an intelligent task scheduling and resource management mechanism, significant efficiency improvements are demonstrated when processing large-scale documents. In actual deployment, this solution keeps system resource utilization within the optimal range, reduces task completion time by more than 40% compared to the traditional first-come, first-served strategy, and effectively avoids system instability caused by resource contention, providing reliable scheduling guarantees for large-scale document processing.

[0132] Furthermore, step S300 involves multi-dimensional quality assessment and cleaning of the identified text data for each original document, including:

[0133] Step S310 involves sequentially performing noise content filtering, duplicate content detection, and language model quality assessment on the identified text data.

[0134] The noisy content filtering function uses regular expression rules and machine learning classifiers to identify and remove noisy text, including headers, footers, and watermarks. Duplicate content detection uses a Locality Sensitive Hash (LSH) algorithm to calculate a LSH value for each text unit and identifies and removes duplicate or highly similar text content by comparing these LSH values. Language model quality assessment is achieved by calling a pre-defined language model, calculating the grammatical correctness score and semantic coherence score of the identified text data, and obtaining a comprehensive quality score based on a weighted combination of these two scores.

[0135] The data cleaning and verification module performs a systematic quality assessment and cleaning process on the identified text data. This process executes three key processing stages sequentially: The noise content filtering stage utilizes a collaborative mechanism of regular expression rules and a machine learning classifier. Regular expression rules are used to identify structured noise with fixed patterns, such as headers, footers, page numbers, and watermarks. The machine learning classifier uses a trained feature model to detect meaningless character combinations and random noise text. The combination of these two methods achieves comprehensive coverage and effective cleaning of different types of noise content. The duplicate content detection stage uses a Locality Sensitive Hash (LSH) algorithm to generate a LSH value representing the content characteristics of each text unit. A hash index library is built to achieve rapid comparison of text similarity. When text content with a hash value similarity exceeding a preset threshold is detected, it is marked as duplicate or highly similar content and removed from the data stream. In the language model quality assessment stage, a pre-trained large language model is used to perform in-depth analysis of the text. The model calculates the grammatical correctness score and semantic coherence score of the text. The grammatical correctness score assesses the degree to which the text conforms to grammatical rules, while the semantic coherence score measures the logical consistency of the text in the context. Finally, the two scores are combined into a comprehensive quality score through preset weighting coefficients to form a quantitative evaluation index of text quality.

[0136] By implementing this multi-layered quality control process, significant advantages were demonstrated when processing large-scale documents containing complex layouts and diverse content types. In actual testing, the solution successfully removed over 95% of the noise content from the original recognized text, achieved a duplicate content recognition accuracy exceeding 98%, and effectively identified semantically incoherent text fragments through language model evaluation. This resulted in the final output of standardized text data achieving a high level of grammatical correctness and semantic integrity, providing a clean and high-quality structured corpus foundation for training large language models.

[0137] Furthermore, the output in step S300, which conforms to the quality level, includes normalized text data, including:

[0138] Step S321: Based on the comprehensive quality score and the results of noise content filtering and duplicate content detection, the identified text data is divided into three quality levels.

[0139] Texts with a comprehensive quality score greater than the first threshold and passing noise filtering and duplicate content detection are rated as Grade A; texts with a comprehensive quality score greater than the second threshold but not greater than the first threshold and passing noise filtering are rated as Grade B; and texts with a comprehensive quality score greater than the third threshold but not greater than the second threshold but failing duplicate content detection are rated as Grade C.

[0140] A three-tiered quality grading system is established based on multi-dimensional quality assessment results, using preset quality thresholds to accurately classify the identified text data. First, three key quality threshold parameters are set: the first threshold serves as the entry threshold for high-quality text, the second threshold as the identification standard for medium-quality text, and the third threshold as the screening baseline for basic-quality text. In practice, texts that simultaneously meet the conditions of a comprehensive quality score greater than the first threshold, passing noise content filtering, and passing duplicate content detection are rated as Grade A. These texts are characterized by grammatical correctness, semantic coherence, and no redundant content. Texts with a comprehensive quality score greater than the second threshold but less than the first threshold and passing noise content filtering are rated as Grade B. While these texts may have slightly insufficient semantic coherence, they guarantee basic grammatical correctness. Texts with a comprehensive quality score greater than the third threshold but less than the second threshold and failing duplicate content detection are rated as Grade C. Texts at this level typically contain duplicate content or semantically incomplete issues. The entire grading process employs an automated decision-making mechanism to ensure that each text receives an accurate quality rating.

[0141] Step S322: Output only the texts rated A and B as normalized text data.

[0142] Precise data output control is implemented based on the quality grading results. A data output filtering rule is established based on the three-level quality grading results, ensuring that only texts rated A and B are included in the standardized text data output range, while C-level texts are excluded from the final output. This filtering mechanism ensures that the output data possesses both high-quality content and broad data coverage. A-level texts provide core high-quality corpus for training large language models, while B-level texts serve as an effective supplement to expand the diversity of training data. During the output process, each text is labeled with its corresponding quality level and detailed quality assessment parameters are recorded, providing a reference for data weighting in subsequent training processes.

[0143] By implementing a rigorous quality grading and screening mechanism, optimal data utilization was achieved while ensuring data quality. In practical applications, this scheme ensured that the final output standardized text data met high standards in terms of grammatical correctness, semantic coherence, and content uniqueness. Simultaneously, the tiered output strategy effectively balanced the relationship between data quality and data volume. This quality control method significantly improved the overall quality of training data for large language models, reduced the negative impact of low-quality data on model training, and lowered the cost of subsequent manual data cleaning, providing a reliable data foundation for building high-quality language models.

[0144] Accordingly, please refer to Figure 2 A second aspect of this invention provides a multimodal document data processing system for training large language models, which identifies multimodal documents based on the aforementioned multimodal document data processing method for training large language models, including:

[0145] The data receiving module 1 is used to receive several original documents in various formats, extract the structural information of each original document, and identify the text region and image region of each original document based on the structural information.

[0146] OCR recognition module 2 is used to perform optical character recognition on text and image regions using a parallel OCR recognition engine based on GPU acceleration and heterogeneous computing, and generate the corresponding original document recognition text data.

[0147] Data processing module 3 is used to perform multi-dimensional quality assessment and cleaning on the identified text data of each original document, and output standardized text data that meets the quality level.

[0148] Data verification module 4 is used to store standardized text data in a predefined structure to a distributed knowledge base and to verify the copyright and compliance of the standardized text data.

[0149] Accordingly, a third aspect of the present invention provides an electronic device, including: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the above-described multimodal document data processing method for training large language models.

[0150] Accordingly, a fourth aspect of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-described multimodal document data processing method for training large language models.

[0151] The embodiments of the present invention aim to protect a method and system for processing multimodal document data for training large language models, which has the following effects:

[0152] 1. By constructing a parallel processing architecture based on GPU acceleration and heterogeneous computing, a qualitative leap in the efficiency of multimodal document data processing was achieved; a multi-threaded scheduler was used to dynamically decompose the OCR recognition task into parallel subtasks, and combined with mixed precision computing and dynamic memory management technology, the advantages of GPU hardware parallel computing were fully utilized, enabling a processing speed of 3-5 images per second, which is more than 500% faster than the traditional single-threaded processing method, while improving resource utilization by more than 75%, thus completely solving the efficiency bottleneck problem in the preparation of large language model training data.

[0153] 2. By establishing a multi-level intelligent quality control system, the quality and reliability of the output text data are significantly improved. The system integrates a triple guarantee mechanism of noise filtering, duplicate content detection and language model quality assessment. The confidence threshold strategy based on region type adaptation ensures that different content adopts differentiated quality control standards. Combined with a three-level quality rating system, the proportion of A-level quality in the output data is increased from 67% in traditional methods to over 94%, providing high-quality, low-noise structured corpus for training large language models.

[0154] 3. It possesses excellent scalability and stability, capable of meeting the needs of petabyte-scale large-scale data processing; through the intelligent batch processing scheduling module, it dynamically adjusts the task execution strategy based on the system resource status, and combined with the distributed knowledge base storage architecture and containerized deployment scheme, it achieves elastic scaling of processing scale; actual deployment shows that while maintaining high throughput, memory usage efficiency is improved by 37% and manual intervention costs are reduced by 80%, providing reliable technical support for enterprise-level large-scale document data processing.

[0155] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0157] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0158] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for processing multi-modal document data for large language model training, the method comprising: The method comprises the following steps: Receiving a plurality of original documents in various formats, extracting the structural information of each original document, identifying the text area and image area of each original document based on the structural information; Using a parallel OCR recognition engine based on GPU acceleration and heterogeneous computing to perform optical character recognition on the text area and the image area, and generating recognized text data of the corresponding original document; Performing multi-dimensional quality assessment and cleaning on the recognized text data of each original document, and outputting standardized text data that meets the quality level; Storing the standardized text data in a distributed knowledge base according to a predefined structure, and performing copyright and compliance testing on the standardized text data; The method of using a parallel OCR recognition engine based on GPU acceleration and heterogeneous computing to perform optical character recognition on the text area and the image area, and generating recognized text data of the corresponding original document, comprises: Preprocessing each text area and image area to obtain a corresponding to-be-recognized image; Combining all to-be-recognized images corresponding to the text area and the image area of a plurality of original documents into a plurality of to-be-recognized image batches, and decomposing each to-be-recognized image batch into a plurality of independent OCR recognition subtasks, each of which corresponds to a to-be-recognized image; Based on a heterogeneous computing platform comprising a CPU and a GPU, a plurality of independent OCR recognition subtasks are parallelly distributed to a plurality of computing threads for processing by a multi-thread scheduler; Calling an optical character recognition model based on an improved deep convolutional neural network to process each OCR recognition subtask, and using a sequence recognition module based on a Transformer architecture to decode and correct the output results of the optical character recognition model to generate intermediate text data and its corresponding confidence score; Filtering the intermediate text data according to a preset confidence threshold, removing the intermediate text data with a confidence score less than the preset confidence threshold, and aggregating the intermediate text data with a confidence score greater than or equal to the preset confidence threshold into recognized text data of the corresponding original document.

2. The multi-modal document data processing method for large language model training according to claim 1, wherein, The method of combining all to-be-recognized images corresponding to the text area and the image area of a plurality of original documents into a plurality of to-be-recognized image batches comprises: Real-time monitoring of the current available video memory capacity and computing core utilization rate of the GPU; Based on the current available video memory capacity and computing core utilization rate, dynamically adjusting the batch size of each to-be-recognized image batch; According to the batch size, the to-be-recognized images are combined into a plurality of to-be-recognized image batches, and the number of images in each to-be-recognized image batch matches the real-time processing capacity of the GPU.

3. The multi-modal document data processing method for large language model training according to claim 1, wherein, The method of calling an optical character recognition model based on an improved deep convolutional neural network to process each OCR recognition subtask comprises: Parallelly invoke at least two different optical character recognition models to recognize the same image to be recognized, each model outputting a recognition result containing a character sequence and corresponding character-level confidence; Align the character sequences output by each optical character recognition model based on a dynamic time warping algorithm to establish a character position correspondence; For each aligned character position, compare the characters and corresponding character-level confidence recognized by all optical character recognition models at the character position; Select the character with the highest character-level confidence at the character position as the character at the character position in the final recognition sequence; Combine the selected characters at all character positions in order to obtain the final recognition result of the image to be recognized.

4. The multi-modal document data processing method for large language model training according to claim 1, wherein, The filtering of the intermediate text data according to the confidence threshold includes: Identifying the original region type corresponding to the intermediate text data, the region type including general text, mathematical formula, and professional term text; Dynamically selecting a corresponding confidence threshold according to the region type, the first confidence threshold corresponding to the mathematical formula and professional term text region being lower than the second confidence threshold corresponding to the general text region; Filtering the intermediate text data generated by the original region type based on the confidence threshold corresponding to each region type.

5. The multi-modal document data processing method for large language model training according to claim 1, wherein, The calling of the optical character recognition model based on the improved deep convolutional neural network to process each OCR recognition subtask includes: Reserving the weight parameters of the input layer, output layer, and batch normalization layer in the deep convolutional neural network as single-precision floating-point numbers, while converting the weight parameters of the intermediate convolutional layer to half-precision floating-point numbers to obtain a mixed-precision model; Performing optical character recognition inference calculation of the mixed-precision model on the GPU, wherein the weight parameters in half-precision floating-point format are accelerated using the dedicated half-precision calculation unit of the GPU.

6. The multi-modal document data processing method for large language model training according to any one of claims 1-5, characterized in that, Before using the parallel OCR recognition engine based on GPU acceleration and heterogeneous computing, the following steps are further included: Calculating a priority score based on at least one of the document size, format complexity, and user-specified priority of each original document; According to the priority score, sorting the several original documents in a processing queue; Real-time detection of CPU utilization and memory usage; Based on the CPU utilization and memory usage, dynamically adjusting the rate and concurrent number of submitting original documents from the processing queue to the parallel OCR recognition engine.

7. The multi-modal document data processing method for large language model training according to any one of claims 1-5, characterized in that, The multi-dimensional quality assessment and cleaning of the recognized text data of each original document includes: Sequentially performing noise content filtering, duplicate content detection, and language model quality assessment on the recognized text data; The noise content filtering is based on a regular expression rule and a machine learning classifier to identify and clean up noise text including a header, a footer, and a watermark; the duplicate content detection is based on a local sensitive hashing algorithm to calculate a local sensitive hash value for each text unit and identify and remove duplicate or highly similar text content by comparing the local sensitive hash values; and the language model quality evaluation is achieved by calling a preset language model to calculate a grammatical correctness score and a semantic coherence score of the recognized text data and obtain a comprehensive quality score based on a weighted combination of the two scores.

8. The multi-modal document data processing method for large language model training according to claim 7, characterized in that, The output normalized text data conforming to a quality level includes: The recognized text data is divided into three quality levels based on the comprehensive quality score and results of the noise content filtering and the duplicate content detection; wherein text with a comprehensive quality score greater than a first threshold and passing the noise filtering and the duplicate content detection is rated as A level, text with a comprehensive quality score greater than a second threshold and not greater than the first threshold and passing the noise filtering is rated as B level, and text with a comprehensive quality score greater than a third threshold and not greater than the second threshold but not passing the duplicate content detection is rated as C level; only text rated as A level and B level is output as the normalized text data. 9.A multi-modal document data processing system for large language model training, characterized in that, The multi-modal document data processing method for large language model training according to any one of claims 1-8 is used to recognize multi-modal documents, including: a data receiving module configured to receive a plurality of original documents in various formats, extract structural information of each original document, and identify text regions and image regions of each original document based on the structural information; an OCR recognition module configured to use a parallel OCR recognition engine based on GPU acceleration and heterogeneous computing to perform optical character recognition on the text regions and the image regions, and generate recognized text data of the corresponding original document; a data processing module configured to perform multi-dimensional quality evaluation and cleaning on the recognized text data of each original document, and output normalized text data conforming to a quality level; a data verification module configured to store the normalized text data to a distributed knowledge base according to a predefined structure, and perform copyright and compliance verification on the normalized text data.

Images