Multi-modal document data processing method and system oriented to large language model training

Abstract

The invention discloses a multi-modal document data processing method and system for large language model training, and the method comprises the steps: receiving a plurality of original documents in various formats, extracting the structure information of each original document, and recognizing a text region and an image region of each original document based on the structure information; performing optical character recognition on the text region and the image region by adopting a parallel OCR (Optical Character Recognition) engine based on GPU (Graphics Processing Unit) acceleration and heterogeneous calculation to generate recognition text data of the corresponding original document; performing multi-dimensional quality evaluation and cleaning on the recognition text data of each original document, and outputting normalized text data; and storing the standardized text data into a distributed knowledge base according to a predefined structure, and performing copyright and compliance test on the standardized text data. By adopting a parallel OCR recognition engine based on GPU acceleration and heterogeneous calculation, efficient and high-precision batch processing of multi-modal documents is realized, and the processing speed, the recognition accuracy and the data quality are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-modal data processing, in particular to a multi-modal document data processing method and system for large language model training. BACKGROUND

[0002] With the rapid development of generative artificial intelligence technology, natural language processing technology represented by large language models has become the core driving force for promoting the intelligent transformation of industries. Such models have strong language understanding and generation capabilities through pre-training on massive high-quality text corpus, and their performance directly depends on the scale and quality of the training data. Currently, mainstream large language models usually require TB-level or even PB-level structured text data for training, however, the data available in the real world mostly exists in the form of unstructured multi-modal documents, including native PDF, PPT presentations, scanned image files and other formats. These documents contain rich text information, but how to efficiently, accurately and batch convert them into high-quality corpus suitable for large language model training, the existing technical solutions face systematic technical challenges.

[0003] In terms of processing efficiency, the existing document processing technology has obvious bottlenecks. Traditional document processing tools mostly use single-threaded serial processing mode, and when processing scanned PDF documents and PPT files with complex layout, the processing speed is generally only 0.5 to 1 image per second. When faced with the demand for processing TB-level large-scale document data, the entire processing cycle can take several months, which completely cannot meet the timeliness requirements of data supply for the rapid iteration training of large language models. This processing efficiency bottleneck seriously restricts the development cycle and iteration speed of large language models, and breakthrough technical solutions are needed to improve the throughput of the entire data processing process.

[0004] In terms of recognition accuracy, traditional optical character recognition technology has significant limitations. When processing complex layout, low-quality scanned documents, content containing mathematical formulas, and multi-language mixed documents, the recognition accuracy of traditional OCR technology is generally below 85%. Especially when processing academic papers, technical documents and other documents containing a large number of professional terms, special symbols and complex tables, the recognition error rate is even as high as 20% or more. These recognition errors will directly lead to a decrease in the quality of training data, and then affect the understanding ability and generation quality of large language models, how to improve the text recognition accuracy in complex scenarios has become a technical problem to be solved.

[0005] In terms of computing resource utilization, existing systems have a serious problem of insufficient resource utilization. Most traditional OCR processing tools are still based on central processors for serial computing, failing to fully utilize the parallel computing advantages of special hardware such as graphics processors. This architecture design leads to the idle and waste of high-performance computing resources such as GPUs, low overall system throughput, and the inability to fully utilize the performance potential of modern hardware devices. At the same time, existing systems lack effective resource scheduling and management mechanisms, and cannot dynamically optimize according to task characteristics and system load, further reducing resource utilization efficiency.

[0006] In terms of data quality control, existing technical solutions lack a systematic quality assurance mechanism. Traditional processing procedures often only focus on text extraction itself, ignoring quality evaluation and cleaning of the extraction results. This leads to a large amount of noise data (such as headers and footers, watermark marks, meaningless characters), repeated content, and format errors mixed into the final training set, seriously affecting the training effect and generalization ability of large language models. The lack of a unified quality standard and grading system makes it difficult to quantitatively evaluate data quality and also unable to accurately control data quality according to different application scenarios.

[0007] In terms of system scalability, existing solutions cannot meet the needs of large-scale data processing. Traditional architectures lack effective distributed processing capabilities and cannot support parallel processing of PB-level data. At the same time, the system has obvious shortcomings in load balancing, fault tolerance processing, etc. This limitation of scalability makes it difficult for existing technologies to meet the stringent requirements of large language model training for data size, becoming another technical bottleneck restricting the development of artificial intelligence technology. SUMMARY

[0008] The purpose of the embodiments of the present application is to provide a multi-modal document data processing method and system for large language model training, which realizes efficient and high-precision batch processing of multi-modal documents by using a parallel OCR recognition engine based on GPU acceleration and heterogeneous computing, combined with multi-dimensional quality evaluation and cleaning mechanism, significantly improving data processing speed, recognition accuracy and data quality, effectively meeting the preparation needs of large language model training for high-quality structured corpus.

[0009] To solve the above technical problems, the first aspect of the embodiments of the present application provides a multi-modal document data processing method for large language model training, comprising the following steps: Receive a plurality of original documents in various formats, extract the structure information of each original document, and identify the text area and image area of each original document based on the structure information; Use a parallel OCR recognition engine based on GPU acceleration and heterogeneous computing to perform optical character recognition on the text area and the image area, and generate recognized text data for the corresponding original document; performing multi-dimensional quality assessment and cleaning on the recognized text data of each of the original documents, and outputting normalized text data conforming to a quality level; storing the normalized text data into a distributed knowledge base according to a predefined structure, and performing copyright and compliance inspection on the normalized text data.

[0010] Further, the parallel OCR recognition engine based on GPU acceleration and heterogeneous computing is used to perform optical character recognition on the text regions and the image regions, and to generate recognized text data of the corresponding original documents, including: performing preprocessing on each of the text regions and the image regions to obtain a corresponding to-be-recognized image; combining all the to-be-recognized images corresponding to the text regions and the image regions of a plurality of original documents into a plurality of to-be-recognized image batches, and decomposing each of the to-be-recognized image batches into a plurality of independent OCR recognition subtasks, each of which corresponds to a to-be-recognized image; based on a heterogeneous computing platform including a CPU and a GPU, parallelly assigning a plurality of independent OCR recognition subtasks to a plurality of computing threads for processing through a multi-thread scheduler; calling an optical character recognition model based on an improved deep convolutional neural network to process each of the OCR recognition subtasks, calling a sequence recognition module based on a Transformer architecture to decode and correct a text sequence of an output result of the optical character recognition model, and generating intermediate text data and a corresponding confidence score thereof; filtering the intermediate text data according to a preset confidence threshold, eliminating the intermediate text data with a confidence score less than the preset confidence threshold, and aggregating the intermediate text data with a confidence score greater than or equal to the preset confidence threshold into recognized text data of the corresponding original document.

[0011] Further, the combining of all the to-be-recognized images corresponding to the text regions and the image regions of a plurality of original documents into a plurality of to-be-recognized image batches includes: monitoring a current available video memory capacity and a computing core utilization rate of the GPU in real time; based on the current available video memory capacity and the computing core utilization rate, dynamically adjusting a batch size of each of the to-be-recognized image batches; combining the to-be-recognized images into a plurality of to-be-recognized image batches according to the batch size, and matching an image quantity of each of the to-be-recognized image batches to a real-time processing capability of the GPU.

[0012] Further, the calling is based on an improved deep convolutional neural network-based optical character recognition model to process each of the OCR recognition sub-tasks, including: Parallelly calling at least two different optical character recognition models to recognize the same to-be-recognized image, each model outputting a recognition result containing a character sequence and a corresponding character-level confidence; Aligning the character sequences output by each of the optical character recognition models based on a dynamic time warping algorithm to establish a character position correspondence; For each aligned character position, comparing the characters and corresponding character-level confidences recognized by all the optical character recognition models at the character position; Selecting the character with the highest character-level confidence at the character position as the character at the character position in the final recognition sequence; Combining the selected characters at all character positions in order to obtain the final recognition result of the to-be-recognized image.

[0013] Further, the filtering of the intermediate text data according to the confidence threshold includes: Recognizing the original region type corresponding to the intermediate text data, the region type including ordinary text, mathematical formula, and professional term text; Dynamically selecting a corresponding confidence threshold according to the region type, the first confidence threshold corresponding to the mathematical formula and professional term text region being lower than the second confidence threshold corresponding to the ordinary text region; Filtering the intermediate text data generated by the original region type based on the confidence threshold corresponding to each of the region types.

[0014] Further, the calling is based on an improved deep convolutional neural network-based optical character recognition model to process each of the OCR recognition sub-tasks, including: Reserving the weight parameters of the input layer, output layer, and batch normalization layer in the deep convolutional neural network as single-precision floating-point number format, while converting the weight parameters of the intermediate convolutional layer to half-precision floating-point number format to obtain a mixed-precision model; Performing optical character recognition inference calculation of the mixed-precision model on the GPU, wherein the weight parameters in half-precision floating-point number format are accelerated by using the special half-precision calculation unit of the GPU.

[0015] Further, before the adoption of the parallel OCR recognition engine based on GPU acceleration and heterogeneous computing, it further includes: Based on at least one of the document size, format complexity, and user-specified priority of each of the original documents, a priority score is calculated; According to the priority score, the plurality of original documents are sorted in a processing queue; Real-time detection of CPU utilization and memory usage; Based on the CPU utilization and the memory usage, dynamically adjusting the rate and the number of concurrent submissions of original documents from the processing queue to the parallel OCR recognition engine.

[0016] Further, the identified text data of each original document is subjected to multi-dimensional quality assessment and cleaning, including: Sequentially performing noise content filtering, duplicate content detection and language model quality assessment on the identified text data; Wherein, the noise content filtering is based on regular expression rules and machine learning classifiers to identify and clean noise text including headers, footers, watermarks; the duplicate content detection is based on a local sensitive hash algorithm to calculate a local sensitive hash value for each text unit, and by comparing the local sensitive hash values, to identify and remove duplicate or highly similar text content; the language model quality assessment is implemented by calling a pre-set language model to calculate the grammatical correctness score and the semantic coherence score of the identified text data, and based on the weighted combination of the two, to obtain a comprehensive quality score.

[0017] Further, the output of the normalized text data conforming to the quality level includes: Based on the comprehensive quality score and the results of noise content filtering and duplicate content detection, the identified text data is divided into three quality levels; Wherein, the text with a comprehensive quality score greater than a first threshold and passing noise filtering and duplicate content detection is rated as A level, the text with a comprehensive quality score greater than a second threshold and not greater than the first threshold and passing noise filtering is rated as B level, and the text with a comprehensive quality score greater than a third threshold and not greater than the second threshold but not passing duplicate content detection is rated as C level; Only the text rated as A level and B level is output as the normalized text data.

[0018] Correspondingly, a second aspect of the embodiment of the present application provides a multi-modal document data processing system for large language model training, which is based on the above-mentioned multi-modal document data processing method for large language model training to recognize multi-modal documents, including: A data receiving module is configured to receive a plurality of original documents in various formats, extract the structure information of each original document, and identify the text area and image area of each original document based on the structure information; an OCR recognition module configured to employ a GPU-accelerated and heterogeneous-computing-based parallel OCR recognition engine to perform optical character recognition on the text region and the image region to generate recognized text data of the original document; a data processing module configured to perform multi-dimensional quality assessment and cleaning on the recognized text data of each original document to output normalized text data meeting a quality level; a data verification module configured to store the normalized text data to a distributed knowledge base according to a predefined structure and perform copyright and compliance verification on the normalized text data.

[0019] Correspondingly, a third aspect of the embodiment of the present application provides an electronic device, comprising: at least one processor; and a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned multi-modal document data processing method for large language model training.

[0020] Correspondingly, a fourth aspect of the embodiment of the present application provides a computer-readable storage medium having computer instructions stored thereon, the instructions being executed by a processor to implement the above-mentioned multi-modal document data processing method for large language model training.

[0021] The above technical solutions of the embodiment of the present application have the following beneficial technical effects: 1. By constructing a parallel processing architecture based on GPU acceleration and heterogeneous computing, a qualitative leap in multi-modal document data processing efficiency is achieved; an OCR recognition task is dynamically decomposed into parallel subtasks by using a multi-threaded scheduler, combined with mixed-precision calculation and dynamic memory management technology, to fully utilize the parallel computing advantages of GPU hardware, so that the processing speed reaches 3-5 images per second, which is more than 500% higher than the traditional single-threaded processing method, and the system resource utilization rate is increased by more than 75%, completely solving the efficiency bottleneck problem in the large language model training data preparation process. 2. By establishing a multi-level intelligent quality control system, the quality and reliability of the output text data are significantly improved; three protection mechanisms of noise filtering, repeated content detection and language model quality assessment are integrated, a confidence threshold strategy based on region type is used to ensure that different contents adopt differentiated quality control standards, and a three-level quality rating system is used to increase the proportion of A-level quality output data from 67% of the traditional method to more than 94%, providing high-quality and low-noise structured corpus for large language model training; 3. It has excellent scalability and stability, can meet the demand of PB-level large-scale data processing; through the intelligent batch processing scheduling module, the task execution strategy is dynamically adjusted based on the system resource state, combined with the distributed knowledge base storage architecture and the containerized deployment scheme, the elastic expansion of the processing scale is realized; actual deployment shows that while maintaining high throughput, memory usage efficiency is improved by 37%, manual intervention cost is reduced by 80%, providing reliable technical support for enterprise-level large-scale document data processing. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is the multi-modal document data processing method flowchart for large language model training provided by the embodiment of the present application; Figure 2 is the multi-modal document data processing system module block diagram for large language model training provided by the embodiment of the present application.

[0023] Reference signs: 1, data receiving module, 2, OCR identification module, 3, data processing module, 4, data verification module. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical scheme and advantages of the present application clearer and more intelligible, the present application will be further described in detail below in conjunction with specific embodiments and with reference to the drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present application. In addition, in the following description, the description of well-known structures and techniques is omitted to avoid unnecessary confusion of the concept of the present application.

[0025] Please refer to Figure 1 The first aspect of the embodiment of the present application provides a multi-modal document data processing method for large language model training, comprising the following steps: Step S100, receiving a plurality of original documents in various formats, extracting the structure information of each original document, and identifying the text area and image area of each original document based on the structure information.

[0026] Through the multi-modal document loading module, a plug-in architecture design is adopted, and multiple open source libraries such as PyMuPDF, pdfplumber, python-pptx and the like are integrated, and unified processing of formats such as PDF, PPT, PPTX, DOC, DOCX and the like is supported. Through a document type automatic recognition algorithm, an optimal analysis strategy is intelligently selected, and the input original document is deeply analyzed, not only to extract document structure information such as page layout and chapter structure, but also to extract metadata such as creation time, author and title. Based on the extracted structure information, a layout analysis algorithm based on deep learning is used, combined with an improved EAST text detection algorithm and a U-Net semantic segmentation network, to accurately identify the text area, image area and table area in the document, laying a foundation for subsequent processing.

[0027] In step S200, a parallel OCR recognition engine based on GPU acceleration and heterogeneous computing is used to perform optical character recognition on the text area and the image area, to generate recognized text data of the corresponding original document.

[0028] First, image normalization preprocessing is performed on each recognized text area and image area, including size normalization, contrast enhancement and noise suppression, to generate standardized images to be recognized. Subsequently, according to the current available video memory capacity and computing core utilization rate of the GPU, the batch size is dynamically adjusted, and all the images to be recognized of multiple original documents are combined into several batches of images to be recognized. Each batch is decomposed into multiple independent OCR recognition sub-tasks, which are distributed to multiple computing threads for processing in parallel on a heterogeneous computing platform composed of CPU and GPU through a multi-thread scheduler.

[0029] During processing, an optical character recognition model based on an improved deep convolutional neural network is called, which uses a mixed precision computing strategy to convert the model weights from single-precision floating-point format to half-precision floating-point format to speed up inference, and uses a sequence recognition module based on the Transformer architecture for text sequence decoding and error correction to generate intermediate text data and its corresponding confidence score. Finally, the intermediate text data is filtered according to a preset confidence threshold, and the filtered text content is aggregated into recognized text data of the corresponding original document.

[0030] In step S300, the recognized text data of each original document is subjected to multi-dimensional quality evaluation and cleaning, and standardized text data meeting the quality level is output.

[0031] First, the identified text data is sequentially subjected to noise content filtering, repeated content detection and language model quality evaluation. Noise content filtering is based on regular expression rules and machine learning classifiers to identify and clean up noise text including headers, footers, watermarks and meaningless characters. Repeated content detection calculates a local sensitive hash value for each text unit based on a local sensitive hash algorithm, and identifies and removes repeated or highly similar text content by comparing the hash values. Language model quality evaluation is achieved by calling a pre-trained large language model, calculating the grammatical correctness score and semantic coherence score of the text data, and obtaining a comprehensive quality score based on the weighted combination of the two. According to the comprehensive quality score and the results of noise filtering and repeated detection, the text data is divided into three quality levels A, B and C, and only the text rated as A and B is output as standardized text data.

[0032] In step S400, the standardized text data is stored in a distributed knowledge base according to a predefined structure, and the standardized text data is subjected to copyright and compliance verification.

[0033] The above steps are completed by the knowledge base storage module and the legality verification module, support multiple database backends including PostgreSQL, MongoDB, Elasticsearch and Neo4j, use knowledge graph technology to organize text data, realize efficient storage and retrieval of PB-level data, and integrate vector database to support semantic retrieval. In terms of legality verification, digital fingerprint technology is used to detect copyrighted content, and deep learning models are used for content safety audit to ensure the legality and safety of training data and avoid copyright disputes and non-compliant content entering the training process.

[0034] Through the above steps, the present application realizes the all-round optimization of the multi-modal document data processing flow. In actual tests, the time for processing 10TB multi-format document data is shortened from 720 hours of traditional methods to 144 hours, the OCR recognition accuracy is improved from 85% to 98.2%, the data quality A level ratio reaches 94%, the system resource utilization is improved by 75%, the labor cost is reduced by more than 80%, and the preparation efficiency and quality of large language model training data are significantly improved, providing a solid data foundation support for the development of artificial intelligence industry.

[0035] Specifically, the parallel OCR recognition engine based on GPU acceleration and heterogeneous computing in step S200 is used to perform optical character recognition on text regions and image regions to generate recognized text data of the corresponding original document, including: In step S210, each text region and image region is preprocessed to obtain a corresponding to-be-recognized image.

[0036] The image normalization preprocessing is performed on each text area and image area to obtain the standardized to-be-recognized image. The preprocessing process includes size normalization operation, contrast enhancement processing, noise suppression operation, and automatic conversion of multiple image formats. The preprocessing module supports automatic conversion of multiple image formats, ensures that the subsequent OCR recognition engine can process image data from various sources, and lays a foundation for high-quality text recognition.

[0037] In step S220, the to-be-recognized images corresponding to the text areas and image areas of a plurality of original documents are combined into a plurality of to-be-recognized image batches, and each to-be-recognized image batch is divided into a plurality of independent OCR recognition subtasks, each of which corresponds to a to-be-recognized image.

[0038] Based on the system resource monitoring result, the to-be-recognized images from a plurality of original documents are combined into an optimized to-be-recognized image batch. The current available video memory capacity and core utilization rate of the GPU are monitored in real time, and the number of images in each batch is dynamically adjusted according to the resource status to ensure that the batch size is accurately matched with the real-time processing capacity of the GPU. Each to-be-recognized image batch is further divided into a plurality of independent OCR recognition subtasks, each of which corresponds to a to-be-recognized image, and a fine-grained parallel processing unit is established to provide a basis for load balancing of the subsequent heterogeneous computing platform.

[0039] In step S230, based on the heterogeneous computing platform composed of CPU and GPU, a plurality of independent OCR recognition subtasks are parallelly distributed to a plurality of computing threads for processing by a multi-thread scheduler.

[0040] Parallel processing scheduling is implemented on the heterogeneous computing platform composed of CPU and GPU. Based on the task queue and load balancing algorithm, the multi-thread scheduler parallelly distributes a plurality of independent OCR recognition subtasks to a plurality of computing threads for processing. The scheduler dynamically adjusts the task distribution strategy according to the real-time load state of each computing thread to ensure full utilization of CPU and GPU computing resources, while avoiding overload of a single computing node, and to achieve optimal distribution of computing tasks on the heterogeneous platform.

[0041] In step S240, an optical character recognition model based on an improved deep convolutional neural network is called to process each OCR recognition subtask, and a sequence recognition module based on a Transformer architecture is called to decode the output result of the optical character recognition model and correct errors to generate intermediate text data and its corresponding confidence score.

[0042] An optical character recognition model based on an improved deep convolutional neural network is called to process each OCR recognition subtask. The recognition model uses an improved ResNet-152 residual network architecture to extract multi-scale feature representations of images, implements mixed-precision calculation inside the network, and converts model weights from single-precision floating-point format to half-precision floating-point format to speed up the inference process. On this basis, a sequence recognition module based on the Transformer attention mechanism decodes the preliminary recognition results and corrects the context, accurately recognizes mixed English and Chinese text through an encoder-decoder architecture, and generates a corresponding confidence score for each recognition result to form intermediate text data containing complete semantic information.

[0043] Step S250, filtering the intermediate text data according to the confidence preset threshold, removing the intermediate text data with a confidence score less than the confidence preset threshold, and aggregating the intermediate text data with a confidence score greater than or equal to the confidence preset threshold as the recognized text data of the corresponding original document.

[0044] The quality of the intermediate text data is filtered according to the dynamically adjustable confidence preset threshold. The confidence threshold is dynamically adjusted according to the type of the original region corresponding to the recognized text. A relatively low confidence threshold is used for document regions containing mathematical formulas or professional terms to retain more professional content, and a relatively high confidence threshold is used for ordinary text regions to ensure text quality. The filtering process removes all intermediate text data with a confidence score lower than the threshold for the corresponding region type, and only retains high-reliability recognized content. Finally, these filtered text data are aggregated according to the original document to which they belong, generating the final recognized text data of the corresponding original document.

[0045] Through the above complete parallel OCR processing process, significant technical effects are achieved in practical applications. In processing large-scale multi-format documents, the processing speed of the traditional method is improved from 0.8 images per second to 3-5 images per second, the recognition accuracy is improved from 85.3% to 98.2%, and the system throughput reaches 15.6 GB / hour, and the memory usage efficiency is improved to 89%, providing a high-quality, large-scale text corpus data basis for large language model training.

[0046] Further, the combination of all the to-be-recognized images of the text regions and image regions of the plurality of original documents in step S220 into a plurality of to-be-recognized image batches comprises: Step S221, real-time monitoring of the current available video memory capacity and computing core utilization rate of the GPU.

[0047] The hardware state parameters of the GPU are monitored in real time by an integrated performance monitoring module. This module continuously tracks the current available video memory capacity, computing core utilization, temperature, and power consumption of the GPU, and establishes a complete device state profile. The monitoring data is updated at a millisecond level, providing real-time decision-making basis for subsequent dynamic resource scheduling. In actual operation, when processing academic paper documents containing a large number of high-resolution image scan pages, the fluctuation of video memory occupancy can be accurately perceived, ensuring the accuracy of resource allocation.

[0048] In step S222, based on the current available video memory capacity and computing core utilization, the batch size of each batch of images to be recognized is dynamically adjusted.

[0049] Based on the real-time monitoring of GPU resource state data, the dynamic memory management unit and the parallel computing scheduler work together to dynamically adjust the batch size of each batch of images to be recognized. The adjustment algorithm takes into account the balance between the current available video memory capacity and the computing core utilization, and appropriately increases the batch size to improve parallelism when the available video memory is sufficient and the computing core utilization is low, and correspondingly reduces the batch size when the video memory is tight and the computing core is saturated. A prediction model based on historical processing data is also established, which can predict resource demand according to document type characteristics and achieve forward-looking batch size optimization.

[0050] In step S223, according to the batch size, the images to be recognized are combined into several batches of images to be recognized, and the number of images in each batch of images to be recognized matches the real-time processing capacity of the GPU.

[0051] According to the dynamically adjusted batch size, the images to be recognized are intelligently combined into several processing batches. The combination process uses an optimized load balancing algorithm to ensure that the number of images in each batch matches the real-time processing capacity of the GPU. For document pages containing complex tables and mathematical formulas, the number of images in a single batch is automatically reduced to ensure processing quality; while for simple pages dominated by pure text, the batch size is appropriately increased to improve throughput efficiency. All batches are arranged in the task queue according to the processing priority, waiting for GPU resource scheduling.

[0052] Through the above fine batch scheduling mechanism, significant technical effects have been achieved in actual deployment. When processing a large-scale document set containing 500,000 academic papers, this scheme makes the GPU resource utilization rate stable at more than 85%, which is more than 40% higher than the traditional fixed batch size method, while avoiding processing interruptions caused by insufficient video memory. The overall throughput of the system reaches 15.6 GB / hour, providing stable data processing capacity guarantee for large-scale language model training.

[0053] Further, the calling in step S240 processes each OCR recognition subtask based on an optical character recognition model improved by a deep convolutional neural network, including: Step S2411, at least two different optical character recognition models are called in parallel to recognize the same to-be-recognized image, and each model outputs a recognition result containing a character sequence and a corresponding character-level confidence.

[0054] At least two optical character recognition models with different network structures are called in parallel through a parallel processing architecture to recognize the same to-be-recognized image. These models are based on respective independent deep convolutional neural network architectures, respectively extract image features from different dimensions, and complete character recognition. Each model not only outputs a complete character sequence recognition result in the processing process, but also generates a corresponding character-level confidence score for each character position in the sequence to form a recognition result set containing fine-grained reliability indicators. This multi-model parallel processing mechanism effectively utilizes the recognition advantages brought by model diversity and provides rich decision-making basis for subsequent result fusion.

[0055] Step S2412, align the character sequence output by each optical character recognition model based on a dynamic time warping algorithm to establish a character position correspondence.

[0056] The dynamic time warping algorithm is used to accurately align the character sequences output by multiple optical character recognition models. This algorithm calculates the minimum cumulative distance between different sequences to establish the optimal correspondence between character positions, effectively solving the alignment problem caused by inconsistent recognition result lengths or character segmentation differences. When processing document images containing mixed Chinese and English content or complex layout formats, this alignment method can accurately match the characters recognized by each model at different positions, establishing a unified coordinate framework for character-level confidence comparison.

[0057] Step S2413, for each aligned character position, compare the characters and corresponding character-level confidences recognized by all optical character recognition models at the character position.

[0058] Based on the aligned character position correspondence, the recognition results of all optical character recognition models at each character position are analyzed in detail. For each character position, the specific characters and corresponding character-level confidences recognized by each model at the position are compared to establish a complete character-confidence mapping table. This process can accurately grasp the candidate character set at each character position and its reliability evaluation, providing data support for selecting the optimal recognition result.

[0059] Step S2414, select the character with the highest character-level confidence at the character position as the character at the character position in the final recognition sequence.

[0060] Based on the character-level confidence comparison results, the optimal recognition result is selected for each character position. By comparing the confidence scores of all candidate characters at the same character position, the character with the highest confidence score is selected as the final recognition character at that position. This confidence-based selection mechanism ensures that the most reliable recognition result is adopted for each character position, effectively improving the individual character recognition accuracy.

[0061] In step S2415, the selected characters at all character positions are combined in order to obtain the final recognition result of the image to be recognized.

[0062] The optimal characters selected at all character positions are combined in the original sequence order to construct the complete final recognition result. This combination process maintains the spatial order and contextual relationship of characters in the original image, ensuring the coherence and readability of the output text. Through this fine-grained character-level fusion strategy, the advantages of multiple OCR models can be fully utilized, significantly improving the overall recognition accuracy while maintaining the integrity of the text structure.

[0063] In actual tests, this scheme reduces the character recognition error rate of complex layouts to less than 50% of the traditional single model method, and the recognition accuracy is particularly improved when processing technical documents containing special symbols and mixed fonts, providing a high-quality text data basis for subsequent text quality evaluation and knowledge base construction.

[0064] Further, the filtering of intermediate text data according to the preset confidence threshold in step S250 includes: In step S251, the original region type corresponding to the intermediate text data is recognized, including ordinary text, mathematical formula, and professional term text.

[0065] By analyzing the context features and layout information of the intermediate text data, the corresponding original region type is accurately recognized. Based on the region annotation information provided by the intelligent segmentation module, the text content is analyzed based on semantic features, and the text region is divided into three types: ordinary text, mathematical formula, and professional term text. For mathematical formula regions, special symbols, subscript structure, and formula layout features are detected for recognition; for professional term text regions, domain dictionary matching and term density analysis are used for determination; the remaining text regions are classified as ordinary text regions. This refined region classification provides accurate type identification for subsequent differentiated quality control.

[0066] In step S252, the corresponding confidence threshold is dynamically selected according to the region type, and the first confidence threshold corresponding to the mathematical formula and professional term text region is lower than the second confidence threshold corresponding to the ordinary text region.

[0067] According to the identified region type, a corresponding confidence threshold is dynamically selected to establish differentiated quality control standards. A relatively low first confidence threshold is set for mathematical formula and professional term text regions, because such content is difficult to identify, but preserving complete professional content is more important for training data quality; a relatively high second confidence threshold is set for ordinary text regions to ensure the accuracy and standardization of basic text. A threshold configuration table is maintained, which contains confidence threshold parameters corresponding to each region type based on a large amount of experimental data. These threshold parameters can be dynamically adjusted according to the needs of specific application scenarios to achieve flexible configuration of quality control strategies.

[0068] In step S253, the intermediate text data generated by the original region type is filtered based on the corresponding confidence threshold of each region type.

[0069] Based on the confidence threshold corresponding to each region type, the intermediate text data generated by the corresponding region is subjected to accurate filtering operation. For mathematical formula regions, the first confidence threshold is used for filtering to retain more text content that may contain important formula information; for professional term text regions, the first confidence threshold is also used to ensure the integrity of professional terms; for ordinary text regions, a higher second confidence threshold is used for strict filtering to eliminate unreliable recognition results. During the filtering process, the confidence score of the intermediate text data is checked piece by piece, and only the data with a score greater than or equal to the threshold value of the corresponding region type is retained to form high-quality recognized text data output.

[0070] By implementing this region type-based differentiated filtering strategy, the relationship between text retention rate and recognition accuracy is effectively balanced when processing complex academic documents. This scheme significantly improves the integrity of professional content recognition, while ensuring the accuracy of ordinary text, so that the final output of the recognized text data maintains the integrity of professional terms and mathematical formulas, while the accuracy of ordinary text reaches a higher level, providing a comprehensive and accurate text corpus basis for large language model training.

[0071] Further, in step S240, an optical character recognition model based on an improved deep convolutional neural network is called to process each OCR recognition subtask, including: In step S2421, the weight parameters of the input layer, output layer and batch normalization layer in the deep convolutional neural network are retained as single-precision floating point number format, while the weight parameters of the intermediate convolutional layer are converted to half-precision floating point number format to obtain a mixed-precision model.

[0072] The model precision conversion operation is performed to construct a mixed precision recognition model suitable for GPU acceleration. The process first analyzes the sensitivity of each layer in the deep convolutional neural network to numerical precision, determines that the input layer, output layer and batch normalization layer need to maintain high numerical precision due to the involvement of data standardization and final output, etc. key calculations, so the weight parameters of these layers are kept in single precision floating point format. At the same time, a large number of intermediate convolutional layers in the recognition network are not sensitive to changes in numerical precision, and their weight parameters are converted to half-precision floating point format. The conversion process is implemented through an automatic precision selector, which automatically identifies the layers that need to be converted according to the network structure and calculation characteristics, and applies loss scaling technology during the conversion process to maintain the stability of numerical calculations by dynamically adjusting the range of gradient values, ensuring that the model precision does not decrease significantly due to format conversion.

[0073] Step S2422, performing optical character recognition inference calculation of the mixed precision model on the GPU, wherein the weight parameters in half-precision floating point format are accelerated using the dedicated half-precision calculation unit of the GPU.

[0074] Deploy and execute the optical character recognition inference calculation of the mixed precision model on the GPU. After loading the mixed precision model, the dedicated half-precision calculation unit of the GPU (such as the tensor core) performs parallel calculation acceleration on the weight parameters in half-precision floating point format, greatly improving the throughput of matrix multiplication and convolution operations. At the same time, the dynamic precision coordinator manages the data transfer between layers of different precision to ensure that the input and output data between single-precision layers and half-precision layers can be correctly converted and connected, maintaining the integrity of the computation graph. When performing the recognition task, the utilization rate of the GPU operation unit is also monitored in real time, and the calculation task allocation strategy is dynamically adjusted according to the load to avoid performance bottlenecks caused by precision mixing, maximizing the speed advantage of mixed precision calculation.

[0075] By implementing the mixed precision calculation strategy, the processing efficiency is significantly improved while maintaining the recognition accuracy. In actual tests, this scheme improves the inference speed of the deep learning model by 3-5 times, and the GPU calculation core utilization rate reaches more than 85%. At the same time, due to the reduction of memory usage caused by half-precision calculation, larger batches of image data can be processed, further improving the overall throughput. This optimization enables significant performance improvement without increasing hardware costs when processing large-scale documents, while reducing energy consumption and providing reliable protection for long-term stable operation.

[0076] Further, before the step S200 of adopting the parallel OCR recognition engine based on GPU acceleration and heterogeneous computing, it further includes: Step S201, calculating a priority score based on at least one of the document size, format complexity and user specified priority of each original document.

[0077] The priority of each original document is evaluated by the task priority assessment module, and the priority score is calculated based on three dimensions of document size, format complexity and user-specified priority. The document size parameter is quantified according to the actual storage capacity of the file, the format complexity parameter is obtained by analyzing the document structure level, image density and layout complexity, and the user-specified priority is determined according to the urgency set by the business requirement. The three parameters are combined into a unified priority score by using a weighted algorithm, in which the user-specified priority has the highest weight, the format complexity has the second highest weight, and the document size has the lowest weight, ensuring that the task scheduling considers both business requirements and processing efficiency.

[0078] In step S202, according to the priority score, a plurality of original documents are sorted in a processing queue.

[0079] According to the calculated priority score, all original documents to be processed are sorted to build a processing queue. The sorting algorithm uses descending order arrangement, and the document with the highest priority score is placed at the front of the queue to ensure that important documents can be prioritized in the processing flow. The processing queue uses a dynamic data structure, which supports the insertion of new processing tasks at any time during system operation without affecting the existing queue order, while recording the metadata information and priority score of each document to provide decision basis for subsequent resource scheduling.

[0080] In step S203, the CPU utilization and memory usage are detected in real time.

[0081] The performance monitoring module detects the key resource indicators in real time, including CPU utilization and memory usage. The monitoring module collects system resource data at a frequency of seconds, establishes a resource usage trend model, and accurately reflects the current load state of the system. When the CPU utilization continuously exceeds 80% or the memory usage exceeds 85%, a resource warning signal is generated, providing a trigger condition for subsequent processing rate adjustment to ensure that the system always runs in a stable state.

[0082] In step S204, based on the CPU utilization and memory usage, the rate and number of concurrent original documents submitted from the processing queue to the parallel OCR recognition engine are dynamically adjusted.

[0083] Based on the real-time resource monitoring data, the document submission strategy is dynamically adjusted. When the system resource usage is low, the intelligent batch processing scheduling module increases the rate and number of concurrent original documents submitted from the processing queue to the parallel OCR recognition engine, making full use of idle computing resources; when the resource usage approaches the warning threshold, the submission rate is correspondingly reduced, and the number of concurrent tasks is reduced to prevent system overload. The adjustment process adopts a gradual strategy to avoid sharp fluctuations in resource usage, ensuring smooth progress of the processing task.

[0084] By implementing intelligent task scheduling and resource management mechanism, significant efficiency improvement is shown in processing large-scale documents. In actual deployment, the scheme makes the system resource utilization rate remain in the optimal interval, and the task completion time is shortened by more than 40% compared with the traditional first-come-first-served strategy, while effectively avoiding the system instability phenomenon caused by resource competition, providing reliable scheduling guarantee for large-scale document processing.

[0085] Further, the multi-dimensional quality evaluation and cleaning of the recognized text data of each original document in step S300 includes: In step S310, noise content filtering, repeated content detection and language model quality evaluation are sequentially performed on the recognized text data.

[0086] Among them, noise content filtering is based on regular expression rules and machine learning classifiers to identify and clean noise text including headers, footers, watermarks. Repeated content detection is based on local sensitive hashing algorithm to calculate local sensitive hash value for each text unit, and through comparing local sensitive hash values, repeated or highly similar text content is identified and removed. Language model quality evaluation is realized by calling a pre-set language model, the grammatical correctness score and semantic coherence score of the recognized text data are calculated, and a comprehensive quality score is obtained based on the weighted combination of the two.

[0087] Through the data cleaning verification module, a systematic quality evaluation and cleaning process is performed on the recognized text data. The process sequentially performs three key processing stages: the noise content filtering stage is based on the cooperative working mechanism of regular expression rules and machine learning classifiers, the regular expression rules are used to identify structured noise such as headers, footers, page numbers and watermark marks with fixed patterns, and the machine learning classifier detects meaningless character combinations and random noise text through the trained feature model, and the combination of the two realizes comprehensive coverage and effective cleaning of different types of noise content. The repeated content detection stage uses the local sensitive hashing algorithm to generate a local sensitive hash value representing the content characteristics of each text unit, and through the construction of a hash index library, the rapid comparison of text similarity is realized, when the hash value similarity exceeds the preset threshold, the text content is marked as repeated or highly similar content and removed from the data stream. The language model quality evaluation stage calls a pre-trained large language model to analyze the text in depth, the model calculates the grammatical correctness score and semantic coherence score of the text respectively, the grammatical correctness score evaluates the degree of text compliance with grammar rules, and the semantic coherence score measures the logical consistency of text in the context, finally the two score components are combined into a comprehensive quality score through a preset weight coefficient, forming a quantitative evaluation index of text quality.

[0088] By implementing this multi-level quality control process, significant advantages are shown in processing large-scale documents containing complex layouts and multiple types of content. In actual testing, the scheme successfully removed noise content from the original recognized text by more than 95%, and the repeated content recognition accuracy was more than 98%, while the language model evaluation effectively identified text fragments with semantic discontinuity, making the final output of standardized text data achieve a high level in grammatical correctness and semantic integrity, providing a clean and high-quality structured corpus basis for large language model training.

[0089] Further, the output of the standardized text data in step S300 includes: Step S321, based on the comprehensive quality score and the results of noise content filtering and repeated content detection, the recognized text data is divided into three quality levels.

[0090] Among them, the text with a comprehensive quality score greater than the first threshold and passing noise filtering and repeated content detection is rated as A level, the text with a comprehensive quality score greater than the second threshold and not greater than the first threshold and passing noise filtering is rated as B level, and the text with a comprehensive quality score greater than the third threshold and not greater than the second threshold but not passing repeated content detection is rated as C level.

[0091] Based on the multi-dimensional quality evaluation results, a three-level quality grading system is established, and the recognized text data is accurately graded through preset quality thresholds. First, three key quality threshold parameters are set: the first threshold as the access threshold for high-quality text, the second threshold as the recognition standard for medium-quality text, and the third threshold as the screening baseline for basic-quality text. In the specific implementation process, for the text that meets the three conditions of comprehensive quality score greater than the first threshold, passing noise content filtering and passing repeated content detection, it is rated as A level, which has the characteristics of grammatical specification, semantic coherence and no redundant content; for the text with a comprehensive quality score greater than the second threshold but not greater than the first threshold and passing noise content filtering, it is rated as B level, which although slightly insufficient in semantic coherence but ensures basic grammatical correctness; for the text with a comprehensive quality score greater than the third threshold but not greater than the second threshold and not passing repeated content detection, it is rated as C level, which usually contains repeated content or incomplete semantic problems. The entire grading process adopts an automated decision-making mechanism to ensure that each text can be accurately quality rated.

[0092] Step S322, only output the text rated as A level and B level as standardized text data.

[0093] According to the quality grading result, accurate data output control is performed. Based on the three-level quality grading result, a data output screening rule is established, only the texts rated as A and B are included in the standardized text data output range, and the C level text is excluded from the final output. This screening mechanism ensures that the output data has both high-quality content and large data coverage. A-level text provides core high-quality corpus for large language model training, and B-level text serves as an effective supplement to expand the diversity of training data. In the output process, each text is labeled with a corresponding quality level identifier, and detailed quality evaluation parameters are recorded to provide a reference for data weighting in subsequent training processes.

[0094] By implementing a strict quality grading and screening mechanism, optimal data utilization is achieved while ensuring data quality. In practical applications, this scheme enables the final output of standardized text data to meet high standards in terms of grammatical correctness, semantic coherence, and content uniqueness, while effectively balancing the relationship between data quality and data volume through the grading output strategy. This quality control method significantly improves the overall quality level of large language model training data, reduces the negative impact of low-quality data on model training, and reduces the cost of subsequent data cleaning manual intervention, providing a reliable data foundation for building high-quality language models.

[0095] Correspondingly, please refer to Figure 2 The second aspect of the embodiment of the present application provides a multi-modal document data processing system for large language model training, which recognizes multi-modal documents based on the above-mentioned multi-modal document data processing method for large language model training, comprising: A data receiving module 1 is used to receive a plurality of original documents in various formats, extract the structure information of each original document, and identify the text area and image area of each original document based on the structure information; An OCR recognition module 2 is used to use a parallel OCR recognition engine based on GPU acceleration and heterogeneous computing to perform optical character recognition on the text area and image area, and generate recognized text data of the corresponding original document; A data processing module 3 is used to perform multi-dimensional quality evaluation and cleaning on the recognized text data of each original document, and output standardized text data meeting the quality level; A data verification module 4 is used to store the standardized text data to a distributed knowledge base according to a predefined structure, and perform copyright and compliance verification on the standardized text data.

[0096] Correspondingly, a third aspect of the embodiment of the present application provides an electronic device, comprising: at least one processor; and a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned multi-modal document data processing method for large language model training.

[0097] Correspondingly, a fourth aspect of the embodiment of the present application provides a computer readable storage medium having stored thereon computer instructions, which, when executed by a processor, implement the above-mentioned multi-modal document data processing method for large language model training.

[0098] The embodiment of the present application aims to protect a multi-modal document data processing method and system for large language model training, which has the following effects: 1. By constructing a parallel processing architecture based on GPU acceleration and heterogeneous computing, the efficiency of multi-modal document data processing is qualitatively improved; the OCR recognition task is dynamically decomposed into parallel sub-tasks by using a multi-thread scheduler, combined with mixed precision calculation and dynamic memory management technology, the advantages of GPU hardware parallel computing are fully utilized, the processing speed reaches 3-5 images per second, which is more than 500% higher than the traditional single-thread processing method, and the resource utilization is improved by more than 75%, completely solving the efficiency bottleneck problem in the process of preparing large language model training data; 2. By establishing a multi-level intelligent quality control system, the quality and reliability of the output text data are significantly improved; integrating noise filtering, repeated content detection and language model quality evaluation triple protection mechanism, based on the confidence threshold strategy of regional type self-adaption to ensure that different contents adopt differentiated quality control standards, combined with a three-level quality rating system, the proportion of A-level quality of output data is improved from 67% of traditional method to more than 94%, providing high-quality, low-noise structured corpus for large language model training; 3. It has excellent scalability and stability, and can meet the demand of PB-level large-scale data processing; through the intelligent batch processing scheduling module, the task execution strategy is dynamically adjusted based on the system resource state, combined with the distributed knowledge base storage architecture and containerized deployment scheme, the elastic expansion of processing scale is realized; actual deployment shows that while maintaining high throughput, memory usage efficiency is improved by 37%, and manual intervention cost is reduced by 80%, providing reliable technical support for enterprise-level large-scale document data processing.

[0099] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0100] The present application is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to embodiments of the application. It should be understood that each flow and / or block in the flowchart and / or block diagrams, and a combination of flows and / or blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flowchart and / or block diagram block or blocks.

[0101] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions means which implement the function specified in the flowchart and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flowchart and / or block diagram block or blocks.

[0102] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flowchart and / or block diagram block or blocks.

[0103] Finally, it should be noted that the above-mentioned embodiments are merely intended for describing and illustrating, not limiting, the technical solutions of the present application. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and any modification or equivalent replacement without departing from the spirit and scope of the present application should be covered in the protection scope of the claims of the present application.

Claims

1. A method for processing multimodal document data for training large language models, characterized in that, Includes the following steps: Receive several original documents in various formats, extract the structural information of each original document, and identify the text region and image region of each original document based on the structural information; A parallel OCR recognition engine based on GPU acceleration and heterogeneous computing is used to perform optical character recognition on the text region and the image region to generate the corresponding recognized text data of the original document; The identified text data of each original document is subjected to multi-dimensional quality assessment and cleaning, and standardized text data that meets the quality level is output. The standardized text data is stored in a distributed knowledge base according to a predefined structure, and the standardized text data is then checked for copyright and compliance.

2. The multimodal document data processing method for training large language models according to claim 1, characterized in that, The parallel OCR recognition engine, based on GPU acceleration and heterogeneous computing, performs optical character recognition on the text region and the image region to generate corresponding recognized text data of the original document, including: Each of the text regions and the image regions is preprocessed to obtain a corresponding image to be recognized; The text regions and image regions of several original documents are combined into several batches of images to be recognized, and each batch of images to be recognized is decomposed into multiple independent OCR recognition subtasks, with each OCR recognition subtask corresponding to an image to be recognized. Based on a heterogeneous computing platform consisting of CPU and GPU, multiple independent OCR recognition subtasks are distributed in parallel to multiple computing threads for processing through a multi-thread scheduler. The optical character recognition model based on an improved deep convolutional neural network is invoked to process each OCR recognition subtask. The sequence recognition module based on the Transformer architecture performs text sequence decoding and error correction on the output of the optical character recognition model to generate intermediate text data and its corresponding confidence score. The intermediate text data is filtered according to a preset confidence threshold, and the intermediate text data with a confidence score lower than the preset confidence threshold is removed. The intermediate text data with a confidence score greater than or equal to the preset confidence threshold are then aggregated into the recognition text data of the corresponding original document.

3. The multimodal document data processing method for training large language models according to claim 2, characterized in that, The step of combining all the images to be identified corresponding to the text regions and image regions of several original documents into several batches of images to be identified includes: Real-time monitoring of the GPU's current available video memory capacity and computing core utilization; Based on the currently available video memory capacity and computing core utilization, the batch size of each batch of images to be identified is dynamically adjusted; Based on the batch size, the images to be identified are combined into several batches of images to be identified, and the number of images in each batch of images to be identified is matched with the real-time processing capability of the GPU.

4. The multimodal document data processing method for training large language models according to claim 2, characterized in that, The invocation of an optical character recognition model based on an improved deep convolutional neural network to process each OCR recognition subtask includes: At least two different optical character recognition models are invoked in parallel to recognize the same image to be recognized, and each model outputs a recognition result containing a character sequence and its corresponding character-level confidence. The character sequences output by each optical character recognition model are aligned based on the dynamic time warping algorithm to establish the correspondence between character positions; For each aligned character position, compare the characters recognized by all optical character recognition models at that character position with their corresponding character-level confidence scores; The character with the highest character-level confidence at the specified character position is selected as the character at that position in the final recognition sequence; The selected characters at all positions are combined in sequence to obtain the final recognition result of the image to be recognized.

5. The multimodal document data processing method for training large language models according to claim 2, characterized in that, The filtering of the intermediate text data based on a preset confidence threshold includes: Identify the original region type corresponding to the intermediate text data, where the region type includes ordinary text, mathematical formulas, and technical terminology text; The corresponding confidence threshold is dynamically selected based on the region type. The first confidence threshold for the mathematical formula and technical term text region is lower than the second confidence threshold for the ordinary text region. The intermediate text data generated from the original region type is filtered based on the confidence threshold corresponding to each region type.

6. The multimodal document data processing method for training large language models according to claim 2, characterized in that, The invocation of an optical character recognition model based on an improved deep convolutional neural network to process each OCR recognition subtask includes: The weight parameters of the input layer, output layer and batch normalization layer in the deep convolutional neural network are retained in single-precision floating-point format, while the weight parameters of the intermediate convolutional layers are converted to half-precision floating-point format to obtain a mixed-precision model. Optical character recognition inference computation of the mixed-precision model is performed on the GPU, wherein the weight parameters in half-precision floating-point format are accelerated using the dedicated half-precision computing unit of the GPU.

7. The multimodal document data processing method for training large language models according to any one of claims 1-6, characterized in that, Before adopting the parallel OCR recognition engine based on GPU acceleration and heterogeneous computing, it also includes: A priority score is calculated based on at least one of the following parameters for each original document: document size, format complexity, and user-specified priority. Based on the priority scores, the plurality of original documents are sorted into a processing queue; Real-time monitoring of CPU utilization and memory usage; Based on the CPU utilization and memory usage, the rate and number of concurrent submissions of original documents from the processing queue to the parallel OCR recognition engine are dynamically adjusted.

8. The multimodal document data processing method for training large language models according to any one of claims 1-6, characterized in that, The process of performing multi-dimensional quality assessment and cleaning on the identified text data of each original document includes: The identified text data is subjected to noise content filtering, duplicate content detection, and language model quality assessment in sequence. The noise content filtering is based on regular expression rules and machine learning classifiers to identify and clean up noisy text, including headers, footers, and watermarks. The duplicate content detection is based on the Locality Sensitive Hash (LSH) algorithm to calculate the LSH value for each text unit, and identifies and removes duplicate or highly similar text content by comparing the LSH values. The language model quality assessment is achieved by calling a preset language model to calculate the grammatical correctness score and semantic coherence score of the identified text data, and obtains a comprehensive quality score based on the weighted combination of the two.

9. The multimodal document data processing method for training large language models according to claim 8, characterized in that, The output consists of normalized text data that meets the quality level, including: Based on the comprehensive quality score and the results of noise content filtering and duplicate content detection, the identified text data is divided into three quality levels; Among them, texts with a comprehensive quality score greater than the first threshold and passing noise filtering and duplicate content detection are rated as Grade A; texts with a comprehensive quality score greater than the second threshold but not greater than the first threshold and passing noise filtering are rated as Grade B; and texts with a comprehensive quality score greater than the third threshold but not greater than the second threshold but failing duplicate content detection are rated as Grade C. Only the texts rated A and B are output as the normalized text data.

10. A multimodal document data processing system for training large language models, characterized in that, The method for processing multimodal document data based on any one of claims 1-9 for training large language models is used to identify multimodal documents, including: A data receiving module is used to receive several original documents in multiple formats, extract the structural information of each original document, and identify the text region and image region of each original document based on the structural information. The OCR recognition module is used to perform optical character recognition on the text region and the image region using a parallel OCR recognition engine based on GPU acceleration and heterogeneous computing, and generate corresponding recognition text data of the original document. The data processing module is used to perform multi-dimensional quality assessment and cleaning on the identified text data of each original document, and output standardized text data that meets the quality level. The data verification module is used to store the standardized text data in a distributed knowledge base according to a predefined structure, and to perform copyright and compliance verification on the standardized text data.

Images