Methods, devices, computer equipment, and storage media for extracting information from aviation material documents.

Abstract

This application relates to a method, apparatus, computer equipment, and storage medium for extracting information from aviation material documents. The method includes: inputting document data to be identified into multiple large Virtual Models (VLMs) to obtain VLM output datasets; the document data to be identified includes unstructured heterogeneous aviation material data; inputting the document data to be identified into an OCR model to obtain OCR output data; and inputting the VLM output dataset and the OCR output data into a fine-tuned small VLM model to obtain target recognition information for the document data to be identified. This application can effectively improve the reliability of aviation material document information recognition.

Description

Technical Field

[0001] This application relates to the field of visual language large model technology, and in particular to a method, apparatus, computer equipment and storage medium for extracting information from aviation material documents. Background Technology

[0002] In the field of aviation material information management, efficient and accurate processing of massive amounts of data is a key foundation for achieving business digitalization and intelligence. This field involves a wide variety of data types, which often exist in unstructured forms, such as image files or PDF documents containing basic aircraft information, customs declaration documents, and detailed maintenance records.

[0003] However, when traditional information extraction models are applied to extract aviation material data, their output content exhibits significant instability and uncontrollability (such as hallucinations, format chaos, and ignoring key constraints). The model's performance fluctuates greatly, making it difficult to meet the high reliability requirements of aviation material business. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer equipment, and storage medium for extracting aviation material document information that can effectively improve the reliability of identification, in order to address the above-mentioned technical problems.

[0005] Firstly, in one example, this application provides a method for extracting information from aviation material documents, including:

[0006] The document data to be identified is input into multiple VLM large models to obtain the VLM output dataset; the document data to be identified includes unstructured heterogeneous data of aviation materials;

[0007] Input the document data to be recognized into the OCR model to obtain the OCR output data;

[0008] Input the VLM output dataset and OCR output data into the fine-tuned VLM mini-model to obtain target recognition information for the document data to be recognized.

[0009] In one embodiment, the multiple VLM large models include a first VLM large model, a second VLM large model, and a third VLM large model with different architecture types;

[0010] The method also includes:

[0011] Obtain the meta dataset;

[0012] The images in the metadata set are input into the OCR model to obtain the first OCR output data;

[0013] The meta-dataset is input into the first VLM large model, the second VLM large model, and the third VLM large model respectively to obtain the fine-tuning training dataset.

[0014] Based on the first OCR output data and the fine-tuning training dataset, the pre-trained small model is fine-tuned in multiple stages to obtain the fine-tuned VLM small model.

[0015] In one embodiment, the pre-trained small model is a first VLM small model; based on the first OCR output data and the fine-tuning training dataset, the pre-trained small model undergoes multi-stage fine-tuning training to obtain the fine-tuned VLM small model, including:

[0016] Based on the fine-tuned training dataset, the visual-language alignment module and language module in the first VLM mini-model are fine-tuned to obtain the second VLM mini-model.

[0017] Based on the fine-tuned training dataset and the first OCR output data, the language module in the second VLM mini-model is fine-tuned to obtain the third VLM mini-model.

[0018] Based on the fine-tuned training dataset and the first OCR output data, the language module in the third VLM mini-model is fine-tuned to obtain the fine-tuned VLM mini-model.

[0019] In one embodiment, based on a fine-tuned training dataset, the visual-language alignment module and the language module in the first VLM mini-model are fine-tuned to obtain a second VLM mini-model, including:

[0020] The fine-tuning training dataset is labeled to indicate the correctness of the model recognition results for each element in the fine-tuning training dataset; where the model recognition results characterize whether the VLM large model correctly recognizes document images;

[0021] Based on the labeled fine-tuned training dataset, the language module of the first VLM mini-model is fine-tuned using the LoRA fine-tuning mode, and the visual-language alignment module of the first VLM mini-model is fine-tuned using the full fine-tuning mode to obtain the second VLM mini-model.

[0022] In one embodiment, based on the labeled fine-tuned training dataset, the language module of the first VLM mini-model is fine-tuned using LoRA fine-tuning mode, and the visual-language alignment module of the first VLM mini-model is fine-tuned using full fine-tuning mode to obtain the second VLM mini-model, including:

[0023] Based on the labeled fine-tuned training dataset, the LoRA fine-tuning mode is used to fine-tune the Attention layer in the language module;

[0024] Based on the first loss function, the visual-language alignment module is fine-tuned using the full fine-tuning mode to obtain the fine-tuned parameters of the visual-language alignment module;

[0025] The LoRA layer parameters of the first VLM mini-model are merged according to a preset merging formula;

[0026] The first VLM mini-model is updated with the fine-tuned visual-language alignment module parameters to obtain the second VLM mini-model.

[0027] In one embodiment, based on the fine-tuning training dataset and the first OCR output data, the language module in the second VLM mini-model is fine-tuned to obtain a third VLM mini-model, including:

[0028] The first fine-tuning training data is obtained based on the fine-tuning training dataset and the first OCR output data;

[0029] Based on the first fine-tuning training data, the language module of the second VLM mini-model is fine-tuned and trained using the DoRA fine-tuning mode and the second loss function to obtain the third VLM mini-model.

[0030] In one embodiment, the language module in the third VLM mini-model is fine-tuned based on the fine-tuned training dataset and the first OCR output data to obtain the fine-tuned VLM mini-model, including:

[0031] The real answers in the fine-tuning training dataset are standardized, and the second fine-tuning training data is obtained based on the standardized fine-tuning training dataset and the first OCR output data.

[0032] Based on the third loss function and the second fine-tuned training data, the LoRA fine-tuning mode is used to fine-tune the Attention layer of the language module of the third VLM mini-model to obtain the fine-tuned VLM mini-model; wherein, the third loss function is calculated based on the GRPO algorithm combined with business rules.

[0033] Secondly, in one example, this application provides an aviation material document information extraction device, comprising:

[0034] The VLM large model processing module is used to input the document data to be identified into multiple VLM large models to obtain the VLM output dataset; the document data to be identified includes unstructured heterogeneous data of aviation materials;

[0035] The OCR model processing module is used to input the document data to be recognized into the OCR model and obtain the OCR output data.

[0036] The recognition module is used to input the VLM output dataset and OCR output data into the fine-tuned VLM mini-model to obtain target recognition information for the document data to be recognized.

[0037] Thirdly, in one example, this application provides a computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method as described in any of the first aspects.

[0038] Fourthly, in one example, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the first aspects.

[0039] The aforementioned method, apparatus, computer equipment, and storage medium for extracting aviation material document information involve inputting the document data to be identified into multiple large VLM models to obtain VLM output datasets; inputting the document data to be identified into an OCR model to obtain OCR output data; and inputting the VLM output dataset and OCR output data into a fine-tuned small VLM model to finally obtain target recognition information for the document data to be identified. This approach effectively improves the reliability of aviation material document information recognition. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is an application environment diagram of the aviation material document information extraction method in one embodiment;

[0042] Figure 2 This is a flowchart illustrating a method for extracting aviation material document information in one embodiment;

[0043] Figure 3 This is a flowchart illustrating the process of obtaining a fine-tuned VLM mini-model in one embodiment;

[0044] Figure 4 This is an image of a document containing aviation material information in a real-world application scenario, as shown in one embodiment.

[0045] Figure 5 This is a flowchart illustrating the process of obtaining a fine-tuned VLM mini-model through multi-stage fine-tuning in one embodiment.

[0046] Figure 6This is a schematic diagram of the process for obtaining the second VLM small model in one embodiment;

[0047] Figure 7 This is a schematic diagram of the process for obtaining the second VLM small model in another embodiment;

[0048] Figure 8 This is a schematic diagram of the process for obtaining a third VLM mini-model in one embodiment;

[0049] Figure 9 This is a flowchart illustrating the process of obtaining a fine-tuned VLM mini-model in another embodiment;

[0050] Figure 10 This is a structural block diagram of an aviation material document information extraction device in one embodiment;

[0051] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0053] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various models or data information, but these models or data information are not limited by these terms. These terms are only used to distinguish the first model or data information from the second model or data information. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the solutions, or any combination of multiple solutions.

[0054] In the field of aviation materials information technology, a large amount of data needs to be digitized. This data includes aircraft information, customs information, maintenance records, and more. Aviation materials data mainly exists in the form of image files (such as scanned copies and photos) or PDF (Portable Document Format) documents. Due to the diverse sources of aviation materials data (such as different suppliers, maintenance workshops, and regulatory agencies), even information describing similar objects (such as engine maintenance dates) often has significant differences in its specific representation format and layout. Therefore, it is necessary to first standardize these heterogeneous data according to established business rules and standards, and convert them into a unified structured expression form in order to effectively complete the digitization and provide solid data support for subsequent core business operations such as aviation materials procurement, inventory management, maintenance scheduling, and airworthiness tracking.

[0055] Currently, recognition methods such as Optical Character Recognition (OCR) and Vision-Language Model (VLM) are used for automated document data extraction.

[0056] Traditional OCR technology excels at recognizing printed text with uniform fonts and layouts, demonstrating high accuracy when handling well-formatted and standardized printed text. However, its limitation lies in its lack of understanding of deep semantic information within images. When faced with complex table structures common in aviation documents (such as nested headers and merged cells) and scenarios requiring an understanding of contextual semantic relationships for accurate data extraction, traditional OCR performance falls short, failing to meet the demands of aviation material data recognition and often resulting in incomplete or incorrect extraction of key information from some documents. Furthermore, traditional OCR technology suffers a significant drop in accuracy when dealing with image quality degradation (such as blurriness, low resolution, shadow occlusion, and background interference). Its recognition capability is highly dependent on standardized page layout and printing quality. When faced with the diverse fonts (such as handwriting, special symbols, and mixed languages), complex page layouts (such as multi-column layout, mixed text and graphics, and nested tables) or non-standard document forms (such as tilted shooting, binding obstruction, and stamp coverage) commonly found in aviation documents, it is very easy to miss text areas, make character segmentation errors, and misidentify content, resulting in failure to extract key information.

[0057] While traditional Visual-Text (VLM) models theoretically possess stronger image and text understanding and reasoning capabilities, their output exhibits significant instability and uncontrollability when applied to aviation material data extraction (e.g., illusions, formatting issues, ignoring key constraints). The model's performance fluctuates greatly, making it difficult to guarantee the high accuracy and reliability required for business operations. Furthermore, they face significant application bottlenecks in real-world scenarios: ① Blind spots and uneven capabilities: The recognition capabilities of traditional VLM models are limited by the coverage and training methods of their training data. Different VLM models show significant differences in performance on specific tasks (i.e., "strong areas" and "weak areas"), lacking stable and reliable overall recognition capabilities. For example, some VLM models can correctly recognize and extract the same content, while others cannot. ② Uncontrollable output structure: The output of VLM models is typically unstructured free text or JSON formats, and the organization of the output data is often inconsistent and unpredictable.

[0058] Furthermore, both traditional OCR technology and traditional Visual-Text (VLM) models are limited to recognizing and reconstructing raw data content from images. They lack the ability to intelligently transform and enhance data according to specific business scenario requirements, and cannot automatically execute the standardized operations necessary in the aviation materials field. For example, they are insufficient in areas such as data format standardization (e.g., converting various date formats "20250620", "20-Jun-2025", "6 / 20 / 25" into the standard "YYYY / MM / DD" format), explicit identification of units of measurement (e.g., automatically adding "kg" or "lb" labels after recognizing weight values), and standardized representation of missing values ​​(e.g., uniformly marking missing fields in images with "N / A"). This lack of adaptability to business rules means that the raw data extracted by these traditional technologies cannot directly meet the requirements of warehousing or downstream business systems (such as MRO, ERP), still requiring significant human resources for secondary processing and verification.

[0059] The shortcomings of these traditional OCR technologies and traditional visual-text models collectively mean that existing automated solutions struggle to meet actual business requirements in the data extraction and warehousing stages of the aviation materials sector. This inevitably necessitates significant manual intervention for data verification, error correction, and supplementary entry. Particularly noteworthy is the need to input critical information from core business systems (such as ERP, Enterprise Resource Planning, and MRO systems) databases. This information includes aircraft / engine serial numbers, part numbers (P / N), batch / lot numbers, airworthiness directives (AD), and maintenance work order numbers—often lengthy strings of numbers or alphanumeric combinations lacking significant semantic features. Manually verifying this information is not only extremely inefficient, creating a bottleneck in the process, but also highly susceptible to input errors due to visual fatigue or distraction, posing potential risks to subsequent material traceability, maintenance compliance, and even flight safety.

[0060] To address the aforementioned issues, this application provides a method for extracting information from aviation material documents.

[0061] The aviation material document information extraction method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0062] In one exemplary embodiment, such as Figure 2 As shown, a method for extracting information from aviation material documents is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S202 to S206. Wherein:

[0063] Step S202: Input the document data to be recognized into multiple VLM large models respectively to obtain the VLM output dataset.

[0064] The document data to be identified may include unstructured heterogeneous aviation material data. In some examples, unstructured heterogeneous aviation material data can characterize aviation material data with diverse data sources, different representation formats, and layouts. For example, heterogeneous aviation material data includes engine maintenance dates from different suppliers.

[0065] For example, the VLM output dataset may include recognition data output by multiple large VLM models after extracting image data from the document data to be recognized. Optionally, the aforementioned multiple large VLM models may be three specified large VLM models, including: Kimi-VL-A3B-Thinking, Qwen2.5-VL-72B-Instruct, and google / gemma-3-27b-it.

[0066] Specifically, the server can input the document data to be recognized into Kimi-VL-A3B-Thinking, Qwen2.5-VL-72B-Instruct, and google / gemma-3-27b-it respectively, and obtain the VLM output dataset containing the output data of the above three models.

[0067] Step S204: Input the document data to be recognized into the OCR model to obtain OCR output data.

[0068] OCR models can be used to perform image data recognition on input document data. After inputting the document data into the OCR model, the OCR output data is obtained, which is then saved in HTML format. In some examples, the PP-OCRv5 model can be used.

[0069] Specifically, the server can input the document data to be recognized into the OCR model and obtain the OCR output data corresponding to the document data to be recognized.

[0070] Step S206: Input the VLM output dataset and OCR output data into the fine-tuned VLM mini-model to obtain target recognition information for the document data to be recognized.

[0071] Among them, the fine-tuned VLM mini-model can be used to identify target recognition information for the document data to be recognized based on the VLM output dataset and OCR output data.

[0072] Specifically, the server can input the VLM output dataset and OCR output data into the fine-tuned VLM mini-model to obtain target recognition information for the document data to be recognized.

[0073] The aforementioned method for extracting information from aviation material documents first inputs the document data to be identified into multiple large VLM models to obtain VLM output datasets; then, it inputs the document data to be identified into an OCR model to obtain OCR output data; finally, it inputs the VLM output dataset and the OCR output data into a fine-tuned small VLM model to obtain target recognition information for the document data to be identified. Through this approach, this application constructs a multi-model collaborative recognition mechanism and proposes a collaborative recognition framework that dynamically integrates multiple heterogeneous OCR / VLM large models.

[0074] It is understandable that this application utilizes different technical approaches (such as rule-based traditional OCR and deep learning-based VLM) and large visual models with different architectures (such as dedicated models focusing on text detection, table understanding, or semantic association) in parallel, comprehensively leveraging the recognition advantages of each model in specific scenarios (such as the high accuracy of OCR for standard printed text and the semantic understanding capabilities of VLM), thereby forming a complementary collaborative recognition framework. The multi-model collaborative recognition mechanism of this application can effectively avoid the blind spots and shortcomings of single models in complex aviation documents (such as low-quality images, unconventional layouts, and multilingual content), thus significantly improving the robustness and recognition coverage of the overall system, reducing the false negative rate and misidentification rate from the source, and ultimately improving the reliability of aviation material document information recognition.

[0075] In one embodiment, the multiple VLM large models include a first VLM large model, a second VLM large model, and a third VLM large model with different architecture types. For example... Figure 3 As shown, the method further includes steps S302 to S308. Wherein:

[0076] Step S302: Obtain the metadata dataset.

[0077] In some examples, each piece of metadata in the metadata set may include data such as images, target extraction terms, and true answers; where a true answer refers to accurate and authentic identification information for the target extraction terms in the image.

[0078] For example, a meta-dataset can be represented as follows:

[0079]

[0080] in, Represents the first in the metadata set +1 image, Represents the first in the metadata set Target extraction terms, Then it means in the first In the image, targeting the first The true answer to the target extraction clause.

[0081] It should be noted that the basic data collection scope of the metadata dataset should cover as many application scenarios as possible for aviation materials. For example, the basic data required for the metadata dataset can be collected based on aircraft parameter information, transportation documents, financial statements, etc. Furthermore, the data sources of the basic data for the metadata dataset should be diverse; for example, the basic data of the metadata dataset can come from photographs, scanned images, printed text, etc. In practical applications, data analysis methods can also be used to avoid single-category target extraction clauses. Too much or too little data should be used to ensure a balanced distribution of data for the target extraction terms.

[0082] For example, such as Figure 4 As shown, Figure 4 It provides an image of a document containing aviation material information in a real-world application scenario. This can be used for... Figure 4 The repair date in the document image generates a metadata item in a metadata set, which can be represented as: ;in, Corresponding to Figure 4 The document image shown, This indicates the target extraction terms corresponding to the current metadata. Indicates in Figure 4 The document shown is for The true answer to this objective is the extraction clause.

[0083] Specifically, the server can obtain the metadata required for model training based on various application scenarios of aviation materials.

[0084] Step S304: Input the images in the metadata set into the OCR model to obtain the first OCR output data.

[0085] The first OCR output data can be the HTML format recognition data output by the OCR model for the image in the metadata set.

[0086] Understandably, the OCR model can be selected based on the aviation materials business scenario. Optionally, the PP-OCRv5 model can be used for image data recognition.

[0087] Specifically, the server can input images from the metadata set into the PP-OCRv5 model to obtain the first OCR output data in HTML format from the PP-OCRv5 model.

[0088] Step S306: Input the metadata into the first VLM large model, the second VLM large model, and the third VLM large model respectively to obtain the fine-tuning training dataset.

[0089] Specifically, the server inputs the metadata into the first VLM large model, the second VLM large model, and the third VLM large model respectively, and can obtain the fine-tuned training dataset.

[0090] In some examples, the first, second, and third VLM large models are Kimi-VL-A3B-Thinking, Qwen2.5-VL-72B-Instruct, and google / gemma-3-27b-it, respectively. That is, It can be Kimi-VL-A3B-Thinking, It can be Qwen2.5-VL-72B-Instruct, It can be google / gemma-3-27b-it.

[0091] In one possible implementation, the meta-dataset is input into each VLM large model separately. Then, based on the prompts from the first stage, prompts are given to each large VLM model to obtain a fine-tuned training dataset. The first-stage prompts... It can be represented as follows:

[0092] =

[0093] “You are an information extraction assistant with a background inaviation materials. Please follow the instructions below to extract imagedata.

[0094] Instructions:

[0095] 1. Fully understand the entire image.

[0096] 2. Only the information of the item needs to be extracted.

[0097] Item: ;

[0098] Image: "

[0099] Based on the above prompts and the output data of multiple large VLM models, the following fine-tuning training dataset can be obtained:

[0100]

[0101] in, Represents the first in the metadata set One image, Represents the first in the metadata set Target extraction terms, Then it means in the first In the image, targeting the first The true answer to the target extraction clause Indicates that the first one was applied. A large VLM model Indicates the first The VLM large model is for the first The first image The predicted answer for each target extraction clause.

[0102] Step S308: Based on the first OCR output data and the fine-tuning training dataset, perform multi-stage fine-tuning training on the pre-trained small model to obtain the fine-tuned VLM small model.

[0103] Multi-stage fine-tuning training can include a progressive fine-tuning strategy consisting of three stages: "discrimination → integration → reinforcement".

[0104] Specifically, the server can perform progressive fine-tuning training on the pre-trained small model in three stages: "discrimination → ensemble → reinforcement" based on the first OCR output data and the fine-tuning training dataset, in order to obtain the fine-tuned VLM small model.

[0105] For example, the above three-stage incremental fine-tuning strategy may include: Stage 1 (Discrimination Foundation): Fine-tuning the small model to establish the domain data discrimination capability of the VLM small model; Stage 2 (Integration Optimization): Fine-tuning the VLM small model to achieve intelligent fusion of multi-model results; Stage 3 (Rule Alignment): Using GRPO reinforcement learning combined with business rule reward mechanism to perform final optimization of the VLM small model.

[0106] It is understandable that the above-mentioned paradigm in this application focuses only on a small VLM model with 3B parameters. By focusing on capability development in stages, it is possible to reduce the training cost of the model while ensuring the accuracy of aerospace professional applications, thereby achieving the economic feasibility of industrial-grade implementation.

[0107] In one embodiment, the pre-trained small model is the first VLM small model. For example... Figure 5As shown, based on the first OCR output data and the fine-tuning training dataset, the pre-trained small model undergoes multi-stage fine-tuning training to obtain the fine-tuned VLM small model, including the following steps S402 to S406. Wherein:

[0108] Step S402: Based on the fine-tuning training dataset, fine-tune the visual-language alignment module and the language module in the first VLM mini-model to obtain the second VLM mini-model.

[0109] For example, the first VLM mini-model can be a VLM model with a lightweight parameter set. In some examples, the first VLM mini-model can be Qwen2.5-VL-3B-Instruct.

[0110] The vision-language alignment module is an important component of the Qwen2.5-VL model. It is used to map the features encoded by the vision module to the space of the language module.

[0111] Specifically, the server can fine-tune the visual-language alignment module and language module in Qwen2.5-VL-3B-Instruct based on the fine-tuning training dataset to obtain a second VLM mini-model.

[0112] Step S404: Based on the fine-tuning training dataset and the first OCR output data, fine-tune the language module in the second VLM mini-model to obtain the third VLM mini-model.

[0113] For example, in this fine-tuning stage, the outputs of multiple large VLM models and the first OCR output data (html data) extracted by the ORC model can be used as conditional inputs to accurately supervise the output results, so as to achieve fine-tuning training of the language module in the second small VLM model.

[0114] Specifically, the server can fine-tune the language module in the second VLM mini-model based on the fine-tuning training dataset and the first OCR output data to obtain the third VLM mini-model.

[0115] Step S406: Based on the fine-tuning training dataset and the first OCR output data, fine-tune the language module in the third VLM mini-model to obtain the fine-tuned VLM mini-model.

[0116] For example, the core objective of this fine-tuning training stage is to further train the third VLM small model obtained in the previous stage using the GRPO (Group Relative Policy Optimization) algorithm based on reinforcement learning, so that the data results extracted from the document images can more accurately meet the specific business rules requirements of the aviation materials field.

[0117] Specifically, the server can fine-tune the language module in the third VLM mini-model based on the fine-tuning training dataset and the first OCR output data to obtain the fine-tuned VLM mini-model.

[0118] In one embodiment, such as Figure 6 As shown, based on the fine-tuned training dataset, the visual-language alignment module and language module in the first VLM mini-model are fine-tuned to obtain the second VLM mini-model, including the following steps S502 to S504. Wherein:

[0119] Step S502: Label the fine-tuning training dataset to mark the correctness of the model recognition results of each element in the fine-tuning training dataset.

[0120] Among them, the model recognition results can characterize whether the VLM large model correctly recognizes the document image.

[0121] For example, based on the second-stage prompts, a multimodal large model can be used to label each element in the fine-tuning training dataset. The second-stage prompts... It can be represented as follows:

[0122] =

[0123] “You are a data quality assessment expert in aviation materials. Please strictly judge whether the predicted answer is semantically consistent with the real answer according to the following information and requirements.

[0124] 1. **Input data**:

[0125] - Image: { }

[0126] - Item to be extracted: { }

[0127] - VLM name: { }

[0128] - Real answer: { }

[0129] - Predicted answer: { }

[0130] 2. **Requirements:**

[0131] - The predicted answer is completely consistent with the real answer at the semantic level;

[0132] - Compatible with format differences but reject content deviation;

[0133] - Only reply "Yes" or "No", output "Yes" if semantically consistent, and "No" if not."

[0134] In some examples, the tagged data is recorded as .

[0135] in, Indicates the first The VLM large model for the first The first image The correctness indicator of the predicted answer for each target extraction clause. For example, when the predicted answer... Corresponding real answer When semantics are consistent and content matches, correctness is indicated. It can be marked as correct; when predicting the answer Corresponding real answer Correctness flag when semantic inconsistencies or content discrepancies occur. This can be marked as an error. Alternatively, large multimodal models can be implemented using large models such as Gemini 2.5 Pro.

[0136] In one possible implementation, the labeling of the fine-tuning training dataset can also be achieved using manual annotation. For example, the criteria for manual annotation could be as follows: simply determine whether the predicted answer and the true answer are semantically consistent and content-matching, ignoring differences in presentation, format, units, etc.

[0137] Specifically, the server can label the fine-tuning training dataset using a multimodal large model or manual annotation, thereby marking the correctness of the model recognition results for each element in the fine-tuning training dataset.

[0138] Step S504: Based on the labeled fine-tuning training dataset, fine-tune the language module of the first VLM mini-model using the LoRA fine-tuning mode, and fine-tune the visual-language alignment module of the first VLM mini-model using the full fine-tuning mode to obtain the second VLM mini-model.

[0139] For example, the objects of fine-tuning training in this stage (the first stage) are the visual-language alignment module in the first VLM small model and the first 1-8 Attention layers of the language module in the first VLM small model.

[0140] Optionally, the first VLM mini-model can be Qwen2.5-VL-3B-Instruct.

[0141] Specifically, based on the labeled fine-tuning training dataset, the server can fine-tune the language module of the first VLM mini-model using the LoRA fine-tuning mode, and fine-tune the visual-language alignment module of the first VLM mini-model using the full fine-tuning mode to obtain the second VLM mini-model.

[0142] In one embodiment, such as Figure 7 As shown, based on the labeled fine-tuned training dataset, the language module of the first VLM mini-model is fine-tuned using the LoRA fine-tuning mode, and the visual-language alignment module of the first VLM mini-model is fine-tuned using the full fine-tuning mode to obtain the second VLM mini-model, including the following steps S602 to S608. Wherein:

[0143] Step S602: Based on the labeled fine-tuning training dataset, fine-tune the Attention layer in the language module using the LoRA fine-tuning mode.

[0144] For example, taking the first VLM small model using Qwen2.5-VL-3B-Instruct as an example, for the language module in Qwen2.5-VL-3B-Instruct, the LoRA (Low-Rank Adaptation) fine-tuning mode can be used to fine-tune the parameter part of ["q_proj","v_proj","k_proj","o_proj"] of the Attention layer. It can be understood that the Attention layer includes three components: Q, K, and V. When performing LoRA fine-tuning, the Attention layer will add matrices A and B to the projection matrices of Q, V, K, and O, respectively; where ["q_proj","v_proj","k_proj","o_proj"] are the projection matrices of Q, V, K, and O, respectively.

[0145] Specifically, based on the labeled fine-tuned training dataset, the server can use the LoRA fine-tuning mode to fine-tune the Attention layer in the language module of the first VLM small model.

[0146] Step S604: Based on the first loss function, the vision-language alignment module is fine-tuned using the full fine-tuning mode to obtain the fine-tuned vision-language alignment module parameters.

[0147] For example, taking the first VLM small model using Qwen2.5-VL-3B-Instruct as an example, regarding the parameters of the vision-language alignment module in Qwen2.5-VL-3B-Instruct... It can be fine-tuned using the full-scale fine-tuning mode.

[0148] In some examples, according to Equation 1 as shown below, the first loss function used for fine-tuning training can be expressed as:

[0149] (Formula 1)

[0150] in, This indicates the weights for fine-tuning the visual-language alignment module. This indicates fine-tuning the LoRa weights of the language module, where N represents the batch size during fine-tuning training, and T represents the order of the output tokens. This indicates the output probability of the current token. Represents contextual input (corresponding image and) ), Represents the nth training data The target token, The cross-entropy loss represents the conditional probability of the model output at time t. This is used to constrain the amount of change in parameters at each layer of the vision-language alignment module after fine-tuning; and As a loss function adjustment factor, in practical applications, it can make... =0.5, .

[0151] Specifically, based on the first loss function, the server can fine-tune the visual-language alignment module of the first VLM small model using a full fine-tuning mode to obtain the fine-tuned visual-language alignment module parameters.

[0152] Step S606: Merge the LoRA layer parameters of the first VLM small model according to the preset merging formula.

[0153] For example, according to Formula 2 below, the preset merging formula can be expressed as:

[0154] (Formula 2)

[0155] in, This represents the first 8 Attention layers that are fine-tuned in this stage. Indicates the initial weights. This indicates the adjustment of hyperparameters. This indicates that the ranks of matrices A and B have been newly adjusted. This indicates the Lora weights for fine-tuning the language module. Optionally, .

[0156] Specifically, the server can merge the LoRA layer parameters of the first VLM mini-model according to the preset merging formula shown in Formula 2.

[0157] Step S608: Update the current visual-language alignment module parameters of the first VLM mini-model using the fine-tuned visual-language alignment module parameters to obtain the second VLM mini-model.

[0158] Specifically, the server can directly use the fine-tuned visual-language alignment module parameters to update the original visual-language alignment module parameters of the first VLM mini-model, thereby obtaining the second VLM mini-model.

[0159] In one embodiment, such as Figure 8 As shown, based on the fine-tuning training dataset and the first OCR output data, the language module in the second VLM mini-model is fine-tuned and trained to obtain the third VLM mini-model, including the following steps S702 to S704. Wherein:

[0160] Step S702: Obtain the first fine-tuning training data based on the fine-tuning training dataset and the first OCR output data.

[0161] For example, the first fine-tuning training data can be derived from ( , ,context_text, It consists of ) and context_text, which is composed of the following prompt:

[0162] context_text=

[0163] “You are a data integration and extraction assistant specializing inaviation materials. You need to make intelligent decisions based on multiple visual language models (VLM) to extract specified items from the same image, as well as OCR recognition results, and accurately output the datacorresponding to the terms to be extracted in the image.

[0164] **Input data * *:

[0165] Image: { };

[0166] Item to be extracted: { };

[0167] 3. Model {VLM1.name} Answer: { };

[0168] 4. Model {VLM2.name} Answer: { };

[0169] 5. Model {VLM3.name} Answer: { };

[0170] 6. ORC recognition: {html}”

[0171] In context_text, VLM1.name is the name of the corresponding VLM model, and html is the first OCR output data output by the OCR model.

[0172] Specifically, the server can construct the first fine-tuning training data based on the fine-tuning training dataset and the first OCR output data.

[0173] Step S704: Based on the first fine-tuning training data, the language module of the second VLM mini-model is fine-tuned and trained using the DoRA fine-tuning mode and the second loss function to obtain the third VLM mini-model.

[0174] For example, this stage (second stage) fine-tuning can use the outputs of multiple large VLM models and the HTML data extracted by the ORC model as conditional inputs to accurately supervise the output results, thereby fine-tuning the language module in the second small VLM model.

[0175] Specifically, based on the first fine-tuning training data, the server can fine-tune the language module of the second VLM mini-model using the DoRA fine-tuning mode and the second loss function to obtain the third VLM mini-model.

[0176] In some examples, taking the second VLM small model as Qwen2.5-VL-3B-Instruct after the first stage of fine-tuning, this stage uses the DoRA (Weight-Decomposed Low-Rank Adaptation) fine-tuning mode and the second loss function to further fine-tune the parameter part ["q_proj","v_proj","k_proj","o_proj","up_proj","down_proj"] of the Attention layer of the language module of Qwen2.5-VL-3B-Instruct; where "up_proj" and "down_proj" represent the up projection matrix and down projection matrix in the FFN module of Attention, respectively.

[0177] According to Formula 3, the second loss function can be expressed as:

[0178] (Formula 3)

[0179] in, Indicates the weight. This refers to the dora parameter used for the second stage of fine-tuning. This indicates the order in which the tokens are output. Indicates the output. Indicates context, This indicates the model output before time t. For each layer of matrices A and B, the regularization terms are... This represents the adjustment factor of the loss function for control. The percentage of this loss, This represents the model layer being fine-tuned, where A and B are matrices A and B added to the Attention layer during fine-tuning. In practical applications, this can be...

[0180] Furthermore, according to Formula 4 below, the weights of DoRA can be expressed as:

[0181] (Formula 4)

[0182] in, This indicates the weights used for DoRA fine-tuning in the second phase. These are the Attention layer weights after fine-tuning in the first stage. These are the dora parameters that are fine-tuned in the second stage. , The F-norm is used to prevent model collapse during training and improve the stability of the fine-tuning process. In practical applications, r=4 can be used for fine-tuning training in this stage, where k represents the original matrix size, which is determined by the initial parameters. Furthermore, after fine-tuning, the AB weights of the Attention layer of the language module can be merged with the original weights to obtain the third VLM mini-model.

[0183] In one embodiment, such as Figure 9 As shown, based on the fine-tuning training dataset and the first OCR output data, the language module in the third VLM mini-model is fine-tuned to obtain the fine-tuned VLM mini-model, including the following steps S802 to S804. Wherein:

[0184] Step S802: Standardize the real answers in the fine-tuning training dataset, and obtain the second fine-tuning training data based on the standardized fine-tuning training dataset and the first OCR output data.

[0185] It should be noted that the core objective of this stage (the third stage) of fine-tuning training is to further train the visual-language model (i.e., the third VLM small model) obtained in the second stage, so that the data results extracted from document images can more accurately meet the specific business rules requirements in the aviation materials field.

[0186] It is understandable that, in order to achieve the above goals, the key point is to make targeted modifications to the representation of the model output data. This modification aims to ensure that the data representation can not only strictly meet the needs of downstream business systems, but also effectively adapt to the training mechanism of the GRPO (Group Relative Policy Optimization) reinforcement learning algorithm.

[0187] Specifically, the server can standardize the real answers in the fine-tuning training dataset and obtain the second fine-tuning training data based on the standardized fine-tuning training dataset and the first OCR output data.

[0188] For example, taking the third VLM small model as Qwen2.5-VL-3B-Instruct after the first and second stage fine-tuning processes, the following is an example: The real answer in metadata After standardization, it can be modified into "data + <tag>The above standardized rewriting method not only unifies the data style, but more importantly, it provides a structured and quantifiable evaluation basis for the reward function of the GRPO algorithm.

[0189] In some examples, the true answers in the fine-tuning training dataset can be standardized based on a predefined metadata annotation format. These predefined metadata annotation formats include:

[0190] ① For data with units in the actual answers: Add a label containing the unit after the numerical value, for example: "Maximum Take Off Weight": 77000 <tag> kg< / tag> ;

[0191] ② For time data in actual answers: use the YYYY / MM / DD format and add a label, for example: "INS DATE": 2025 / 06 / 20 <tag> date< / tag> ;

[0192] ③ For data in the actual answer that has no unit or special identifier: Add an empty label pair after the data value, for example: "Serial Number": 340-000-826-0 <tag>< / tag> ;

[0193] ④ For missing data items in the images of the actual answers: Use N / A to explicitly indicate missing items and add empty label pairs, for example: "MSN": N / A <tag>< / tag> ;

[0194] ⑤ For other data information in the actual answer that requires further explanation, you can... <tag>< / tag> Interval annotation.

[0195] Furthermore, the metadata rewritten using the aforementioned preset metadata annotation format can be... , denoted as standardized metadata .

[0196] In some examples, aside from this standardization process performed on the true answers in the original metadata before inputting them into the model, the training data used in this stage of fine-tuning is completely identical to the dataset used in the second stage of fine-tuning. That is, the second fine-tuning training data can be derived from (… , ,context_text, )composition.

[0197] Step S804: Based on the third loss function and the second fine-tuned training data, the Attention layer of the language module of the third VLM mini-model is fine-tuned using the LoRA fine-tuning mode to obtain the fine-tuned VLM mini-model.

[0198] The third loss function is calculated based on the GRPO algorithm combined with business rules.

[0199] It should be noted that, due to the instability of the GRPO fine-tuning process, this stage (the third stage) uses the LoRA fine-tuning mode to fine-tune the language model part of Qwen2.5-VL-3B-Instruct in order to fine-tune the parameters of the ["q_proj","v_proj","k_proj","o_proj","up_proj","down_proj"] module of its Attention layer.

[0200] Specifically, based on the third loss function and the second fine-tuning training data, the server can use the LoRA fine-tuning mode to fine-tune the Attention layer of the language module of the third VLM small model.

[0201] For example, according to Equation 5 below, the third loss function can be expressed as:

[0202] (Formula 5)

[0203] in, The LoRa weights are fine-tuned in the third stage; The group represents the amount of data used in each training iteration; This indicates the weights for the third-stage fine-tuning. The output function under the strategy For the second phase (old weights) in The output function under the strategy For the clipping function, , Here, q is the hyperparameter, and q is the model input. For model output, To fine-tune the parameters, The original weights (i.e., the weights after the second stage of fine-tuning) are in Output distribution under the strategy The strategy represents the sampling strategy (e.g., the next token is selected based on the highest probability). In some examples, , Hyperparameters can be adaptively adjusted based on different stages of training.

[0204] Furthermore, according to Formula 6 below, the third loss function can be calculated. part:

[0205] (Formula 6)

[0206] Furthermore, the fine-tuning parameters of the third loss function can be calculated according to Formula 7 below. :

[0207] (Formula 7)

[0208] in, It can be calculated based on preset business rules. This indicates the calculation of the mean. This indicates variance calculation.

[0209] In some examples, the preset business rules may include the following Rules 1 through Rules 4:

[0210] Rule 1: When the model output is... When they are completely identical, 10 points were awarded; among them, <tag>The previous characters were exactly the same, so it's worth 5 points. <tag>* 5 points are awarded for correct characters within the *<\tag> tag.

[0211] Rule 2: When the length of the characters output by the model is... When the lengths are the same, 6 points were scored; among them, <tag>The preceding characters have the same length. 3 points; <tag>* 3 points are awarded if the length of characters within the `<\tag>` tag is equal to the length of the input character; if the model output length is equal to the input character length, the score is 3 points. If the length deviation is more than twice, then -6 points.

[0212] Rule 3: When the model outputs characters containing complete... <tag>When *<\tag>, 3 points.

[0213] Rule 4: (This is incomplete and likely refers to a specific rule or rule.) <tag>Maewa <tag>The two parts of data within *<\tag> are checked. If these two parts contain confusion errors involving {'O' and 'o'; 'l' and '1'; 'B' and '8'; 'S' and '5'}, then... -0.5 points; if there is confusion between uppercase and lowercase letters in these two parts of the data, -0.5 points will be awarded; each of the above deduction items will only be deducted once.

[0214] In some examples, this stage (stage three) of fine-tuning training may also include the following training details:

[0215] Training Details ①: In the early stages of training, reward scoring is only applied to Rules 1 to 3 of the aforementioned business rules. A reject sample mechanism is used to prevent consistent reward scores across the entire group, and the hyperparameters in the loss function are adjusted accordingly. =0.5, =1.0, to smooth the optimization process of the model.

[0216] Training Detail ②: When the loss function to be trained tends to stabilize (e.g., the loss value changes by less than 5% before two epochs), or when a certain number of epochs are reached, the Rule4 scoring rule in the above business rules is added, and the hyperparameters in the loss function are adjusted. =0.8, =0.5 to broaden the learning range of model weights, thereby increasing the model's fitting ability. It should be noted that at this stage, a reject sample mechanism can be used to avoid the entire group consisting of either high-scoring or low-scoring samples.

[0217] For example, a rejection sample mechanism can be used to exclude invalid or inefficient training samples. For instance, if the data and reward scores are consistent across all training batches, these samples are discarded and resampled until the model training requirements are met.

[0218] Understandably, the scoring details of the aforementioned business rules for extracting aviation material data, in addition to Rules 1 to 4, can be further supplemented with other scoring details based on the actual inference results of the model and business needs to further improve the accuracy of the model's data extraction. In some examples, if the magnitude of the reward value in the business rules causes instability in the reinforcement learning training process, the reward scores in the business rules can be normalized to improve the stability of the training process.

[0219] In one embodiment, after the fine-tuned VLM small model is trained using the methods of the above embodiments, the actual reasoning process for extracting information from the document data to be recognized may include the following steps A1 to A3. Wherein:

[0220] Step A1, put the document data to be recognized Input the VLM1, VLM2, and VLM3 models respectively, and obtain the corresponding model output data;

[0221] Step A2, then take the document data to be recognized. Input the ORC model to obtain the corresponding HTML recognition results;

[0222] Step A3: Input the context_text from step S702 as the prompt word into the pre-trained Qwen2.5-VL-3B-Instruct model (i.e., the fine-tuned VLM mini-model) to obtain the document data to be recognized. The target identification information. For example, the specific content of context_text can be represented as follows:

[0223] "You are a data integration and extraction assistant specializing inaviation materials. You need to make intelligent decisions based on multiple visual language models (VLM) to extract specified items from the same image, as well as OCR recognition results, and accurately output the datacorresponding to the terms to be extracted in the image.

[0224] **Input data * *:

[0225] Image: { };

[0226] Item to be extracted: { };

[0227] 3. Model {VLM1.name} Answer: { };

[0228] 4. Model {VLM2.name} Answer: { };

[0229] 5. Model {VLM3.name} Answer: { };

[0230] 6. ORC recognition: {html}”

[0231] It is understood that by using this application to extract aviation material document information, at least the following beneficial technical effects can be achieved:

[0232] ①This application achieves a breakthrough in accuracy through multi-model collaborative recognition:

[0233] This application effectively integrates the high-precision character recognition capabilities of traditional OCR engines with the deep semantic understanding advantages of multiple large-scale visual-language models (VLM models) by constructing a dynamic heterogeneous model fusion architecture. Furthermore, this dynamic heterogeneous model fusion architecture employs an intelligent weighted decision-making mechanism, which can: (a) preserve the stability of OCR models in the recognition of standardized printed text; (b) maximize the utilization of the specific advantages of different VLM models in scenarios such as table parsing, multilingual processing, blurred images, and multi-image source adaptation; and (c) effectively suppress recognition errors between models through cross-validation. Based on real experimental data, the technical solution of this application can improve the accuracy of key field recognition to 86.7% on the aviation material document test set, which is 12.5 percentage points higher than the traditional single optimal model, greatly alleviating the systemic recognition defects caused by the limitations of a single model.

[0234] ②This application employs a progressive training paradigm to achieve fine-tuning of a low-cost image data extraction model:

[0235] This application uses the lightweight Qwen2.5-VL-3B-Instruct (a small VLM model with 3B parameters) as a hub to distill the aerospace domain knowledge of billions of VLMs into a small model, reducing training costs to a level acceptable to small and medium-sized enterprises, thereby achieving a sharp reduction in model capability transfer costs. Secondly, this application innovatively designs structured outputs (such as unified representations). <tag> kg< / tag> Unit labeling, date standardization, and N / A missing value marking enable seamless integration between the model output and the aviation materials business system. This significantly reduces preprocessing time before data entry, allowing for deep reinforcement learning using business rules. Furthermore, the structured output of this application provides quantifiable reward signals for GRPO reinforcement learning (such as output character length, label integrity score, and unit compliance detection), bridging the gap between aviation materials data extraction and GRPO reinforcement learning, thus achieving efficient alignment of reinforcement learning.

[0236] ③ This application creates significant application value in the field of aviation materials through a closed-loop technology:

[0237] The technical solution of this application constructs a closed-loop technology of "multi-source input - intelligent decision-making - rule output", which can realize fully automatic digital processing of complex documents (such as maintenance records / customs documents) in the field of aviation materials, and provide highly reliable data support for the MRO system of airlines.

[0238] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0239] Based on the same inventive concept, this application also provides an aviation material document information extraction device for implementing the above-mentioned aviation material document information extraction method. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations of the one or more aviation material document information extraction device embodiments provided below can be found in the limitations of the aviation material document information extraction method described above, and will not be repeated here.

[0240] In one exemplary embodiment, such as Figure 10 As shown, an aviation material document information extraction device 900 is provided, the device 900 includes:

[0241] The VLM large model processing module 902 is used to input the document data to be identified into multiple VLM large models to obtain the VLM output dataset; the document data to be identified includes unstructured heterogeneous data of aviation materials;

[0242] OCR model processing module 904 is used to input the document data to be recognized into the OCR model and obtain OCR output data.

[0243] The recognition module 906 is used to input the VLM output dataset and OCR output data into the fine-tuned VLM mini-model to obtain target recognition information for the document data to be recognized.

[0244] In one embodiment, the plurality of VLM large models include a first VLM large model, a second VLM large model, and a third VLM large model with different architecture types; the device 900 also includes:

[0245] The metadata acquisition module is used to acquire metadata datasets;

[0246] The first OCR output data acquisition module is used to input the image from the metadata set into the OCR model to obtain the first OCR output data.

[0247] The fine-tuning training dataset acquisition module is used to input the metadata into the first VLM large model, the second VLM large model, and the third VLM large model respectively to obtain the fine-tuning training dataset.

[0248] The fine-tuning training module is used to perform multi-stage fine-tuning training on the pre-trained small model based on the first OCR output data and the fine-tuning training dataset, so as to obtain the fine-tuned VLM small model.

[0249] In one embodiment, the fine-tuning training module is further configured to fine-tune the visual-language alignment module and the language module in the first VLM mini-model based on the fine-tuning training dataset to obtain a second VLM mini-model; fine-tune the language module in the second VLM mini-model based on the fine-tuning training dataset and the first OCR output data to obtain a third VLM mini-model; and fine-tune the language module in the third VLM mini-model based on the fine-tuning training dataset and the first OCR output data to obtain a fine-tuned VLM mini-model.

[0250] In one embodiment, the fine-tuning training module is further used to label the fine-tuning training dataset to mark the correctness of the model recognition results of each element in the fine-tuning training dataset; wherein, the model recognition results characterize whether the VLM large model correctly recognizes the document image; based on the labeled fine-tuning training dataset, the language module of the first VLM small model is fine-tuned using the LoRA fine-tuning mode, and the visual-language alignment module of the first VLM small model is fine-tuned using the full fine-tuning mode to obtain the second VLM small model.

[0251] In one embodiment, the fine-tuning training module is further configured to fine-tune the Attention layer in the language module using the LoRA fine-tuning mode based on the labeled fine-tuning training dataset; fine-tune the visual-language alignment module using the full fine-tuning mode according to the first loss function to obtain the fine-tuned visual-language alignment module parameters; merge the LoRA layer parameters of the first VLM mini-model according to a preset merging formula; and update the current visual-language alignment module parameters of the first VLM mini-model using the fine-tuned visual-language alignment module parameters to obtain the second VLM mini-model.

[0252] In one embodiment, the fine-tuning training module is further configured to obtain first fine-tuning training data based on the fine-tuning training dataset and the first OCR output data; and to fine-tune the language module of the second VLM mini-model based on the first fine-tuning training data using the DoRA fine-tuning mode and the second loss function to obtain the third VLM mini-model.

[0253] In one embodiment, the fine-tuning training module is further used to standardize the real answers in the fine-tuning training dataset, and obtain the second fine-tuning training data based on the standardized fine-tuning training dataset and the first OCR output data; based on the third loss function and the second fine-tuning training data, the LoRA fine-tuning mode is used to fine-tune the Attention layer of the language module of the third VLM mini-model to obtain the fine-tuned VLM mini-model; wherein, the fine-tuning parameters of the third loss function are calculated by the GRPO algorithm based on business rules.

[0254] Each module in the aforementioned aviation material document information extraction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0255] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data such as documents to be identified. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a method for extracting information from aviation material documents.

[0256] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0257] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0258] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0259] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0260] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0261] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0262] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0263] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.< / tag> < / tag> < / tag> < / tag> < / tag> < / tag> < / tag> < / tag>

Claims

1. A method for extracting information from aviation material documents, characterized in that, include: The document data to be identified is input into multiple VLM large models to obtain the VLM output dataset; the document data to be identified includes unstructured heterogeneous data of aviation materials; The multiple VLM large models include a first VLM large model, a second VLM large model, and a third VLM large model with different architecture types; The document data to be recognized is input into the OCR model to obtain the OCR output data; The VLM output dataset and the OCR output data are input into the finely tuned VLM mini-model to obtain target recognition information for the document data to be recognized. The method further includes: Obtain the meta dataset; The images in the metadata set are input into the OCR model to obtain the first OCR output data; The meta-dataset is input into the first VLM large model, the second VLM large model, and the third VLM large model respectively to obtain the fine-tuning training dataset; Based on the first OCR output data and the fine-tuned training dataset, the pre-trained small model is subjected to multi-stage fine-tuning training to obtain the fine-tuned VLM small model; the pre-trained small model is the first VLM small model. The step of performing multi-stage fine-tuning training on the pre-trained small model based on the first OCR output data and the fine-tuning training dataset to obtain the fine-tuned VLM small model includes: Based on the fine-tuned training dataset, the visual-language alignment module and language module in the first VLM mini-model are fine-tuned to obtain the second VLM mini-model. The first fine-tuning training data is obtained based on the fine-tuning training dataset and the first OCR output data; Based on the first fine-tuning training data, the language module of the second VLM mini-model is fine-tuned and trained using the DoRA fine-tuning mode and the second loss function to obtain the third VLM mini-model. The real answers in the fine-tuning training dataset are standardized, and the second fine-tuning training data is obtained based on the standardized fine-tuning training dataset and the first OCR output data. Based on the third loss function and the second fine-tuning training data, the attention layer of the language module of the third VLM mini-model is fine-tuned using the LoRA fine-tuning mode to obtain the fine-tuned VLM mini-model; wherein, the third loss function is calculated based on the GRPO algorithm combined with business rules.

2. The method according to claim 1, characterized in that, The second VLM mini-model is obtained by fine-tuning the visual-language alignment module and the language module in the first VLM mini-model based on the fine-tuned training dataset, including: The fine-tuned training dataset is labeled to indicate the correctness of the model recognition results for each element in the fine-tuned training dataset; wherein, the model recognition results characterize whether the VLM large model correctly recognizes the document image; Based on the labeled fine-tuned training dataset, the language module of the first VLM mini-model is fine-tuned using the LoRA fine-tuning mode, and the visual-language alignment module of the first VLM mini-model is fine-tuned using the full fine-tuning mode, to obtain the second VLM mini-model.

3. The method according to claim 2, characterized in that, The second VLM mini-model is obtained by fine-tuning the language module of the first VLM mini-model using LoRA fine-tuning mode and the visual-language alignment module of the first VLM mini-model using full fine-tuning mode, based on the labeled fine-tuned training dataset. The second VLM mini-model includes: Based on the labeled fine-tuned training dataset, the Attention layer in the language module is fine-tuned using the LoRA fine-tuning mode; Based on the first loss function, the visual-language alignment module is fine-tuned using a full-scale fine-tuning mode to obtain the fine-tuned visual-language alignment module parameters; The LoRA layer parameters of the first VLM mini-model are merged according to a preset merging formula; The first VLM mini-model is updated with the fine-tuned visual-language alignment module parameters to obtain the second VLM mini-model.

4. An aviation material document information extraction device, characterized in that, include: The VLM large model processing module is used to input the document data to be identified into multiple VLM large models to obtain the VLM output dataset; the document data to be identified includes unstructured heterogeneous aviation material data; the multiple VLM large models include a first VLM large model, a second VLM large model, and a third VLM large model with different architecture types; The OCR model processing module is used to input the document data to be recognized into the OCR model to obtain OCR output data. The recognition module is used to input the VLM output dataset and the OCR output data into the fine-tuned VLM mini-model to obtain target recognition information for the document data to be recognized. The device further includes a metadata acquisition module, a first OCR output data acquisition module, a fine-tuning training dataset acquisition module, and a fine-tuning training module; The metadata acquisition module is used to acquire metadata. The first OCR output data acquisition module is used to input the image in the metadata set into the OCR model to obtain the first OCR output data; The fine-tuning training dataset acquisition module is used to input the meta-dataset into the first VLM large model, the second VLM large model and the third VLM large model respectively to obtain the fine-tuning training dataset; The fine-tuning training module is used to perform multi-stage fine-tuning training on the pre-trained small model based on the first OCR output data and the fine-tuning training dataset to obtain the fine-tuned VLM small model; the pre-trained small model is the first VLM small model. The fine-tuning training module is also used to fine-tune the visual-language alignment module and the language module in the first VLM mini-model based on the fine-tuning training dataset to obtain the second VLM mini-model. Based on the fine-tuning training dataset and the first OCR output data, the language module in the second VLM mini-model is fine-tuned and trained to obtain the third VLM mini-model; based on the fine-tuning training dataset and the first OCR output data, the language module in the third VLM mini-model is fine-tuned and trained to obtain the fine-tuned VLM mini-model. The fine-tuning training module is also used to obtain the first fine-tuning training data based on the fine-tuning training dataset and the first OCR output data; Based on the first fine-tuning training data, the language module of the second VLM mini-model is fine-tuned and trained using the DoRA fine-tuning mode and the second loss function to obtain the third VLM mini-model. The fine-tuning training module is also used to standardize the real answers in the fine-tuning training dataset, and obtain the second fine-tuning training data based on the standardized fine-tuning training dataset and the first OCR output data. Based on the third loss function and the second fine-tuning training data, the attention layer of the language module of the third VLM mini-model is fine-tuned using the LoRA fine-tuning mode to obtain the fine-tuned VLM mini-model; wherein, the third loss function is calculated based on the GRPO algorithm combined with business rules.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

Images