A method and apparatus for extracting parameter data

Abstract

This application provides a parameter data extraction method and apparatus, relating to the field of artificial intelligence technology. The method can perform unified image processing on description files, including PDF files, and automatically and accurately extract parameter data of the object to be extracted under multiple target data categories based on the content type tags and parameter category information of the target area, combined with the image-text alignment capability of a multimodal large model. This effectively improves the accuracy and completeness of data extraction and enhances cross-modal fusion capabilities.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a method and apparatus for extracting parameter data. Background Technology

[0002] With the widespread adoption of digital office work, Portable Document Format (PDF) has become the mainstream format for document exchange. However, while maintaining visual consistency, PDF documents also contain a large amount of unstructured information, such as data tables, technical images, statistical charts, and structural diagrams, which are difficult for machines to automatically recognize and understand due to their inherent characteristic of "content not being directly computable." This information is of great value for applications such as knowledge extraction, automatic archiving, and intelligent search. Therefore, how to extract data from PDF documents with high accuracy and intelligence is an urgent problem to be solved. Summary of the Invention

[0003] In view of this, this application provides a parameter data extraction method and apparatus to solve the problem of difficulty in extracting PDF document data in the prior art.

[0004] Specifically, this application is implemented through the following technical solution:

[0005] Firstly, this application provides a method for extracting parameter data, including:

[0006] Obtain a description file of the object to be extracted, and convert the description file into image data; the description file includes a portable document format PDF file;

[0007] At least one target region is identified from the image data, and a content type label corresponding to the target region is determined; the content type label includes tables and / or images;

[0008] For any target region, determine at least one target parameter category contained within the target region from multiple parameter categories of the object to be extracted;

[0009] The target region, the content type label corresponding to the target region, and the target parameter category corresponding to the target region are input into the multimodal large model to obtain the parameter data of the object to be extracted under various target data categories output by the multimodal large model.

[0010] Optionally, determining at least one target parameter category contained within the target region from multiple parameter categories of the object to be extracted includes:

[0011] The target region is input into a trained first computer vision classification model to determine the probability value of each parameter category contained in the target region;

[0012] Based on the probability value, at least one type of target data contained within the target area is determined.

[0013] Optionally, identifying at least one target region from the image data and determining the content type tag corresponding to the target region includes:

[0014] The image data is input into a trained second computer vision classification model to obtain the target region output by the second computer vision classification model and the content type label corresponding to the target region.

[0015] Optionally, the multimodal large model is used for:

[0016] Based on the content type tags corresponding to the target region, the target parameter categories corresponding to the target region, and the pre-input task description information, multimodal alignment semantic analysis is performed on the target region to obtain semantic analysis information;

[0017] Based on the task description information and the semantic analysis information, the parameter data of the object to be extracted under various target data categories are output.

[0018] Optionally, the multimodal large model includes a text sub-model, a visual sub-model, and a multimodal alignment sub-model;

[0019] The text sub-model is used to extract text from the description file to obtain text information corresponding to the description file; to clean, denoise, and segment the text information to obtain multiple text blocks corresponding to the text information; to retrieve target text blocks related to the task description information from the text blocks based on the task description information; the target text blocks are used to provide context information for the multimodal alignment sub-model.

[0020] The visual sub-model is used for semantic analysis of the image in the target region;

[0021] The multimodal alignment sub-model is used to perform multimodal alignment and fusion based on the semantic analysis information of the target text block and the image.

[0022] Optionally, the visual sub-model is also used for:

[0023] If image association indication information exists in the target text block, semantic recognition is performed on the target image corresponding to the image association indication information; the semantic recognition information of the target image is used to provide context information for the multimodal alignment sub-model.

[0024] Optionally, the method further includes:

[0025] Before the multimodal alignment sub-model outputs the multimodal alignment and fusion parameter data, the semantic credibility of the target region is determined by image-text consistency detection;

[0026] If the semantic credibility is below the threshold, multimodal alignment and fusion are performed again through a cross-regional comparison mechanism.

[0027] Optionally, the multimodal large model further includes a retrieval submodule;

[0028] The retrieval submodule is used for:

[0029] If the text sub-model or the visual sub-model identifies an undefined semantic, a retrieval instruction for the undefined semantic is generated;

[0030] Based on the search instruction, the undefined semantics are searched using an external knowledge base to obtain search information; the search information is used to provide context information for the multimodal alignment sub-model.

[0031] Optionally, the method further includes:

[0032] Determine the metadata corresponding to the description file;

[0033] The metadata and parameter data of the object to be extracted under various target data categories are integrated and correlated to obtain structured parameter information.

[0034] Secondly, this application also provides a parameter data extraction device, comprising:

[0035] The acquisition module is used to acquire a description file of the object to be extracted and convert the description file into image data; the description file includes a portable document format PDF file.

[0036] The recognition module is used to identify at least one target region from the image data and determine the content type label corresponding to the target region; the content type label includes tables and / or images;

[0037] The determination module is used to determine, for any target region, at least one target parameter category contained in the target region from multiple parameter categories of the object to be extracted;

[0038] The extraction module is used to input the target region, the content type tag corresponding to the target region, and the target parameter category corresponding to the target region into the multimodal large model to obtain the parameter data of the object to be extracted under various target data categories output by the multimodal large model.

[0039] Optionally, the determining module is used to:

[0040] The target region is input into a trained first computer vision classification model to determine the probability value of each parameter category contained in the target region;

[0041] Based on the probability value, at least one type of target data contained within the target area is determined.

[0042] Optionally, the identification module is used for:

[0043] The image data is input into a trained second computer vision classification model to obtain the target region output by the second computer vision classification model and the content type label corresponding to the target region.

[0044] Optionally, the multimodal large model is used for:

[0045] Based on the content type tags corresponding to the target region, the target parameter categories corresponding to the target region, and the pre-input task description information, multimodal alignment semantic analysis is performed on the target region to obtain semantic analysis information;

[0046] Based on the task description information and the semantic analysis information, the parameter data of the object to be extracted under various target data categories are output.

[0047] Optionally, the multimodal large model includes a text sub-model, a visual sub-model, and a multimodal alignment sub-model;

[0048] The text sub-model is used to extract text from the description file to obtain text information corresponding to the description file; to clean, denoise, and segment the text information to obtain multiple text blocks corresponding to the text information; to retrieve target text blocks related to the task description information from the text blocks based on the task description information; the target text blocks are used to provide context information for the multimodal alignment sub-model.

[0049] The visual sub-model is used for semantic analysis of the image in the target region;

[0050] The multimodal alignment sub-model is used to perform multimodal alignment and fusion based on the semantic analysis information of the target text block and the image.

[0051] Optionally, the visual sub-model is also used for:

[0052] If image association indication information exists in the target text block, semantic recognition is performed on the target image corresponding to the image association indication information; the semantic recognition information of the target image is used to provide context information for the multimodal alignment sub-model.

[0053] Optionally, the multimodal large model is used for:

[0054] Before the multimodal alignment sub-model outputs the multimodal alignment and fusion parameter data, the semantic credibility of the target region is determined by image-text consistency detection;

[0055] If the semantic credibility is below the threshold, multimodal alignment and fusion are performed again through a cross-regional comparison mechanism.

[0056] Optionally, the multimodal large model further includes a retrieval submodule;

[0057] The retrieval submodule is used for:

[0058] If the text sub-model or the visual sub-model identifies an undefined semantic, a retrieval instruction for the undefined semantic is generated;

[0059] Based on the search instruction, the undefined semantics are searched using an external knowledge base to obtain search information; the search information is used to provide context information for the multimodal alignment sub-model.

[0060] Optionally, the device further includes an integration module for:

[0061] Determine the metadata corresponding to the description file;

[0062] The metadata and parameter data of the object to be extracted under various target data categories are integrated and correlated to obtain structured parameter information.

[0063] Thirdly, embodiments of this application also provide a computer device, which includes a processor and a memory. The memory stores machine-readable instructions executable by the processor. The processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, they perform the steps of the first aspect above, or any possible implementation of the first aspect.

[0064] Fourthly, optional embodiments of this application also provide a computer-readable storage medium storing a computer program that, when run, performs the steps of the first aspect or any possible implementation of the first aspect.

[0065] The parameter data extraction method and apparatus provided in this application embodiment can perform unified image processing on description files including PDF files, and based on the content type tags and parameter category information of the target area, combined with the image-text alignment capability of the multimodal large model, automatically and accurately extract the parameter data of the object to be extracted under multiple target data categories, effectively improving the accuracy and completeness of data extraction and enhancing cross-modal fusion capability. Attached Figure Description

[0066] Figure 1 This is a flowchart illustrating a parameter data extraction method according to an exemplary embodiment of this application;

[0067] Figure 2 This is a schematic diagram illustrating prior information in an exemplary embodiment of this application;

[0068] Figure 3 This is a schematic diagram illustrating structured parameter information in an exemplary embodiment of this application;

[0069] Figure 4 This is a flowchart illustrating another parameter data extraction method according to an exemplary embodiment of this application;

[0070] Figure 5 This is a flowchart illustrating another parameter data extraction method in an exemplary embodiment of this application;

[0071] Figure 6 This is a flowchart illustrating yet another parameter data extraction method according to an exemplary embodiment of this application;

[0072] Figure 7 This is a schematic diagram of a parameter data extraction device shown in an exemplary embodiment of this application;

[0073] Figure 8 This is a schematic diagram of a computer device illustrated in an exemplary embodiment of this application. Detailed Implementation

[0074] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0075] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0076] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0077] Research has found that while PDF documents maintain visual consistency, their inherent characteristic of "content that cannot be directly calculated" makes it difficult for machines to automatically recognize and understand the large amount of unstructured information they contain, such as data tables, technical images, statistical charts, and structural diagrams.

[0078] In view of this, this application provides a parameter data extraction method and apparatus, which can perform unified image processing on description files including PDF files, and automatically and accurately extract parameter data of the object to be extracted under multiple target data categories based on the content type tags and parameter category information of the target area, combined with the image-text alignment capability of the multimodal large model, effectively improving the accuracy and completeness of data extraction and enhancing cross-modal fusion capability.

[0079] The deficiencies of the existing technical solutions are the result of the inventor's practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this application below should be considered as the inventor's contributions to this application.

[0080] To facilitate understanding of this embodiment, the application scenarios of the parameter data extraction method disclosed in this application embodiment will first be introduced. The execution subject of the parameter data extraction method provided in this application embodiment can be a computer device. In some possible implementations, the parameter data extraction method can be implemented by a processor calling computer-readable instructions stored in memory.

[0081] See Figure 1 The diagram shown is a flowchart illustrating a parameter data extraction method according to an exemplary embodiment of this application. The method includes steps S101 to S104, wherein:

[0082] S101. Obtain the description file of the object to be extracted, and convert the description file into image data; the description file includes a portable document format PDF file.

[0083] S102. Identify at least one target region from the image data and determine the content type label corresponding to the target region; the content type label includes tables and / or images;

[0084] S103. For any target region, determine at least one target parameter category contained within the target region from multiple parameter categories of the object to be extracted;

[0085] S104. Input the target region, the content type label corresponding to the target region, and the target parameter category corresponding to the target region into the multimodal large model to obtain the parameter data of the object to be extracted under various target data categories output by the multimodal large model.

[0086] The following will provide a detailed explanation of each step in the parameter data extraction method provided in this application.

[0087] Regarding S101:

[0088] In this step, the description file can refer to a file containing descriptive information about the object to be extracted. This descriptive information can include text, images, tables, etc. Specifically, the description file can include, but is not limited to, image files, text files, PDF files, Word files, scanned documents, photographs, etc.

[0089] In some embodiments, users can submit keywords to the system, which can then retrieve relevant descriptive documents from a document library based on those keywords. This document library may include both online and local resources.

[0090] After obtaining the description file, it can be converted into image data. For example, each page of a PDF file can be directly read and rendered as an image.

[0091] Regarding S102:

[0092] In one implementation, a trained second computer vision classification model is used to perform region detection and classification on the image. This model can be based on convolutional neural networks (CNN), visual Transformer (ViT), YOLO series models, or other visual recognition structures.

[0093] For example, the second computer vision classification model can be an improved YOLOv11 model, capable of detecting and locating key regions of specific types in image data, and outputting the coordinate information and category label of the target region. The category label can represent the type of content in the region, such as image, table, paragraph, formula, etc. Specifically, the image data can be input into the trained second computer vision classification model to obtain the target region output by the second computer vision classification model, and the content type label corresponding to the target region.

[0094] For example, the second computer vision classification model can be trained through the following steps:

[0095] First, test data, including multi-page PDF images, is obtained. The Labelme annotation tool is used to finely annotate the target regions in the images. The annotation process strictly adheres to the principle of "close to the target edge, with no redundant background," and uses a unified English label system to define target categories, including: table, figure, paragraph, and abandon. A corresponding annotation file is generated for each image. Then, based on the constructed annotation dataset, YOLOv11 pre-trained weight parameters are loaded, and the dataset is divided into training and validation sets in an 8:2 ratio. Through model fine-tuning and continuous optimization of the model parameters, an improved version of YOLOv11 weight parameters suitable for PDF document element detection is obtained, capable of accurately identifying tables, images, and other targets in PDFs.

[0096] Regarding S103:

[0097] In this step, a pre-trained computer vision classification model can be used. The target region is input into the model, which outputs the probability values ​​of each parameter category. The parameter category within the target region is then determined based on the probability threshold. Examples include: geometric parameters (such as length, width, and angle), performance parameters (such as speed, power, and weight), electrical parameters (such as voltage and current), physical quantity parameters (such as pressure and temperature), and image annotation parameters (such as the values ​​corresponding to indicator arrows). By determining the parameter categories corresponding to a region, a comprehensive and in-depth understanding of the data contained within the target region can be achieved, improving the accuracy of subsequent multimodal inference and making the inference process more targeted.

[0098] For example, image data can be cropped according to the coordinate information of the target region, and a multi-label classification network with YOLOv11 as the backbone can be used to extract the category probability vector of the target region as the semantic branch feature of the target region for use by downstream multimodal large models.

[0099] For example, the object to be extracted can be a processor device, and the parameter categories can include: body height, pin outline height, top and bottom pin spacing, pin spacing, device pin length, device pin width, device body length, device body width, total number of pins, etc.

[0100] In some possible implementations, the first computer vision classification model can be trained through the following steps:

[0101] First, test data was acquired, including multiple pages of image data. Then, the Labelme annotation tool was used to perform fine-grained annotation on each page of images. During the annotation process, the rule of "close to the target edge, no redundant background" was followed, and each image was divided into multiple category regions to ensure that the annotation results accurately reflect the target content. Nine categories were defined, covering the common feature parameters of the device. Subsequently, a multi-label classification network based on the YOLOv11 classification model was constructed, with input images uniformly scaled to 640×640 resolution. 500 rounds of training were performed on a self-built dataset, with an initial learning rate set to 0.001. Finally, through multiple rounds of training, optimized model parameters were obtained, which can accurately output the category probability vector of the image, serving as the image semantic features for use by multimodal large models.

[0102] In the above category probability vector, 0 and 1 can be used to indicate whether the parameter category at that position exists in the target region, or [0, 1] can be used to indicate the probability value that the parameter category at that position exists in the target region.

[0103] Regarding S104:

[0104] In this step, the multimodal large model can be implemented using CLIP, BLIP-2, GPT-4V, Qwen2.5-VL-72B series models, or a self-developed large-scale model. For example, the multimodal large model can output parameter data through the following mechanism:

[0105] Select the appropriate internal decoding link based on content type tags; prompt the model to focus on specific types of semantic information within the region based on parameter category; combine the association capabilities of text and visual modules to achieve image-text alignment and extract target data; output the parameter data of the object to be extracted under various target data categories, including textual parameter values, units, dimensions, parameter names, etc.

[0106] In some embodiments, the multimodal large model can perform multimodal alignment semantic analysis on the target region based on the content type label corresponding to the target region, the target parameter category corresponding to the target region, and the pre-input task description information to obtain semantic analysis information; then, based on the task description information and the semantic analysis information, it outputs the parameter data of the object to be extracted under various target data categories.

[0107] For example, a multimodal large model can select a visual resolution strategy based on content type tags. For instance, if the target area is a table, the visual submodule will focus on identifying cell structure, row and column relationships, and text content; if the target area is an image, the model will prioritize graphic elements, directional arrows, local annotation symbols, etc.

[0108] Multimodal large models can also focus on specific semantic regions based on the target parameter category. For example, different parameter categories (such as "geometric dimension parameters" and "physical quantity parameters") correspond to different semantic features. The model will automatically increase the weight of relevant information during the inference process, thereby improving the accuracy of the analysis.

[0109] Multimodal large models can also generate multimodal aligned semantic analysis information. For example, visual analysis results can be aligned and fused with text semantic embedding vectors to obtain semantic analysis vectors that can accurately describe the structure, related information, and potential meaning of the target region. Here, text semantic embedding vectors can refer to the text information extracted from the descriptive data.

[0110] After obtaining the semantic analysis results, the multimodal large model can perform parameter semantic recognition on the semantic analysis information; then extract the numerical values, units, labels and other information corresponding to the parameter items, and output structured parameter data according to the format specified in the task description information.

[0111] For example, a multimodal large model can take structured prior information of each target region (such as an image or a table) as input. This structured prior information may include content type labels, relevant semantic descriptions, and task prompts, etc., as contextual guidance for the multimodal large model to help it complete semantic association and annotation tasks.

[0112] See Figure 2 The diagram shown illustrates the prior information provided in an exemplary embodiment of this application. The "content" of the "system" section can be task prompts (i.e., task description information), the "content" of the "user" section is the target region and its corresponding target parameter category, the "content" of the "assistant" section can be auxiliary information obtained from historical data or template information provided by the user, and the content of the "images" section is an image file of the target region cropped from image data.

[0113] In some embodiments, the method can also determine the metadata of the description file and integrate and associate the metadata and the parameter data of the object to be extracted under various target data categories to obtain structured parameter information.

[0114] Metadata can include readable text information, index information, title information, remarks information, name information, etc., from the description file. Multimodal large models can extract the necessary information from the metadata and associate it with the parameter data to form structured parameter information in a standard format.

[0115] See Figure 3 The diagram shown is a schematic representation of structured parameter information provided in an exemplary embodiment of this application. This structured parameter information can be output in JSON format.

[0116] In one possible implementation, the multimodal large model includes a text sub-model, a visual sub-model, and a multimodal alignment sub-model.

[0117] The text sub-model is used to extract text from the description file to obtain text information corresponding to the description file; to clean, denoise, and segment the text information to obtain multiple text blocks corresponding to the text information; to retrieve target text blocks related to the task description information from the text blocks based on the task description information; and the target text blocks are used to provide context information for the multimodal alignment sub-model.

[0118] The visual sub-model is used for semantic analysis of the image in the target region;

[0119] The multimodal alignment sub-model is used to perform multimodal alignment and fusion based on the semantic analysis information of the target text block and the image.

[0120] The text sub-model is primarily used to extract text information from description files, providing linguistic context support for multimodal understanding. Optional methods include OCR extraction of text from scanned documents or images, PDF text layer parsing, and direct parsing of text formats.

[0121] Since the extracted text often contains noise or non-standard formatting issues, the text sub-model performs operations such as denoising (e.g., removing page break symbols, redundant spaces, and garbled characters), standard formatting, and paragraph division according to semantic structure.

[0122] Subsequently, the text sub-model can perform semantic retrieval on all text blocks based on task description information (such as "extract voltage parameters" and "identify structure dimensions"), filtering out target text blocks relevant to the task. These target text blocks are used to provide semantic context for the multimodal alignment sub-model as part of prior information, enabling the model to more accurately understand the meaning corresponding to the image regions.

[0123] Visual sub-models can extract visual semantic information from image regions, such as table structure analysis, graphic content recognition, image annotation parsing, and layout relationship analysis. Visual sub-models can be implemented based on CNN, ViT, DETR, or YOLO series models.

[0124] In some cases, the "image association indication information" (such as "image association indication information") identified in the text sub-model Figure 2 (As shown in the image) and (see the area indicated by the arrow in the image below) can be used to assist the visual sub-model in locating the target image.

[0125] For example, the visual sub-model can perform semantic recognition on the target image corresponding to the image association indication information when there is image association indication information in the target text block; the semantic recognition information of the target image is used to provide context information for the multimodal alignment sub-model.

[0126] Multimodal alignment sub-models can be used to fuse semantic features from text sub-models and visual sub-models. Their main functions can include:

[0127] 1. Mapping text and image features to a unified semantic space. The multimodal alignment sub-model enables image semantics and text semantics to be similarly measured and associated in the same space through a joint embedding space.

[0128] 2. Perform cross-modal feature fusion. Through an attention mechanism, keywords in the target text block are mapped to visual semantics in the image region, thereby strengthening the parameter correspondence.

[0129] 3. Output fused semantic representation. The fused semantic vector is used to guide the final parameter data generation module, ensuring that the output parameter content has accurate semantic basis and logical relationship.

[0130] Thus, based on steps S101 to S104, the multimodal large model can be split into a way for the large model to work in collaboration with traditional models, which enhances the understanding of semantic information of text and image data and improves the correlation between text and image semantics, thereby improving the accuracy of recognition parameter data.

[0131] In some implementations, the multimodal large model further includes a retrieval submodule for information association and cross-modal cue enhancement. This module can provide auxiliary information to the multimodal alignment submodel by retrieving relevant content from external knowledge bases, historical text information, local context, and cross-regional content, thereby improving the accuracy of parameter extraction and the interpretability of inference. The retrieval submodule plays a crucial role in the parameter inference process by comprehensively considering visual data, textual data, and task description information to achieve accurate retrieval and matching of relevant data fragments.

[0132] For example, when the text sub-model or the visual sub-model identifies undefined semantics, the retrieval sub-module can generate a retrieval instruction for the undefined semantics; based on the retrieval instruction, the undefined semantics are retrieved using an external knowledge base to obtain retrieval information; the retrieval information is used to provide contextual information for the multimodal alignment sub-model.

[0133] On the visual sub-model side, for example, if the visual sub-model recognizes an uncommon mechanical part drawing, it will automatically generate a text query, such as "3D model drawing, name and purpose of metal part with hexagonal screw holes and annular grooves," and retrieve the corresponding information. On the text-based large model side, for example, when analyzing a medical report and seeing the abbreviation "CADASIL" for a rare disease, the model will construct a query "full name of CADASIL, symptoms and latest treatment options," and similarly retrieve the corresponding information. When the retrieval sub-module receives a query, it will search in a pre-defined vector database (composed of corresponding data from the current description file and / or historical description files) or the network, and then feed the retrieved knowledge, after data augmentation, to the large model.

[0134] In some implementations, in order to further improve the accuracy of parameter data extraction of multimodal large models in complex document scenarios, before the multimodal alignment sub-model outputs the final fusion result, a text-image consistency detection can be performed on the target region to evaluate the semantic credibility of the target region.

[0135] For example, before the multimodal alignment sub-model outputs the multimodal alignment and fusion parameter data, the multimodal large model can determine the semantic credibility of the target region through image-text consistency detection; if the semantic credibility is lower than the threshold, multimodal alignment and fusion are re-performed through a cross-regional comparison mechanism.

[0136] Among them, image-text consistency detection is used to determine whether the image content and its associated text blocks support each other semantically. For example, when the text sub-model and the visual sub-model perform semantic extraction on the same target region, the following situations may occur:

[0137] (1) The image content is inconsistent with the text description. For example, the image shows a diagram of a pressure sensor structure, while the adjacent text describes it as a current parameter.

[0138] (2) Blurred parameter names or fonts in the table lead to a decrease in the confidence level of the visual model recognition;

[0139] (3) Excessively long text blocks cause the context to contain information unrelated to the region, reducing the quality of semantic alignment;

[0140] (4) The image contains compression artifacts, shadows, reflections, etc., which cause the visual semantics to deviate from the true content.

[0141] In its implementation, the image-text consistency detection module can construct a consistency score based on the following features:

[0142] The semantics of the target text block extracted by the text sub-model; the semantics of the image region by the visual sub-model; the pre-alignment features of the multimodal alignment sub-model; the degree of matching between text word meaning, parameter units, and symbol features in the image; and prior knowledge of parameter categories (such as the degree of conflict between the probability of "mA" appearing in the table and the probability of "voltage symbol" appearing in the image).

[0143] The system obtains semantic credibility based on a consistency score. When the semantic credibility is higher than a preset threshold, the multimodal alignment sub-model directly outputs the fused parameter data. If the semantic credibility is lower than the threshold, the system executes a cross-regional comparison mechanism to enhance the reliability of the final semantic decision.

[0144] Among them, the cross-region comparison mechanism is an information supplementation method based on the overall document structure. When the semantic credibility of a target region is insufficient, the system will select other regions with spatial or semantic relationships with it for comparison, such as: adjacent table regions on the same page; the occurrence positions of the same parameter category in different regions; the relationship structure between multiple images under the title or chapter; and the contextual information of the same parameter name appearing in multiple places throughout the text.

[0145] Cross-region comparison mechanisms can employ the following strategies: utilize semantic distance calculation between regions to establish a multimodal embedding for the reference region; retrieve reference data of the same type or category as the target region from other regions; reconstruct visual and textual semantics based on cross-region attention weights; and compare the differences with the reference region to correct erroneous parameter candidate values ​​or unit inferences.

[0146] The reference information obtained through comparison can be re-input into the multimodal alignment sub-model, enabling it to perform multimodal fusion on the target region again, thereby improving the accuracy and consistency of the final output parameter data.

[0147] The aforementioned text-image consistency detection and cross-regional comparison mechanisms significantly improve the system's robustness to complex, ambiguous, and mismatched documents, especially for scanned documents, old PDFs, multilingual mixed documents, or source files with weakened text-image relationships.

[0148] See Figure 4The diagram shown is a flowchart illustrating another data extraction method according to an exemplary embodiment of this application. In this method, the retrieval module retrieves the user's requirements, obtains a description file of the object to be extracted, and converts it into an image; the PDF parsing module detects multiple target regions and their corresponding content type tags from the image data of the description file; the target detection module determines the target parameter category corresponding to the target region; and provides this information for use by the subsequent image understanding module. The image understanding module processes the chart information to generate prior information, which is then processed by the multimodal semantic understanding module to obtain semantic understanding output. Finally, the result integration module outputs the integrated result.

[0149] See Figure 5 The following is a flowchart illustrating another data extraction method according to an exemplary embodiment of this application. In this method, a multimodal document is first input, and the text information (obtained through OCR and layout analysis) and image information (JPG or PNG images) are retrieved through a retrieval enhancement generation module (i.e., a retrieval submodule). Then, through modality recognition, the text information and image information are processed using a large text model and a large visual model, respectively, and a unified semantic understanding and output are obtained through cross-modal semantic alignment and fusion.

[0150] See Figure 6 The diagram shown is a flowchart illustrating another data extraction method according to an exemplary embodiment of this application. In this method, a multimodal document is first input, and modality recognition is performed on the document content. JPG or PNG images are input into a visual large model and a retrieval enhancement generation module (i.e., a retrieval submodule). The text input values ​​obtained through OCR and layout analysis are then processed by the text large model and the retrieval enhancement generation module. The semantic information generated by the above modules is then subjected to cross-modal semantic alignment and fusion to obtain a unified semantic understanding and output.

[0151] The parameter data extraction method provided in this application embodiment can perform unified image processing on description files including PDF files, and based on the content type tags and parameter category information of the target area, combined with the image-text alignment capability of the multimodal large model, automatically and accurately extract the parameter data of the object to be extracted under multiple target data categories, effectively improving the accuracy and completeness of data extraction and enhancing cross-modal fusion capability.

[0152] See Figure 7 The diagram shown is a schematic representation of a parameter data extraction apparatus according to an exemplary embodiment of this application. The apparatus includes:

[0153] The acquisition module 710 is used to acquire a description file of the object to be extracted and convert the description file into image data; the description file includes a portable document format PDF file.

[0154] The recognition module 720 is used to identify at least one target region from the image data and determine the content type label corresponding to the target region; the content type label includes tables and / or images;

[0155] The determining module 730 is used to determine, for any target region, at least one target parameter category contained in the target region from multiple parameter categories of the object to be extracted;

[0156] The extraction module 740 is used to input the target region, the content type tag corresponding to the target region, and the target parameter category corresponding to the target region into the multimodal large model to obtain the parameter data of the object to be extracted under various target data categories output by the multimodal large model.

[0157] Optionally, the determining module 730 is used for:

[0158] The target region is input into a trained first computer vision classification model to determine the probability value of each parameter category contained in the target region;

[0159] Based on the probability value, at least one type of target data contained within the target area is determined.

[0160] Optionally, the identification module 720 is used for:

[0161] The image data is input into a trained second computer vision classification model to obtain the target region output by the second computer vision classification model and the content type label corresponding to the target region.

[0162] Optionally, the multimodal large model is used for:

[0163] Based on the content type tags corresponding to the target region, the target parameter categories corresponding to the target region, and the pre-input task description information, multimodal alignment semantic analysis is performed on the target region to obtain semantic analysis information;

[0164] Based on the task description information and the semantic analysis information, the parameter data of the object to be extracted under various target data categories are output.

[0165] Optionally, the multimodal large model includes a text sub-model, a visual sub-model, and a multimodal alignment sub-model;

[0166] The text sub-model is used to extract text from the description file to obtain text information corresponding to the description file; to clean, denoise, and segment the text information to obtain multiple text blocks corresponding to the text information; to retrieve target text blocks related to the task description information from the text blocks based on the task description information; the target text blocks are used to provide context information for the multimodal alignment sub-model.

[0167] The visual sub-model is used for semantic analysis of the image in the target region;

[0168] The multimodal alignment sub-model is used to perform multimodal alignment and fusion based on the semantic analysis information of the target text block and the image.

[0169] Optionally, the visual sub-model is also used for:

[0170] If image association indication information exists in the target text block, semantic recognition is performed on the target image corresponding to the image association indication information; the semantic recognition information of the target image is used to provide context information for the multimodal alignment sub-model.

[0171] Optionally, the multimodal large model is used for:

[0172] Before the multimodal alignment sub-model outputs the multimodal alignment and fusion parameter data, the semantic credibility of the target region is determined by image-text consistency detection;

[0173] If the semantic credibility is below the threshold, multimodal alignment and fusion are performed again through a cross-regional comparison mechanism.

[0174] Optionally, the multimodal large model further includes a retrieval submodule;

[0175] The retrieval submodule is used for:

[0176] If the text sub-model or the visual sub-model identifies an undefined semantic, a retrieval instruction for the undefined semantic is generated;

[0177] Based on the search instruction, the undefined semantics are searched using an external knowledge base to obtain search information; the search information is used to provide context information for the multimodal alignment sub-model.

[0178] Optionally, the device further includes an integration module 750 for:

[0179] Determine the metadata corresponding to the description file;

[0180] The metadata and parameter data of the object to be extracted under various target data categories are integrated and correlated to obtain structured parameter information.

[0181] The parameter data extraction device provided in this application embodiment can perform unified image processing on description files including PDF files, and based on the content type tags and parameter category information of the target area, combined with the image-text alignment capability of the multimodal large model, automatically and accurately extract the parameter data of the object to be extracted under multiple target data categories, effectively improving the accuracy and completeness of data extraction and enhancing cross-modal fusion capability.

[0182] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0183] This application also provides a computer device, such as... Figure 8 The diagram shown is a schematic representation of a computer device structure according to an exemplary embodiment of this application. The computer device includes:

[0184] A processor 81 and a memory 82; the memory 82 stores machine-readable instructions executable by the processor 81, and the processor 81 executes the machine-readable instructions stored in the memory 82. When the machine-readable instructions are executed by the processor 81, the processor 81 performs the following steps:

[0185] Obtain a description file of the object to be extracted, and convert the description file into image data; the description file includes a portable document format PDF file;

[0186] At least one target region is identified from the image data, and a content type label corresponding to the target region is determined; the content type label includes tables and / or images;

[0187] For any target region, determine at least one target parameter category contained within the target region from multiple parameter categories of the object to be extracted;

[0188] The target region, the content type label corresponding to the target region, and the target parameter category corresponding to the target region are input into the multimodal large model to obtain the parameter data of the object to be extracted under various target data categories output by the multimodal large model.

[0189] Optionally, determining at least one target parameter category contained within the target region from multiple parameter categories of the object to be extracted includes:

[0190] The target region is input into a trained first computer vision classification model to determine the probability value of each parameter category contained in the target region;

[0191] Based on the probability value, at least one type of target data contained within the target area is determined.

[0192] Optionally, identifying at least one target region from the image data and determining the content type tag corresponding to the target region includes:

[0193] The image data is input into a trained second computer vision classification model to obtain the target region output by the second computer vision classification model and the content type label corresponding to the target region.

[0194] Optionally, the multimodal large model is used for:

[0195] Based on the content type tags corresponding to the target region, the target parameter categories corresponding to the target region, and the pre-input task description information, multimodal alignment semantic analysis is performed on the target region to obtain semantic analysis information;

[0196] Based on the task description information and the semantic analysis information, the parameter data of the object to be extracted under various target data categories are output.

[0197] Optionally, the multimodal large model includes a text sub-model, a visual sub-model, and a multimodal alignment sub-model;

[0198] The text sub-model is used to extract text from the description file to obtain text information corresponding to the description file; to clean, denoise, and segment the text information to obtain multiple text blocks corresponding to the text information; to retrieve target text blocks related to the task description information from the text blocks based on the task description information; the target text blocks are used to provide context information for the multimodal alignment sub-model.

[0199] The visual sub-model is used for semantic analysis of the image in the target region;

[0200] The multimodal alignment sub-model is used to perform multimodal alignment and fusion based on the semantic analysis information of the target text block and the image.

[0201] Optionally, the visual sub-model is also used for:

[0202] If image association indication information exists in the target text block, semantic recognition is performed on the target image corresponding to the image association indication information; the semantic recognition information of the target image is used to provide context information for the multimodal alignment sub-model.

[0203] Optionally, the processor 81 is also configured to perform:

[0204] Before the multimodal alignment sub-model outputs the multimodal alignment and fusion parameter data, the semantic credibility of the target region is determined by image-text consistency detection;

[0205] If the semantic credibility is below the threshold, multimodal alignment and fusion are performed again through a cross-regional comparison mechanism.

[0206] Optionally, the multimodal large model further includes a retrieval submodule;

[0207] The retrieval submodule is used for:

[0208] If the text sub-model or the visual sub-model identifies an undefined semantic, a retrieval instruction for the undefined semantic is generated;

[0209] Based on the search instruction, the undefined semantics are searched using an external knowledge base to obtain search information; the search information is used to provide context information for the multimodal alignment sub-model.

[0210] Optionally, the processor 81 is also configured to perform:

[0211] Determine the metadata corresponding to the description file;

[0212] The metadata and parameter data of the object to be extracted under various target data categories are integrated and correlated to obtain structured parameter information.

[0213] The aforementioned memory 82 includes a main memory 821 and an external memory 822. The main memory 821, also known as internal memory, is used to temporarily store the computational data in the processor 81, as well as the data exchanged with external memory 822 such as a hard disk. The processor 81 exchanges data with the external memory 822 through the main memory 821.

[0214] The specific execution process of the above instructions can be referred to the steps of the parameter data extraction method described in the embodiments of this application, and will not be repeated here.

[0215] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0216] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the parameter data extraction method described in the above method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0217] This application also provides a computer program product, including a computer program / instruction, which, when executed by the computer program / instruction processor, implements the parameter data extraction method provided in the various embodiments of this application.

[0218] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0219] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0220] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0221] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0222] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0223] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0224] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method of parameter data extraction, characterized by, The method includes: Obtain a description file of the object to be extracted, and convert the description file into image data; the description file includes a portable document format PDF file; At least one target region is identified from the image data, and a content type label corresponding to the target region is determined; the content type label includes tables and / or images; For any target region, determine at least one target parameter category contained within the target region from multiple parameter categories of the object to be extracted; The target region, the content type tag corresponding to the target region, and the target parameter category corresponding to the target region are input into the multimodal large model to obtain the parameter data of the object to be extracted under various target data categories output by the multimodal large model. The multimodal large model is used for: Based on the content type tags corresponding to the target region, the target parameter categories corresponding to the target region, and the pre-input task description information, multimodal alignment semantic analysis is performed on the target region to obtain semantic analysis information; Based on the task description information and the semantic analysis information, output the parameter data of the object to be extracted under various target data categories; The multimodal large model includes a text sub-model, a visual sub-model, and a multimodal alignment sub-model; The text sub-model is used to extract text from the description file to obtain text information corresponding to the description file; to clean, denoise, and segment the text information to obtain multiple text blocks corresponding to the text information; to retrieve target text blocks related to the task description information from the text blocks based on the task description information; the target text blocks are used to provide context information for the multimodal alignment sub-model. The visual sub-model is used for semantic analysis of the image in the target region; The multimodal alignment sub-model is used to perform multimodal alignment and fusion based on the semantic analysis information of the target text block and the image; The method further includes: Before the multimodal alignment sub-model outputs the multimodal alignment and fusion parameter data, the semantic credibility of the target region is determined by image-text consistency detection; If the semantic credibility is below the threshold, multimodal alignment and fusion are re-performed through a cross-regional comparison mechanism. The cross-regional comparison mechanism is used to select other regions with spatial or semantic association with the target region for comparison when the semantic credibility of the target region is insufficient, so as to obtain reference data. The reference data is used to re-perform multimodal alignment and fusion.

2. The method of claim 1, wherein, Determining at least one target parameter category contained within the target region from multiple parameter categories of the object to be extracted includes: The target region is input into a trained first computer vision classification model to determine the probability value of each parameter category contained in the target region; Based on the probability value, at least one type of target data contained within the target area is determined.

3. The method according to claim 1, characterized in that, The step of identifying at least one target region from the image data and determining the content type tag corresponding to the target region includes: The image data is input into a trained second computer vision classification model to obtain the target region output by the second computer vision classification model and the content type label corresponding to the target region.

4. The method according to claim 1, characterized in that, The visual sub-model is also used for: If image association indication information exists in the target text block, semantic recognition is performed on the target image corresponding to the image association indication information; the semantic recognition information of the target image is used to provide context information for the multimodal alignment sub-model.

5. The method according to claim 1 or 4, characterized in that, The multimodal large model also includes a retrieval submodule; The retrieval submodule is used for: If the text sub-model or the visual sub-model identifies an undefined semantic, a retrieval instruction for the undefined semantic is generated; Based on the search instruction, the undefined semantics are searched using an external knowledge base to obtain search information; the search information is used to provide context information for the multimodal alignment sub-model.

6. The method according to claim 1, characterized in that, The method further includes: Determine the metadata corresponding to the description file; The metadata and parameter data of the object to be extracted under various target data categories are integrated and correlated to obtain structured parameter information.

7. A parameter data extraction device, characterized in that, The device includes: The acquisition module is used to acquire a description file of the object to be extracted and convert the description file into image data; the description file includes a portable document format PDF file. The recognition module is used to identify at least one target region from the image data and determine the content type label corresponding to the target region; the content type label includes tables and / or images; The determination module is used to determine, for any target region, at least one target parameter category contained in the target region from multiple parameter categories of the object to be extracted; The extraction module is used to input the target region, the content type tag corresponding to the target region, and the target parameter category corresponding to the target region into the multimodal large model to obtain the parameter data of the object to be extracted under various target data categories output by the multimodal large model; The multimodal large model is used for: Based on the content type tags corresponding to the target region, the target parameter categories corresponding to the target region, and the pre-input task description information, multimodal alignment semantic analysis is performed on the target region to obtain semantic analysis information; Based on the task description information and the semantic analysis information, output the parameter data of the object to be extracted under various target data categories; The multimodal large model includes a text sub-model, a visual sub-model, and a multimodal alignment sub-model; The text sub-model is used to extract text from the description file to obtain text information corresponding to the description file; to clean, denoise, and segment the text information to obtain multiple text blocks corresponding to the text information; to retrieve target text blocks related to the task description information from the text blocks based on the task description information; the target text blocks are used to provide context information for the multimodal alignment sub-model. The visual sub-model is used for semantic analysis of the image in the target region; The multimodal alignment sub-model is used to perform multimodal alignment and fusion based on the semantic analysis information of the target text block and the image; The multimodal large model is used for: Before the multimodal alignment sub-model outputs the multimodal alignment and fusion parameter data, the semantic credibility of the target region is determined by image-text consistency detection; If the semantic credibility is below the threshold, multimodal alignment and fusion are re-performed through a cross-regional comparison mechanism. The cross-regional comparison mechanism is used to select other regions with spatial or semantic association with the target region for comparison when the semantic credibility of the target region is insufficient, so as to obtain reference data. The reference data is used to re-perform multimodal alignment and fusion.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 6.

Images