Construction and evaluation method of manufacturing industry multi-mode question and answer data set

Abstract

The invention discloses a construction and evaluation method of a manufacturing industry multi-modal question and answer data set. The construction and evaluation method comprises the following steps: S100: constructing a corpus of manufacturing industry standard documents; s200, respectively carrying out metadata annotation on the standard documents in the corpus; s300, multi-modal element decoupling separation is carried out, multi-modal heterogeneous elements are extracted, text recognition is carried out, and recognition results are rearranged according to an original standard document layout; s400, post-processing the extracted text, inputting the post-processed text into a language model, carrying out term extraction according to a cue word template, and constructing a global manufacturing industry term dictionary by adopting the extracted terms; s500, constructing an initial question and answer pair seed set; and S600, performing batch question and answer pair generation and answer reasoning step generation by adopting a multi-modal model, and constructing a multi-modal manufacturing industry standard question and answer data set. According to the method, efficient and accurate construction of the manufacturing industry multi-modal question and answer data set can be realized, and a reliable data basis can be provided for training and evaluation of a subsequent multi-modal large model in a manufacturing industry scene.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of term data set construction, and particularly relates to a manufacturing multi-modal question and answer data set construction and evaluation method. BACKGROUND

[0002] Manufacturing is undergoing a deepening transformation from traditional digitization to intelligence. In product design, process planning, quality control, and safety management, enterprises rely heavily on various technical documents such as national standards, industry standards, equipment manuals, and inspection procedures. These documents often contain structural diagrams, flowcharts, statistical charts, structured tables, and large amounts of professional text, with high information density and complex modal forms. At present, the understanding and retrieval of these documents still mainly rely on manual review, which is not only inefficient and costly, but also difficult to meet the real-time and automated needs of new business scenarios such as intelligent quality inspection and intelligent diagnosis.

[0003] To improve document processing efficiency, various document intelligence and multi-modal question and answer models have emerged in recent years, which can jointly model images and text to achieve automatic recognition and question answering of documents such as bills, academic papers, and presentations. These models usually rely on general multi-modal data sets for training and have achieved good results in natural image understanding and general document question answering tasks. However, when these models are directly migrated to the manufacturing industry, they often encounter problems such as parameter reading errors, unit identification inaccuracies, and professional term understanding biases, making it difficult to support accurate question answering and complex reasoning of manufacturing documents, and limiting the further application of intelligent document systems in manufacturing enterprises.

[0004] The main reasons for the above problems are: first, strong modal heterogeneity. Manufacturing documents contain various modalities such as technical drawings, functional structure diagrams, flowcharts, tables, and paragraph texts. Key information is often distributed across modalities, such as a part size, assembly clearance, or inspection condition, which is usually found in both schematic diagram annotations and clause texts. Existing general data sets that focus on single-page images or single modalities cannot cover such complex cross-modal associations; second, domain specificity is prominent. Manufacturing documents contain a large number of low-frequency professional terms such as "axial play", "tolerance band", and "impactor", and these terms often rely on specific context for correct interpretation, making it difficult for models pre-trained on general corpora to learn these fine-grained semantics.

[0005] The current mainstream multi-modal question answering model generally adopts the training paradigm of "large-scale image-text pre-training + downstream multi-modal question answering / instruction data fine-tuning": in the pre-training stage, the alignment relationship between visual features and text representations is learned by using large-scale image and text paired data (such as image-text description, web image-text, scene image with text, etc.); in the downstream stage, supervised fine-tuning or instruction fine-tuning is further performed on the specially constructed multi-modal question answering data set or multi-modal instruction data set to improve the question answering ability and instruction following ability of the model. In order to better use the general multi-modal model in the manufacturing industry, it is necessary to utilize the generalization ability of the pre-trained multi-modal model, inject specific knowledge of the manufacturing industry into the model, and adapt the model to the target domain to better migrate to the manufacturing industry scene.

[0006] Although deep learning and general pre-training models have made great achievements in many fields, model fine-tuning in the manufacturing industry requires a large amount of multi-modal data. However, in terms of existing general multi-modal data sets, in terms of image types, they are mostly from life scenes or general fields, lacking structural diagrams, schematic diagrams and process diagrams commonly seen in manufacturing documents; in terms of annotation granularity, they focus more on factual question answering and short text answers, lacking sufficient annotation of high-precision information involving key parameters, units and tolerance ranges; in terms of context modeling, most data sets take a single page or local area as the basic unit, rarely considering joint reasoning needs across pages, chapters or even documents; in terms of content understanding, there is generally a lack of specialized modeling of manufacturing industry terminology.

[0007] Since the manufacturing industry scenario usually involves various physical quantities and their corresponding units, such as machining size, material compressive strength, equipment operating temperature and other parameters, numerical deviation may directly lead to production accidents or product scrap, so the manufacturing industry scenario is extremely sensitive to "numerical value + unit", and general indicators cannot reflect the risk; and the manufacturing industry scenario often involves process lists, list lists, etc., which need to avoid missing items and redundant items, and also need to ensure accurate element matching, and general indicators cannot directly measure; at the same time, general indicators cannot determine whether the terminology used in the semantically correct answer is correct, which is necessary in the manufacturing industry scenario with high specification requirements. SUMMARY

[0008] The purpose of the present application is to provide a method for constructing and evaluating a manufacturing multi-modal question answering data set, which decouples and separates the manufacturing standard documents in the national standard full-text disclosure system, annotates questions and answers for complex schematic diagrams, structural diagrams and process diagrams using multi-modal large models, and then checks and screens the data to construct a new data set that meets specific requirements, providing benchmark support for intelligent document understanding in the manufacturing industry.

[0009] In one aspect, the application provides a method for constructing a manufacturing multi-modal question and answer dataset, comprising: S100: constructing a corpus of manufacturing standard documents; S200: respectively annotating the metadata of the standard documents in the corpus; S300: respectively decoupling and separating the multi-modal elements of the standard documents in the corpus, extracting text, table, formula, image multi-modal heterogeneous elements, and respectively performing text recognition, and rearranging the recognition results according to the original standard document layout; wherein the images include but are not limited to schematic diagrams, structural diagrams, process diagrams; S400: S400: post-processing the extracted text, and inputting the post-processed text into a language model according to the prompt word template to extract terms, and constructing a global manufacturing terminology dictionary using the extracted terms; S500: for a document page containing a table, formula or image, based on the image adjacent paragraphs and the global manufacturing terminology dictionary, a dynamic context window mechanism is constructed, and the text segments related to the semantics of the image or table are arranged as context supplement information; then a plurality of task type questions are manually annotated, and the answer types are divided into numerical type, list type and text type according to the target application scenario, forming an initial question and answer pair seed set; S600: inputting the decoupled and separated single image or table, the context supplement information supplemented by the dynamic context window mechanism, the global manufacturing terminology dictionary, and the question and answer pair seed set into a multi-modal model, performing batch question and answer pair generation and answer reasoning step generation, and constructing a multi-modal manufacturing standard question and answer dataset.

[0010] Specifically, step S100 further comprises: S110: downloading PDF format manufacturing standard documents from the national standard full text disclosure system to obtain a standard document original set; S120: selecting manufacturing standard documents with high graphic-text mixing degree from the standard document original set; The sub-step S120 is specifically: for each manufacturing standard document, perform: cutting the manufacturing standard document into pictures according to pages, prompting the multi-modal model to select pictures containing two or more elements of structural diagrams, schematic diagrams, statistical diagrams, process diagrams, tables, and text, and if the proportion of such pictures exceeds a preset proportion threshold, the manufacturing standard document is a manufacturing standard document with high graphic-text mixing degree; S130: forming and saving each selected manufacturing standard document into a PDF file to constitute a corpus of manufacturing standard documents.

[0011] Further, in step S120, the multi-modal model Qwen2.5-VL-7B is used for screening.

[0012] Further, the metadata in step S200 includes the field, the standard name, the standard number, the keyword, the publication date, and the implementation date.

[0013] Specifically, the extracted text is post-processed, including: S410: Preliminary filtering of obviously meaningless or noise characters by matching common garbled character features and abnormal characters through regular expressions; S420: Taking the candidate character and its front and back text fragments and performing semantic association analysis to determine whether the candidate character hits the industry special character dictionary, and if not, deleting the candidate character; wherein the industry special character dictionary is obtained by manually sorting symbol-related national standards and contains special symbols and units.

[0014] On the other hand, the application provides an evaluation method of a manufacturing multi-modal question and answer data set, which is used to quality test the constructed multi-modal manufacturing standard question and answer data set, including: After removing the answers and reasoning processes of the samples in the multi-modal manufacturing standard question and answer data set, input the multi-modal model to obtain the model predicted answers and their new reasoning processes; Input the model predicted answers and their new reasoning processes and the answers and reasoning processes in the multi-modal manufacturing standard question and answer data set into the language model for consistency checking; When the language model judges that they are consistent, it is recorded as "pass"; when it is judged as "inconsistent", it is recorded as "fail".

[0015] Specifically, the multi-modal model is a GPT-4o model.

[0016] Specifically, the language model is a Deepseek-V3 model.

[0017] Compared with the prior art, the application has the following advantages and beneficial effects: Due to the particularity of the manufacturing industry scene, the existing multi-modal question and answer data set construction method is not applicable to the manufacturing industry scene, and the application provides a multi-modal question and answer data set construction method for the manufacturing industry scene. The construction method of the application can realize efficient and accurate construction of the manufacturing multi-modal question and answer data set, and also systematically complete multi-modal analysis of manufacturing standard documents, term extraction, question and answer pair generation, and fine quality evaluation, which can provide reliable data basis for subsequent training and evaluation of multi-modal large models in the manufacturing industry scene. BRIEF DESCRIPTION OF DRAWINGS

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating the construction method of this application; Figure 2 This is a block diagram illustrating the principle of the estimation and evaluation methods in the embodiments of this application. Detailed Implementation

[0020] The technical solution and effects of this application will be clearly and completely described below with reference to specific embodiments. Obviously, the described specific embodiments are only a part of the specific embodiments of this application, and not all of them. Based on the specific embodiments in this application, all other specific embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] The method for constructing a multimodal question-answering dataset for the manufacturing industry provided in this application can be found in [link to specific implementation details]. Figure 1 The steps include: S100: Construct a corpus of standard documents for the manufacturing industry; This step further includes: S110: Download the manufacturing standard documents in PDF format from the National Standards Full-Text Disclosure System to obtain the original set of standard documents; To ensure the industry representativeness and diversity of the corpus, standard documents from multiple sub-fields, including mechanical manufacturing, mechanical systems and general components, road vehicle engineering, shipbuilding materials, new energy, and equipment manufacturing, were selected in terms of data coverage. S120: Select manufacturing standard documents with high text-image mixing from the original set of standard documents; In this embodiment, the multimodal model Qwen2.5-VL-7B is used to filter the original set of standard documents, identifying PDF documents with high text-image mixing. It should be noted that the PDF documents here refer to manufacturing standard documents in PDF format from the original set of standard documents.

[0022] Specifically, for each PDF document, the following steps are performed: the PDF document is split into images by page, and a multimodal model is prompted to filter images that contain two or more of the following elements: structural diagrams, schematic diagrams, statistical charts, process diagrams, tables, and text. If the proportion of such images exceeds a preset threshold, then the PDF document is a PDF document with a high degree of image and text mixing.

[0023] Above, the proportion of such pictures refers to the proportion of such pictures in all the split pictures. Further, in the present embodiment, the proportion threshold is preset to 40%.

[0024] In addition, in order to increase sample diversity, part of the manufacturing standard documents with low graphic-text mixing degree are reserved.

[0025] The manufacturing standard document with low graphic-text mixing degree is opposite to the manufacturing standard document with low graphic-text mixing degree, that is, the proportion of such pictures is not higher than the preset proportion threshold.

[0026] S130: Forming PDF files of each manufacturing standard document screened out respectively and saving, to constitute a corpus of manufacturing standard documents.

[0027] S200: Metadata labeling is performed on each standard document in the corpus; In order to facilitate subsequent retrieval of associated context, the metadata includes but is not limited to field, standard name, standard number, keyword, publication date, implementation date, to form a structured index of manufacturing standard documents.

[0028] S300: Multi-modal element decoupling separation is performed on each standard document in the corpus, and text, table, formula, image multi-modal heterogeneous elements are extracted, and text recognition is performed respectively, and the recognition results are rearranged according to the original standard document layout; wherein the image includes but is not limited to schematic diagram, structure diagram, process diagram; After the metadata labeling is completed, each standard document in PDF format is parsed by page, including using a multi-modal element decoupling separation method to analyze the layout of the standard document, and detecting and separating multi-modal heterogeneous elements such as text, table, formula, image, etc.

[0029] In the present embodiment, the specific steps of the specific implementation manner of the present step are as follows: S310: Based on the DocLayout-YOLO model, the layout analysis and detection of the pages of the standard document are performed respectively, and the image, table, text, title, legend, formula, etc. Document element area is located; S320: In view of the complexity of the formula area, the Unimernet_tiny model is used to recognize the formula area, and the recognized formula is converted into LaTeX format; S330: According to the boundary box coordinates obtained by detection, the page image is cropped to generate table sub-graph, text sub-graph, formula sub-graph, and sub-graph of schematic diagram, structure diagram, process diagram; S340: The Chinese OCR model of paddleocr is used to perform OCR recognition on the text area to obtain the text content; S350: Rearrange and organize the recognition results of the text, title, table, formula, etc. according to the original document layout to finally generate the corresponding Markdown format document.

[0030] S400: Post-processing the extracted text, and inputting the post-processed text into a language model to extract terms according to a prompt word template, and constructing a global manufacturing industry term dictionary using the extracted terms; Specifically, the prompt word template is used to screen concepts with clear and professional meanings in the industrial manufacturing context, including equipment or machine names, processing technology, manufacturing process, materials, parts, models, process parameters, technical indicators, quality control, detection methods, industry abbreviations, etc. Term extraction is performed according to the concepts respectively.

[0031] In natural language processing, post-processing of text refers to cleaning the text to improve text quality. In this specific embodiment, the post-processing includes: using a rule-based method to remove meaningless characters such as garbled characters, special characters, and format redundant symbols, while retaining key technical symbols.

[0032] The post-processing of the text uses a double error correction method of "semantic association + rule matching", including: S410: Match common garbled character features and abnormal characters through regular expressions to preliminarily filter out obviously meaningless or noise characters; S420: Take the candidate character and its surrounding text fragments and perform semantic association analysis to determine whether the candidate character hits the industry special character dictionary. If not, delete the candidate character to avoid the traditional rule filtering from deleting key technical symbols such as φ, ±, ≥, etc. The industry special character dictionary is obtained by manually sorting symbols related to national standards and contains special symbols and units.

[0033] S500: For document pages containing tables, formulas, or images, based on image adjacent paragraphs and the global manufacturing industry term dictionary, a dynamic context window mechanism is constructed to organize text fragments related to the semantics of the image or table as context supplement information. Then, manually annotate multiple task types of questions, and according to the target application scenario, divide the answer types into numerical type (Quant), list type (List), and text type (Text) to form an initial question and answer pair seed set.

[0034] In this embodiment, the task types include DocVQA, VIE, DLA, and DCS, etc.

[0035] The above-mentioned text fragments related to the semantics of the image or table are organized as context supplement information, which specifically includes: segmenting the text according to paragraphs, and organizing text fragments that refer to the table or image, and text fragments with high semantic similarity with the table or image title as context supplement information.

[0036] The schematic diagram, structural diagram, process diagram and table obtained in step three are associated with the text content and term dictionary obtained in steps four and five, a dynamic context window is constructed to enhance the semantic interpretability of the image and table, and context support is provided for subsequent question and answer pair construction.

[0037] S600: The single image or table obtained by decoupling separation, the context supplement information supplemented by the dynamic context window mechanism, the global manufacturing industry term dictionary, and the question and answer pair seed set are input into a multi-modal model to perform batch question and answer pair generation and answer reasoning step generation, thereby constructing a multi-modal manufacturing industry standard question and answer data set.

[0038] Specifically, the seed question and answer pairs are labeled by a domain expert based on the dynamic context window and the global manufacturing industry term dictionary. The problem types cover four task forms of document visual question answering (DocVQA), key information extraction (VIE), document layout analysis (DLA), and document content understanding (DCS). The answer types include numerical type, list type, and text type. Different labeling specifications are formulated for different problem types and answer types.

[0039] S700: Quality inspection is performed on the question and answer pairs generated in step S600. Specifically, a three-level verification system of “model pre-verification → consistency check → manual review” is adopted to ensure the accuracy and reliability of the generated question and answer pairs.

[0040] To ensure the quality of the constructed data set, the present application constructs a data verification process based on the multi-modal model and the language model, which includes: First, the samples in the multi-modal manufacturing industry standard question and answer data set are input into the multi-modal model after the answers and reasoning processes are removed, to obtain the model predicted answers and new reasoning processes. In this embodiment, the multi-modal model is a GPT-4o model. Then, the model predicted answers and new reasoning processes are input into the language model together with the answers and reasoning processes in the multi-modal manufacturing industry standard question and answer data set for consistency checking. In this embodiment, the language model used for consistency checking is a Deepseek-V3 model. When the language model determines that the model answers are consistent with the reference answers, it is recorded as “pass”, and a portion of the samples are randomly sampled for manual review. When it is determined that they are inconsistent, all corresponding samples are submitted to the human for item-by-item checking and modification.

[0041] After the above verification process, the generated question and answer pairs are screened and corrected to form the final manufacturing industry multi-modal document understanding high-quality data set ManuStdVQA. Embodiment

[0042] This embodiment uses an independent test set containing 3958 samples, with the following sample distribution for numerical, list, and text classes: 997 numerical samples, 821 list samples, and 2140 text samples.

[0043] This embodiment designs a differentiated scoring mechanism based on the characteristics of each type of answer: (1) Numerical tasks Given that manufacturing scenarios typically involve various physical quantities and their corresponding units, such as machining dimensions, material compressive strength, and equipment operating temperature, numerical deviations can directly lead to production accidents or product scrapping. Furthermore, general-purpose models are prone to unit confusion or misuse during processing. Therefore, in addition to employing precise numerical matching, this embodiment further introduces a unit consistency verification mechanism. The specific calculation method is as follows: ; in, The unit consistency score is obtained through a rule-based matching mechanism: 1 when the unit matches successfully, and 0 otherwise. The score represents an exact match of numerical values, also calculated using a rule-based approach, with the same value selection rules as... Consistent, and This item is used to reflect the consistency between numerical values ​​and unit matching. It is only 1 when both are correct, otherwise 0, thus preventing the model from receiving a non-zero score when only the unit is correct. The weight parameter is less than 1.

[0044] The aforementioned mechanism means that a perfect score can only be obtained when the predicted value and the actual value are completely identical in both value and unit. However, this stringent criterion may lead to some reasonable results being misjudged—for example, when the difference between the predicted value and the actual value is only eliminated by unit conversion or by the difference between Chinese and English, it may be classified as a score of 0. To address this, for samples with a score of 0, this application further introduces Deepseek-V3 for secondary verification: the model determines whether the predicted value can be converted into the actual value; if the determination result is "yes," the final score is corrected to 1; otherwise, the score remains unchanged.

[0045] (2) List-based tasks For list-type data, such as assembly process lists and parts material lists, it is necessary to ensure that the list length is consistent with the standard answer to avoid omissions and redundancies, while also ensuring accurate matching of core elements. Therefore, the evaluation mechanism comprehensively considers both length consistency and element accuracy. Let the actual list length be... The length of the predicted value list is Define length consistency score For: when hour, 1; otherwise, 0. The basis for this setting is that the matching degree of the list length can intuitively reflect the comprehensive retrieval ability of the model to visual information and the complete control ability to image details. If the lengths are inconsistent, it means that the model has a fundamental deviation in the integrity or accuracy of information extraction, so directly This item is scored as 0.

[0046] Then the element accuracy rate is calculated , which is defined as the proportion of the number of elements in the predicted value list that are completely matched with the elements in the true value list to the total number of elements , where represents the number of elements predicted correctly. Among them, is used to reflect the consistency of length and accuracy rate, so as to avoid the model from obtaining higher evaluation when it performs well in a single dimension, and are weight parameters less than 1. The calculation formula of the final score is .

[0047] (3) Text type task For text data such as technical document interpretation, troubleshooting instructions, etc., the core is to convey accurate semantics, and term expression deviation can be corrected by industry common sense, but semantic loss directly affects use. Therefore, the evaluation mechanism combines semantic similarity judgment and term consistency verification, and the specific calculation method is as follows: First, the Deepseek-V3 model is introduced to score the semantic similarity. This model judges the matching degree based on the semantic content of the true value and the predicted value. If they are semantically consistent, the semantic correctness score is set to 1, otherwise it is 0. Second, define the term consistency score to verify the consistency of the terms used in the predicted value and the terms in the question. If there is a term loss, mismatch or semantic deviation (non-synonymous word replacement situation), then 0, otherwise 1. The final score is obtained by weighting the above two items: , is a weight parameter less than 1, and if there is no term in the question, it is 0.

[0048] In the model training phase, the experiment is based on a single NVIDIA A100 graphics card, and the LoRA parameter fine-tuning method is used to train the Qwen2.5-VL-7B model. The specific parameter settings of the training process are as follows: the training data set contains 35,560 samples, the total training rounds are 2; the batch size is set to 16; the rank parameter of LoRA is set to 16; the initial learning rate is configured as and cosine learning rate decay strategy is adopted to dynamically adjust the learning rate to optimize the model convergence process. Table 1 is the evaluation score of the model on each type of sample and the comprehensive weighted score after taking the average of three experiments.

[0049] Table 1 is the evaluation score of the model on each type of sample and the comprehensive weighted score after taking the average of three experiments. Model Numerical class sample List class sample Text class sample Synthesis Janus-Pro-7B 5.71 3.33 34.19 20.61 Molmo-7B-D-0924 19.73 7.10 40.95 28.58 llava-ov-qwen2-7b 28.48 8.87 43.16 32.35 MiniCPM-V-2.6 56.56 51.77 72.49 64.18 SAIL-VL-1d6-8B 56.18 54.85 77.66 67.06 InternVL3-8B 64.15 60.68 70.87 67.52 GLM-4V-9B 60.57 60.37 77.53 69.70 MiniCPM-O-2.6 62.63 60.47 79.12 71.10 Qwen2.5-VL-3B 64.14 63.32 80.25 72.68 InternVL3-9B 68.85 65.76 79.46 73.95 Ovis2-8B 70.27 67.61 83.58 76.91 Qwen2.5-VL-7B 78.51 68.82 81.59 78.16 Lora-Qwen2.5-VL-7B 83.34 75.50 84.54 82.36 Lora-Qwen2.5-VL-7B is the trained model of the present embodiment. As can be seen from Table 1, the overall performance of the model is improved by 4.20%, and the improvement in numerical and list type tasks is significant, while the improvement in text type tasks is weak.

[0050] The above has described the embodiments of the present application, the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles, practical application or improvement of technology in the market of the embodiments, or to enable other ordinary skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for constructing a multimodal question-answering dataset for manufacturing, characterized in that, include: S100: Construct a corpus of standard documents for the manufacturing industry; S200: Perform metadata annotation on standard documents in the corpus; S300: Perform multimodal element decoupling and separation on standard documents in the corpus, extract multimodal heterogeneous elements such as text, tables, formulas, and images, and perform text recognition on each. Then, rearrange the recognition results according to the original standard document layout. Among them, images include, but are not limited to, schematic diagrams, structural diagrams, and process diagrams. S400: Post-process the extracted text and input the post-processed text into the language model to extract terms according to the prompt word template. Use the extracted terms to build a global manufacturing terminology dictionary. S500: For document pages containing tables, formulas, or images, a dynamic context window mechanism is built based on adjacent paragraphs of the image and a global manufacturing terminology dictionary. Text fragments semantically related to the image or table are organized into supplementary context information. Then, questions of various task types are manually labeled, and the answer types are divided into numerical, list, and text types according to the target application scenario to form an initial seed set of question-answer pairs. S600: Input the decoupled individual images or tables, supplemented contextual information through a dynamic context window mechanism, a global manufacturing terminology dictionary, and a seed set of question-answer pairs into the multimodal model, perform batch question-answer pair generation and answer reasoning step generation, and construct a multimodal manufacturing standard question-answer dataset.

2. The construction method as described in claim 1, characterized in that: Step S100 further includes: S110: Download the manufacturing standard documents in PDF format from the National Standards Full-Text Disclosure System to obtain the original set of standard documents; S120: Select manufacturing standard documents with high text-image mixing from the original set of standard documents; Sub-step S120 specifically involves: performing the following steps on each manufacturing standard document: dividing the manufacturing standard document into images by page, prompting a multimodal model to filter images that simultaneously contain two or more elements from structural diagrams, schematic diagrams, statistical charts, process diagrams, tables, and text. If the proportion of such images exceeds a preset threshold, then the manufacturing standard document is a manufacturing standard document with a high degree of image-text mixing. S130: The selected manufacturing standard documents are generated into PDF files and saved to form a corpus of manufacturing standard documents.

3. The construction method as described in claim 1, characterized in that: The metadata mentioned in step S200 includes the domain, standard name, standard number, keywords, publication date, and implementation date.

4. The construction method as described in claim 2, characterized in that: In step S120, the multimodal model Qwen2.5-VL-7B is used for screening.

5. The construction method as described in claim 1, characterized in that: The post-processing of the extracted text includes: S410: Use regular expressions to match common garbled character features and abnormal characters, and perform preliminary filtering of obviously meaningless or noisy characters; S420: Take candidate characters and the text fragments before and after them and perform semantic association analysis to determine whether the candidate characters match the industry special character dictionary. If they do not match, the candidate characters are deleted; The industry special character dictionary is obtained by manually compiling national standards related to symbols and includes special symbols and units.

6. An evaluation method for a multimodal question-answering dataset in the manufacturing industry, characterized in that: Used to perform quality inspection on the multimodal manufacturing standard question-and-answer dataset constructed according to any one of claims 1 to 5, including: After removing the answers and reasoning processes from the samples in the multimodal manufacturing standard question-and-answer dataset, the samples are input into the multimodal model to obtain the model's predicted answers and its new reasoning processes. The model's predicted answer and its new reasoning process are input into the language model along with the answers and reasoning processes from the multimodal manufacturing standard question-answering dataset for consistency verification. When the language model determines that the language model agrees, it is recorded as "pass"; when it determines that the language model disagrees, it is recorded as "fail".

7. The evaluation method as described in claim 6, characterized in that: The multimodal model is the GPT-4o model.

8. The evaluation method as described in claim 6, characterized in that: The language model is the Deepseek-V3 model.

Images