A method and device for scheme detection of complex text-in-image documents

Abstract

The application provides a scheme detection method and device for a complex mixed file, and the method comprises the following steps: extracting standard review items and standard review contents in a standard manual and performing structural processing to obtain a standard review file; converting a file to be detected into an image to be detected, using OCR, an initial recognition model obtained by pre-training, and a matching model to extract a multi-scale candidate region image to be reviewed from the image to be detected, and extracting text information of the multi-scale candidate image to obtain candidate region text information; inputting the multi-scale candidate region image and the standard review file into a lightweight hybrid twin network obtained by pre-training to perform image feature matching, and outputting a matching feature image; inputting the matching feature image, the standard review file, and the candidate region text information into a multi-modal large model obtained by pre-training to perform compliance analysis, and outputting a detection result; and the method can significantly improve the automation and intelligence of the review of the complex mixed file.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent manufacturing, and particularly relates to a scheme detection method and device for a complex graphic-text mixed file. BACKGROUND

[0002] In the engineering, manufacturing, power and other industries, the icons in design drawings, product manuals, assembly documents, etc. (usually in PDF, CAD and other formats) have strict standard and specification requirements, and their standardization and consistency are of great significance to project safety and quality. In order to ensure the standardization and consistency of the above-mentioned files, it is often necessary to check each item of the scheme in the file according to the specifications in the manual, and check its existence, position accuracy, image integrity, rationality of matching and consistency of style.

[0003] The traditional design file review method relies on manual page-by-page checking and comparison with standards, which is low in efficiency and strong in subjectivity, and is prone to omissions or misjudgments in icon matching, position judgment, detail review and other aspects, and is difficult to meet the current high-standard and batch review requirements. To improve efficiency, some systems attempt to use OCR (Optical Character Recognition), image template matching, CAD automatic review plug-ins and other technologies to realize intelligent review of drawings. For example, patent text CN118334694A discloses a mechanical design drawing review method, which includes: performing character recognition on the mechanical design drawing uploaded by the user to obtain all character data, and converting each character data into text data to obtain all text data; converting each text data into standard text data according to a pre-defined mechanical design drawing standard to obtain all standard text data; judging whether each standard text data is error data based on a pre-established mechanical design drawing judgment rule library; and generating error prompt information according to the standard text data when any standard text data is error data. This method is based on the OCR and image template matching scheme, does not have semantic understanding ability and cannot handle complex scenarios, and has a high risk of recognition failure when there are problems such as icon or text rotation, scaling, deformation, incompleteness, unclear standard requirement semantics, etc.

[0004] The patent CN116563879A discloses a method and system for recognizing multiple lines of text and / or multi-angle text in electrical drawings, relating to the field of image analysis. The method for recognizing multiple lines of text and multi-angle text in electrical drawings includes: detecting the area where the multiple lines of text are located through the YOLOv5 target detection algorithm; retaining the coordinates of the recognition result and extracting the area where the multiple lines of text are located; performing single-line text detection on the extracted multiple-line text area through the YOLOv5 target detection algorithm again; splicing the single-line text belonging to the same multiple-line text area; rotating the detected single-line text three times, i.e. 90 degrees, 180 degrees, and 270 degrees, and saving the results; recognizing the text in four directions through OCR and selecting the text recognition result in the direction with the highest confidence as the output result; if the confidence is lower than the threshold, repeating the detection of the area where the multiple lines of text are located, and if the confidence is higher than the threshold, taking it as the output result. This method is based on the YOLO visual detection model and can train and recognize fixed categories of reviews, greatly improving the positioning accuracy and stability. However, it has the problems of high training cost and poor generalization, and it needs to be retrained for each additional category and does not support semantic analysis and free language review. SUMMARY

[0005] The present application provides a scheme detection method and device for complex graphic-text mixed files, which can improve the accuracy of complex graphic-text mixed file review.

[0006] A scheme detection method for complex graphic-text mixed files, comprising:

[0007] extracting standard review items and standard review content in a standard manual and performing structured processing to obtain a standard review file;

[0008] converting a file to be detected into an image to be detected, using OCR, an initial recognition model obtained by pre-training, and a matching model to extract a multi-scale candidate region image to be reviewed from the image to be detected, and extracting text information of the multi-scale candidate image to obtain candidate region text information;

[0009] inputting the multi-scale candidate region image and the standard review file into a lightweight hybrid twin network obtained by pre-training to perform image feature matching and output a matching feature image;

[0010] inputting the matching feature image, the standard review file, and the candidate region text information into a multi-modal large model obtained by pre-training to perform compliance analysis and output a detection result.

[0011] Further, the standard manual includes structured data and unstructured data;

[0012] The standard review items and standard review contents in the standard manual are extracted and structured to obtain a standard review file, including:

[0013] For the structured data, a first standard review item is identified based on the standard manual, and graphic and text information about the first standard review item is extracted line by line as first standard review contents. For each line of extracted graphic and text information, text information is extracted as first text additional information by using OCR. The first standard review item, the first standard review contents, and the corresponding first text additional information are structured to obtain the standard review file.

[0014] For the unstructured data, the standard manual is converted into a picture, layout analysis is performed on the picture by using a pre-trained recognition model, a graphic and text content module is extracted, a second standard review item is identified according to the graphic and text content module, and a region in the graphic and text content module corresponding to the second standard review item is identified as second standard review contents. Text information in the graphic and text content module is extracted as second text additional information by using OCR. The second standard review item, the second standard review contents, and the corresponding second text additional information are structured to obtain the standard review file.

[0015] Further, a plurality of scale candidate region images to be reviewed are extracted from the to-be-detected image by using OCR, a pre-trained initial recognition model, and a matching model, including:

[0016] The to-be-detected image is input into the initial recognition model for layout analysis, and a drawing region and a text region are extracted as a coarse positioning region.

[0017] OCR is used to recognize text in the coarse positioning region to obtain a text recognition result.

[0018] A standard review item in the standard review file is extracted.

[0019] The text recognition result, the standard review item, and the coarse positioning region are input into the matching model. The matching model performs at least one operation of keyword matching, regular matching, and semantic similarity matching on the text recognition result and the standard review item, determines a region in the coarse positioning region that matches the standard review item, and performs multi-scale feature extraction on the matched region by using a sliding window to obtain the plurality of scale candidate region images.

[0020] Further, the lightweight hybrid twin network comprises a backbone network module and a twin network module; the backbone network module comprises an input layer, a plurality of stacked feature extraction modules, and an output layer; the twin network module comprises a feature projection module, an MLP similarity calculation module, and an output module; the feature extraction module comprises a lightweight convolutional layer, a Transformer layers module, and a feature fusion module.

[0021] Further, the multi-scale candidate region image and the standard review file are input into a lightweight hybrid twin network obtained by pre-training for image feature matching, and a matching feature image is output, comprising:

[0022] The multi-scale candidate region image and the standard review file are sequentially subjected to feature extraction, global modeling, and feature fusion by the feature extraction module, the Transformer layers module, and the feature fusion module, and output fusion multi-scale candidate region features and fusion standard features;

[0023] The feature projection module maps the fusion multi-scale candidate region features and the fusion standard features into the same space, and then performs similarity calculation through the MLP similarity calculation module to obtain the matching probability of the fusion multi-scale candidate region features and the fusion standard features; the output module filters the fusion multi-scale candidate region features according to the matching probability and outputs a matching feature image.

[0024] Further, the input layer receives the multi-scale candidate region image and the standard review file and sends them to a plurality of feature extraction modules; the lightweight convolutional layer in the feature extraction module uses standard convolution to extract features from the multi-scale candidate region image and the standard review file respectively, obtaining multi-scale candidate region features and standard features; the Transformer layers module is used for global modeling of the multi-scale candidate region features and the standard features respectively, obtaining global multi-scale candidate region features and global standard features at different levels; the feature fusion module is used for feature fusion of the global multi-scale candidate region features and the global standard features at different levels, and outputs fusion multi-scale candidate region features and fusion standard features through the output layer; the feature projection module is used for mapping the fusion multi-scale candidate region features and the fusion standard features into the same space; the MLP similarity calculation module is used for similarity calculation of the fusion multi-scale candidate region features and the fusion standard features in the same space, obtaining the matching probability of the fusion multi-scale candidate region features and the fusion standard features; the output module is used for filtering the fusion multi-scale candidate region features according to the matching probability and a preset probability threshold, and outputting the fusion multi-scale candidate region features with a matching probability exceeding the preset probability threshold as a matching feature image.

[0025] Further, the method further comprises: training the lightweight hybrid twin network according to a pre-constructed training sample;

[0026] The training sample is constructed, including:

[0027] Collecting industrial standard document data and converting the data into image data, and performing image-text extraction and cropping based on the image data to obtain standard sample patches;

[0028] A sliding window is used to extract candidate patches from the sample to-be-inspected file;

[0029] Selecting a candidate patch corresponding to the standard sample patch as a positive sample, and grouping the positive sample and the corresponding standard sample patch into a positive sample pair;

[0030] Selecting a candidate patch not corresponding to the standard sample patch as a negative sample, and grouping the negative sample and the standard sample patch into a negative sample pair.

[0031] Further, the training of the lightweight hybrid twin network comprises:

[0032] Repeat the following steps until the stop condition is met:

[0033] Input the positive sample pair and the negative sample pair into the backbone network module to extract features, output a first sample feature vector and a second sample feature vector;

[0034] Input the first sample feature vector and the second sample feature vector into the twin network module to perform similarity calculation and threshold judgment and output a judgment result, and find the most similar negative sample to the positive sample pair as a third sample feature vector during the training process;

[0035] Calculate the cosine loss according to the first sample feature vector and the second sample feature vector, calculate the binary cross entropy loss according to the output result of the twin network module, and calculate the triplet loss according to the first sample feature vector, the second sample feature vector and the third sample feature vector;

[0036] Obtain a comprehensive loss according to the cosine loss, the binary cross entropy loss and the triplet loss, and update the parameters of the lightweight hybrid twin network according to the comprehensive loss.

[0037] Further, the multi-modal large model comprises a dynamic Prompt module and a Qwen2.5-VL model, and the dynamic Prompt module is used to construct a prompt word according to the candidate region text information and the standard review file;

[0038] The Qwen2.5-VL model is used for analysis based on the prompt word and the matching feature image, and outputs a detection result of whether the review content corresponding to the review item in the file to be detected is compliant.

[0039] Further, the multi-modal large model is pre-trained, including:

[0040] Industrial standard document data is collected and converted into image data, and image-text extraction is performed based on the image data to obtain sample image-text data. The sample image-text data is integrated and processed with the corresponding review standard to obtain a large model sample.

[0041] A pre-trained Qwen2.5-VL model is obtained and injected with LoRA parameters, and the main parameters of the pre-trained Qwen2.5-VL model are frozen.

[0042] The following steps are repeatedly executed until the stopping condition is met:

[0043] The large model sample is input into the pre-trained Qwen2.5-VL model, and the output result of the pre-trained Qwen2.5-VL model is obtained. The output result of the pre-trained Qwen2.5-VL model is added with artificial feedback data to form a supervised sample, and a reward model is constructed.

[0044] The supervised sample is input into the reward model for training, and the LoRA parameters of the pre-trained Qwen2.5-VL model are fine-tuned based on the output result of the reward model.

[0045] A scheme detection device for complex image-text mixed files, comprising:

[0046] A standard extraction module is used to extract standard review items and standard review content in a standard manual and perform structured processing to obtain a standard review file.

[0047] A candidate region extraction module is used to convert a file to be detected into a to-be-detected image, and uses OCR, an initial recognition model obtained by pre-training, and a matching model to extract a multi-scale candidate region image to be reviewed from the to-be-detected image, and extract text information of the multi-scale candidate image to obtain candidate region text information.

[0048] A matching module is used to input the multi-scale candidate region image and the standard review file into a pre-trained lightweight hybrid twin network to perform image feature matching, and output a matching feature image.

[0049] A detection module is used to input the matching feature image, the standard review file, and the candidate region text information into a pre-trained multi-modal large model for compliance analysis, and output a detection result.

[0050] Further, the standard manual includes structured data and unstructured data;

[0051] The standard extraction module extracts standard review items and standard review contents in the standard manual and performs structured processing to obtain a standard review file, including:

[0052] For the structured data, a first standard review item is identified based on the standard manual, and graphic and text information about the first standard review item is extracted line by line as first standard review contents. For each line of graphic and text information extracted, text information is extracted as first text additional information using OCR. The first standard review item, the first standard review contents, and the corresponding first text additional information are structured processed to obtain the standard review file.

[0053] For the unstructured data, the standard manual is converted into a picture, layout analysis is performed on the picture using a pre-trained recognition model, a graphic and text content module is extracted, a second standard review item is identified according to the graphic and text content module, and a region in the graphic and text content module corresponding to the second standard review item is identified as second standard review contents. Text information in the graphic and text content module is extracted as second text additional information using OCR. The second standard review item, the second standard review contents, and the corresponding second text additional information are structured processed to obtain the standard review file.

[0054] Further, a candidate region extraction module extracts a multi-scale candidate region image to be reviewed from the to-be-detected image using OCR, a pre-trained initial recognition model, and a matching model, including:

[0055] The to-be-detected image is input to the initial recognition model for layout analysis to extract a graphic region and a text region as a coarse positioning region.

[0056] OCR is used to recognize text in the coarse positioning region to obtain a text recognition result.

[0057] A standard review item in the standard review file is extracted.

[0058] The text recognition result, the standard review item, and the coarse positioning region are input to the matching model. The matching model performs at least one of keyword matching, regular matching, and semantic similarity matching on the text recognition result and the standard review item to determine a region in the coarse positioning region that matches the standard review item. A multi-scale feature of the matched region is extracted using a sliding window to obtain the multi-scale candidate region image.

[0059] Further, the lightweight hybrid twin network comprises a backbone network module and a twin network module; the backbone network module comprises an input layer, a plurality of stacked feature extraction modules, and an output layer; the twin network module comprises a feature projection module, an MLP similarity calculation module, and an output module; the feature extraction module comprises a lightweight convolutional layer, a Transformer layers module, and a feature fusion module.

[0060] Further, the matching module inputs the multi-scale candidate region image and the standard review file into a lightweight hybrid twin network obtained by pre-training for image feature matching, and outputs a matching feature image, comprising:

[0061] The multi-scale candidate region image and the standard review file are sequentially subjected to feature extraction, global modeling, and feature fusion by the feature extraction module, the Transformer layers module, and the feature fusion module, and output fusion multi-scale candidate region features and fusion standard features;

[0062] The feature projection module maps the fusion multi-scale candidate region features and the fusion standard features into the same space, and then performs similarity calculation by the MLP similarity calculation module to obtain the matching probability of the fusion multi-scale candidate region features and the fusion standard features; the output module filters the fusion multi-scale candidate region features according to the matching probability, and outputs the matching feature image.

[0063] Further, the matching module is further used for:

[0064] The input layer is controlled to receive the multi-scale candidate region image and the standard review document and send to a plurality of feature extraction modules, a light convolutional layer in the feature extraction module adopts a standard convolution to respectively perform feature extraction on the multi-scale candidate region image and the standard review document, to obtain multi-scale candidate region features and standard features, the Transformer layers module is used for respectively performing global modeling on the multi-scale candidate region features and the standard features, to obtain global multi-scale candidate region features and global standard features at different levels, the feature fusion module is used for performing feature fusion on the global multi-scale candidate region features and the global standard features at different levels, and outputs the fused multi-scale candidate region features and the fused standard features through an output layer; the feature projection module is used for mapping the fused multi-scale candidate region features and the fused standard features into the same space, the MLP similarity calculation module is used for performing similarity calculation on the fused multi-scale candidate region features and the fused standard features in the same space, to obtain a matching probability of the fused multi-scale candidate region features and the fused standard features, and the output module is used for screening the fused multi-scale candidate region features according to the matching probability and a preset probability threshold, and outputting the fused multi-scale candidate region features with the matching probability exceeding the preset probability threshold as a matching feature image.

[0065] Further, the device further comprises a first training module configured to train the light hybrid twin network according to pre-constructed training samples.

[0066] The training samples are constructed, including:

[0067] The industrial standard document data is collected and converted into image data, and image-text extraction and cropping are performed based on the image data to obtain standard sample blocks.

[0068] A sliding window is used to extract candidate blocks from the sample review document.

[0069] The candidate blocks corresponding to the standard sample blocks are selected as positive samples, and the positive samples and the corresponding standard sample blocks are combined to form a positive sample pair.

[0070] The candidate blocks not corresponding to the standard sample blocks are selected as negative samples, and the negative samples and the standard sample blocks are combined to form a negative sample pair.

[0071] Further, the first training module trains the light hybrid twin network, including:

[0072] The following steps are repeatedly executed until a stop condition is met:

[0073] The positive sample pair and the negative sample pair are input into the backbone network module for feature extraction, to output a first sample feature vector and a second sample feature vector.

[0074] inputting the first sample feature vector and the second sample feature vector into the twin network module for similarity calculation and threshold judgment and outputting a judgment result, and finding a negative sample most similar to the positive sample pair as a third sample feature vector in a training process;

[0075] calculating a cosine loss according to the first sample feature vector and the second sample feature vector, calculating a binary cross entropy loss according to an output result of the twin network module, and calculating a triplet loss according to the first sample feature vector, the second sample feature vector and the third sample feature vector;

[0076] obtaining a comprehensive loss according to the cosine loss, the binary cross entropy loss and the triplet loss, and updating parameters of the lightweight hybrid twin network according to the comprehensive loss through back propagation.

[0077] Further, the multi-modal large model comprises a dynamic Prompt module and a Qwen2.5-VL model, the dynamic Prompt module is configured to construct a prompt word according to the candidate region text information and the standard review file;

[0078] the Qwen2.5-VL model is configured to analyze based on the prompt word and the matching feature image, and output a detection result of whether the review content corresponding to the corresponding review item in the file to be detected is compliant.

[0079] Further, the device further comprises a second training module configured to pre-train the multi-modal large model, comprising:

[0080] collecting industrial standard document data and converting the industrial standard document data into image data, performing image-text extraction based on the image data to obtain sample image-text data, integrating and processing the sample image-text data and corresponding review standards to obtain a large model sample;

[0081] obtaining a pre-trained Qwen2.5-VL model and injecting LoRA parameters, and freezing main parameters of the pre-trained Qwen2.5-VL model;

[0082] repeating the following steps until a stop condition is met:

[0083] inputting the large model sample into the pre-trained Qwen2.5-VL model and obtaining an output result of the pre-trained Qwen2.5-VL model, adding artificial feedback data to the output result of the pre-trained Qwen2.5-VL model to form a supervised sample, and constructing a reward model;

[0084] The supervision sample is input into the reward model for training, and the LoRA parameters of the pre-trained Qwen2.5-VL model are fine-tuned based on the output result of the reward model.

[0085] The scheme detection method and device for complex graphic-text mixed files provided by the application have at least the following beneficial effects:

[0086] (1) The multimodal large model (Qwen2.5-VL) is introduced to realize complex graphic-text semantic understanding, which can automatically identify, match and understand the elements such as icons, symbols, texts and structures in the scheme, improve the accuracy of scheme detection and reduce the risk of missed detection, and significantly improve the automation and intelligence of the examination of complex graphic-text mixed files;

[0087] (2) A lightweight hybrid twin network is constructed, global and local features are fused, and the representation ability of the block feature is significantly enhanced, a dynamic similarity measurement module composed of a lightweight MLP and a GELU activation function is designed, the robustness of the model to factors such as scale difference, style change and printing error is enhanced, and finally a multiple loss function combination mechanism is used to optimize the model weight, further improving the representation ability of the model to the scheme features, adapting to the multi-scale, multi-angle and multi-style variants of the elements in the scheme;

[0088] (3) The OCR recognition information, image features and examination item information are fused to improve the semantic accuracy, a dynamic Prompt construction mechanism is used to constrain the model to generate content, control the context length, avoid multi-task interference, and can be flexibly adapted to different file types and quickly integrated into an industrial platform;

[0089] (4) By uploading the examination standards in Excel / Word format, the examination template is dynamically generated, the document can be flexibly replaced to adapt to different custom standards, the industrial drawing dataset can be automatically built by building tools, and the model is fine-tuned by combining LoRA and RLHF technology, the model knowledge system is continuously improved, and the understanding ability of the enterprise standard is continuously enhanced;

[0090] (5) The explainability and traceability of the file examination task are enhanced, the manual correction workload and examination period are greatly reduced, the overall process is automatically executed, the examination and annotation of the complete file scheme can be completed in an average of minutes, the examination report is automatically generated, the examination result includes structured output, model judgment reason, credibility score and intuitive visualization on the drawing, which assists manual confirmation, greatly reduces manual cost, and improves the efficiency of large engineering project drawing examination. BRIEF DESCRIPTION OF DRAWINGS

[0091] Figure 1 The flowchart of one embodiment of the scheme detection method for complex graphic-text mixed files provided by the application.

[0092] Figure 2 A structural schematic diagram of an embodiment of a lightweight hybrid twin network provided by the present application for a scheme detection method for complex graphic-text mixed files.

[0093] Figure 3 A structural schematic diagram of an embodiment of a feature extraction module provided by the present application for a scheme detection method for complex graphic-text mixed files.

[0094] Figure 4 A structural schematic diagram of an embodiment of a multi-modal large model provided by the present application for a scheme detection method for complex graphic-text mixed files.

[0095] Figure 5 A flowchart of an embodiment of a scheme detection device provided by the present application for complex graphic-text mixed files. DETAILED DESCRIPTION

[0096] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in combination with the drawings in the specification and specific embodiments.

[0097] REFERENCE Figure 1 In some embodiments, a scheme detection method for complex graphic-text mixed files is provided, comprising:

[0098] S1, extracting standard review items and standard review contents in a standard manual and performing structured processing to obtain a standard review file;

[0099] S2, converting a file to be detected into an image to be detected, using OCR, an initial recognition model obtained by pre-training, and a matching model to extract a multi-scale candidate region image to be reviewed from the image to be detected, and extracting text information of the multi-scale candidate image to obtain candidate region text information;

[0100] S3, inputting the multi-scale candidate region image and the standard review file into a lightweight hybrid twin network obtained by pre-training to perform image feature matching, and outputting a matching feature image;

[0101] S4, inputting the matching feature image, the standard review file, and the candidate region text information into a multi-modal large model obtained by pre-training to perform compliance analysis, and outputting a detection result.

[0102] Specifically, in step S1, the standard manual can include structured data and unstructured data, specifically, standard examination items and corresponding standard examination contents. The structured data is, for example, an Excel table, and the unstructured data is, for example, a Word or PDF document. For the structured data, a first standard examination item is identified based on the standard manual, and graphic and text information is extracted line by line as first standard examination contents. For each line of extracted graphic and text information, OCR is used to extract text information as first text additional information, and the first standard examination item, the first standard examination contents, and the corresponding first text additional information are structured to obtain a standard examination file with uniform format.

[0103] For the unstructured data, the standard manual is converted into a picture, the picture is input into a pre-trained recognition model for layout analysis, and title, paragraph, drawing, table, and other graphic and text content modules are extracted. The second standard examination item is identified according to the graphic and text content modules, the second standard examination item is associated with the corresponding graphic and text content in the graphic and text content modules using position relationship and semantic similarity, the second standard examination contents are obtained, the standard examination item number, name, text description, and specification requirements in the graphic and text content modules are extracted as second text additional information based on OCR technology, the second standard examination item, the second standard examination contents, and the corresponding second text additional information are structured to obtain a standard examination file with uniform format.

[0104] In summary, the final standard examination file contains standard examination items, corresponding graphic and text information / graphic and text content modules (i.e., standard examination contents), and text additional information, and the file format is json.

[0105] Further, in step S2, an initial recognition model and a matching model obtained by pre-training are used to extract a multi-scale candidate region image to be examined from the to-be-detected image, including:

[0106] S21, inputting the to-be-detected image into the initial recognition model for layout analysis to extract a drawing area and a text area as a coarse positioning area;

[0107] S22, using OCR to recognize the text in the coarse positioning area to obtain a text recognition result;

[0108] S23, extracting a standard examination item in the standard examination file;

[0109] S24, input the text recognition result, the standard review item and the coarse positioning region to the matching model, the matching model performs at least one operation of keyword matching, regular matching, semantic similarity matching on the text recognition result and the standard review item, determines the region in the coarse positioning region matched with the standard review item, and performs multi-scale feature extraction on the matched region by using a sliding window to obtain the multi-scale candidate region image.

[0110] Specifically, in step S21, the initial recognition model can be a neural network model, which performs layout analysis on the input to-be-detected image, extracts text regions such as titles and paragraphs, and paper regions as coarse positioning regions.

[0111] Further, in step S23, the matching model performs keyword matching on the text recognition result and the standard review item, judges whether the text recognition result contains a keyword related to the review item, if the keyword is contained, positions the position of the keyword in the coarse positioning region, and extracts the region at the position by using a sliding window to obtain the multi-scale candidate region.

[0112] Further, the matching model can also perform regular matching on the text recognition result and the standard review item, define a related regular expression, judge whether the text recognition result matches the standard review item, if matched, position the matched region according to the position of the corresponding text in the coarse positioning region, and perform multi-scale feature extraction by using a sliding window to obtain the multi-scale candidate region image.

[0113] Further, the matching model can also perform semantic similarity matching on the text recognition result and the standard review item, calculate the semantic similarity degree between the two, so as to judge whether the text recognition result matches the review item, if matched, position the matched region according to the position of the corresponding text in the coarse positioning region, and perform multi-scale feature extraction by using a sliding window to obtain the multi-scale candidate region image.

[0114] Specifically, different window sizes, sliding steps and overlap ratios can be set to obtain the multi-scale candidate region image.

[0115] In some embodiments, one, two or three operations of keyword matching, regular matching and semantic similarity matching can be performed on the text recognition result and the standard review item, and the results obtained by the two or three operations can be integrated to obtain the multi-scale candidate region image.

[0116] The obtained multi-scale candidate region is a relevant region in the to-be-detected file corresponding to the standard review item.

[0117] Further, referenceFigure 2 and Figure 3 In step S3, the lightweight hybrid twin network comprises a backbone network module 1 and a twin network module 2; the backbone network module 1 comprises an input layer 11, a plurality of stacked feature extraction modules 12 and an output layer 13; the twin network module 2 comprises a feature projection module 21, an MLP similarity calculation module 22 and an output module 23; the feature extraction module 12 comprises a lightweight convolutional layer 121, a Transformer layers module 122 and a feature fusion module 123;

[0118] In the detection process, the multi-scale candidate region image and the standard review file are input into the lightweight hybrid twin network obtained by pre-training for image feature matching, and a matching feature image is output, comprising:

[0119] The multi-scale candidate region image and the standard review file are sequentially subjected to feature extraction, global modeling and feature fusion by the feature extraction module, the Transformer layers module and the feature fusion module, and output fusion multi-scale candidate region features and fusion standard features;

[0120] The feature projection module maps the fusion multi-scale candidate region features and the fusion standard features into the same space, and then the similarity calculation module is used for similarity calculation to obtain the matching probability of the fusion multi-scale candidate region features and the fusion standard features; the output module filters the fusion multi-scale candidate region features according to the matching probability, and outputs a matching feature image.

[0121] Specifically, the input layer 11 receives the multi-scale candidate region image and the standard review file and sends to a plurality of feature extraction modules, and specifically can also send the image content in the multi-scale candidate region image and the standard review file to the plurality of feature extraction modules. The lightweight convolutional layer 121 in the feature extraction module 12 uses standard convolution to respectively extract features of the multi-scale candidate region image and the standard review file, specifically extracts features based on the image content in the standard review file, obtains multi-scale candidate region features and standard features, the Transformer layers module 122 is used for respectively modeling the global multi-scale candidate region features and the standard features, and obtains global multi-scale candidate region features and global standard features at different levels. The feature fusion module 123 is used for feature fusion of the global multi-scale candidate region features and the global standard features at different levels, and outputs the fused multi-scale candidate region features and the fused standard features through the output layer 13. The feature projection module 21 maps the fused multi-scale candidate region features and the fused standard features to the same space, the MLP similarity calculation module 22 is used for similarity calculation of the fused multi-scale candidate region features and the fused standard features in the same space, and obtains a matching probability of the fused multi-scale candidate region features and the fused standard features. The output module 23 is used for screening the fused multi-scale candidate region features according to the matching probability and a preset probability threshold, and outputs the fused multi-scale candidate region features with a matching probability exceeding the preset probability threshold as a matching feature image.

[0122] Specifically, the output matching feature image is one or more than one region image in the multi-scale candidate region image that is most similar to the standard image in the standard review file.

[0123] Specifically, the backbone network module 1 is mainly used for extracting complete and high-quality features to provide a basis for subsequent similarity calculation. Specifically, the input layer 11 receives the multi-scale candidate region image and the standard review file and sends to a plurality of feature extraction modules 12. The lightweight convolutional layer 121 in the feature extraction module uses standard convolution to extract image features, which is suitable for extracting details such as structural lines, frames, symbols, etc. in the image. The Transformer layers module 122 is used for respectively modeling the global multi-scale candidate region features and the standard features, which can understand the structure of the image features as a whole, such as the spatial and semantic relationship between multiple parts in the symbol, and obtains global multi-scale candidate region features and global standard features at different levels. The feature fusion module 123 is used for feature fusion of the global multi-scale candidate region features and the global standard features at different levels, and integrates the fused multi-scale candidate region features and the fused standard features into a fixed-dimensional vector through the output layer 13 and outputs. The plurality of stacked feature extraction modules 12 are helpful for simultaneously recognizing large blocks and small blocks.

[0124] Further, the feature projection module 21 maps the fused multi-scale candidate region features and the fused standard features into the same space, so that the features with similar semantics are closer in distance, for subsequent similarity calculation. The MLP similarity calculation module 22 is used to learn a nonlinear mapping relationship (mapping from a feature pair to a matching probability) that matches, adapts to the nonlinear relationship between complex patches, thereby realizing similarity calculation on the fused multi-scale candidate region features and the fused standard features in the same space, obtaining the matching probability of the fused multi-scale candidate region features and the fused standard features, generating a binary classification probability, and the output module 23 is used to screen the fused multi-scale candidate region features according to the matching probability and a preset probability threshold, and outputs the screened fused multi-scale candidate region features as matching feature images.

[0125] In summary, the lightweight hybrid twin network is constructed based on the backbone network module of the Transformer layer and the twin network module of the learnable similarity measurement mechanism. The lightweight hybrid twin network introduces the Transformer mechanism as the feature extraction backbone. Compared with traditional pure CNN models such as ResNet and MobileNet, the backbone network module in the embodiment adopts a hybrid structure of convolution and Transformer, which takes into account both local details and global relationships, and the built-in lightweight self-attention mechanism is more suitable for modeling the structure of technical drawings and symbols, with low parameter quantity and fast inference speed, which is suitable for industrial deployment.

[0126] Traditional twin networks use static Euclidean distance or cosine similarity, which performs poorly in cases of scale differences, style changes, and printing errors. To break through this limitation, a learnable similarity measurement subnetwork is designed in the embodiment and applied in the MLP similarity calculation module to more flexibly express the semantic relationship between patches. The similarity measurement subnetwork can be represented as:

[0127] (1)

[0128] wherein, f 1 and f 2 represent two feature vectors output by the backbone network module, Sigmoid represents the Sigmoid activation function, MLP represents a lightweight multilayer perceptron composed of two fully connected layers and a GELU activation function, and Sim( f 1, f 2) represents the similarity calculation result. This structure allows the model to automatically learn the nonlinear similarity relationship between images during training, significantly enhancing the robustness to factors such as scale differences, style changes, and printing errors.

[0129] In the inference stage, the structured standard review file is paired with the multi-scale candidate region image as input to the lightweight hybrid twin network, and the matching probability is output through feature extraction, MLP and Sigmoid classifier, and screening is performed based on the threshold. In some embodiments, non-maximum suppression and Top-K high similarity mechanism can also be used on the screening results of all scales to obtain the final matching feature image. All review items and corresponding candidate region images are integrated using a unified JSON data structure to form data that is convenient for model automatic processing and batch review.

[0130] Further, in some embodiments, the method further comprises: training the lightweight hybrid twin network according to pre-constructed training samples;

[0131] Wherein, the training sample is constructed, comprising:

[0132] Collecting industrial standard document data and converting it into image data, and performing image-text extraction and cropping based on the image data to obtain standard sample patches;

[0133] Using a sliding window to extract candidate patches from the sample to-be-reviewed file;

[0134] Selecting the candidate patches corresponding to the standard sample patches as positive samples, and grouping the positive samples with the corresponding standard sample patches to form positive sample pairs;

[0135] Selecting candidate patches that do not correspond to the standard sample patches as negative samples, and grouping the negative samples with the standard sample patches to form negative sample pairs.

[0136] Collecting industrial standard document data, including design drawings, product manuals, assembly documents, etc. to construct a standard industrial document review dataset. The samples are organized in JSON format and can be used for training of lightweight hybrid twin networks and multi-modal large models. The dataset covers multi-scale, multi-category, and multi-angle patch samples, and provides positive and negative sample annotations, OCR content, and patch position markers, with high industrial generality and generalization. The positive samples are standard image matching regions, and the negative samples are random mixed images.

[0137] In some embodiments, the positive sample pairs and the negative sample pairs can be integrated into a dataset, and the lightweight hybrid twin network is trained based on the dataset.

[0138] Further, the training of the lightweight hybrid twin network comprises:

[0139] Repeat the following steps until the stop condition is met:

[0140] Input the positive sample pairs and the negative sample pairs to the backbone network module for feature extraction, and output first sample feature vectors and second sample feature vectors;

[0141] inputting the first sample feature vector and the second sample feature vector into the twin network module to perform similarity calculation and threshold judgment and outputting a judgment result, and finding a negative sample most similar to the positive sample pair as a third sample feature vector in a training process;

[0142] calculating a cosine loss according to the first sample feature vector and the second sample feature vector, calculating a binary cross entropy loss according to an output result of the twin network module, and calculating a triplet loss according to the first sample feature vector, the second sample feature vector and the third sample feature vector;

[0143] obtaining a comprehensive loss according to the cosine loss, the binary cross entropy loss and the triplet loss, and updating parameters of the lightweight hybrid twin network according to the comprehensive loss through back propagation.

[0144] Specifically, the loss function design part of the lightweight hybrid twin network adopts a multiple loss function combination mechanism, which can improve the discrimination ability of the model to the positive and negative sample pairs and the embedding space structure expression ability. It can be expressed as:

[0145] ; (2)

[0146] wherein, Loss represents a comprehensive loss, L TRIP represents a triplet loss, L COSINE represents a cosine loss, and L BCE represents a binary cross entropy loss, λ1, λ2 and λ3 represent weights of the triplet loss, the cosine loss and the binary cross entropy loss respectively, and each part is defined as follows:

[0147] Triplet loss (Triplet Loss):

[0148] ; (3)

[0149] wherein, f 1, f 2, f 3 represent the first sample feature vector, the second sample feature vector and the third sample feature vector respectively.

[0150] Cosine similarity loss (Cosine Loss):

[0151] ; (4)

[0152] wherein, respectively represent Euclidean norms of the first sample vector and the second sample vector.

[0153] Binary cross entropy loss (BCE):

[0154] (5)

[0155] wherein, y t represents the true label, y p represents the output result of the twin network module, used to supervise the consistency of the similarity output and the true label.

[0156] Further, with reference to Figure 4 , the multi-modal large model comprises a dynamic Prompt module 3 and a Qwen2.5-VL model 4.

[0157] The dynamic Prompt module is used to construct a prompt word according to the candidate region text information and the standard review file; specifically, the candidate region text information and the standard review file are constructed into a standardized graphic-text data format, and input into the dynamic Prompt module for construction of the prompt word.

[0158] The Qwen2.5-VL model is used to analyze based on the prompt word and the matching feature image; in some embodiments, the prompt word and the matching feature image can be constructed into a standardized graphic-text data format and input into the Qwen2.5-VL model, and the detection result of whether the review content corresponding to the corresponding review item in the file to be detected is compliant is output.

[0159] Specifically, the dynamic Prompt module is constructed for precise control and low hallucination in multi-modal drawing review. It is a candidate region driven dynamic Prompt construction mechanism, which constructs a dynamic Prompt by extracting matching feature image pairing data, dynamically constructs a minimum context prompt word, realizes a fine control instruction structure, and enhances the model instruction understanding and review targeting. The specific way is to automatically call templates to generate standard item content and set different template subclasses for different types of items (graphics / text / labels), so that the model follows specific instructions to enhance review accuracy, significantly shortens the token length, avoids historical interference and model hallucination.

[0160] The final detection result outputs each review task in the format of serial number, review item, candidate region information, review result, confidence and judgment reason structure, and all results are aggregated at the task level to generate a task overview and detailed explanation. A standard review report generation mechanism can also be built to complete automatic arrangement of report content, support multiple formats (JSON, PDF, Excel) export, and meet the quality system record requirements.

[0161] In addition, the image visualization result can be superimposed, and all review candidate regions in the original file are highlighted, with green boxes for compliant regions and red boxes for non-compliant regions. The problem regions are sorted according to the inference reliability, and the region color is rendered according to the risk level (red-orange-yellow-green grading annotation), guiding artificial review of error-prone regions in priority.

[0162] Further, the multi-modal large model is pre-trained, including:

[0163] Collect industrial standard document data and convert it into image data, perform image-text extraction based on the image data to obtain sample image-text data, integrate and process the sample image-text data with the corresponding review standards to obtain a large model sample.

[0164] Obtain a pre-trained Qwen2.5-VL model and inject LoRA parameters, and freeze the main parameters of the pre-trained Qwen2.5-VL model. The pre-trained Qwen2.5-VL model is from the official.

[0165] Repeat the following steps until the stopping condition is met:

[0166] Input the large model sample into the pre-trained Qwen2.5-VL model, and obtain the output result of the pre-trained Qwen2.5-VL model, add artificial feedback data to the output result of the pre-trained Qwen2.5-VL model to form a supervised sample, and construct a reward model.

[0167] Input the supervised sample into the reward model for training, and train the pre-trained Qwen2.5-VL model based on the output result of the reward model, and fine-tune the LoRA parameters of the pre-trained Qwen2.5-VL model. The output result of the reward model is used as a reinforcement learning signal to fine-tune the LoRA parameters of the pre-trained Qwen2.5-VL model.

[0168] The multi-modal review capability enhancement training method based on LoRA and RLHF realizes a substantial improvement in the structural review generalization capability, accuracy and review consistency of the model in the industrial drawing scene by introducing LoRA and RLHF based on the pre-trained multi-modal large model (Qwen2.5-VL).

[0169] The addition of LoRA allows fine-tuning of the image-text alignment and review reasoning capability with only a small amount of parameter training (about 1% of the original parameters). The core idea is to freeze the main parameters of the pre-trained Qwen2.5-VL model, and only insert trainable weight pairs A, B of rank r in the Query and Value weight matrices in the multi-head attention mechanism to realize low-resource fine-tuning of the model in the industrial drawing review task, i.e.

[0170] (6)

[0171] where W represents the weight matrix of the pre-trained Qwen2.5-VL model, AB is the product of a pair of low-rank matrices, and W' is the adjusted weight matrix. The training data is in the chat_template format, and the fine-tuned model can automatically reason according to the drawing and standard specification problems, with stronger non-compliance detection and review standard understanding capabilities.

[0172] The RLHF (Reinforcement Learning with Human Feedback) method is introduced, which learns the reward function through artificial preference and optimizes the model behavior based on the function to improve its review ability and consistency. It can include the following steps: obtaining the output result of the pre-trained Qwen2.5-VL model, adding artificial feedback data (including candidate reply ranking and compliance score) to the output result to form a supervised sample; constructing a reward model based on the Qwen2.5-VL model, and completing preference modeling through Pairwise Ranking Loss; using the Proximal Policy Optimization algorithm (PPO) to optimize and train the pre-trained Qwen2.5-VL model to maximize the human feedback preference score.

[0173] The Pairwise Ranking Loss is defined as follows:

[0174] (7)

[0175] where r1, r2 represent the reward model output, Lrank represents the loss of the reward model, and σ represents the Sigmoid function. When r1≥r2, the preference response score is higher, and the loss tends to 0. The goal is to let the reward model learn to distinguish "better" and "worse" model answers.

[0176] In the process of optimizing and training the pre-trained Qwen2.5-VL model using the Proximal Policy Optimization algorithm (PPO), the advantage function is calculated by comparing the ratio of the output probabilities of the current policy and the old policy, and the reward score. The objective function is as follows:

[0177] (8)

[0178] where π0(y|x) and π old (y|x) represent the generation probability of the current policy and the old policy for answer y, respectively, R(x,y) represents the output of the reward model, which is used to evaluate the quality of answer y, clip is a clipping function used to limit the policy update range, and b(x) is a baseline function, is a hyperparameter used to control the clipping range.

[0179] Reference Figure 5 The application provides a scheme detection device for a complex picture-text mixed file, comprising:

[0180] A standard extraction module 201 is configured to extract standard review items and standard review content in a standard manual and perform structural processing to obtain a standard review file.

[0181] A candidate region extraction module 202 is configured to convert a file to be detected into an image to be detected, extract a multi-scale candidate region image to be reviewed from the image to be detected by using OCR, an initial recognition model obtained by pre-training and a matching model, and extract text information of the multi-scale candidate image to obtain candidate region text information.

[0182] A matching module 203 is configured to input the multi-scale candidate region image and the standard review file into a lightweight hybrid twin network obtained by pre-training to perform image feature matching and output a matching feature image.

[0183] A detection module 204 is configured to input the matching feature image, the standard review file and the candidate region text information into a multi-modal large model obtained by pre-training to perform compliance analysis and output a detection result.

[0184] Further, the standard manual comprises structured data and unstructured data.

[0185] The standard extraction module 201 extracts standard review items and standard review content in a standard manual and performs structural processing to obtain a standard review file, comprising:

[0186] For the structured data, a first standard review item is identified based on the standard manual, and picture-text information about the first standard review item is extracted as first standard review content row by row and item by item. For each row of extracted picture-text information, text information is extracted as first text additional information by using OCR. The first standard review item, the first standard review content and the corresponding first text additional information are subjected to structural processing to obtain the standard review file.

[0187] For the unstructured data, the standard manual is converted into a picture, a recognition model obtained by pre-training is used to perform layout analysis on the picture, a picture-text content module is extracted, a second standard review item is identified according to the picture-text content module, and a region in the picture-text content module corresponding to the second standard review item is identified as second standard review content. Text information in the picture-text content module is extracted as second text additional information by using OCR. The second standard review item, the second standard review content and the corresponding second text additional information are subjected to structural processing to obtain the standard review file.

[0188] Further, the candidate region extraction module 202 extracts the multi-scale candidate region image to be examined from the to-be-detected image by using OCR, an initial recognition model obtained by pre-training, and a matching model, including:

[0189] inputting the to-be-detected image into the initial recognition model for layout analysis, and extracting a graph region and a text region as a coarse positioning region;

[0190] performing text recognition on the coarse positioning region by using OCR to obtain a text recognition result;

[0191] extracting a standard examination item in the standard examination file;

[0192] inputting the text recognition result, the standard examination item, and the coarse positioning region into the matching model, and performing at least one operation of keyword matching, regular matching, and semantic similarity matching on the text recognition result and the standard examination item by using the matching model to determine a region in the coarse positioning region that matches the standard examination item, and performing multi-scale feature extraction on the matched region by using a sliding window to obtain the multi-scale candidate region image.

[0193] Further, the lightweight hybrid twin network includes a backbone network module and a twin network module; the backbone network module includes an input layer, a plurality of stacked feature extraction modules, and an output layer; the twin network module includes a feature projection module, an MLP similarity calculation module, and an output module; and the feature extraction module includes a lightweight convolutional layer, a Transformer layers module, and a feature fusion module.

[0194] Further, the matching module 203 inputs the multi-scale candidate region image and the standard examination file into a lightweight hybrid twin network obtained by pre-training to perform image feature matching, and outputs a matching feature image, including:

[0195] the multi-scale candidate region image and the standard examination file are sequentially subjected to feature extraction, global modeling, and feature fusion by using the feature extraction module, the Transformer layers module, and the feature fusion module, and output fusion multi-scale candidate region features and fusion standard features;

[0196] the feature projection module maps the fusion multi-scale candidate region features and the fusion standard features to the same space, and then performs similarity calculation by using the MLP similarity calculation module to obtain a matching probability of the fusion multi-scale candidate region features and the fusion standard features; and the output module filters the fusion multi-scale candidate region features according to the matching probability, and outputs the matching feature image.

[0197] Further, the matching module 203 is further used for:

[0198] controlling the input layer to receive the multi-scale candidate region image and the standard review file and send to a plurality of feature extraction modules, the light convolutional layer in the feature extraction module adopts standard convolution to respectively perform feature extraction on the multi-scale candidate region image and the standard review file, to obtain multi-scale candidate region features and standard features, the Transformer layers module is used for respectively performing global modeling on the multi-scale candidate region features and the standard features, to obtain global multi-scale candidate region features and global standard features at different levels, the feature fusion module is used for performing feature fusion on the global multi-scale candidate region features and the global standard features at different levels, and outputting fused multi-scale candidate region features and fused standard features through an output layer; the feature projection module is used for mapping the fused multi-scale candidate region features and the fused standard features into the same space, the MLP similarity calculation module is used for performing similarity calculation on the fused multi-scale candidate region features and the fused standard features in the same space, to obtain a matching probability of the fused multi-scale candidate region features and the fused standard features, and the output module is used for screening the fused multi-scale candidate region features according to the matching probability and a preset probability threshold, and outputting the fused multi-scale candidate region features with a matching probability exceeding the preset probability threshold as matching feature images.

[0199] Further, the device further comprises a first training module configured to train the light hybrid twin network according to pre-constructed training samples.

[0200] The first training module is configured to construct the training samples.

[0201] The industrial standard document data is collected and converted into image data, and image-text extraction and cropping are performed based on the image data to obtain standard sample blocks.

[0202] The candidate blocks are extracted from the sample review files by using a sliding window.

[0203] The candidate blocks corresponding to the standard sample blocks are selected as positive samples, and the positive samples and the corresponding standard sample blocks are combined to form a positive sample pair.

[0204] The candidate blocks not corresponding to the standard sample blocks are selected as negative samples, and the negative samples and the standard sample blocks are combined to form a negative sample pair.

[0205] Further, the first training module trains the light hybrid twin network, including:

[0206] The following steps are repeatedly executed until a stop condition is met:

[0207] inputting the positive sample pair and the negative sample pair into the backbone network module to perform feature extraction, and outputting a first sample feature vector and a second sample feature vector;

[0208] inputting the first sample feature vector and the second sample feature vector into the twin network module to perform similarity calculation and threshold judgment and outputting a judgment result, and finding, in a training process, a negative sample most similar to the positive sample pair as a third sample feature vector;

[0209] calculating a cosine loss according to the first sample feature vector and the second sample feature vector, calculating a binary cross entropy loss according to an output result of the twin network module, and calculating a triplet loss according to the first sample feature vector, the second sample feature vector and the third sample feature vector;

[0210] obtaining a comprehensive loss according to the cosine loss, the binary cross entropy loss and the triplet loss, and updating parameters of the lightweight hybrid twin network according to the comprehensive loss through back propagation.

[0211] Further, the multi-modal large model comprises a dynamic Prompt module and a Qwen2.5-VL model, the dynamic Prompt module is configured to construct a prompt word according to the candidate region text information and the standard review file;

[0212] The Qwen2.5-VL model is configured to analyze based on the prompt word and the matching feature image, and output a detection result of whether the review content corresponding to the corresponding review item in the file to be detected is compliant.

[0213] Further, the device further comprises a second training module configured to pre-train the multi-modal large model, comprising:

[0214] Collecting industrial standard document data to convert into image data, performing image-text extraction based on the image data to obtain sample image-text data, integrating and processing the sample image-text data with corresponding review standards to obtain a large model sample;

[0215] Obtaining a pre-trained Qwen2.5-VL model and injecting LoRA parameters, and freezing main parameters of the pre-trained Qwen2.5-VL model;

[0216] repeating the following steps until a stop condition is met:

[0217] inputting the large model sample into the pre-trained Qwen2.5-VL model, and obtaining an output result of the pre-trained Qwen2.5-VL model, adding artificial feedback data to the output result of the pre-trained Qwen2.5-VL model to form a supervised sample, and constructing a reward model;

[0218] The supervision sample is input into the reward model for training, and the LoRA parameters of the pre-trained Qwen2.5-VL model are fine-tuned based on the output result of the reward model.

[0219] The scheme detection method and device for complex graphic-text mixed files provided by the above embodiments at least have the following beneficial effects:

[0220] (1) The multi-modal large model (Qwen2.5-VL) is introduced to realize complex graphic-text semantic understanding, which can automatically identify, match and understand the elements such as icons, symbols, texts and structures in the scheme, improve the accuracy of detection and reduce the risk of missed detection, and significantly improve the automation and intelligence of the examination of complex graphic-text mixed files;

[0221] (2) A lightweight hybrid twin network is constructed to significantly enhance the block feature representation capability by fusing global and local features, a dynamic similarity measurement module composed of a lightweight MLP and a GELU activation function is designed to enhance the robustness of the model to factors such as scale difference, style change and printing error, and finally a multiple loss function combination mechanism is used to optimize the model weight, further improving the feature representation capability of the model to the scheme, adapting to the multi-scale, multi-angle and multi-style variants of the elements in the scheme;

[0222] (3) The OCR recognition information, image features and examination item information are fused to improve the semantic accuracy, a dynamic Prompt construction mechanism is used to constrain the model to generate content, control the context length, avoid multi-task interference, and can be flexibly adapted to different file types and quickly integrated into an industrial platform;

[0223] (4) By uploading the examination standards in Excel / Word format, the examination template is dynamically generated, the document can be flexibly replaced to adapt to different custom standards, the industrial drawing dataset can be automatically built by building tools, and the model is fine-tuned by LoRA and RLHF technology, the model knowledge system is continuously improved, and the understanding ability of enterprise standards is continuously enhanced;

[0224] (5) The explainability and traceability of the file examination task are enhanced, the manual correction workload and examination period are greatly reduced, the overall process is automatically executed, the complete file scheme can be examined and labeled in an average of minutes, the examination report is automatically generated, the examination result includes structured output, model judgment reason, credibility score and intuitive visualization on the drawing, which assists manual confirmation, greatly reduces manual cost, and improves the efficiency of large engineering project drawing examination.

[0225] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the foregoing description without departing from the spirit and scope of the application. Accordingly, it is intended that the appended claims be interpreted as including all such variations and modifications as fall within the spirit and scope of the application. It is further intended that the disclosure of all such modifications and variations be included within the scope of the application, the terms used herein being defined solely for purposes of the description being applied thereto unless otherwise indicated.

Claims

1. A method for detecting a scheme of a complex composite document, characterized in that, The method comprises the following steps: extracting and structuring the standard review items and the standard review content in the standard manual to obtain a standard review file; converting the file to be detected into an image to be detected, and extracting a multi-scale candidate region image to be reviewed from the image to be detected by using OCR, an initial recognition model obtained by pre-training, and a matching model: inputting the image to be detected into the initial recognition model to perform layout analysis, and extracting a graph region and a text region as a coarse positioning region; performing text recognition on the coarse positioning region by using OCR to obtain a text recognition result; extracting the standard review items in the standard review file; inputting the text recognition result, the standard review items, and the coarse positioning region into the matching model, and performing at least one operation of keyword matching, regular matching, and semantic similarity matching on the text recognition result and the standard review items by using the matching model to determine the region in the coarse positioning region that matches the standard review items, and performing multi-scale feature extraction on the matched region by using a sliding window to obtain the multi-scale candidate region image; and extracting the text information of the multi-scale candidate image to obtain candidate region text information; inputting the multi-scale candidate region image and the standard review file into a lightweight hybrid twin network obtained by pre-training to perform image feature matching, and outputting a matching feature image; inputting the matching feature image, the standard review file, and the candidate region text information into a multi-modal large model obtained by pre-training to perform compliance analysis, and outputting a detection result.

2. The method of claim 1, wherein, The standard manual comprises structured data and unstructured data; extracting and structuring the standard review items and the standard review content in the standard manual to obtain a standard review file, comprising: for the structured data, identifying a first standard review item based on the standard manual, and extracting graphic and text information about the first standard review item as first standard review content line by line and item by item; for each line of extracted graphic and text information, extracting text information as first text additional information by using OCR; structuring the first standard review item, the first standard review content, and the corresponding first text additional information to obtain the standard review file; for the unstructured data, converting the standard manual into a picture, performing layout analysis on the picture by using a recognition model obtained by pre-training, extracting a graphic and text content module, identifying a second standard review item based on the graphic and text content module, and identifying a region in the graphic and text content module corresponding to the second standard review item as second standard review content; extracting text information in the graphic and text content module as second text additional information by using OCR; and structuring the second standard review item, the second standard review content, and the corresponding second text additional information to obtain the standard review file.

3. The method of claim 1, wherein, The lightweight hybrid twin network comprises a backbone network module and a twin network module; the backbone network module comprises an input layer, a plurality of stacked feature extraction modules, and an output layer; the twin network module comprises a feature projection module, an MLP similarity calculation module, and an output module; the feature extraction module comprises a lightweight convolutional layer, a Transformer layers module, and a feature fusion module.

4. The method of claim 3, wherein, The multi-scale candidate region image and the standard review file are input into a lightweight hybrid twin network obtained by pre-training for image feature matching, and a matching feature image is output, comprising: The input layer receives the multi-scale candidate region image and the standard review file and sends them to a plurality of feature extraction modules. The lightweight convolutional layer in the feature extraction module extracts features from the multi-scale candidate region image and the standard review file respectively using standard convolution to obtain multi-scale candidate region features and standard features. The Transformer layers module is used to model the global multi-scale candidate region features and the global standard features at different levels respectively. The feature fusion module is used to fuse the global multi-scale candidate region features and the global standard features at different levels, and outputs the fused multi-scale candidate region features and the fused standard features through the output layer. The feature projection module is used to map the fused multi-scale candidate region features and the fused standard features to the same space. The MLP similarity calculation module is used to calculate the similarity of the fused multi-scale candidate region features and the fused standard features in the same space to obtain the matching probability of the fused multi-scale candidate region features and the fused standard features. The output module is used to filter the fused multi-scale candidate region features according to the matching probability and a preset probability threshold, and outputs the fused multi-scale candidate region features with a matching probability exceeding the preset probability threshold as the matching feature image.

5. The method of claim 3, wherein, The method further comprises training the lightweight hybrid twin network according to pre-constructed training samples; wherein the training samples are constructed, comprising: Collecting industrial standard document data and converting it into image data, performing image-text extraction and cropping based on the image data to obtain standard sample patches; Extracting candidate patches from sample review files using a sliding window; Selecting candidate patches corresponding to the standard sample patches as positive samples, and grouping the positive samples and the corresponding standard sample patches into positive sample pairs; Selecting candidate patches not corresponding to the standard sample patches as negative samples, and grouping the negative samples and the standard sample patches into negative sample pairs.

6. The method of claim 5, wherein, Training the lightweight hybrid twin network, comprising: repeating the following steps until the stop condition is met: inputting the positive sample pairs and the negative sample pairs into the backbone network module for feature extraction to output first sample feature vectors and second sample feature vectors; inputting the first sample feature vector and the second sample feature vector into the twin network module to perform similarity calculation and threshold judgment and outputting a judgment result, and finding a negative sample most similar to the positive sample pair as a third sample feature vector in a training process; calculating a cosine loss according to the first sample feature vector and the second sample feature vector, calculating a binary cross entropy loss according to an output result of the twin network module, and calculating a triplet loss according to the first sample feature vector, the second sample feature vector and the third sample feature vector; obtaining a comprehensive loss according to the cosine loss, the binary cross entropy loss and the triplet loss, and updating parameters of the lightweight hybrid twin network according to the comprehensive loss through back propagation.

7. The method of claim 5, wherein, The multi-modal large model comprises a dynamic Prompt module and a Qwen2.5-VL model, the dynamic Prompt module is configured to construct a prompt word according to the candidate region text information and the standard review file; The Qwen2.5-VL model is configured to analyze based on the prompt word and the matching feature image, and output a detection result of whether the review content corresponding to the corresponding review item in the file to be detected is compliant.

8. The method of claim 7, wherein, The multi-modal large model is pre-trained, comprising: industrial standard document data is collected and converted into image data, image-text extraction is performed based on the image data to obtain sample image-text data, the sample image-text data is integrated and processed with the corresponding review standard to obtain a large model sample; a pre-trained Qwen2.5-VL model is obtained and LoRA parameters are injected, and the main parameters of the pre-trained Qwen2.5-VL model are frozen; the following steps are repeatedly executed until a stop condition is met: the large model sample is input into the pre-trained Qwen2.5-VL model, and an output result of the pre-trained Qwen2.5-VL model is obtained, artificial feedback data is added to the output result of the pre-trained Qwen2.5-VL model to form a supervised sample, and a reward model is constructed; the supervised sample is input into the reward model for training, and the LoRA parameters of the pre-trained Qwen2.5-VL model are fine-tuned based on the output result of the reward model.

9. A scheme detection device for complex mixed text and image documents, characterized in that, comprising: a standard extraction module configured to extract standard review items and standard review content in a standard manual and perform structured processing to obtain a standard review file; a candidate region extraction module configured to convert a file to be detected into a to-be-detected image, extract a multi-scale candidate region image to be reviewed from the to-be-detected image by using OCR, an initial recognition model obtained through pre-training and a matching model, input the to-be-detected image into the initial recognition model for layout analysis, and extract a drawing region and a text region as a coarse positioning region; The OCR is used to perform character recognition on the coarse positioning area to obtain a character recognition result; a standard examination item in the standard examination file is extracted; the character recognition result, the standard examination item, and the coarse positioning area are input into the matching model, the matching model performs at least one operation of keyword matching, regular matching, and semantic similarity matching on the character recognition result and the standard examination item, determines an area in the coarse positioning area that matches the standard examination item, and performs multi-scale feature extraction on the matched area using a sliding window to obtain a multi-scale candidate area image; and extracts character information of the multi-scale candidate image to obtain candidate area character information; The matching module is configured to input the multi-scale candidate area image and the standard examination file into a lightweight hybrid twin network obtained through pre-training to perform image feature matching and output a matching feature image. The detection module is configured to input the matching feature image, the standard examination file, and the candidate area character information into a multi-modal large model obtained through pre-training to perform compliance analysis and output a detection result.

Images