Scheme detection method and device for complex image-text mixed file

Abstract

The invention provides a scheme detection method and device for a complex image-text mixed file, and the method comprises the steps: extracting standard review items and standard review contents in a standard manual, and carrying out the structural processing, and obtaining a standard review file; converting a to-be-detected file into a to-be-detected image, extracting a to-be-examined multi-scale candidate region image from the to-be-detected image by using an initial recognition model and a matching model obtained by OCR and pre-training, and extracting character information of the multi-scale candidate image to obtain character information of the candidate region; inputting the multi-scale candidate region image and the standard review file into a lightweight hybrid twin network obtained by pre-training for image feature matching, and outputting a matched feature image; inputting the matched feature image, the standard review file and the candidate area text information into a multi-modal large model obtained by pre-training for compliance analysis, and outputting a detection result; according to the method, the automation and intelligence degree of review of the complex image-text mixed file can be remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing technology, and in particular to a method and apparatus for detecting complex mixed text and graphics documents. Background Technology

[0002] In industries such as engineering, manufacturing, and power, icons in design drawings, product manuals, assembly documents, etc. (usually in PDF, CAD, etc.) have strict standards and specifications. Their standardization and consistency are of great importance to project safety and quality. To ensure the standardization and consistency of the above documents, it is often necessary to check the existence, location accuracy, image integrity, reasonable combination, and style consistency of each item in the document against the specifications in the manual.

[0003] Traditional design document review methods rely on manual page-by-page checking and comparison with standards, which is inefficient and highly subjective. This is especially true in areas such as icon matching, location judgment, and detail review, where omissions or misjudgments are common, making it difficult to meet the current high-standard, batch review requirements. To improve efficiency, some systems are attempting to utilize technologies such as OCR (Optical Character Recognition), image template matching, and CAD automatic review plugins to achieve intelligent drawing review. For example, patent text CN118334694A discloses a method for reviewing mechanical design drawings, including: performing character recognition on user-uploaded mechanical design drawings to obtain all character data, and converting each character data into text data to obtain all text data; converting each text data into standard text data according to predefined mechanical design drawing standards to obtain all standard text data; determining whether each standard text data is erroneous based on a pre-established mechanical design drawing judgment rule base; and generating an error message based on any standard text data when it is erroneous. This method is based on OCR and image template matching schemes. It lacks semantic understanding capabilities and cannot handle complex scenarios. When problems such as icon or text rotation, scaling, deformation, incompleteness, or unclear semantic requirements arise, there is a high risk of recognition failure.

[0004] Patent CN116563879A discloses a method and system for recognizing multi-line and / or multi-angle text in electrical drawings, relating to the field of image analysis. This method for recognizing multi-line and multi-angle text in electrical drawings includes: detecting the region containing multi-line text using the YOLOv5 object detection algorithm; preserving the coordinates of the recognition results and extracting the region containing the multi-line text; performing single-line text detection on the extracted multi-line text region using the YOLOv5 object detection algorithm; concatenating single-line text belonging to the same multi-line text region; rotating the detected single-line text image three times (90 degrees, 180 degrees, and 270 degrees) and saving the result; recognizing text in four directions using OCR and selecting the recognition result of the direction with the highest confidence as the output result; if the confidence is below a threshold, repeatedly detecting the region containing the multi-line text; if the confidence is above the threshold, using the result as the output. This method is based on the YOLO visual detection model, which can be trained to recognize fixed review categories, significantly improving positioning accuracy and stability. However, it suffers from high training costs and poor generalization. Each additional category requires retraining and it does not support semantic parsing or free language censorship. Summary of the Invention

[0005] This invention provides a method and apparatus for detecting complex mixed text and image documents, which can improve the accuracy of reviewing complex mixed text and image documents.

[0006] A scheme detection method for complex mixed text and image files includes: Extract the standard review entries and content from the standard manual and perform structured processing to obtain the standard review documents; The file to be detected is converted into an image to be detected. OCR, a pre-trained initial recognition model and a matching model are used to extract multi-scale candidate region images to be reviewed from the image to be detected, and the text information of the multi-scale candidate images is extracted to obtain candidate region text information. The multi-scale candidate region image and the standard review file are input into a pre-trained lightweight hybrid Siamese network for image feature matching, and the matching feature image is output. The matching feature image, the standard review document, and the candidate region text information are input into a pre-trained multimodal large model for compliance analysis, and the detection results are output.

[0007] Furthermore, the standard manual includes both structured and unstructured data; Extract the standard review entries and content from the standards manual and perform structured processing to obtain the standard review documents, including: For the structured data, the first standard review item is identified based on the standard manual, and the graphic and textual information about the first standard review item is extracted line by line as the first standard review content. For each line of graphic and textual information extracted, text information is extracted using OCR as the first text supplementary information. The first standard review item, the first standard review content and the corresponding first text supplementary information are structured to obtain the standard review document. For the unstructured data, the standard manual is converted into an image. A pre-trained recognition model is used to analyze the layout of the image and extract the text and image content modules. Based on the text and image content modules, the second standard review items are identified, and the areas in the text and image content modules that correspond to the second standard review items are identified as the second standard review content. OCR is used to extract the text information in the text and image content modules as the second text supplementary information. The second standard review items, the second standard review content, and the corresponding second text supplementary information are then structured to obtain the standard review document.

[0008] Furthermore, OCR, a pre-trained initial recognition model, and a matching model are used to extract multi-scale candidate region images for review from the image to be detected, including: The image to be detected is input into the initial recognition model for layout analysis, and the drawing area and text area are extracted as coarse positioning areas. OCR is used to perform text recognition on the coarsely located area to obtain the text recognition result; Extract the standard review entries from the aforementioned standard review documents; The text recognition result, standard review items, and coarse localization region are input into the matching model. The matching model performs at least one operation among keyword matching, regular expression matching, and semantic similarity matching on the text recognition result and the standard review items to determine the region in the coarse localization region that matches the standard review items. The matching region is then subjected to multi-scale feature extraction using a sliding window to obtain the multi-scale candidate region image.

[0009] Furthermore, the lightweight hybrid Siamese network includes a backbone network module and a Siamese network module; the backbone network module includes an input layer, multiple stacked feature extraction modules, and an output layer; the Siamese network module includes a feature projection module, an MLP similarity calculation module, and an output module; the feature extraction module includes lightweight convolutional layers, a Transformer layers module, and a feature fusion module.

[0010] Further, the multi-scale candidate region image and the standard review document are input into a pre-trained lightweight hybrid Siamese network for image feature matching, outputting a matched feature image, including: The multi-scale candidate region image and the standard review document are sequentially processed by the feature extraction module, the Transformer layers module, and the feature fusion module for feature extraction, global modeling, and feature fusion, outputting fused multi-scale candidate region features and fused standard features; The feature projection module maps the fused multi-scale candidate region features and the fused standard features into the same space. Then, the MLP similarity calculation module performs similarity calculation to obtain the matching probability of the fused multi-scale candidate region features and the fused standard features. The output module filters the fused multi-scale candidate region features according to the matching probability and outputs the matching feature image.

[0011] Further, the input layer receives the multi-scale candidate region image and the standard review document and sends them to multiple feature extraction modules. The lightweight convolutional layers in these feature extraction modules use standard convolution to extract features from the multi-scale candidate region image and the standard review document, respectively, to obtain multi-scale candidate region features and standard features. The Transformer... The layers module is used to globally model the multi-scale candidate region features and standard features respectively, to obtain global multi-scale candidate region features and global standard features at different levels. The feature fusion module is used to fuse the global multi-scale candidate region features and global standard features at different levels, and output the fused multi-scale candidate region features and fused standard features through the output layer. The feature projection module is used to map the fused multi-scale candidate region features and fused standard features to the same space. The MLP similarity calculation module is used to calculate the similarity between the fused multi-scale candidate region features and fused standard features in the same space, to obtain the matching probability of the fused multi-scale candidate region features and fused standard features. The output module is used to filter the fused multi-scale candidate region features according to the matching probability and a preset probability threshold, and output the fused multi-scale candidate region features with a matching probability exceeding the preset probability threshold as the matching feature image.

[0012] Furthermore, the method also includes: training the lightweight hybrid Siamese network based on pre-constructed training samples; The construction of training samples includes: Collect industry standard document data and convert it into image data. Based on the image data, perform image and text extraction and cropping to obtain standard sample image blocks. A sliding window is used to extract candidate patches from the sample documents to be reviewed; Select candidate patches corresponding to standard sample patches as positive samples, and form positive sample pairs with the corresponding standard sample patches; Candidate patches that do not correspond to standard sample patches are used as negative samples, and the negative samples are paired with standard sample patches to form negative sample pairs.

[0013] Further, the lightweight hybrid Siamese network is trained, including: Repeat the following steps until the stopping condition is met: The positive and negative sample pairs are input into the backbone network module for feature extraction, and the first sample feature vector and the second sample feature vector are output. The first sample feature vector and the second sample feature vector are input into the Siamese network module for similarity calculation and threshold judgment, and the judgment result is output. During the training process, the negative sample most similar to the positive sample pair is found as the third sample feature vector. The cosine loss is calculated based on the first sample feature vector and the second sample feature vector; the binary cross-entropy loss is calculated based on the output of the Siamese network module; and the triplet loss is calculated based on the first sample feature vector, the second sample feature vector, and the third sample feature vector. The combined loss is obtained based on the cosine loss, binary cross-entropy loss, and triplet loss. The parameters of the lightweight hybrid twin network are then updated by backpropagation based on the combined loss.

[0014] Furthermore, the multimodal large model includes a dynamic Prompt module and a Qwen2.5-VL model. The dynamic Prompt module is used to construct prompt words based on the text information of the candidate regions and the standard review document. The Qwen2.5-VL model is used to analyze the prompt words and the matching feature images, and output the detection results of whether the review content corresponding to the review items in the file to be detected is compliant.

[0015] Furthermore, the multimodal large model is pre-trained, including: Collect industry standard document data and convert it into image data. Based on the image data, extract the text and images to obtain sample text and images data. Integrate and process the sample text and images data with the corresponding review standards to obtain a large model sample. Obtain the pre-trained Qwen2.5-VL model and inject LoRA parameters, then freeze the main parameters of the pre-trained Qwen2.5-VL model; Repeat the following steps until the stopping condition is met: The large model samples are input into the pre-trained Qwen2.5-VL model, and the output results of the pre-trained Qwen2.5-VL model are obtained. Human feedback data is added to the output results of the pre-trained Qwen2.5-VL model to form supervised samples, and a reward model is constructed. The supervised samples are input into the reward model for training, and the LoRA parameters of the pre-trained Qwen2.5-VL model are fine-tuned based on the output of the reward model.

[0016] A scheme detection device for complex mixed text and image documents, comprising: The standard extraction module is used to extract standard review entries and standard review content from the standard manual and perform structured processing to obtain standard review documents; The candidate region extraction module is used to convert the file to be detected into an image to be detected, and to extract multi-scale candidate region images to be examined from the image to be detected using OCR, a pre-trained initial recognition model and a matching model, and to extract text information from the multi-scale candidate images to obtain candidate region text information. The matching module is used to input the multi-scale candidate region image and the standard review file into a pre-trained lightweight hybrid Siamese network to perform image feature matching and output a matching feature image. The detection module is used to input the matching feature image, the standard review document, and candidate region text information into a pre-trained multimodal large model for compliance analysis and output the detection results.

[0017] Furthermore, the standard manual includes both structured and unstructured data; The standard extraction module extracts the standard review entries and content from the standard manual and performs structured processing to obtain the standard review documents, including: For the structured data, the first standard review item is identified based on the standard manual, and the graphic and textual information about the first standard review item is extracted line by line as the first standard review content. For each line of graphic and textual information extracted, text information is extracted using OCR as the first text supplementary information. The first standard review item, the first standard review content and the corresponding first text supplementary information are structured to obtain the standard review document. For the unstructured data, the standard manual is converted into an image. A pre-trained recognition model is used to analyze the layout of the image and extract the text and image content modules. Based on the text and image content modules, the second standard review items are identified, and the areas in the text and image content modules that correspond to the second standard review items are identified as the second standard review content. OCR is used to extract the text information in the text and image content modules as the second text supplementary information. The second standard review items, the second standard review content, and the corresponding second text supplementary information are then structured to obtain the standard review document.

[0018] Furthermore, the candidate region extraction module uses OCR, a pre-trained initial recognition model, and a matching model to extract multi-scale candidate region images for review from the image to be detected, including: The image to be detected is input into the initial recognition model for layout analysis, and the drawing area and text area are extracted as coarse positioning areas. OCR is used to perform text recognition on the coarsely located area to obtain the text recognition result; Extract the standard review entries from the aforementioned standard review documents; The text recognition result, standard review items, and coarse localization region are input into the matching model. The matching model performs at least one operation among keyword matching, regular expression matching, and semantic similarity matching on the text recognition result and the standard review items to determine the region in the coarse localization region that matches the standard review items. The matching region is then subjected to multi-scale feature extraction using a sliding window to obtain the multi-scale candidate region image.

[0019] Furthermore, the lightweight hybrid Siamese network includes a backbone network module and a Siamese network module; the backbone network module includes an input layer, multiple stacked feature extraction modules, and an output layer; the Siamese network module includes a feature projection module, an MLP similarity calculation module, and an output module; the feature extraction module includes lightweight convolutional layers, a Transformer layers module, and a feature fusion module.

[0020] Further, the matching module inputs the multi-scale candidate region image and the standard review document into a pre-trained lightweight hybrid Siamese network for image feature matching, and outputs a matching feature image, including: The multi-scale candidate region image and the standard review document are sequentially processed by the feature extraction module, the Transformer layers module, and the feature fusion module for feature extraction, global modeling, and feature fusion, outputting fused multi-scale candidate region features and fused standard features; The feature projection module maps the fused multi-scale candidate region features and the fused standard features into the same space. Then, the MLP similarity calculation module performs similarity calculation to obtain the matching probability of the fused multi-scale candidate region features and the fused standard features. The output module filters the fused multi-scale candidate region features according to the matching probability and outputs the matching feature image.

[0021] Furthermore, the matching module is also used for: The input layer receives the multi-scale candidate region image and the standard review document and sends them to multiple feature extraction modules. The lightweight convolutional layers in these feature extraction modules use standard convolution to extract features from the multi-scale candidate region image and the standard review document, respectively, to obtain multi-scale candidate region features and standard features. The Transformer... The layers module is used to globally model the multi-scale candidate region features and standard features respectively, to obtain global multi-scale candidate region features and global standard features at different levels. The feature fusion module is used to fuse the global multi-scale candidate region features and global standard features at different levels, and output the fused multi-scale candidate region features and fused standard features through the output layer. The feature projection module is used to map the fused multi-scale candidate region features and fused standard features to the same space. The MLP similarity calculation module is used to calculate the similarity between the fused multi-scale candidate region features and fused standard features in the same space, to obtain the matching probability of the fused multi-scale candidate region features and fused standard features. The output module is used to filter the fused multi-scale candidate region features according to the matching probability and a preset probability threshold, and output the fused multi-scale candidate region features with a matching probability exceeding the preset probability threshold as the matching feature image.

[0022] Furthermore, the device also includes a first training module for training the lightweight hybrid Siamese network based on pre-constructed training samples; The construction of training samples includes: Collect industry standard document data and convert it into image data. Based on the image data, perform image and text extraction and cropping to obtain standard sample image blocks. A sliding window is used to extract candidate patches from the sample documents to be reviewed; Select candidate patches corresponding to standard sample patches as positive samples, and form positive sample pairs with the corresponding standard sample patches; Candidate patches that do not correspond to standard sample patches are used as negative samples, and the negative samples are paired with standard sample patches to form negative sample pairs.

[0023] Furthermore, the first training module trains the lightweight hybrid Siamese network, including: Repeat the following steps until the stopping condition is met: The positive and negative sample pairs are input into the backbone network module for feature extraction, and the first sample feature vector and the second sample feature vector are output. The first sample feature vector and the second sample feature vector are input into the Siamese network module for similarity calculation and threshold judgment, and the judgment result is output. During the training process, the negative sample most similar to the positive sample pair is found as the third sample feature vector. The cosine loss is calculated based on the first sample feature vector and the second sample feature vector; the binary cross-entropy loss is calculated based on the output of the Siamese network module; and the triplet loss is calculated based on the first sample feature vector, the second sample feature vector, and the third sample feature vector. The combined loss is obtained based on the cosine loss, binary cross-entropy loss, and triplet loss. The parameters of the lightweight hybrid twin network are then updated by backpropagation based on the combined loss.

[0024] Furthermore, the multimodal large model includes a dynamic Prompt module and a Qwen2.5-VL model. The dynamic Prompt module is used to construct prompt words based on the text information of the candidate regions and the standard review document. The Qwen2.5-VL model is used to analyze the prompt words and the matching feature images, and output the detection results of whether the review content corresponding to the review items in the file to be detected is compliant.

[0025] Furthermore, the device also includes a second training module for pre-training the multimodal large model, including: Collect industry standard document data and convert it into image data. Based on the image data, extract the text and images to obtain sample text and images data. Integrate and process the sample text and images data with the corresponding review standards to obtain a large model sample. Obtain the pre-trained Qwen2.5-VL model and inject LoRA parameters, then freeze the main parameters of the pre-trained Qwen2.5-VL model; Repeat the following steps until the stopping condition is met: The large model samples are input into the pre-trained Qwen2.5-VL model, and the output results of the pre-trained Qwen2.5-VL model are obtained. Human feedback data is added to the output results of the pre-trained Qwen2.5-VL model to form supervised samples, and a reward model is constructed. The supervised samples are input into the reward model for training, and the LoRA parameters of the pre-trained Qwen2.5-VL model are fine-tuned based on the output of the reward model.

[0026] The method and apparatus for detecting complex mixed text and image documents provided by this invention have at least the following beneficial effects: (1) Introducing a multimodal large model (Qwen2.5-VL) to achieve semantic understanding of complex graphics and text, it can automatically identify, match and understand elements such as icons, symbols, text and structures in the scheme, improve the accuracy of scheme detection and reduce the risk of missed detection, and significantly improve the automation and intelligence of review of complex graphics and text mixed documents; (2) A lightweight hybrid Siamese network is constructed to integrate global and local features to significantly enhance the ability to represent map features. A dynamic similarity measurement module composed of a lightweight MLP and GELU activation function is designed to enhance the robustness of the model to factors such as scale differences, style changes, and printing errors. Finally, a multi-loss function combination mechanism is used to optimize the model weights, which further improves the model's ability to represent the features of the scheme and adapts to the multi-scale, multi-angle, and multi-style variations of various elements in the scheme. (3) It integrates OCR recognition information, image features and review item information to improve semantic accuracy. It adopts a dynamic Prompt construction mechanism to constrain the content generated by the model, control the context length, avoid multi-task interference, and can flexibly adapt to different file types. It can be quickly integrated into industrial platforms. (4) By uploading the review standards in Excel / Word format, the review template can be dynamically generated. The document can be flexibly changed to adapt to different custom standards. It supports the automated construction of industrial drawing datasets through the construction tool, and combines LoRA and RLHF technologies to fine-tune the model, continuously improve the model knowledge system, and continuously enhance the understanding of enterprise standards. (5) Enhance the interpretability and traceability of document review tasks, significantly reduce the workload of manual proofreading and review cycle. The entire process is automated, and the review and annotation of the complete document plan can be completed in an average of a few minutes. The review report is automatically generated. The review results include structured output, model judgment reasons, credibility score and intuitive visualization on the drawings, which assists in manual confirmation, greatly reduces labor costs and improves the efficiency of drawing review for large-scale engineering projects. Attached Figure Description

[0027] Figure 1 This is a flowchart of one embodiment of the scheme detection method for complex mixed text and image files provided by the present invention.

[0028] Figure 2 This is a schematic diagram of the structure of a lightweight hybrid twin network in the scheme detection method for complex mixed text and image documents provided by the present invention.

[0029] Figure 3 This is a schematic diagram of the structure of a feature extraction module in an embodiment of the scheme detection method for complex mixed text and image files provided by the present invention.

[0030] Figure 4 This is a schematic diagram of the structure of one embodiment of the multimodal large model in the scheme detection method for complex mixed text and image files provided by the present invention.

[0031] Figure 5 This is a flowchart of one embodiment of the scheme detection device for complex mixed text and image documents provided by the present invention. Detailed Implementation

[0032] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0033] refer to Figure 1 In some embodiments, a scheme detection method for complex mixed text and image files is provided, including: S1. Extract the standard review items and standard review content from the standard manual and perform structured processing to obtain the standard review documents; S2. Convert the file to be detected into an image to be detected. Use OCR, a pre-trained initial recognition model and a matching model to extract multi-scale candidate region images to be reviewed from the image to be detected, and extract the text information of the multi-scale candidate images to obtain candidate region text information. S3. Input the multi-scale candidate region image and the standard review document into a pre-trained lightweight hybrid Siamese network to perform image feature matching and output the matching feature image. S4. Input the matching feature image, the standard review document, and the candidate region text information into the pre-trained multimodal large model for compliance analysis and output the detection results.

[0034] Specifically, in step S1, the standard manual may include structured and unstructured data, specifically standard review items and corresponding standard review content. Structured data includes, for example, Excel spreadsheets, while unstructured data includes, for example, Word and PDF documents. For structured data, the first standard review item is identified based on the standard manual, and the text and image information is extracted line by line as the first standard review content. For each extracted line of text and image information, OCR is used to extract text information as the first text supplementary information. Then, the first standard review item, the first standard review content, and the corresponding first text supplementary information are processed into a structured format to obtain a standard review document with a uniform format.

[0035] For unstructured data, the standard manual is converted into an image, which is then input into a pre-trained recognition model for layout analysis. Titles, paragraphs, drawings, tables, and other textual content modules are extracted. Second standard review items are identified based on these content modules. Positional relationships and semantic similarity methods are used to associate the second standard review items with the corresponding textual content in the content modules to obtain the second standard review content. Based on OCR technology, textual information such as the standard review item number, name, text description, and specification requirements are extracted from the content modules as second textual supplementary information. The second standard review items, second standard review content, and corresponding second textual supplementary information are then structured to obtain a standardized standard review document.

[0036] In summary, the final standard review document contains the standard review items, the corresponding graphic information / graphic content modules (i.e., the standard review content), and additional text information, and the file format is JSON.

[0037] Further, in step S2, OCR, a pre-trained initial recognition model, and a matching model are used to extract multi-scale candidate region images for review from the image to be detected, including: S21. Input the image to be detected into the initial recognition model to perform layout analysis, and extract the drawing area and text area as coarse positioning areas; S22. Use OCR to perform text recognition on the coarsely located area to obtain text recognition results; S23. Extract the standard review entries from the standard review document; S24. Input the text recognition result, standard review items, and coarse localization region into the matching model. The matching model performs at least one operation among keyword matching, regular expression matching, and semantic similarity matching on the text recognition result and the standard review items to determine the region in the coarse localization region that matches the standard review items. Then, it uses a sliding window to extract multi-scale features from the matched region to obtain the multi-scale candidate region image.

[0038] Specifically, in step S21, the initial recognition model can be a neural network model, which performs layout analysis on the input image to be detected, extracts text areas such as titles and paragraphs, as well as drawing areas as coarse positioning areas.

[0039] Further, in step S23, the matching model performs keyword matching on the text recognition result and the standard review item, and determines whether the text recognition result contains keywords related to the review item. If the keyword is contained, the location of the keyword in the coarse localization area is located, and the sliding window is used to extract the region at the location to obtain the multi-scale candidate region.

[0040] Furthermore, the matching model can also perform regular expression matching on the text recognition result and the standard review item, define relevant regular expressions, determine whether the text recognition result matches the standard review item, and if they match, locate the matching area according to the position of the corresponding text in the coarse localization area, and use a sliding window to perform multi-scale feature extraction to obtain the multi-scale candidate region image.

[0041] Furthermore, the matching model can also perform semantic similarity matching between the text recognition result and the standard review item, calculate the degree of similarity between the two at the semantic level, and thus determine whether the text recognition result matches the review item. If they match, the matching area is located according to the position of the corresponding text in the coarse localization area, and multi-scale feature extraction is performed using a sliding window to obtain the multi-scale candidate region image.

[0042] Specifically, multi-scale candidate region images can be obtained by setting different window sizes, sliding steps, and overlap ratios.

[0043] In some embodiments, one of the following operations—keyword matching, regular expression matching, and semantic similarity matching—can be performed on the text recognition results and standard review items. Alternatively, two or three operations can be performed separately, and the results obtained from each operation can be integrated to obtain a multi-scale candidate region image.

[0044] The obtained multi-scale candidate regions are the relevant regions in the file to be detected that correspond to the standard review entries.

[0045] Further, refer to Figure 2 and Figure 3 In step S3, the lightweight hybrid Siamese network includes a backbone network module 1 and a Siamese network module 2; the backbone network module 1 includes an input layer 11, multiple stacked feature extraction modules 12, and an output layer 13; the Siamese network module 2 includes a feature projection module 21, an MLP similarity calculation module 22, and an output module 23; the feature extraction module 12 includes a lightweight convolutional layer 121, a Transformer layers module 122, and a feature fusion module 123; During the detection process, the multi-scale candidate region image and the standard review document are input into a pre-trained lightweight hybrid Siamese network for image feature matching, and the output is a matched feature image, including: The multi-scale candidate region image and the standard review document are sequentially processed by the feature extraction module, the Transformer layers module, and the feature fusion module for feature extraction, global modeling, and feature fusion, outputting fused multi-scale candidate region features and fused standard features; The feature projection module maps the fused multi-scale candidate region features and the fused standard features into the same space. Then, the MLP similarity calculation module performs similarity calculation to obtain the matching probability of the fused multi-scale candidate region features and the fused standard features. The output module filters the fused multi-scale candidate region features according to the matching probability and outputs the matching feature image.

[0046] Specifically, the input layer 11 receives the multi-scale candidate region image and the standard review file and sends them to multiple feature extraction modules. Alternatively, it can send the image content of the multi-scale candidate region image and the standard review file to multiple feature extraction modules. The lightweight convolutional layer 121 in the feature extraction module 12 uses standard convolution to extract features from the multi-scale candidate region image and the standard review file respectively. Specifically, feature extraction is performed based on the image content in the standard review file to obtain multi-scale candidate region features and standard features. The Transformerlayers module 122 is used to perform global modeling on the multi-scale candidate region features and standard features respectively to obtain global multi-scale candidate region features and global standard features at different levels. The feature fusion module 123 is used to fuse global multi-scale candidate region features and global standard features at different levels, and outputs the fused multi-scale candidate region features and fused standard features through the output layer 13; the feature projection module 21 maps the fused multi-scale candidate region features and fused standard features to the same space; the MLP similarity calculation module 22 is used to calculate the similarity between the fused multi-scale candidate region features and fused standard features in the same space to obtain the matching probability of the fused multi-scale candidate region features and fused standard features; the output module 23 is used to filter the fused multi-scale candidate region features according to the matching probability and a preset probability threshold, and outputs the fused multi-scale candidate region features with a matching probability exceeding the preset probability threshold as the matching feature image.

[0047] Specifically, the output matching feature image is one or more region images from the multi-scale candidate region images that are most similar to the standard image in the standard review file.

[0048] Specifically, the backbone network module 1 is mainly used to extract complete, high-quality features, providing a foundation for subsequent similarity calculations. Specifically, the input layer 11 receives the multi-scale candidate region image and the standard review file and sends them to multiple feature extraction modules 12. The lightweight convolutional layer 121 in the feature extraction module uses standard convolution to extract image features, suitable for extracting details such as structural lines, frames, and symbols in the image. The Transformer layers module 122 is used to globally model the multi-scale candidate region features and standard features respectively, enabling a holistic understanding of the structure of image features, such as the spatial and semantic relationships between multiple parts of a symbol, obtaining global multi-scale candidate region features and global standard features at different levels. The feature fusion module 123 is used to fuse the global multi-scale candidate region features and global standard features at different levels, and the output layer 13 integrates the fused multi-scale candidate region features and fused standard features into a fixed-dimensional vector and outputs it. Multiple stacked feature extraction modules 12 help to simultaneously identify large and small image patches.

[0049] Furthermore, the feature projection module 21 maps the fused multi-scale candidate region features and the fused standard features to the same space, making semantically similar features closer together for subsequent similarity calculation. The MLP similarity calculation module 22 is used to learn the nonlinear mapping relationship of matching (the mapping from feature pairs to matching probabilities), adapting to the nonlinear relationship between complex image patches, thereby realizing the similarity calculation of the fused multi-scale candidate region features and the fused standard features in the same space, obtaining the matching probability of the fused multi-scale candidate region features and the fused standard features, generating a binary classification probability. The output module 23 is used to filter the fused multi-scale candidate region features according to the matching probability and a preset probability threshold, and output the filtered fused multi-scale candidate region features as the matching feature image.

[0050] In summary, a lightweight hybrid Siamese network is constructed by combining a backbone network module based on Transformer layers with a Siamese network module that can learn a similarity measurement mechanism. This lightweight hybrid Siamese network introduces the Transformer mechanism as the backbone for feature extraction. Compared to traditional pure CNN models such as ResNet and MobileNet, the backbone network module in this embodiment adopts a hybrid structure of convolution and Transformer, which takes into account both local details and global relationships. The built-in lightweight self-attention mechanism is more suitable for modeling the structure of drawing symbols, with low parameter count and fast inference speed, making it suitable for industrial deployment.

[0051] Traditional Siamese networks using static Euclidean distance or cosine similarity perform poorly under conditions of scale differences, style variations, and printing errors. To overcome this limitation, this embodiment designs a learnable similarity metric sub-network, applied to the MLP similarity calculation module, to more flexibly express semantic relationships between graph tiles. The similarity metric sub-network can be represented as: (1) in, f 1 and f 2 represents the two feature vectors output by the backbone network module. Sim( represents the Sigmoid activation function, MLP represents a lightweight multilayer perceptron consisting of two fully connected layers and the GELU activation function, Sim( f 1, f 2) Represents the similarity calculation results. This structure allows the model to automatically learn non-linear similarity relationships between images during training, significantly enhancing robustness to factors such as scale differences, style variations, and printing errors.

[0052] During the inference phase, structured standard review documents are paired with multi-scale candidate region images and input into a lightweight hybrid Siamese network. After feature extraction, MLP, and a sigmoid classifier, matching probabilities are output and filtered based on a threshold. In some embodiments, non-maximum suppression and Top-K high similarity mechanisms can be applied to the filtering results at all scales to obtain the final matching feature image. All review entries and corresponding candidate region images are organized and integrated using a unified JSON data structure to form data that facilitates automatic model processing and batch review.

[0053] Furthermore, in some embodiments, the method further includes: training the lightweight hybrid Siamese network based on pre-constructed training samples; The construction of training samples includes: Collect industry standard document data and convert it into image data. Based on the image data, perform image and text extraction and cropping to obtain standard sample image blocks. A sliding window is used to extract candidate patches from the sample documents to be reviewed; Select candidate patches corresponding to standard sample patches as positive samples, and form positive sample pairs with the corresponding standard sample patches; Candidate patches that do not correspond to standard sample patches are used as negative samples, and the negative samples are paired with standard sample patches to form negative sample pairs.

[0054] This dataset collects standard industry document data, including design drawings, product manuals, and assembly documents, to construct a standard industry document review dataset. The samples are organized in JSON format, suitable for training lightweight hybrid Siamese networks and multimodal large models. The dataset covers multi-scale, multi-class, and multi-angle patch samples, providing positive and negative sample annotations, OCR content, and patch location markers, exhibiting high industrial versatility and generalization. Positive samples are standard image matching regions, while negative samples are randomly confused images.

[0055] In some embodiments, positive and negative sample pairs can be integrated into a single dataset, on which a lightweight hybrid Siamese network can be trained.

[0056] Further, the lightweight hybrid Siamese network is trained, including: Repeat the following steps until the stopping condition is met: The positive and negative sample pairs are input into the backbone network module for feature extraction, and the first sample feature vector and the second sample feature vector are output. The first sample feature vector and the second sample feature vector are input into the Siamese network module for similarity calculation and threshold judgment, and the judgment result is output. During the training process, the negative sample most similar to the positive sample pair is found as the third sample feature vector. The cosine loss is calculated based on the first sample feature vector and the second sample feature vector; the binary cross-entropy loss is calculated based on the output of the Siamese network module; and the triplet loss is calculated based on the first sample feature vector, the second sample feature vector, and the third sample feature vector. The combined loss is obtained based on the cosine loss, binary cross-entropy loss, and triplet loss. The parameters of the lightweight hybrid twin network are then updated by backpropagation based on the combined loss.

[0057] Specifically, the loss function design of the lightweight hybrid Siamese network employs a multi-loss function combination mechanism, which improves the model's ability to distinguish between positive and negative sample pairs and its ability to express the embedding space structure. This can be expressed as: (2) Where Loss represents the overall loss, L TRIP L represents the triplet loss. COSINE L represents cosine loss. BCE Let λ1, λ2, and λ3 represent the weights of the triplet loss, cosine loss, and binary cross-entropy loss, respectively. The definitions of each part are as follows: Triplet Loss: (3) in, f 1. f 2. f 3 represents the feature vector of the first sample, the feature vector of the second sample, and the feature vector of the third sample, respectively.

[0058] Cosine similarity loss: (4) in, Let represent the Euclidean norms of the first sample vector and the second sample vector, respectively.

[0059] Binary cross-entropy loss (BCE): (5) Among them, y t Represents the true label, y p This represents the output of the Siamese network module, used to supervise the consistency between the similarity output and the real label.

[0060] Further, refer to Figure 4 The multimodal large model includes a dynamic Prompt module 3 and a Qwen2.5-VL model 4; The dynamic Prompt module is used to construct prompt words based on the candidate region text information and the standard review document; specifically, it constructs the candidate region text information and the standard review document into a standardized graphic data format and inputs it into the dynamic Prompt module to construct prompt words.

[0061] The Qwen2.5-VL model is used to perform analysis based on the prompt words and the matching feature images. In some embodiments, the prompt words and matching feature images can be constructed into a standardized image and text data format and input into the Qwen2.5-VL model to output the detection result of whether the review content corresponding to the review item in the file to be detected is compliant.

[0062] Specifically, the dynamic Prompt module is constructed for precise control and low-illusion in multimodal drawing review. This is a candidate region-driven dynamic Prompt construction mechanism. By dynamically constructing the Prompt through matching feature image extraction and pairing data, it dynamically constructs minimal contextual cue words to achieve fine-grained control of the instruction structure, enhancing the model's instruction comprehension and review focus. Specifically, it automatically calls templates to generate standard entry content and sets different template subclasses for different types of entries (graphics / text / annotations), prompting the model to follow specific instructions to enhance review accuracy, significantly shorten token length, and avoid historical interference and model illusion.

[0063] The final test results will structure each review task output into a format including sequence number, review item, candidate area information, review result, confidence level, and judgment reason. All results will be aggregated at the task level to generate a task overview and detailed descriptions of each item. A standard review report generation mechanism can also be built to automate report content compilation and support export in multiple formats (JSON, PDF, Excel) to meet the recording requirements of the quality system.

[0064] In addition, image visualization results can be overlaid, highlighting all candidate review areas in the original file, with compliant areas highlighted in green and non-compliant areas highlighted in red. Problem areas are sorted according to the credibility of inferences and the system supports rendering area colors according to risk level (red, orange, yellow, and green grading), guiding manual review of error-prone areas first.

[0065] Furthermore, the multimodal large model is pre-trained, including: Collect industry standard document data and convert it into image data. Based on the image data, extract the text and images to obtain sample text and images data. Integrate and process the sample text and images data with the corresponding review standards to obtain a large model sample. Obtain a pre-trained Qwen2.5-VL model and inject LoRA parameters, then freeze the main parameters of the pre-trained Qwen2.5-VL model; wherein, the pre-trained Qwen2.5-VL model is from the official source.

[0066] Repeat the following steps until the stopping condition is met: The large model samples are input into the pre-trained Qwen2.5-VL model, and the output results of the pre-trained Qwen2.5-VL model are obtained. Human feedback data is added to the output results of the pre-trained Qwen2.5-VL model to form supervised samples, and a reward model is constructed. The supervised samples are input into the reward model for training, and the pre-trained Qwen2.5-VL model is trained based on the output of the reward model, with the LoRA parameters of the pre-trained Qwen2.5-VL model being fine-tuned. Specifically, the output of the reward model is used as a reinforcement learning signal to fine-tune the LoRA parameters of the pre-trained Qwen2.5-VL model.

[0067] A training method for enhancing multimodal review capabilities based on LoRA and RLHF is proposed. By introducing LoRA and RLHF into a pre-trained multimodal large model (Qwen2.5-VL), the generalization ability, accuracy, and consistency of structural review in industrial drawing scenarios are significantly improved.

[0068] The inclusion of LoRA allows for fine-tuning of image-text alignment and review inference capabilities with only a minimal number of parameters (approximately 1% of the original parameters). Its core idea is to freeze the master parameters of the pre-trained Qwen2.5-VL model and insert trainable weight pairs A and B of rank r only into the Query and Value weight matrices of the multi-head attention mechanism, achieving low-resource fine-tuning of the model in industrial drawing review tasks. (6) Where W represents the weight matrix of the pre-trained Qwen2.5-VL model, AB is the product of low-rank matrix pairs, and W' is the adjusted weight matrix. The training data is in chat_template format. The fine-tuned model can automatically perform inference based on drawings and standard specifications, possessing stronger capabilities for non-compliance detection and understanding of review standards.

[0069] This paper introduces the RLHF (Reinforcement Learning with Human Feedback) method, which learns a reward function through human preferences and strengthens the model's behavior based on this function to improve its review capabilities and consistency. The steps may include: obtaining the output of the pre-trained Qwen2.5-VL model; adding human feedback data (including candidate response ranking and compliance scores) to the output to form supervised samples; constructing a reward model based on the Qwen2.5-VL model and completing preference modeling through Pairwise Ranking Loss; and optimizing the pre-trained Qwen2.5-VL model using the Proximal Policy Optimization (PPO) algorithm to maximize the human feedback preference score.

[0070] The Pairwise Ranking Loss is defined as follows: (7) Where r1 and r2 represent the output of the reward model, Lrank represents the loss of the reward model, and σ represents the Sigmoid function. When r1 ≥ r2, the preferred response score is higher, and the loss approaches 0. The goal is to teach the reward model to distinguish between "better" and "worse" model responses.

[0071] During the optimization training of the pre-trained Qwen2.5-VL model using the Proximal Policy Optimization (PPO) algorithm, the advantage function is calculated by comparing the ratio of the output probabilities of the current policy to those of the old policy, combined with the reward score. The objective function is shown below: (8) Where π0(y|x) and π old (y|x) represent the probabilities of generating the answer y under the current policy and the old policy, respectively; R(x,y) represents the output of the reward model, used to evaluate the quality of the answer y; clip is the cutoff function, used to limit the policy update magnitude; and b(x) is the baseline function. This is a hyperparameter used to control the cutoff range.

[0072] refer to Figure 5 A scheme detection device for complex mixed text and image documents is provided, comprising: The standard extraction module 201 is used to extract standard review entries and standard review content from the standard manual and perform structured processing to obtain standard review documents; The candidate region extraction module 202 is used to convert the file to be detected into an image to be detected, and to extract multi-scale candidate region images to be examined from the image to be detected using OCR, a pre-trained initial recognition model and a matching model, and to extract text information from the multi-scale candidate images to obtain candidate region text information. Matching module 203 is used to input the multi-scale candidate region image and the standard review document into a pre-trained lightweight hybrid Siamese network for image feature matching and output a matching feature image; The detection module 204 is used to input the matching feature image, the standard review document, and candidate region text information into a pre-trained multimodal large model for compliance analysis and output the detection results.

[0073] Furthermore, the standard manual includes both structured and unstructured data; The standard extraction module 201 extracts the standard review entries and content from the standard manual and performs structured processing to obtain the standard review documents, including: For the structured data, the first standard review item is identified based on the standard manual, and the graphic and textual information about the first standard review item is extracted line by line as the first standard review content. For each line of graphic and textual information extracted, text information is extracted using OCR as the first text supplementary information. The first standard review item, the first standard review content and the corresponding first text supplementary information are structured to obtain the standard review document. For the unstructured data, the standard manual is converted into an image. A pre-trained recognition model is used to analyze the layout of the image and extract the text and image content modules. Based on the text and image content modules, the second standard review items are identified, and the areas in the text and image content modules that correspond to the second standard review items are identified as the second standard review content. OCR is used to extract the text information in the text and image content modules as the second text supplementary information. The second standard review items, the second standard review content, and the corresponding second text supplementary information are then structured to obtain the standard review document.

[0074] Furthermore, the candidate region extraction module 202 uses OCR, a pre-trained initial recognition model, and a matching model to extract multi-scale candidate region images for review from the image to be detected, including: The image to be detected is input into the initial recognition model for layout analysis, and the drawing area and text area are extracted as coarse positioning areas. OCR is used to perform text recognition on the coarsely located area to obtain the text recognition result; Extract the standard review entries from the aforementioned standard review documents; The text recognition result, standard review items, and coarse localization region are input into the matching model. The matching model performs at least one operation among keyword matching, regular expression matching, and semantic similarity matching on the text recognition result and the standard review items to determine the region in the coarse localization region that matches the standard review items. The matching region is then subjected to multi-scale feature extraction using a sliding window to obtain the multi-scale candidate region image.

[0075] Furthermore, the lightweight hybrid Siamese network includes a backbone network module and a Siamese network module; the backbone network module includes an input layer, multiple stacked feature extraction modules, and an output layer; the Siamese network module includes a feature projection module, an MLP similarity calculation module, and an output module; the feature extraction module includes lightweight convolutional layers, a Transformer layers module, and a feature fusion module.

[0076] Further, the matching module 203 inputs the multi-scale candidate region image and the standard review document into a pre-trained lightweight hybrid Siamese network for image feature matching, and outputs a matching feature image, including: The multi-scale candidate region image and the standard review document are sequentially processed by the feature extraction module, the Transformer layers module, and the feature fusion module for feature extraction, global modeling, and feature fusion, outputting fused multi-scale candidate region features and fused standard features; The feature projection module maps the fused multi-scale candidate region features and the fused standard features into the same space. Then, the MLP similarity calculation module performs similarity calculation to obtain the matching probability of the fused multi-scale candidate region features and the fused standard features. The output module filters the fused multi-scale candidate region features according to the matching probability and outputs the matching feature image.

[0077] Furthermore, the matching module 203 is also used for: The input layer receives the multi-scale candidate region image and the standard review document and sends them to multiple feature extraction modules. The lightweight convolutional layers in these feature extraction modules use standard convolution to extract features from the multi-scale candidate region image and the standard review document, respectively, to obtain multi-scale candidate region features and standard features. The Transformer... The layers module is used to globally model the multi-scale candidate region features and standard features respectively, to obtain global multi-scale candidate region features and global standard features at different levels. The feature fusion module is used to fuse the global multi-scale candidate region features and global standard features at different levels, and output the fused multi-scale candidate region features and fused standard features through the output layer. The feature projection module is used to map the fused multi-scale candidate region features and fused standard features to the same space. The MLP similarity calculation module is used to calculate the similarity between the fused multi-scale candidate region features and fused standard features in the same space, to obtain the matching probability of the fused multi-scale candidate region features and fused standard features. The output module is used to filter the fused multi-scale candidate region features according to the matching probability and a preset probability threshold, and output the fused multi-scale candidate region features with a matching probability exceeding the preset probability threshold as the matching feature image.

[0078] Furthermore, the device also includes a first training module for training the lightweight hybrid Siamese network based on pre-constructed training samples; The construction of training samples includes: Collect industry standard document data and convert it into image data. Based on the image data, perform image and text extraction and cropping to obtain standard sample image blocks. A sliding window is used to extract candidate patches from the sample documents to be reviewed; Select candidate patches corresponding to standard sample patches as positive samples, and form positive sample pairs with the corresponding standard sample patches; Candidate patches that do not correspond to standard sample patches are used as negative samples, and the negative samples are paired with standard sample patches to form negative sample pairs.

[0079] Furthermore, the first training module trains the lightweight hybrid Siamese network, including: Repeat the following steps until the stopping condition is met: The positive and negative sample pairs are input into the backbone network module for feature extraction, and the first sample feature vector and the second sample feature vector are output. The first sample feature vector and the second sample feature vector are input into the Siamese network module for similarity calculation and threshold judgment, and the judgment result is output. During the training process, the negative sample most similar to the positive sample pair is found as the third sample feature vector. The cosine loss is calculated based on the first sample feature vector and the second sample feature vector; the binary cross-entropy loss is calculated based on the output of the Siamese network module; and the triplet loss is calculated based on the first sample feature vector, the second sample feature vector, and the third sample feature vector. The combined loss is obtained based on the cosine loss, binary cross-entropy loss, and triplet loss. The parameters of the lightweight hybrid twin network are then updated by backpropagation based on the combined loss.

[0080] Furthermore, the multimodal large model includes a dynamic Prompt module and a Qwen2.5-VL model. The dynamic Prompt module is used to construct prompt words based on the text information of the candidate regions and the standard review document. The Qwen2.5-VL model is used to analyze the prompt words and the matching feature images, and output the detection results of whether the review content corresponding to the review items in the file to be detected is compliant.

[0081] Furthermore, the device also includes a second training module for pre-training the multimodal large model, including: Collect industry standard document data and convert it into image data. Based on the image data, extract the text and images to obtain sample text and images data. Integrate and process the sample text and images data with the corresponding review standards to obtain a large model sample. Obtain the pre-trained Qwen2.5-VL model and inject LoRA parameters, then freeze the main parameters of the pre-trained Qwen2.5-VL model; Repeat the following steps until the stopping condition is met: The large model samples are input into the pre-trained Qwen2.5-VL model, and the output results of the pre-trained Qwen2.5-VL model are obtained. Human feedback data is added to the output results of the pre-trained Qwen2.5-VL model to form supervised samples, and a reward model is constructed. The supervised samples are input into the reward model for training, and the LoRA parameters of the pre-trained Qwen2.5-VL model are fine-tuned based on the output of the reward model.

[0082] The method and apparatus for detecting complex mixed text and image documents provided in the above embodiments have at least the following beneficial effects: (1) Introducing a multimodal large model (Qwen2.5-VL) to achieve semantic understanding of complex graphics and text, it can automatically identify, match and understand elements such as icons, symbols, text and structures in the scheme, improve the accuracy of detection and reduce the risk of missed detection, and significantly improve the automation and intelligence of reviewing complex graphics and text mixed documents; (2) A lightweight hybrid Siamese network is constructed to integrate global and local features to significantly enhance the ability to represent map features. A dynamic similarity measurement module composed of a lightweight MLP and GELU activation function is designed to enhance the robustness of the model to factors such as scale differences, style changes, and printing errors. Finally, a multi-loss function combination mechanism is used to optimize the model weights, which further improves the model's ability to represent the features of the scheme and adapts to the multi-scale, multi-angle, and multi-style variations of various elements in the scheme. (3) It integrates OCR recognition information, image features and review item information to improve semantic accuracy. It adopts a dynamic Prompt construction mechanism to constrain the content generated by the model, control the context length, avoid multi-task interference, and can flexibly adapt to different file types. It can be quickly integrated into industrial platforms. (4) By uploading the review standards in Excel / Word format, the review template can be dynamically generated. The document can be flexibly changed to adapt to different custom standards. It supports the automated construction of industrial drawing datasets through the construction tool, and combines LoRA and RLHF technologies to fine-tune the model, continuously improve the model knowledge system, and continuously enhance the understanding of enterprise standards. (5) Enhance the interpretability and traceability of document review tasks, significantly reduce the workload of manual proofreading and review cycle. The entire process is automated, and the review and annotation of the complete document plan can be completed in an average of a few minutes. The review report is automatically generated. The review results include structured output, model judgment reasons, credibility score and intuitive visualization on the drawings, which assists in manual confirmation, greatly reduces labor costs and improves the efficiency of drawing review for large-scale engineering projects.

[0083] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A method for detecting schemes in complex mixed text and image documents, characterized in that, include: Extract the standard review entries and content from the standard manual and perform structured processing to obtain the standard review documents; The file to be detected is converted into an image to be detected. OCR, a pre-trained initial recognition model and a matching model are used to extract multi-scale candidate region images to be reviewed from the image to be detected, and the text information of the multi-scale candidate images is extracted to obtain candidate region text information. The multi-scale candidate region image and the standard review file are input into a pre-trained lightweight hybrid Siamese network for image feature matching, and the matching feature image is output. The matching feature image, the standard review document, and the candidate region text information are input into a pre-trained multimodal large model for compliance analysis, and the detection results are output.

2. The method according to claim 1, characterized in that, The standard manual includes structured and unstructured data; Extract the standard review entries and content from the standards manual and structure them to obtain the standard review documents, including: For the structured data, the first standard review item is identified based on the standard manual, and the graphic and textual information about the first standard review item is extracted line by line as the first standard review content. For each line of graphic and textual information extracted, text information is extracted using OCR as the first text supplementary information. The first standard review item, the first standard review content and the corresponding first text supplementary information are structured to obtain the standard review document. For the unstructured data, the standard manual is converted into an image. A pre-trained recognition model is used to analyze the layout of the image and extract the text and image content modules. Based on the text and image content modules, the second standard review items are identified, and the areas in the text and image content modules that correspond to the second standard review items are identified as the second standard review content. OCR is used to extract the text information in the text and image content modules as the second text supplementary information. The second standard review items, the second standard review content, and the corresponding second text supplementary information are then structured to obtain the standard review document.

3. The method according to claim 1, characterized in that, The image to be detected is processed using OCR, a pre-trained initial recognition model, and a matching model to extract multi-scale candidate regions for review, including: The image to be detected is input into the initial recognition model for layout analysis, and the drawing area and text area are extracted as coarse positioning areas. OCR is used to perform text recognition on the coarsely located area to obtain the text recognition result; Extract the standard review entries from the aforementioned standard review documents; The text recognition result, standard review items, and coarse localization region are input into the matching model. The matching model performs at least one operation among keyword matching, regular expression matching, and semantic similarity matching on the text recognition result and the standard review items to determine the region in the coarse localization region that matches the standard review items. The matching region is then subjected to multi-scale feature extraction using a sliding window to obtain the multi-scale candidate region image.

4. The method according to claim 1, characterized in that, The lightweight hybrid Siamese network includes a backbone network module and a Siamese network module; the backbone network module includes an input layer, multiple stacked feature extraction modules, and an output layer; the Siamese network module includes a feature projection module, an MLP similarity calculation module, and an output module; the feature extraction module includes lightweight convolutional layers, a Transformer layers module, and a feature fusion module.

5. The method according to claim 4, characterized in that, The multi-scale candidate region image and the standard review file are input into a pre-trained lightweight hybrid Siamese network for image feature matching, and the output is a matched feature image, including: The multi-scale candidate region image and the standard review document are sequentially processed by the feature extraction module, the Transformer layers module, and the feature fusion module for feature extraction, global modeling, and feature fusion, outputting fused multi-scale candidate region features and fused standard features; The feature projection module maps the fused multi-scale candidate region features and the fused standard features into the same space. Then, the MLP similarity calculation module performs similarity calculation to obtain the matching probability of the fused multi-scale candidate region features and the fused standard features. The output module filters the fused multi-scale candidate region features according to the matching probability and outputs the matching feature image.

6. The method according to claim 4, characterized in that, The method further includes: training the lightweight hybrid Siamese network based on pre-constructed training samples; The construction of training samples includes: Collect industry standard document data and convert it into image data. Based on the image data, perform image and text extraction and cropping to obtain standard sample image blocks. A sliding window is used to extract candidate patches from the sample documents to be reviewed; Select candidate patches corresponding to standard sample patches as positive samples, and form positive sample pairs with the corresponding standard sample patches; Candidate patches that do not correspond to standard sample patches are used as negative samples, and the negative samples are paired with standard sample patches to form negative sample pairs.

7. The method according to claim 6, characterized in that, Training the lightweight hybrid Siamese network includes: Repeat the following steps until the stopping condition is met: The positive and negative sample pairs are input into the backbone network module for feature extraction, and the first sample feature vector and the second sample feature vector are output. The first sample feature vector and the second sample feature vector are input into the Siamese network module for similarity calculation and threshold judgment, and the judgment result is output. During the training process, the negative sample most similar to the positive sample pair is found as the third sample feature vector. The cosine loss is calculated based on the first sample feature vector and the second sample feature vector; the binary cross-entropy loss is calculated based on the output of the Siamese network module; and the triplet loss is calculated based on the first sample feature vector, the second sample feature vector, and the third sample feature vector. The combined loss is obtained based on the cosine loss, binary cross-entropy loss, and triplet loss. The parameters of the lightweight hybrid twin network are then updated by backpropagation based on the combined loss.

8. The method according to claim 6, characterized in that, The multimodal large model includes a dynamic Prompt module and a Qwen2.5-VL model. The dynamic Prompt module is used to construct prompt words based on the text information of the candidate regions and the standard review document. The Qwen2.5-VL model is used to analyze the prompt words and the matching feature images, and output the detection results of whether the review content corresponding to the review items in the file to be detected is compliant.

9. The method according to claim 8, characterized in that, Pre-training the multimodal large model includes: Collect industry standard document data and convert it into image data. Based on the image data, extract the text and images to obtain sample text and images data. Integrate and process the sample text and images data with the corresponding review standards to obtain a large model sample. Obtain the pre-trained Qwen2.5-VL model and inject LoRA parameters, then freeze the main parameters of the pre-trained Qwen2.5-VL model; Repeat the following steps until the stopping condition is met: The large model samples are input into the pre-trained Qwen2.5-VL model, and the output results of the pre-trained Qwen2.5-VL model are obtained. Human feedback data is added to the output results of the pre-trained Qwen2.5-VL model to form supervised samples, and a reward model is constructed. The supervised samples are input into the reward model for training, and the LoRA parameters of the pre-trained Qwen2.5-VL model are fine-tuned based on the output of the reward model.

10. A scheme detection device for complex mixed text and image documents, characterized in that, include: The standard extraction module is used to extract standard review entries and standard review content from the standard manual and perform structured processing to obtain standard review documents; The candidate region extraction module is used to convert the file to be detected into an image to be detected, and to extract multi-scale candidate region images to be examined from the image to be detected using OCR, a pre-trained initial recognition model and a matching model, and to extract text information from the multi-scale candidate images to obtain candidate region text information. The matching module is used to input the multi-scale candidate region image and the standard review file into a pre-trained lightweight hybrid Siamese network to perform image feature matching and output a matching feature image. The detection module is used to input the matching feature image, the standard review document, and candidate region text information into a pre-trained multimodal large model for compliance analysis and output the detection results.

Images