Engineering drawing automatic bubble labeling and information extraction method based on hybrid visual model
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JIANGSU DAOYUNYIN TECH CO LTD
- Filing Date
- 2026-02-27
- Publication Date
- 2026-05-12
AI Technical Summary
Existing automatic bubble annotation methods for engineering drawings cannot effectively distinguish between measured items and reference values, and traditional OCR technology has low recognition accuracy when processing rotated text and complex layouts, resulting in logical errors frequently appearing in the inspection list.
A hybrid vision model is adopted, combining rotating target detection technology with fine-tuned visual language model (VLM) and optical character recognition (OCR) technology. The rotating target detection model is used for image preprocessing, VLM is used to understand the semantic structure of drawing elements, and OCR is used for verification to generate accurate structured data.
It enables precise location and attribute classification of features that need to be controlled in engineering drawings, reduces the logical error rate, and improves recognition accuracy and robustness. In particular, it significantly improves the system's recognition accuracy when processing complex tabular data.
Smart Images
Figure CN122024271A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method for automatic bubble annotation and information extraction in engineering drawings based on a hybrid vision model, belonging to the field of computer vision technology. Background Technology
[0002] In digital manufacturing and quality management processes, "ballooning" of engineering drawings is a crucial step. This process requires numbering every feature on the drawing, such as dimensions, tolerances, and annotations, and extracting their attributes to generate an inspection plan.
[0003] Existing academic research typically categorizes drawing elements into GD&T, surface roughness, chamfers, and thread types. While this classification based on "geometric feature type" is detailed, it does not directly correspond to the "control attributes" in quality inspection. For example, in ballooning logic, the processing logic for dimensions with tolerances (requiring measurement) and reference dimensions (for reference only) is completely different, but existing inspection models often conflate them.
[0004] Traditional OCR technologies (such as Tesseract) perform poorly when processing rotated text, special symbols, and complex layouts in engineering drawings; simple OCR cannot understand the semantic structure of dimensions and often requires complex post-processing rules that are self-compiled. Summary of the Invention
[0005] The technical problem to be solved by the present invention is to construct a new classification and detection system that combines rotating target detection technology with fine-tuning VLM and OCR technology to achieve accurate positioning, attribute classification and structured extraction of features that need to be controlled in drawings.
[0006] To address the aforementioned technical problems, this invention proposes a method for automatic bubble annotation and information extraction in engineering drawings based on a hybrid visual model, comprising the following steps:
[0007] S1. Input the original image data of the engineering drawings;
[0008] S2. Based on the rotating target detection model, the original image data is detected and classified to obtain detection result data including multiple detection units. Each detection unit includes the coordinate information of the oriented bounding box, the category of the drawing element, and the category confidence.
[0009] The drawing elements include measurement items, theoretical values, reference values, text descriptions, and tables. Measurement items include all dimensions with tolerances, theoretical values correspond to the theoretically correct dimensions marked with boxes, and reference values correspond to the reference dimensions marked with brackets.
[0010] S3. Perform information extraction processing on the image region corresponding to each detection unit:
[0011] Based on the category of the detection unit, the corresponding region image data is extracted from the original image data;
[0012] If the category is a measurement item, theoretical value, or reference value, the region image data is input into the fine-tuned visual language model (VLM) to obtain the first structured data and the corresponding generated confidence score.
[0013] If the category is text description, then the region image data is input into the Visual Language Model (VLM) to obtain the second structured data and the corresponding generation confidence.
[0014] If the category is a table, the regional image data will be processed sequentially through the Optical Character Recognition (OCR) module and the Large Language Model (LLM) to obtain the third structured data.
[0015] S4. Perform confidence verification processing on the first and second structured data:
[0016] When the generated confidence level is lower than the preset threshold, the optical character recognition (OCR) module is called to recognize the corresponding region image data to obtain OCR verification data, and the first structured data or the second structured data is corrected accordingly to obtain the corrected structured data.
[0017] When the generated confidence level is higher than or equal to the preset threshold, the first structured data or the second structured data is directly used as the corrected structured data.
[0018] S5. Based on the spatial location information of the detection units categorized as measurement items in the detection result data, assign a unique bubble number to each measurement item detection unit according to the preset spatial sorting rules, and generate number mapping data;
[0019] S6. The corrected structured data, the third structured data, and the detection result data are correlated and matched to obtain the enhanced detection record; then the enhanced detection record is integrated with the number mapping data to generate the final output result including the bubble annotation layer and the structured detection list.
[0020] The rotating target detection model in S2 is a one-stage target detection network that supports rotating bounding box prediction. Its output is the oriented bounding box parameters (cx,cy,w,h,θ), where (cx,cy) are the center point coordinates, (w,h) are the width and height, and θ is the rotation angle.
[0021] The test results are obtained through the following steps:
[0022] S21. Standardize the original image data to form the model input tensor;
[0023] S22. Extract image features through a rotating target detection model and perform multi-task prediction to obtain raw prediction data including oriented bounding box parameters, class probabilities, and target confidence.
[0024] S23. Perform confidence screening, frame decoding, and non-maximum suppression on the original prediction data to obtain a list of candidate detection units;
[0025] S24. Encapsulate the candidate detection unit list into structured detection result data.
[0026] As mentioned above, the non-maximum suppression processing in S23 calculates the intersection-union ratio of the rotated frame.
[0027] As mentioned above, in S3, based on the Visual Language Model (VLM), during the generation of structured data, the prediction probability of key tokens in the output sequence is aggregated and calculated to obtain the generation confidence of the first or second structured data.
[0028] As mentioned above, the first structured data in S3 is a structured object including nominal value, tolerance, geometric symbol, and quantity field; the second structured data is a structured object including the recognition text content; and the third structured data is key-value pair structured data extracted from OCR text through the Large Language Model (LLM).
[0029] The confidence verification process in S4 also includes comparing the OCR verification data with the first or second structured data to be verified at the character level or semantic level, and performing automatic correction or conflict arbitration based on the comparison results.
[0030] The above-mentioned S5 preset spatial sorting rules are sorting rules from left to right, from top to bottom, or clockwise.
[0031] The clockwise surround sorting rule in S5, as described above, includes the following steps:
[0032] Determine the center point of the drawing or area;
[0033] Calculate the angle between the reference point of each measurement unit and the center point;
[0034] Number them in ascending order of angle.
[0035] As mentioned above, in S6, the association matching uses the detection unit ID of each detection unit in the detection result data as an index to find and merge the corresponding corrected structured data or third structured data to form an enhanced detection record.
[0036] As described above, in S6, all measurement item records are filtered from the integrated data, and key information is extracted and arranged according to the preset report format to generate a structured inspection list;
[0037] Based on the location information of each record in the integrated data, bubble annotation graphics with bubble numbers are drawn at the corresponding positions on the drawing, thereby generating a bubble annotation layer.
[0038] This invention has positive effects:
[0039] (1) By directly distinguishing between measurement items and reference values during the testing stage, this invention avoids the logical error of mistakenly including reference dimensions in the testing list from the source, and the classification is more accurate;
[0040] (2) The present invention uses a fine-tuned visual language model to replace simple OCR, which can directly understand engineering semantics such as size, tolerance, quantity and output structured data, saves the step of complex post-processing of messy OCR results, reduces the error rate and achieves efficient structured information extraction.
[0041] (3) This invention constructs a hybrid recognition architecture with VLM as the main component and OCR as the auxiliary component, and introduces a confidence-based verification mechanism. It combines the semantic understanding advantage of VLM with the character recognition stability of OCR, especially for complex table data, which improves the overall robustness and recognition accuracy of the system. Attached Figure Description
[0042] The invention will now be further described with reference to the accompanying drawings.
[0043] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation
[0044] Example 1
[0045] See Figure 1 This invention provides a technical solution: an automatic bubble annotation and information extraction method for engineering drawings based on a hybrid visual model, comprising the following steps:
[0046] S1. Input the original image data of the engineering drawings;
[0047] S2. Based on the rotating target detection model, the original image data is detected and classified to obtain detection result data including location, category and confidence level;
[0048] The detection result data contains multiple detection units, each of which includes directional bounding box coordinate information, drawing element category, category confidence level, and detection unit ID;
[0049] The drawing elements include measurement items, theoretical values, reference values, text descriptions, and tables;
[0050] Measurement items include all dimensions with tolerances; theoretical values correspond to the theoretically correct dimensions indicated by boxes; reference values correspond to the reference dimensions indicated by brackets.
[0051] The specific steps for detection and classification are as follows:
[0052] 1. After scaling the image to the model input size, normalize the pixel values to form the model input tensor.
[0053] 2. An improved YOLO-OBB model that supports rotated bounding box prediction is used to extract image features. Then, a multi-task prediction is performed using the rotating detection head, DetectionHead. The rotating detection head predicts the orientation bounding box parameters, class probability, and target confidence for each preset anchor point, which together construct the original prediction data.
[0054] The above-mentioned oriented bounding box parameters are represented as (cx,cy,w,h,θ), where (cx,cy) are the coordinates of the center point, (w,h) are the width and height of the box, and θ is the rotation angle of the box.
[0055] The above category probabilities are represented by a vector of length 5, which corresponds to the confidence scores of the five categories: measurement item, theoretical value, reference value, text description, and table.
[0056] The target confidence level mentioned above represents the overall confidence level that a target exists at that location.
[0057] 3. First, perform confidence level screening based on the original prediction data:
[0058] First, most background or low-quality predictions are filtered out based on the target confidence level and category probability;
[0059] Next, the bounding box is decoded to convert the predicted offset parameters (Δcx, Δcy, Δw, Δh, Δθ) into absolute orientation bounding box coordinate information in the image coordinate system.
[0060] Then, non-maximum suppression is performed to suppress predicted boxes with high overlap and belonging to the same category, retaining the most reliable one to solve the problem of duplicate detection. For rotated boxes, the intersection-union ratio of the rotated boxes needs to be calculated.
[0061] Finally, after filtering and deduplication, a list of candidate detection units is obtained. Each unit contains the decoded oriented bounding box and its corresponding most likely class and score.
[0062] 4. Based on the candidate detection unit list, encapsulate each unit in the list into a structured format, and finally output the detection result data. The detection result data is a data structure, where each element represents a detection unit. Each detection unit includes the coordinate information of the oriented bounding box, the drawing element category, and the category confidence score.
[0063] S3. Based on the detection results data, perform information extraction processing on the image region corresponding to each detection unit:
[0064] For the coordinates of the oriented bounding box of each detection unit, the corresponding region image data is extracted from the original image data:
[0065] If it is a measurement item, theoretical value, or reference value, the region image data is input into the fine-tuned visual language model (VLM) to obtain the first structured data and generate confidence scores.
[0066] If it is a text description, the region image data is input into the Visual Language Model (VLM) to obtain the second structured data and generate confidence scores;
[0067] If it is a table, the regional image data is input into the optical character recognition module to obtain text data, and then the text data is input into the large language model for key-value pair extraction to obtain the third structured data;
[0068] The specific steps for obtaining the first structured data mentioned above are as follows:
[0069] 1. Using the directional bounding box coordinates ([cx,cy,w,h,θ]) in the detection unit, perform affine transformation or rotation cropping on the original image to obtain the corrected region image data.
[0070] 2. Adjust the region image to the input size of the VLM model and normalize it to obtain the preprocessed image tensor.
[0071] 3. Input the preprocessed image tensor into the fine-tuned VLM. The VLM model extracts image features through a visual encoder and then generates a structured JSON string through a text decoder or a specific output header. The JSON string output by the VLM is parsed into a structured object inside the program, such as a Python dictionary, forming the first structured data and generating confidence scores.
[0072] The fields in the first structured data differ depending on the category:
[0073] Measurement items include nominal value, upper tolerance, lower tolerance, tolerance type, geometric symbol, and quantity.
[0074] Theoretical values include theoretical dimension values and basic dimension markings.
[0075] Reference values include reference dimension values and reference markings.
[0076] The steps for obtaining the second structured data are as follows:
[0077] 1. Consistent with the first structured data step described above, a preprocessed image tensor is obtained.
[0078] 2. Using a VLM specifically tuned for text recognition, recognize the text content and output a plain text string.
[0079] 3. The identified text string is encapsulated into a structured JSON object to form the second structured data and generate confidence scores.
[0080] The steps for obtaining the third type of structured data mentioned above are as follows:
[0081] 1. Consistent with the first structured data step described above, regional image data is obtained.
[0082] 2. Input the region image data into the OCR module. The OCR module first performs text detection, obtaining multiple horizontal text boxes and their text content, forming the raw OCR text data. The raw OCR text data is a list, where each element contains the coordinates of the text box and the recognized text.
[0083] 3. Sort the original OCR text data according to the spatial position of the text boxes to form an ordered text sequence.
[0084] 4. Input the ordered text sequence into the Large Language Model (LLM). Using the prompt word "Prompt", the LLM understands the table structure and extracts key-value pairs.
[0085] 5. LLM outputs a structured JSON string representing the key-value pair content of the table.
[0086] 6. Parse the JSON string output by the LLM to form third-party structured data.
[0087] The specific steps for obtaining the confidence score are as follows:
[0088] When VLM generates a JSON string, each token generation step has a probability distribution. For example, when generating the number 1, the model assigns the token 1 with a probability of 0.95; when generating 0, the probability is 0.98.
[0089] The overall confidence level can be approximated as the geometric mean or minimum of the probabilities of all key tokens. The confidence level for each field is calculated by determining the probability of the token sequence corresponding to each field value. The overall confidence level of the JSON is the average of the confidence levels of all key fields. The formula is as follows:
[0090] Conf_vlm=Aggregate(P(token_i|image, prefix token_{1:i-1})), where i is the key token of the structured content generated by traversal.
[0091] S4. Perform confidence verification processing on the first and second structured data obtained in S3:
[0092] Obtain the generation confidence of the Visual Language Model (VLM) for the first and second structured data;
[0093] When the generated confidence level is lower than the preset threshold, the optical character recognition module is called to perform character recognition on the corresponding area image data to obtain OCR verification data.
[0094] The first or second structured data is corrected based on the OCR verification data to obtain the corrected structured data.
[0095] The specific steps for generating the confidence threshold are as follows:
[0096] Read the generation confidence score attached to each structured data;
[0097] The generated confidence score is compared with a preset confidence score threshold.
[0098] When the generated confidence level is greater than or equal to the confidence threshold, the VLM output is considered reliable, directly marked as the final valid data, and proceeds to S6.
[0099] If the generated confidence level is less than the confidence level threshold, the VLM output is considered questionable, triggering the subsequent OCR verification process.
[0100] The specific OCR verification process is as follows:
[0101] 1. OCR character recognition:
[0102] The corresponding region image data is input into the OCR engine, and the OCR engine outputs the OCR-recognized text of that region image, i.e., the OCR verification data.
[0103] For example, for the image "4xφ10±0.1", the OCR might output "4xØ10+0.1".
[0104] 2. Data alignment and comparison:
[0105] Align the structured data output by VLM with the OCR verification data.
[0106] For example, the field values in {"quantity":"4","nominal":"10"} output by VLM can be compared with the "4" and "10" in the OCR string at the character level or semantic level.
[0107] 3. Decision Revision and Generation:
[0108] Automatic correction: For simple, ruleable inconsistencies, such as the number "1" being recognized as the letter "I" by VLM, the incorrect field in the VLM output is automatically replaced with the correct character from the OCR verification data.
[0109] Conflict Arbitration and Generation: For complex conflicts, the system combines the confidence levels of both parties and the logical relationships between fields. For example, the tolerance value should be much smaller than the nominal value for arbitration, or it can be marked as "requiring manual review".
[0110] Finally, a corrected structured data set is generated as the determining information for entering S6.
[0111] S5. Based on the spatial location information of the measurement item detection unit in the detection result data, perform bubble numbering and allocation processing:
[0112] Based on the coordinate information of the directional bounding box of the measurement item detection unit, a unique bubble number is assigned to each measurement item according to the preset spatial sorting rules;
[0113] Generate number mapping data containing the relationship between bubble numbers and the positions of corresponding measurement items.
[0114] The specific steps for numbering bubbles are as follows:
[0115] 1. Traverse all detection units in the detection result data; retain only the detection units whose category label is a measurement item; thus obtaining the set of measurement item detection units;
[0116] 2. Establish a unified coordinate system:
[0117] With the top left corner of the drawing image as the origin (0,0), the positive X-axis direction is to the right and the positive Y-axis direction is downward;
[0118] If the detection unit uses normalized coordinates (0-1), then multiply by the image size to convert to pixel coordinates;
[0119] If the detection unit uses absolute pixel coordinates, then use them directly and output a set of standardized coordinates.
[0120] 3. For the rotating frame OBB, use the center point of the bounding box to determine a stable reference point, then calculate the angle of each detection unit reference point relative to the center point, and finally number them clockwise in order of increasing angle.
[0121] S6. Fuse the data and output the final result:
[0122] The corrected structured data, third-party structured data, and detection result data are correlated and matched; the correlated structured data is integrated with the numbering mapping data; and the final output result, including a bubble annotation layer and a structured detection list, is generated. Details are as follows:
[0123] 1. Data association and matching
[0124] First, obtain the detection result data of S2, the corrected structured data of S4, and the third structured data of S3;
[0125] Then, each detection unit in the detection result data is traversed, and its detection unit ID is used as an index to find the corresponding parsing result in the corrected structured data and the third structured data.
[0126] The found analytical results are combined with the geometric information (bounding boxes and categories) in the detection results data to form an enhanced detection record.
[0127] 2. Integration with number mapping data
[0128] Iterate through the enhanced inspection record list. For records categorized as measurement items, look up the corresponding bubble number in the number mapping data based on its inspection unit ID. Write this bubble number field into the enhanced inspection record list. For records not categorized as measurement items, this field is either empty or assigned a special flag. Finally, output a complete database of numbered drawing elements.
[0129] 3. Generate the final output result
[0130] First, based on the numbered database, all measurement item records are filtered out. Then, according to the standard format of the FAI report, the key information of each measurement item is extracted and arranged to output a structured inspection list.
[0131] Then, based on the numbered database and the coordinates of each record on the original drawing image, a circular bubble label is drawn at the corresponding position. For text descriptions and tabular records, no bubble is drawn, or a different style of label is drawn. Finally, a drawing file with automatically generated bubble numbers is output.
[0132] Example 2
[0133] Based on Example 1, this embodiment further improves the information extraction and processing in step S3 by using a multimodal fusion mechanism.
[0134] The specific steps are as follows:
[0135] S3. Perform information extraction processing on the image region corresponding to each detection unit:
[0136] Based on the category of the detection unit, the corresponding region image data is extracted from the original image data;
[0137] If the category is a measurement item, theoretical value, or reference value, the region image data is input into an improved visual language model (VLM). This model adds a cross-modal attention feature fusion module to the traditional VLM. The specific process is as follows:
[0138] 1. Visual feature extraction: Extract feature maps of the region image through a visual encoder, such as ViT or CNN, to obtain a visual feature tensor;
[0139] 2. Text Feature Extraction: Input the region image into a lightweight OCR module or the text recognition branch built into VLM to identify the character sequence in the image and its corresponding bounding box coordinates, and convert each character or word into a text feature vector through the text encoder BERT;
[0140] 3. Cross-modal attention fusion: The visual feature tensor is flattened into a sequence and input together with the text feature sequence into a multi-head cross-attention layer. The multi-head cross-attention layer calculates the attention weights between visual and text features to achieve deep interaction and alignment between the two types of features, thereby obtaining the fused multimodal features.
[0141] For example, for each visual location, its attention score with all text features is calculated to obtain the enhanced visual features, and similarly, the enhanced text features can be obtained.
[0142] 4. Joint Decoding: The fused multimodal features are weighted and summed, then input into the text decoder to generate structured data token by token. This structured data includes fields such as nominal value, tolerance, geometric symbol, and quantity, and is output in JSON format; details are as follows:
[0143] During the decoding phase, the model generates tokens for the structured data one by one in an autoregressive manner. Suppose that the structured data to be generated consists of N key tokens, and the generation of each token is based on the generated prefix tokens and the fused multimodal features.
[0144] For example, the key token is composed of {, "nominal", :, "10", ,, "tolerance", :, "+0.1",}.
[0145] For the t-th token yt, the model outputs a probability distribution P(yt∣y<t,image,text) through a Softmax layer, selects the token with the highest probability as the output, and records the generation probability pt = maxP(yt) of this token, which is used to reflect the certainty degree of the model for the selected token in the current step.
[0146] In the cross-modal attention fusion module, the hidden state at each decoding step t interacts with visual features and text features to generate an attention weight distribution, which is used to represent the degree of attention to each region of visual features and each position of text features in the current step; the concentration of attention weights is used to measure whether the information relied on by the model during decision-making is clear.
[0147] The attention concentration ct of the token yt is obtained through the following formula:
[0148] ct = 1 - H(a t ) / logM
[0149] where H(at) is the information entropy of the attention weights and M is the length of the attention distribution; the value range of ct is [0, 1], and the larger the ct value, the more concentrated the attention and the more stable and reliable the model decision.
[0150] 5. Generation confidence calculation: During the decoding process, the prediction probabilities of each key token are aggregated to obtain the generation confidence of this structured data, as follows:
[0151] For each decoding step, calculate the contribution degrees of the visual modality and the text modality to the generation of the current token. Let the weighted sum of the visual attention weights (a t vis ) be (w t vis ), and the weighted sum of the text attention weights (a t text ) be (w t text ), satisfying (w t vis + w t text = 1), then the modality contribution degree is the larger of the two: M t = max(w t vis , w t text )
[0152] where (m t ) is larger, indicating that the current token mainly relies on a single modality and the decision-making basis is clear; if (m tIf the value is close to 0.5, it indicates that the model depends on two modes simultaneously, which may pose a risk of inconsistency between modes.
[0153] The visual modality refers to the pixel-level image features of the engineering drawings, that is, the visual information contained in the drawings as two-dimensional images, including the graphic structure, spatial layout, and rotation angle of lines, shapes, textures, symbols, etc.
[0154] The text modality is the character sequence and its semantic information identified in the engineering drawings, that is, the text content marked on the drawings, including numbers, letters, symbols, and the engineering semantics formed by their combination.
[0155] If the primary mode corresponding to the current token, i.e., the mode with the highest contribution, matches the expected mode, then a higher confidence level is assigned. This is achieved through (E(y)). t ) ϵ {vis,text}) Pre-defined modal expectation mapping, thereby calculating the consistency score:
[0156]
[0157] Where 1[.] is an indicator function.
[0158] Combined with the token generation probability (p) t ), attention concentration (c t ) and modal consistency (s t Define token (y) t The overall confidence level of )
[0159]
[0160] in( ϵ [0,1]) is a penalty coefficient used to appropriately reduce the weight of tokens with inconsistent modes.
[0161] Structured data consists of multiple logical fields. Let the set of token indices corresponding to a certain field (f) be (T). f If the field confidence is defined as the geometric mean of the combined confidence of these tokens, the geometric mean is used to reflect the overall reliability of all tokens in the field and is sensitive to low-confidence tokens.
[0162] For the entire structured data object, its overall confidence score is defined as the weighted average of the confidence scores of all key fields. The weights are preset according to the importance of the fields; for example, nominal values and tolerance values have higher weights, while structural symbols have lower weights. The weighted average is calculated using the following formula:
[0163]
[0164] Among them (w) fThe weight is the field weight. Alternatively, you can simply take the minimum confidence score of all key fields to ensure that the overall reliability is not lower than that of the worst field.
[0165] Finally, the generated JSON structured data and overall confidence score (conf) will be used. global ) and confidence scores for each field (conf) (f) field Output them together for use in the aforementioned step S4.
[0166] Example 3
[0167] Based on Example 2, this embodiment proposes an improvement to the confidence verification process in step S4, based on multi-level confidence assessment and multi-source data fusion. By introducing field-level confidence, multi-level threshold judgment, context consistency verification, and an engineering rule base, it achieves refined verification and correction of the structured data generated by VLM. The specific implementation steps are as follows:
[0168] S4. Perform multi-level confidence assessment and fusion verification on the first and second structured data, specifically including the following sub-steps:
[0169] S4.1 During the generation of structured data using VLM, record the generation probability of each key field, such as "nominal value," "tolerance," "geometric symbol," "quantity," and "text content," corresponding to the token sequence. Aggregate these fields to obtain the confidence level (conf) for each. field At the same time, the overall confidence level of the entire structured data is preserved. global ).
[0170] S4.2 Based on the accuracy requirements of the engineering drawings and historical data statistics, three confidence thresholds are preset:
[0171] High threshold (T) high ): This indicates that the field is extremely reliable and can be used directly.
[0172] The middle threshold (T) mid ): This indicates that the field may have a minor error and OCR auxiliary verification needs to be triggered.
[0173] Low threshold (T) low ): This indicates that the field has low confidence and requires context consistency verification, and may be marked for manual review.
[0174] S4.3 For each key field in the structured data :
[0175] If (conf) field ≥ T high If the value is not specified, then the field value is directly marked as the final valid value, without any additional processing.
[0176] If (T) mid ≤ conf field < T high If this occurs, OCR auxiliary verification will be triggered:
[0177] 1. Call the OCR module on the region image corresponding to the detection unit to obtain OCR-recognized text.
[0178] 2. Perform character-level comparison between the OCR text and the field values output by VLM, and calculate the edit distance or character matching rate.
[0179] 3. If the OCR text contains characters that are highly consistent with the VLM output, and the OCR text itself has a high character recognition confidence, then the character at the corresponding position in the OCR result will be used to correct the VLM output; if there is a conflict, such as the number "1" and the letter "I", then the OCR character confidence and the VLM field confidence will be weighted and fused, and the source with the higher confidence will be selected as the final value.
[0180] 4. Record the corrected field values and their combined confidence scores, and take the weighted average of the two confidence scores.
[0181] If (conf) field < T mid If so, then further context consistency checks will be performed:
[0182] 1. Engineering Rule Base Validation: This function calls a pre-defined engineering rule base to check if the field value conforms to common engineering principles. For example, nominal values are typically positive, tolerance values should be much smaller than nominal values, and geometric symbols should match the annotation type. If a rule is violated, automatic correction is triggered.
[0183] For example, replacements or labels can be made based on the OCR results or the values of neighboring similar cells as "potential errors".
[0184] 2. Cross-detection unit consistency verification: For measurement items, check whether their nominal values and tolerances have a reasonable relationship with surrounding similar annotations. For example, different views of the same feature should have the same nominal value. If an anomaly is found, make a comprehensive judgment based on the OCR results and information from adjacent units, and reduce the overall confidence level of that unit.
[0185] 3. If the above verification still cannot determine the result, mark the field as "requires manual review" and highlight it in the final output.
[0186] S4.4 After processing all fields, recalculate the overall confidence score (conf) of the detection unit. finalThis is used as a weighted average of the final confidence scores for each field. If any field is marked as "requires manual review", the entire detection unit is marked as "requires manual review".
[0187] S4.5. For the third structured data obtained by OCR+LLM processing, field-level confidence levels are introduced. The OCR module can output the recognition confidence level of each character, and the LLM can also output the generation probability of each field when generating key-value pairs. Obviously, the above embodiments are merely examples to clearly illustrate the embodiments of the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all embodiments here. However, these obvious variations or modifications derived from the spirit of the present invention are still within the protection scope of the present invention.
Claims
1. A method for automatic bubble annotation and information extraction in engineering drawings based on a hybrid visual model, characterized in that, Includes the following steps: S1. Input the original image data of the engineering drawings; S2. Based on the rotating target detection model, the original image data is detected and classified to obtain detection result data including multiple detection units. Each detection unit includes directional bounding box coordinate information, drawing element category and category confidence. The drawing element categories include measurement items, theoretical values, reference values, text descriptions, and tables; the measurement items include all dimensions with tolerances, the theoretical values correspond to the theoretically correct dimensions marked with boxes, and the reference values correspond to the reference dimensions marked with brackets; S3. Perform information extraction processing on the image region corresponding to each detection unit: Based on the category of the detection unit, the corresponding region image data is extracted from the original image data; If the category is a measurement item, theoretical value, or reference value, the region image data is input into a fine-tuned visual language model (VLM) to obtain the first structured data and the corresponding generated confidence score. If the category is text description, the region image data is input into the Visual Language Model (VLM) to obtain the second structured data and the corresponding generation confidence score. If the category is a table, the regional image data is processed sequentially through the Optical Character Recognition (OCR) module and the Large Language Model (LLM) to obtain the third structured data. S4. Perform confidence verification processing on the first structured data and the second structured data: When the generated confidence level is lower than a preset threshold, the optical character recognition (OCR) module is invoked to recognize the corresponding region image data to obtain OCR verification data, and the first structured data or the second structured data is corrected accordingly to obtain corrected structured data. When the generated confidence level is higher than or equal to the preset threshold, the first structured data or the second structured data is directly used as the corrected structured data. S5. Based on the spatial location information of the detection units categorized as measurement items in the detection result data, assign a unique bubble number to each measurement item detection unit according to a preset spatial sorting rule, and generate number mapping data. S6. The corrected structured data, the third structured data, and the detection result data are correlated and matched to obtain an enhanced detection record; then the enhanced detection record is integrated with the number mapping data to generate a final output result including a bubble annotation layer and a structured detection list.
2. The method for automatic bubble annotation and information extraction in engineering drawings based on a hybrid visual model according to claim 1, characterized in that, The rotating target detection model described in S2 is a one-stage target detection network that supports rotating bounding box prediction. Its output is oriented bounding box parameters (cx,cy,w,h,θ), where (cx,cy) are the coordinates of the center point, (w,h) are the width and height, and θ is the rotation angle. The test result data is obtained through the following steps: S21. Standardize the original image data to form the model input tensor; S22. Extract image features through the rotating target detection model and perform multi-task prediction to obtain raw prediction data including oriented bounding box parameters, class probability and target confidence. S23. Perform confidence screening, frame decoding, and non-maximum suppression processing on the original prediction data to obtain a candidate detection unit list; S24. Encapsulate the candidate detection unit list into structured detection result data.
3. The method for automatic bubble annotation and information extraction in engineering drawings based on a hybrid visual model according to claim 2, characterized in that, The nonmaximum suppression process described in S23 is calculated using the crossover-union ratio of the rotated frame.
4. The method for automatic bubble annotation and information extraction in engineering drawings based on a hybrid visual model according to claim 3, characterized in that, In S3, based on the Visual Language Model (VLM), during the generation of structured data, the predicted probabilities of key tokens in the output sequence are aggregated and calculated to obtain the generation confidence of the first or second structured data.
5. The method for automatic bubble annotation and information extraction in engineering drawings based on a hybrid visual model according to claim 4, characterized in that, In S3, the first structured data is a structured object including nominal value, tolerance, geometric symbol, and quantity fields; the second structured data is a structured object including the recognition of text content; and the third structured data is key-value pair structured data extracted from OCR text by the Large Language Model (LLM).
6. The method for automatic bubble annotation and information extraction in engineering drawings based on a hybrid visual model according to claim 5, characterized in that, The confidence verification process described in S4 further includes comparing the OCR verification data with the first or second structured data to be verified at the character level or semantic level, and performing automatic correction or conflict arbitration based on the comparison results.
7. The method for automatic bubble annotation and information extraction in engineering drawings based on a hybrid visual model according to claim 6, characterized in that, The preset spatial sorting rule described in S5 is a sorting rule from left to right, from top to bottom, or clockwise.
8. The method for automatic bubble annotation and information extraction in engineering drawings based on a hybrid visual model according to claim 7, characterized in that, The clockwise circular sorting rule described in S5 includes the following steps: Determine the center point of the drawing or area; Calculate the angle of the reference point of each measurement item detection unit relative to the center point; Number them in ascending order of angle.
9. The method for automatic bubble annotation and information extraction in engineering drawings based on a hybrid visual model according to claim 8, characterized in that, The association matching described in S6 uses the detection unit ID of each detection unit in the detection result data as an index to find and merge the corresponding corrected structured data or third structured data to form the enhanced detection record.
10. The method for automatic bubble annotation and information extraction in engineering drawings based on a hybrid visual model according to claim 9, characterized in that, In S6, all measurement item records are filtered from the integrated data, and key information is extracted and arranged according to the preset report format to generate the structured inspection list. Based on the location information of each record in the integrated data, a bubble annotation graphic with bubble number is drawn at the corresponding position on the drawing, thereby generating the bubble annotation layer.