Engineering drawing PMI recognition method based on YOLO and multi-modal large model
By using a YOLO-based and multimodal large model approach to preprocess and segment engineering drawings, and combining a visual language model and an OCR engine, the problems of distributed dispersion, symbol recognition, and format differences in PMI recognition of engineering drawings are solved, achieving high-precision and efficient automated PMI recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- WUXI XUELANG DIGITAL TECH CO LTD
- Filing Date
- 2026-04-29
- Publication Date
- 2026-05-29
AI Technical Summary
Existing engineering drawing PMI recognition technologies face problems such as scattered PMI distribution, difficulty in recognizing professional symbols, large differences in annotation formats, and severe interference between lines, resulting in low recognition accuracy, low efficiency, and high cost.
A method based on YOLO and multimodal large models is adopted. A raster image with a unified format is generated through preprocessing. The PMI region is segmented using an object detection model and a rotated box detection model. The recognition and verification are performed by combining a visual language model and a lightweight OCR engine, so as to achieve accurate detection and structured extraction of the PMI region.
It improves the accuracy and efficiency of PMI recognition for engineering drawings, reduces recognition complexity and cost, and achieves reliability and consistency in automated recognition.
Smart Images
Figure CN122116407A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of engineering drawing recognition technology, and more specifically, to a method for PMI recognition of engineering drawings based on YOLO and multimodal large models. Background Technology
[0002] Engineering drawings are the core carriers of design intent and manufacturing requirements in the mechanical manufacturing field. The Product Manufacturing Information (PMI) embedded in them directly determines the processing technology, quality inspection standards, and assembly accuracy of parts. With the development of intelligent manufacturing and digital factories, the automated and intelligent extraction of PMI information from engineering drawings has become an inevitable trend in the industry. However, due to the high complexity and specialization of engineering drawings, existing automated PMI recognition technologies still face many challenges: First, PMIs are scattered and at varying angles on engineering drawings, making it difficult for horizontal bounding box-based target detection methods to accurately capture their entire area; second, PMIs contain a large number of professional symbols and special structures, and Optical Character Recognition (OCR) technology lacks the ability to understand the semantics of these symbols; third, different companies and designers use different annotation formats and expression methods, lacking a unified recognition standard; fourth, dense projection lines, section lines, center lines, and other lines in the drawings cause serious visual interference to PMIs, further increasing the difficulty of accurate extraction.
[0003] To address the aforementioned issues, existing technologies attempt to combine traditional image processing with machine learning for PMI recognition.
[0004] However, existing methods require designing separate feature extraction and classification models for each PMI type, which leads to problems such as complex system architecture, high maintenance costs, and weak generalization ability. Summary of the Invention
[0005] The purpose of this application is to provide a PMI recognition method for engineering drawings based on YOLO and multimodal large models, which can improve the recognition accuracy, efficiency and reliability of product manufacturing information in engineering drawings, and reduce the complexity and cost of automated recognition of product manufacturing information.
[0006] The embodiments of this application are implemented as follows: A first aspect of this application provides a method for PMI recognition of engineering drawings based on YOLO and multimodal large models, the method comprising: Preprocess the engineering drawing files to obtain the raster image corresponding to the engineering drawing files; A pre-built target detection model is used to detect the outer frame boundary, title bar region, and main view region of the raster image, and the raster image is segmented to obtain the title bar region sub-image and the view region sub-image of the raster image. Based on a pre-built rotated bounding box detection model, PMI region detection is performed on the view region sub-image to obtain multiple candidate PMI instances contained in the view region sub-image and the detection information of each candidate PMI instance. The detection information includes: rotated bounding box coordinates. Redundancy is eliminated from multiple candidate PMI instances to obtain at least one target PMI instance. Based on the rotated bounding box coordinates of each target PMI instance, the view region sub-image is cropped by affine transformation to obtain the corrected slice image of each target PMI instance. The corrected slice images of each target PMI instance are input into a pre-built visual language model to obtain the structured fields corresponding to each target PMI instance, and the title bar area sub-image is input into the visual language model to extract general manufacturing information. The structured fields corresponding to each target PMI instance are integrated with general manufacturing information to obtain product manufacturing information from engineering drawing files.
[0007] As one possible implementation, the process of constructing a rotated box detection model is as follows: Large-scale random scaling and random rotation are used to augment the data of the original detection model; The rotated bounding box is modeled as a two-dimensional Gaussian distribution, and the Bhattacharyya distance is calculated based on the target algorithm to determine the bounding box regression loss of the original detection model. The target algorithm includes a dynamic nonlinear aspect ratio penalty factor. Based on the bounding box regression loss, the model parameters of the original detection model are adjusted to obtain the rotated box detection model.
[0008] As one possible implementation, the above-mentioned target algorithm includes: Through formula Determine the dynamic nonlinear aspect ratio penalty factor; Through formula Determine the bounding box regression loss; in, Used to represent the aspect ratio penalty factor for nonlinearity Used to represent the width of the predicted rotation box. Used to represent the height of the predicted rotation box Used to represent preset adjustment hyperparameters, Used to represent the Bach distance between the predicted and actual two-dimensional Gaussian distributions corresponding to the rotated boxes. As one possible implementation, the aforementioned detection information also includes: confidence scores and PMI instance categories. Redundancy is eliminated from multiple candidate PMI instances to obtain at least one target PMI instance, including: Based on the PMI instance category of each candidate PMI instance, the candidate PMI instances are grouped and sorted in descending order of confidence score to obtain multiple initial detection queues. The rotated bounding boxes of each candidate PMI instance in each initial detection queue are mapped to a two-dimensional Gaussian distribution. Starting from the highest confidence score in each initial detection queue, calculate the Bhattachary distance between two rotated bounding boxes to determine the overlap between them. Based on the degree of overlap, remove rotated bounding boxes with low confidence scores; Repeat the iteration until all initial detection queues are cleared to obtain the target detection queue, which includes multiple target rotated bounding boxes; Each candidate PMI instance corresponding to the rotating bounding box of each target in the target detection queue is treated as a target PMI instance.
[0009] As one possible implementation, based on overlap, rotated bounding boxes with low confidence scores are discarded, including: If the overlap is greater than or equal to a preset threshold, the two rotated bounding boxes are identified as duplicate detections, and the rotated bounding box with the lower confidence score is removed.
[0010] As one possible implementation, the corrected sliced images of each target PMI instance are input into a pre-built visual language model to obtain the structured fields corresponding to each target PMI instance, including: Obtain the cross-modal semantic parsing results output by the visual language model; A lightweight character recognition engine is invoked to extract numerical characters from the corrected slice images of each target PMI instance in order to obtain the numerical fields corresponding to each target PMI instance. Based on the numeric fields, the cross-modal semantic parsing results are corrected, and spatial anchoring is performed based on the center coordinates of the character bounding boxes of each numeric character in the numeric fields and the attribute fields contained in the cross-modal semantic parsing results to obtain the structured fields corresponding to each target PMI instance.
[0011] As one possible implementation, the cross-modal semantic parsing results are modified based on the numeric field, including: If the confidence level of a numeric field is higher than the preset security threshold, and the numeric field conflicts with the cross-modal semantic parsing result, then the numeric field in the cross-modal semantic parsing result is corrected based on the numeric field.
[0012] As one possible implementation, based on the rotated bounding box coordinates of each target PMI instance, an affine transformation is performed on the view region sub-image to crop it, resulting in a corrected slice image of each target PMI instance, including: When cropping the rotated bounding boxes of each target PMI instance, each side of the rotated bounding box of each target PMI instance is expanded outward by a preset number of pixels based on the coordinates of the rotated bounding box of each target PMI instance. During the affine transformation process, the tilt angle of each expanded rotated bounding box is corrected to obtain the corrected slice image of each target PMI instance.
[0013] A second aspect of this application provides a computer device comprising: a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the computer program, it implements the steps of the engineering drawing PMI recognition method based on YOLO and multimodal large model described in the first aspect.
[0014] A third aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the engineering drawing PMI recognition method based on YOLO and multimodal large model described in the first aspect.
[0015] The beneficial effects of the embodiments of this application include: This application provides a method for PMI recognition in engineering drawings based on YOLO and a multimodal large model. The method receives engineering drawing files in various formats from a user and preprocesses these files to obtain raster images of the same format. A pre-built object detection model is used to detect the outer frame boundary, title bar region, and main view region of the raster image, and the raster image is segmented to obtain title bar region sub-images and view region sub-images. A pre-built rotated bounding box detection model is used to detect PMI regions in the view region sub-image to obtain multiple candidate PMI instances contained in the view region sub-image, as well as the rotated bounding box coordinates and PMI values of each candidate PMI instance. The system collects detection information such as instance category and confidence score, and performs redundancy removal on multiple candidate PMI instances based on the detection information of each candidate PMI instance to obtain at least one target PMI instance. Based on the rotated bounding box coordinates of each target PMI instance, a full affine transformation is performed on the view area sub-image to obtain the corrected slice image of each target PMI instance. The corrected slice image of each target PMI instance is input into a pre-built visual language model to obtain the structured fields corresponding to each target PMI instance, and feature extraction is performed on the title bar area sub-image to obtain general manufacturing information. The structured fields corresponding to each target PMI instance are integrated with the general manufacturing information to obtain the product manufacturing information of the engineering drawing file. The process involves first converting the engineering drawing file input to the target detection model into a unified format raster image, then segmenting the raster image into regions. An optimized rotation detection model is then used to perform PMI region detection and slice segmentation on the segmented view region sub-images. Finally, a visual language model and a lightweight optical character recognition engine are used to perform PMI content recognition and double verification on the corrected slice images and title bar region sub-images, thereby improving the accuracy and efficiency of product manufacturing information recognition. This approach enhances the accuracy, efficiency, and reliability of product manufacturing information recognition on engineering drawings while reducing the complexity and cost of automated product manufacturing information recognition. Attached Figure Description
[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 An architecture diagram of a product manufacturing information identification system for engineering drawings is provided as an embodiment of this application; Figure 2A flowchart of the first method for PMI recognition of engineering drawings based on YOLO and multimodal large models provided for embodiments of this application; Figure 3 A mapping table for multiple PMI instances is provided for embodiments of this application; Figure 4 A flowchart of a second method for PMI recognition in engineering drawings based on YOLO and multimodal large models, provided for embodiments of this application; Figure 5 A flowchart of the third method for PMI recognition of engineering drawings based on YOLO and multimodal large models provided in this application embodiment; Figure 6 A flowchart of the fourth method for PMI recognition of engineering drawings based on YOLO and multimodal large models provided in this application embodiment; Figure 7 A collaborative architecture diagram of a detection model and a visual language model is provided for embodiments of this application; Figure 8 An example diagram illustrating the identification result of product manufacturing information provided in this application embodiment; Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0019] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
[0020] Traditional PMI extraction relies heavily on manual reading and input. Operators need to review each engineering drawing, identify and transcribe hundreds of manufacturing parameters scattered in the views and title blocks. This not only results in low efficiency but is also affected by subjective experience, visual fatigue, and unclear annotations, leading to a high error rate.
[0021] With the development of intelligent manufacturing and digital factories, the automated and intelligent extraction of PMI information from engineering drawings has become an inevitable trend in the industry. However, due to the high complexity and specialization of engineering drawings, existing automated PMI recognition technologies still face many challenges: First, PMIs are scattered and have varying angles on engineering drawings, making it difficult for horizontal bounding box-based target detection methods to accurately capture their complete areas; second, PMIs contain a large number of professional symbols and special structures, and Optical Character Recognition (OCR) technology lacks the ability to understand the semantics of these symbols; third, different companies and designers use different annotation formats and expression methods, lacking a unified recognition standard; fourth, dense projection lines, section lines, center lines, and other lines in the drawings cause serious visual interference to PMIs, further increasing the difficulty of accurate extraction.
[0022] To address these issues, existing technologies attempt to combine traditional image processing with machine learning for PMI identification. However, these methods require designing separate feature extraction and classification models for each PMI type, leading to complex system architecture, high maintenance costs, and weak generalization capabilities.
[0023] To address this, this application provides a method for PMI recognition in engineering drawings based on YOLO and a multimodal large model. The method involves receiving engineering drawing files provided by a user and preprocessing them to obtain a raster image in a unified format. A pre-built object detection model is used to detect the outer frame boundary, title bar region, and main view region of the raster image, and the raster image is segmented to obtain a title bar region sub-image and a view region sub-image. Based on a pre-built rotating frame detection model, PMI region detection is performed on the view region sub-image to obtain multiple candidate PMI instances contained within the view region sub-image and the PMI instances of each candidate PMI instance. The process involves: detecting redundant PMI instances from multiple candidate instances to obtain at least one target PMI instance; cropping the view region sub-image using affine transformation based on the rotated bounding box coordinates of each target PMI instance to obtain corrected slice images of each target PMI instance; inputting the corrected slice images of each target PMI instance into a pre-built visual language model to obtain the structured fields corresponding to each target PMI instance; inputting the title bar region sub-image into the visual language model to extract general manufacturing information from the engineering drawing file; and integrating the structured fields corresponding to each target PMI instance with the general manufacturing information to obtain the product manufacturing information of the engineering drawing file. This approach improves the accuracy, efficiency, and reliability of product manufacturing information recognition in engineering drawings while reducing the complexity and cost of automated product manufacturing information recognition.
[0024] The following description, in conjunction with the accompanying drawings, provides a detailed explanation of the PMI recognition method for engineering drawings based on YOLO and multimodal large models provided in this application.
[0025] Figure 1 A product manufacturing information identification system architecture diagram for engineering drawings provided in this application is shown below. Figure 1 The product manufacturing information recognition system for engineering drawings provided in this application includes an input layer, a preprocessing layer, a detection layer, a recognition layer, and an output layer. The input layer receives engineering drawing files of various formats input by the user. These files can be vector formats such as Portable Document Format (PDF), Drawing (DWG), and Drawing Exchange Format (DXF), or they can be Portable Network Graphics (PNG), Joint Photographic Experts Group (JPG), or Tagged Image File format. The system can accept files in raster formats such as TIFF (Transformer in Format), as well as scanned documents, photographic images, and other image formats. The preprocessing layer converts user-input engineering image files into raster images of a unified format. This preprocessing layer performs rasterization, grayscale / denoising, and size normalization on the received engineering drawing files, ensuring that the image files entering the detection layer are raster image files of a unified format. The detection layer deploys a target detection model and a rotating frame detection module. It detects and segments the received raster images using a frame segmentation module, a PMI region detection module, and a slice segmentation module, outputting corrected slice images of each PMI instance contained in the raster image to the recognition layer. The recognition layer deploys a visual language model and a lightweight character recognition engine. This layer identifies PMI content, extracts general manufacturing information, and performs verification and integration, outputting structured fields and general manufacturing information to the output layer. The output layer processes the received structured fields and general manufacturing information and outputs product manufacturing information according to the user-specified data pattern, facilitating integration with downstream manufacturing execution systems, process planning systems, and quality inspection systems.
[0026] Specifically, if the engineering drawing file input by the user is in PDF format, the preprocessing layer will use high resolution (such as a resolution of no less than 300 DPI) to rasterize the engineering drawing file to obtain the corresponding raster image; if the engineering drawing file input by the user is in vector format such as DWG or DXF, the preprocessing layer will call the Computer-Aided Design (CAD) conversion engine to convert the engineering drawing file into a raster image; if the engineering drawing file input by the user is in image format such as PNG, JPG, or TIFF, the preprocessing layer will directly normalize the engineering drawing file to obtain the corresponding raster image.
[0027] It should be noted that the preprocessing layer's preprocessing of engineering drawing files specifically includes: resolution normalization (e.g., unifying the resolution of engineering drawing files to 300 DPI), image grayscale conversion, adaptive binarization noise reduction, and size adjustment (e.g., adjusting the short side of the engineering drawing file to no less than 1024 pixels), to ensure that the engineering drawing files input to the detection layer are consistent.
[0028] Optionally, a target detection model is pre-deployed in the frame segmentation module of the detection layer. The detection layer can detect the outer frame boundary, title bar area, and main view area of the received raster image through the frame segmentation module, and segment the view area of the raster image to obtain title bar area sub-images and view area sub-images. The title bar area is usually located in the lower right corner of the engineering drawing and contains general manufacturing information such as materials, heat treatment, and tolerances; the view area contains each projected view and its associated PMI standard information. At the same time, the detection layer can also segment the technical requirements area and the title bar area through the frame segmentation module.
[0029] Furthermore, a rotated bounding box detection model is pre-deployed in the PMI region detection module of the detection layer. The detection layer can perform OBB rotated bounding box detection on the sub-image of the view region through the PMI region detection module, identify various types of candidate PMI instances contained in the view region, and output detection information such as the rotated bounding box coordinates, PMI instance category, and confidence score of each candidate PMI instance.
[0030] Furthermore, the detection layer performs redundancy removal on multiple candidate PMI instances through a slice segmentation module to obtain at least one target PMI instance. It then performs affine transformation cropping on the view region sub-image based on the rotated bounding box coordinates of each target PMI instance and performs rotation correction on each target PMI instance to obtain the corrected slice image corresponding to each target PMI instance.
[0031] Optionally, a Vision-Language Model (VLM) and a lightweight Optical Character Recognition (OCR) engine are pre-deployed in the recognition layer. Guided by structured prompts, the recognition layer uses the VLM to identify the PMI category of each target PMI instance and extracts the structured fields of each target PMI instance from the corrected slice images. Simultaneously, the recognition layer extracts general manufacturing information such as material, heat treatment, general tolerances, surface treatment, drawing number, scale, and version, as well as unstructured process annotations, from the title bar sub-image and the technical requirements area outside the title bar area.
[0032] Furthermore, the recognition layer performs field integrity verification on the structured fields of each target PMI instance based on the OCR engine, and merges and deduplicates the numerical fields extracted by the OCR engine and the cross-modal semantic parsing results output by the VLM model to obtain the final structured fields of each target PMI instance and the general manufacturing information of the engineering image file.
[0033] Optionally, the output layer integrates the structured fields of each target PMI instance, the general manufacturing information of the engineering image file, and the unstructured process annotations, and outputs the product manufacturing information recognition result of the engineering image file according to a preset data format.
[0034] Figure 2 This application provides a flowchart of a PMI recognition method for engineering drawings based on YOLO and multimodal large models. This method can be applied to computer equipment. See [link / reference]. Figure 2 This application provides a method for PMI recognition of engineering drawings based on YOLO and multimodal large models, including: S201. Preprocess the engineering drawing file to obtain the raster image corresponding to the engineering drawing file.
[0035] Optionally, the engineering drawing files in various formats input by the user are preprocessed to convert each page of the engineering drawing file into a raster image of a uniform format. The specific formats of the engineering drawing files can be PDF, DWG, DXF, PNG, JPG, TIFF, scanned documents, photographic images, etc., and this application does not impose any specific limitations on them.
[0036] Optionally, different processing methods are used for engineering drawing files of different formats to obtain raster images of uniform specifications. Specifically, if the engineering drawing file entered by the user is a PDF file, it is rasterized at a resolution of no less than 300 DPI to convert each page into a raster image; if the engineering drawing file entered by the user is a vector format such as DWG or DXF, it is rendered into a raster image through a CAD conversion engine (such as the Teigha library provided by the Open Design Alliance or the AutoCAD core engine), with a rendering resolution of no less than 300 DPI; if the engineering drawing file entered by the user is an existing raster format such as PNG, JPG, or TIFF, it is directly read and normalized.
[0037] Furthermore, the following normalization operations are performed on all raster images: the resolution is uniformly adjusted to 300 DPI; the color images are converted to grayscale; an adaptive thresholding method (such as the Otsu algorithm or the Sauvola algorithm) is used for binarization to reduce noise; and the short side size of the image is adjusted to no less than 1024 pixels (maintaining the aspect ratio), finally resulting in raster images of uniform specifications.
[0038] S202. A pre-built target detection model is used to detect the outer frame boundary, title bar area, and main view area of the raster image, and the raster image is segmented to obtain the title bar area sub-image and the view area sub-image of the raster image.
[0039] Optionally, the object detection model is based on the YOLOv12 architecture and is trained on layout elements of engineering drawings (such as the outer frame, title block, and main view). The training data for the object detection model includes a large number of raster images of engineering drawings labeled with three types of areas: the outer frame, the title block, and the main view.
[0040] Specifically, by inputting the raster image corresponding to the engineering drawing file into a pre-built target detection model, the outer bounding box, title bar area, and main view area of the raster image can be obtained. The outer bounding box is often parallel to the edge of the drawing, so a horizontal rectangle is commonly used to represent it.
[0041] Furthermore, based on the outer drawing frame boundary, the effective area of the entire engineering drawing can be determined. Then, based on the outer drawing frame boundary, the title block area, and the main view area, sub-images of the title block area and the view area are cropped from the raster image corresponding to the engineering drawing file. The view area sub-image refers to all areas within the outer drawing frame except for the title block area, typically including the main view, top view, left view, sectional view, and various PMI annotations.
[0042] S203. Based on the pre-built rotated bounding box detection model, perform PMI region detection on the view region sub-image to obtain multiple candidate PMI instances contained in the view region sub-image and the detection information of each candidate PMI instance. The detection information includes: rotated bounding box coordinates.
[0043] Optionally, the rotating frame detection model is trained on PMI symbols (such as linear dimensions, angular dimensions, radius / diameter, geometric tolerance frames, surface roughness symbols, welding symbols, datum symbols, technical requirement annotations, etc.) based on the YOLOv12 architecture.
[0044] Optionally, by inputting the view region sub-image into a pre-built rotated bounding box detection model, the rotated bounding box coordinates, PMI instance category, and confidence score of each candidate PMI instance contained in the view region sub-image can be obtained. Specifically, the rotated bounding box coordinates can be represented as (x1, y1, w, h, θ), where (x1, y1) identifies the center point coordinates of the rotated bounding box, w represents the width of the rotated bounding box, h represents the height of the rotated bounding box, and θ represents the rotation angle of the rotated bounding box.
[0045] It is worth noting that the PMI instance category is used to describe the type of candidate PMI instance; see [link / reference]. Figure 3 The specific PMI instance categories can be linear dimensioning, angular dimensioning, radius / diameter dimensioning, geometric tolerance frames, surface roughness symbols, welding symbols, datum symbols, technical requirement notes, title block general information, etc., but this application does not make specific limitations on them.
[0046] Optionally, the confidence score is used to assess the reliability of candidate PMI instances; the higher the confidence score, the more reliable the candidate PMI instance.
[0047] Specifically, the rotated box detection model uses a rotated box detection window to slide and detect on a sub-image of the view region, generating a sequence of candidate PMI instances that contains multiple candidate PMI instances.
[0048] S204. Redundancy is eliminated from multiple candidate PMI instances to obtain at least one target PMI instance. Based on the rotated bounding box coordinates of each target PMI instance, affine transformation is performed on the view region sub-image to obtain the corrected slice image of each target PMI instance.
[0049] Optionally, an improved nonmaximum suppression algorithm is used to perform redundancy removal on multiple candidate PMI instances to permanently remove repeatedly detected candidate PMI instances in order to obtain at least one target PMI instance.
[0050] Specifically, based on the rotated bounding box coordinates of each candidate PMI instance, the rotated bounding box corresponding to each candidate PMI instance is determined. Each rotated bounding box is mapped to a two-dimensional Gaussian distribution. The Bhattachary distance between the two-dimensional Gaussian distributions corresponding to the rotated bounding boxes of two candidate PMI instances of the same category is calculated to determine the overlap between the rotated bounding boxes of two candidate PMI instances of the same category, thereby determining whether two candidate PMI instances are duplicate detections. If so, the candidate PMI instance corresponding to the rotated bounding box with the lower confidence score is removed, and the candidate PMI instance corresponding to the remaining candidate bounding box is the target PMI instance.
[0051] Optionally, based on the rotated bounding box coordinates of each target PMI instance, the rotated bounding box corresponding to each target PMI instance is batch-corrected to complete the affine transformation cropping of the view region sub-image, thereby obtaining the corrected slice image corresponding to each target PMI instance.
[0052] Specifically, based on the rotation bounding box coordinates of each target PMI instance, the rotation matrix of each target PMI instance is calculated, and the tilted rectangular region corresponding to each target PMI instance is straightened based on the rotation matrix. The four sides of the straightened rectangular region are then expanded outward by a preset number of pixels (e.g., 5-10 pixels). Based on the expanded rectangular region, the corrected slice image corresponding to each target PMI instance is obtained.
[0053] S205. Input the corrected sliced images of each target PMI instance into the pre-built visual language model to obtain the structured fields corresponding to each target PMI instance, and input the title bar area sub-image into the visual language model to extract general manufacturing information.
[0054] Optionally, the pre-built visual language model calls different cue word templates to generate structured fields for different categories of target PMI instances for PMI content recognition of corrected slice images of different categories of target PMI instances.
[0055] For example, if the target PMI instance A is a linear dimension, the prompt word template called by the visual language model could be: "You are a professional engineering drawing recognition expert. Please carefully analyze the dimension information in the image and output the recognition results strictly according to the following JSON format: {"nominal_value": "nominal value", "upper_tolerance": "upper deviation", "lower_tolerance": "lower deviation", "tolerance_class": "tolerance class", "dimension_type": "linear / diameter / radius / angle", "special_symbol": "special symbol"} Only output JSON, without any explanation."
[0056] Optionally, the visual language model generates structured fields for each target PMI instance based on the corrected sliced image and its corresponding cue word template. Simultaneously, the visual language model extracts general manufacturing information from the title bar region sub-image using specific cue words.
[0057] For example, the prompt word template used by the visual language model to extract general manufacturing information from the title bar area sub-image is: "Extract the following information from the title bar: material specifications, heat treatment requirements, surface treatment requirements, general tolerance grade, drawing scale, drawing number, part name, and version number. Output in JSON format."
[0058] S206. Integrate the structured fields corresponding to each target PMI instance with general manufacturing information to obtain product manufacturing information from engineering drawing files.
[0059] Optionally, see Figure 8 By merging all structured fields output by the visual language model with general manufacturing information and outputting them in a preset data format (such as JSON or XML), the product manufacturing information of the user-input engineering drawing file can be obtained. The integrated product manufacturing information includes overall information about the engineering drawing file (materials, heat treatment, drawing number, etc.) as well as detailed information such as the location, PMI instance type, value, tolerance, and symbols of each target PMI instance.
[0060] Optionally, the product manufacturing information in the engineering drawings can be directly used in downstream manufacturing execution systems, process planning systems, or quality inspection systems.
[0061] In this embodiment, multiple formats of engineering drawing files are received from user input, and the engineering drawing files are preprocessed to obtain raster images of the same format corresponding to the engineering drawing files. A pre-built object detection model is used to detect the outer frame boundary, title bar region, and main view region of the raster image, and the raster image is segmented to obtain title bar region sub-images and view region sub-images of the raster image. A pre-built rotated bounding box detection model is used to perform PMI region detection on the view region sub-image to obtain multiple candidate PMI instances contained in the view region sub-image, as well as the rotated bounding box coordinates, PMI instance category, and confidence score of each candidate PMI instance. The system first detects and eliminates redundant candidate PMI instances based on their detection information to obtain at least one target PMI instance. Then, based on the rotated bounding box coordinates of each target PMI instance, it performs a full-quantization affine transformation cropping on the view region sub-image to obtain corrected slice images of each target PMI instance. The corrected slice images of each target PMI instance are then input into a pre-built visual language model to obtain the structured fields corresponding to each target PMI instance. Feature extraction is then performed on the title bar region sub-image to obtain general manufacturing information. Finally, the structured fields corresponding to each target PMI instance are integrated with the general manufacturing information to obtain the product manufacturing information of the engineering drawing file. Specifically, the engineering drawing file input to the target detection model is first converted into a unified format raster image, and the raster image is segmented into regions. Then, the segmented view region sub-image is subjected to PMI region detection and slice segmentation using an optimized rotation detection model. Finally, the corrected slice images and title bar region sub-image are subjected to PMI content recognition and double verification using a visual language model and a lightweight optical character recognition engine, thereby improving the accuracy and efficiency of product manufacturing information recognition. In this way, the accuracy, efficiency and reliability of product manufacturing information recognition in engineering drawings can be improved, while reducing the complexity and cost of automated recognition of product manufacturing information.
[0062] In one alternative implementation, see [link to implementation details]. Figure 4 The process of constructing the rotated box detection model in this application is as follows: S401. Data augmentation of the original detection model is performed by using large-scale random scaling and random rotation.
[0063] Optionally, to improve the generalization ability of the original YOLOv12 model to PMI symbols of different sizes and orientations in engineering drawing files, online data augmentation is performed on the input image and the corresponding rotated bounding box annotations before training.
[0064] Specifically, for each sub-image of the view area and its labeled rotated bounding boxes, the following operations are performed randomly with a probability of 0.5: Large-scale random scaling: The short side of the image is randomly scaled to 0.8 to 1.5 times the original size, while the long side is scaled at the same ratio. At the same time, the coordinates and dimensions of all rotated bounding boxes are scaled synchronously to simulate the changes in the drawing under different printing resolutions or scanning scales, thereby enhancing the model's robustness to scale changes; Random rotation: The entire image is randomly rotated around the center by an angle ranging from -15° to +15° (which can be adjusted according to the characteristics of the engineering drawing). After the rotation, the orientation of all rotated bounding boxes is updated accordingly, forcing the model to learn rotation-invariant PMI features, effectively dealing with annotations at arbitrary angles in engineering drawings.
[0065] S402. The rotated bounding box is modeled as a two-dimensional Gaussian distribution, and the Bhattacharyya distance is calculated based on the target algorithm to determine the bounding box regression loss of the original detection model. The target algorithm includes a dynamic nonlinear aspect ratio penalty factor.
[0066] Optionally, the rotated bounding box is mapped to a two-dimensional Gaussian distribution, and the Bhattacharyya distance between the predicted rotated bounding box and the true rotated bounding box is calculated based on the target algorithm to determine the bounding box regression loss of the original detection model.
[0067] S403. Adjust the model parameters of the original detection model based on the bounding box regression loss to obtain the rotated box detection model.
[0068] Optionally, the model parameters of the original detection model are adjusted according to the bounding box regression loss so that the loss function of the original detection model is close to the bounding box regression loss, thereby obtaining the rotated box detection model.
[0069] In one optional implementation, the target algorithm includes: Through formula Determine the dynamic nonlinear aspect ratio penalty factor; Through formula Determine the bounding box regression loss; in, Used to represent the aspect ratio penalty factor for nonlinearity Used to represent the width of the predicted rotation box. Used to represent the height of the predicted rotation box Used to represent preset adjustment hyperparameters, Used to represent the Bach distance between the predicted and actual two-dimensional Gaussian distributions corresponding to the rotated boxes.
[0070] Optionally, the Bhattacharyya distance between the predicted and actual rotated boxes can be dynamically adjusted using a nonlinear aspect ratio penalty factor to make the rotated box prediction model better fit the PMI sign. For example, for robust targets with normal aspect ratios, the nonlinear aspect ratio penalty factor is approximately 1, and the loss degenerates into the ordinary probability cross-union loss; for slender targets, the nonlinear aspect ratio penalty factor is greater than 1, the loss is more sensitive to Bhattacharyya distance, amplifying the distribution distance changes caused by angular deviations, making the rotated box detection model more accurately fit the slender PMI sign.
[0071] In an optional implementation, the above detection information further includes: confidence score and PMI instance category, see [link to relevant documentation]. Figure 5 The operation of "redundancy removal from multiple candidate PMI instances to obtain at least one target PMI instance" in step S204 above can be specifically described as follows: S501. Based on the PMI instance category of each candidate PMI instance, group each candidate PMI instance and sort them in descending order according to the confidence score of each candidate PMI instance to obtain multiple initial detection queues.
[0072] Optionally, each candidate PMI instance is grouped according to its PMI instance category, so that candidate PMI instances with the same PMI instance category label are grouped into the same group. The candidate PMI instances are then sorted in descending order according to the confidence scores of each candidate PMI instance included in each PMI instance group to obtain multiple initial detection queues, thereby avoiding the situation where different PMI categories are incorrectly suppressed due to spatial overlap.
[0073] S502. Map the rotated bounding boxes of each candidate PMI instance in each initial detection queue to a two-dimensional Gaussian distribution.
[0074] Optionally, the rotated bounding box of each candidate PMI instance in each initial detection queue is mapped to a two-dimensional Gaussian distribution to transform the discrete rectangular rotated bounding box into a continuous probability distribution. This allows the overlap between two rotated bounding boxes to be efficiently estimated through the similarity between the distributions, without the need for complex polygon intersection-union ratio calculations.
[0075] S503. Starting from the highest confidence score in each initial detection queue, calculate the Barthold distance between the two rotated bounding boxes to determine the overlap between them.
[0076] Optionally, starting from the rotating bounding box corresponding to the highest confidence score in each initial detection queue, the remaining rotating bounding boxes in each initial detection queue are traversed, and the Bhattachary distance between the two rotating bounding boxes is calculated to determine whether the rotating bounding boxes of the two candidate PMI instances point to the same target.
[0077] S504. Based on the degree of overlap, remove rotated bounding boxes with lower confidence scores.
[0078] Optionally, based on the overlap between the two rotated bounding boxes, the rotated bounding boxes obtained from duplicate detections and their corresponding candidate PMI instances are discarded.
[0079] S505. Repeat the iteration until all initial detection queues are cleared to obtain the target detection queue, which includes multiple target rotated bounding boxes.
[0080] Optionally, the target detection queue can be obtained by fusing all the initial detection queues after removing duplicate detections from the rotated bounding boxes.
[0081] S506. Treat each candidate PMI instance corresponding to the rotating bounding box of each target in the target detection queue as a target PMI instance.
[0082] In one optional implementation, step S504 above can specifically be performed as follows: If the overlap is greater than or equal to a preset threshold, the two rotated bounding boxes are identified as duplicate detections, and the rotated bounding box with the lower confidence score is removed.
[0083] Optionally, if the overlap between the rotated bounding boxes of two candidate PMI instances of the same category is greater than or equal to a preset threshold, it indicates that the rotated bounding boxes corresponding to the two candidate PMI instances are duplicate detections, and the candidate PMI instance with the lower confidence score and its corresponding rotated bounding box are removed.
[0084] The preset threshold is a value set by the user in advance. The preset threshold can be 80%, 90%, etc., and this application does not make any specific limitation on it.
[0085] In one alternative implementation, see [link to implementation details]. Figure 6 The operation in step S205 above, "inputting the corrected sliced images of each target PMI instance into a pre-constructed visual language model to obtain the structured fields corresponding to each target PMI instance," can be specifically described as follows: S601. Obtain the cross-modal semantic parsing results output by the visual language model.
[0086] Optionally, the corrected slice image corresponding to each target PMI instance is input into a pre-built visual language model. The visual language model calls the corresponding structured cue word template based on the PMI instance category of each target PMI instance and performs PMI content recognition on the corrected slice image to output the cross-modal semantic parsing results corresponding to each target PMI instance.
[0087] S602. Call the lightweight character recognition engine to extract numerical characters from the corrected slice images of each target PMI instance to obtain the numerical fields corresponding to each target PMI instance.
[0088] Optionally, for the same corrected slice image, a lightweight optical character recognition engine can be invoked to perform independent numerical character extraction.
[0089] Specifically, text detection is performed on the corrected slice image to locate the bounding boxes of all characters in the corrected slice image. Each detected character region is identified, and the character text and its confidence score are output. For numerical characters, the text content, confidence score, and center coordinates of the bounding box are recorded.
[0090] S603. Based on the numeric fields, correct the cross-modal semantic parsing results, and perform spatial anchoring based on the center coordinates of the character bounding boxes of each numeric character in the numeric fields and the attribute fields contained in the cross-modal semantic parsing results, so as to obtain the structured fields corresponding to each target PMI instance.
[0091] Optionally, depending on the field type, it can be determined whether to use numeric fields to correct the cross-modal semantic parsing results. If corrected, spatial anchoring is invoked to establish a one-to-one correspondence between numeric fields and cross-modal semantic parsing results.
[0092] Specifically, when parsing corrected sliced images, the visual language model uses its internal attention mechanism to map the visual features of different regions to different semantic fields.
[0093] Furthermore, although the output of the visual language model is only JSON text, the image region corresponding to each attribute field can be obtained in the following ways: First, the image is divided into several grids. By analyzing the attention weights of the visual language during the generation of each attribute field, the salient region on the image can be located in reverse. This region can be approximated as the sub-region of the image that the field focuses on. Second, the positional information of the character bounding boxes extracted by a lightweight character recognition engine is used, combined with heuristic rules for matching. For example, the nominal value is usually located in the middle or to one side of the dimension line, and the vertical deviation is usually located to the upper right or directly below the nominal value. This can be associated using a pre-defined spatial layout prior (such as "the center ordinate of the deviation character is higher than that of the nominal value character").
[0094] Optionally, each attribute field in the cross-modal semantic parsing results is extracted. For each field, the expected relative position range is defined according to the typical layout of the PMI instance category. All numerical character blocks extracted by the lightweight character recognition engine are traversed, the distance between the center coordinates of each character block and the expected area of the field is calculated, and the character block that is closest and has a confidence level higher than the threshold is selected as the candidate value of the field.
[0095] In an optional implementation, the operation of "correcting the cross-modal semantic parsing result based on the numerical field" in step S603 above can specifically be as follows: If the confidence level of a numeric field is higher than the preset security threshold, and the numeric field conflicts with the cross-modal semantic parsing result, then the numeric field in the cross-modal semantic parsing result is corrected based on the numeric field.
[0096] Optionally, if the attribute field is a numeric field (such as nominal value, upper deviation, lower deviation, etc.), and the confidence level of the numeric field extracted by the lightweight character recognition engine is higher than the preset safety threshold, and the numeric field is inconsistent with the value in the cross-modal semantic parsing result, then the absolute value of the numeric field extracted by the lightweight character recognition engine will be used to cover the corresponding field in the cross-modal semantic parsing result. Conversely, if the confidence level of the numeric field extracted by the lightweight character recognition engine is lower than the preset safety threshold, then the output of the cross-modal semantic parsing result will remain unchanged.
[0097] The preset security threshold is a threshold set by the user in advance. The preset security threshold can be 0.95, etc., and this application does not make specific limitations on it.
[0098] It should be noted that the recognition of professional symbols or letter codes depends on the contextual semantics and symbol morphology. Lightweight character recognition engines have difficulty accurately processing these fields, so there is no need to correct the cross-modal semantic parsing results.
[0099] In an optional implementation, the operation in step S204 above, "performing an affine transformation crop on the view region sub-image based on the rotated bounding box coordinates of each target PMI instance to obtain the corrected slice image of each target PMI instance," can specifically be as follows: When cropping the rotated bounding boxes of each target PMI instance, each side of the rotated bounding box of each target PMI instance is expanded outward by a preset number of pixels based on the coordinates of the rotated bounding box of each target PMI instance. During the affine transformation process, the tilt angle of each expanded rotated bounding box is corrected to obtain the corrected slice image of each target PMI instance.
[0100] Optionally, when cropping the rotated bounding box of each target PMI instance, the coordinates of the rotated bounding box of each target PMI instance are first obtained. In order to avoid the PMI instance being too close to the edge in the cropped slice image, which would cause the loss of edge features or the introduction of cropping boundary artifacts during subsequent recognition, each side of the rotated bounding box is extended outward by a preset number of pixels to reduce character truncation caused by excessive cropping.
[0101] The preset number of pixels can be dynamically adjusted according to the drawing resolution and PMI size; this application does not impose specific limitations on this.
[0102] Furthermore, since PMI instances in engineering drawing files may appear at arbitrary tilt angles, directly cropping the rotated rectangular area will result in a tilted image block, which is not conducive to the recognition of subsequent visual language models. Therefore, it is necessary to perform an affine transformation on the expanded rotated bounding box to correct it into a frontal (horizontal) image.
[0103] In one alternative implementation, see [link to implementation details]. Figure 7 The specific implementation of the YOLO v12 model structure and feature extraction in this application embodiment is as follows: An attention-centered backbone network is used, introducing an area attention mechanism. This mechanism can significantly expand the receptive field, accurately focusing on sparsely distributed but locally densely arranged small PMI symbols (such as roughness and geometric tolerances) in the drawing, while avoiding invalid calculations for large blank areas. Combined with Flash Attention technology, it greatly optimizes the memory access efficiency when processing high-resolution raster drawings. The feature fusion layer (Neck) adopts R-ELAN (Residual Efficient Layer Aggregation Network), which optimizes feature propagation and reuse in deep networks through residual design, ensuring that macroscopic long line segments and microscopic small text features can be efficiently and losslessly fused. The decoupled OBB Head physically separates the classification task and the bounding box regression task, performing predictions through independent branches to handle labeled text tilted at various angles.
[0104] The following describes the computer equipment and computer-readable storage medium used to implement the PMI recognition method for engineering drawings based on YOLO and multimodal large models provided in this application. The specific implementation process and technical effects are described above and will not be repeated below.
[0105] Figure 9 This is a schematic diagram of the structure of a computer device provided in this application. See also: Figure 9 The computer device provided in this application includes a memory 901 and a processor 902. The memory 901 stores a computer program that can run on the processor 902. When the processor 902 executes the computer program, it implements the steps in any of the above method embodiments.
[0106] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the various method embodiments described above.
[0107] Optionally, this application also provides a program product, such as a computer-readable storage medium, including a program that, when executed by a processor, performs any of the above-described embodiments of the PMI recognition method for engineering drawings based on YOLO and multimodal large models.
[0108] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute partial steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0109] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0110] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A method for PMI recognition in engineering drawings based on YOLO and multimodal large models, characterized in that, The method includes: Preprocess the engineering drawing file to obtain the raster image corresponding to the engineering drawing file; A pre-built target detection model is used to detect the outer frame boundary, title bar region, and main view region of the raster image, and the raster image is segmented to obtain the title bar region sub-image and the view region sub-image of the raster image; Based on a pre-built rotated bounding box detection model, PMI region detection is performed on the view region sub-image to obtain multiple candidate PMI instances contained in the view region sub-image and detection information of each candidate PMI instance. The detection information includes: rotated bounding box coordinates. Redundancy is eliminated from the multiple candidate PMI instances to obtain at least one target PMI instance. Based on the rotated bounding box coordinates of each target PMI instance, the view region sub-image is cropped by affine transformation to obtain the corrected slice image of each target PMI instance. The corrected sliced images of each target PMI instance are input into a pre-built visual language model to obtain the structured fields corresponding to each target PMI instance, and the title bar area sub-image is input into the visual language model to extract general manufacturing information. The structured fields corresponding to each target PMI instance are integrated with the general manufacturing information to obtain the product manufacturing information of the engineering drawing file.
2. The method for PMI recognition of engineering drawings based on YOLO and multimodal large models according to claim 1, characterized in that, The process of constructing the rotated box detection model is as follows: Large-scale random scaling and random rotation are used to augment the data of the original detection model; The rotated bounding box is modeled as a two-dimensional Gaussian distribution, and the Bhattacharyya distance is calculated based on the target algorithm to determine the bounding box regression loss of the original detection model. The target algorithm includes a dynamic nonlinear aspect ratio penalty factor. Based on the bounding box regression loss, the model parameters of the original detection model are adjusted to obtain the rotated box detection model.
3. The method for PMI recognition of engineering drawings based on YOLO and multimodal large models according to claim 2, characterized in that, The target algorithm includes: Through formula Determine the dynamic nonlinear aspect ratio penalty factor; Through formula Determine the bounding box regression loss; in, Used to represent the aspect ratio penalty factor for nonlinearity Used to represent the width of the predicted rotation box. Used to represent the height of the predicted rotation box Used to represent preset adjustment hyperparameters, Used to represent the Bach distance between the predicted and actual two-dimensional Gaussian distributions corresponding to the rotated boxes.
4. The method for PMI recognition of engineering drawings based on YOLO and multimodal large models according to claim 1, characterized in that, The detection information also includes: confidence score and PMI instance category. The process of redundantly eliminating the multiple candidate PMI instances to obtain at least one target PMI instance includes: Based on the PMI instance category of each candidate PMI instance, the candidate PMI instances are grouped and sorted in descending order of confidence score to obtain multiple initial detection queues. The rotated bounding boxes of each candidate PMI instance in each initial detection queue are mapped to a two-dimensional Gaussian distribution. Starting from the highest confidence score in each initial detection queue, calculate the Bhattachary distance between two rotated bounding boxes to determine the overlap between them. Based on the overlap, rotated bounding boxes with low confidence scores are removed; Repeat the iteration until all initial detection queues are cleared to obtain a target detection queue, which includes multiple target rotated bounding boxes; Each candidate PMI instance corresponding to the rotating bounding box of a target in the target detection queue is treated as a target PMI instance.
5. The method for PMI recognition of engineering drawings based on YOLO and multimodal large models according to claim 4, characterized in that, The step of removing rotated bounding boxes with low confidence scores based on the overlap includes: If the overlap is greater than or equal to a preset threshold, the two rotated bounding boxes are determined to be duplicate detections, and the rotated bounding box with the lower confidence score is removed.
6. The method for PMI recognition of engineering drawings based on YOLO and multimodal large models according to claim 1, characterized in that, The step of inputting the corrected sliced images of each target PMI instance into a pre-constructed visual language model to obtain the structured fields corresponding to each target PMI instance includes: Obtain the cross-modal semantic parsing results output by the visual language model; A lightweight character recognition engine is invoked to extract numerical characters from the corrected slice images of each target PMI instance in order to obtain the numerical fields corresponding to each target PMI instance. Based on the numerical fields, the cross-modal semantic parsing results are corrected, and spatial anchoring is performed based on the center coordinates of the character bounding boxes of each numerical character in the numerical fields and the attribute fields contained in the cross-modal semantic parsing results to obtain the structured fields corresponding to each target PMI instance.
7. The method for PMI recognition of engineering drawings based on YOLO and multimodal large models according to claim 6, characterized in that, The step of correcting the cross-modal semantic parsing result based on the numerical field includes: If the confidence level of the numerical field is higher than a preset security threshold, and the numerical field conflicts with the cross-modal semantic parsing result, then the numerical field in the cross-modal semantic parsing result is corrected based on the numerical field.
8. The method for PMI recognition of engineering drawings based on YOLO and multimodal large models according to claim 1, characterized in that, The step of performing an affine transformation cropping on the view region sub-image based on the rotated bounding box coordinates of each target PMI instance to obtain the corrected slice image of each target PMI instance includes: When cropping the rotated bounding boxes of each target PMI instance, each side of the rotated bounding box of each target PMI instance is expanded outward by a preset number of pixels based on the coordinates of the rotated bounding box of each target PMI instance. During the affine transformation process, the tilt angle of each expanded rotated bounding box is corrected to obtain the corrected slice image of each target PMI instance.
9. A computer device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor, when executing the computer program, implements the steps of the method according to any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1 to 8.