Technical drawing information extraction device, technical drawing information extraction method, and technical drawing information extraction program

Abstract

Extract information from technical drawings with high accuracy, taking context into consideration. [Solution] The technical drawing information extraction device 10 includes an input unit 11 that receives a technical drawing 20 as input, a division unit 12 that analyzes the technical drawing 20 and divides it into multiple regions based on the type of drawing element, an extraction unit 13 that extracts partial information including semantic information from each of the divided regions using a pre-stored multimodal large-scale language model 131, and an integration unit 14 that integrates the multiple partial information obtained from the multiple regions to form structured information 50 of the entire technical drawing.

Description

Technical Field

[0001] The present invention relates to a technical drawing information extraction device, a technical drawing information extraction method, and a technical drawing information extraction program, and particularly to a technique for performing information extraction that takes context into account with high accuracy from technical drawings such as CAD (Computer Aided Design) drawings.

Background Art

[0002] Conventionally, for information extraction from technical drawings, a method of extracting character information using OCR (Optical Character Recognition) technology has been common. However, although the OCR technology can recognize character strings, it has been difficult to understand the context of what meaning the characters have within the technical drawing. In recent years, with the development of multimodal large language models (LLMs), technologies for integratively understanding image and character information have attracted attention. However, when the entire technical drawing is input into a multimodal large language model at once, there is a problem that the attention mechanism is dispersed due to the huge amount of information, and important information may be overlooked.

[0003] [[ID=ID=18]]In Patent Document 1, a drawing management system includes a user-side terminal device used by a user who places an order for a product, a manufacturer-side terminal device used by a manufacturer who receives an order from the user, and a drawing management server that communicates with the user-side terminal device and the manufacturer-side terminal device via a network communication line. The drawing management server extracts a group of drawing data related to the product specifications from a plurality of drawing data created in the past, and has drawing data analysis means for ranking the extracted group of drawing data in descending order of the matching rate with the product specifications of the product, and drawing list editing means for listing the ranked group of drawing data in order from the one with the highest matching rate. An invention is disclosed in which the group of drawing data listed by the drawing list editing means can be viewed from either the user-side terminal device or the manufacturer-side terminal device.

Prior Art Documents

Patent Documents

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-91769 [Overview of the project] [Problems that the invention aims to solve]

[0005] The drawing analysis system using optical character recognition (OCR) technology described in Patent Document 1 processes the entire drawing and does not adequately address local contextual understanding or accurate extraction of named entities.

[0006] Conventional OCR technology could not understand the contextual meaning of a number like "50" in a technical drawing, such as whether it represented the height or width of a part, or the height of a part positioned horizontally. Furthermore, when the entire technical drawing was input into a multimodal large-scale language model, the complexity of the technical drawing caused attention to become scattered, making it difficult to accurately extract localized important information.

[0007] In particular, multimodal large-scale language models tend to replace visually similar characters with more common characters. Multimodal large-scale language models frequently have problems such as misidentifying the Japanese place name "Ōmika" as the Chinese city name "Dalian," converting the personal name "Sato" to "Sano," or replacing the company-specific part name "XYZ-2024A" with the general "ABC-2024A." Such misidentifications cause critical information loss in quality control and traceability management in the manufacturing industry. Furthermore, conventional technologies have made it difficult to incorporate the tacit knowledge and industry-specific interpretation rules held by experts in the manufacturing sector into the system, making it impossible to achieve practical accuracy.

[0008] Therefore, the present invention aims to extract information from technical drawings with high accuracy, taking context into consideration. [Means for solving the problem]

[0009] To solve the aforementioned problems, the technical drawing information extraction device of the present invention includes an input unit that receives technical drawings as input, and an analysis unit that analyzes the technical drawings. While identifying the type of each part region through object detection processing, Multiple based on the type of drawing element parts A dividing section that divides the region, and each of the divided parts parts For a given domain, using a pre-stored multimodal large-scale language model, context An extraction unit that extracts partial information containing information, and the plurality parts It comprises an integration unit that integrates multiple partial pieces of information obtained from a region to form structured information for the entire technical drawing, The division unit divides a part area containing one or more of the following as independent part areas based on the identified type: title block, main shape, dimensions, material, paint, approver, and bill of materials. The extraction unit applies a customized extraction prompt according to the type identified by the division unit, and for the independent part areas, applies a prompt that includes instructions to extract the unique notation contained in the part area exactly as it is displayed, without replacing the characters in the image with common notation, thereby literally extracting the unique notation contained in the part area. It is characterized by the following:

[0010] The present invention's method for extracting technical drawing information comprises the steps of: an input unit receiving a technical drawing as input; and a division unit performing object detection processing on the technical drawing. While identifying the type of each part region through object detection processing, The aforementioned technical drawing is divided into multiple part areas. Along with this, the step of dividing the part area containing one or more of the following—title field, main shape, dimensions, material, paint, approver, and bill of materials—as an independent part area based on the identified type, The extraction unit performs information extraction processing for each of the multiple part regions using a multimodal large-scale language model, and extracts partial information including contextual information from each part region. The extraction unit applies a customized extraction prompt according to the type identified by the division unit, and for the independent part region, applies a prompt that includes an instruction to extract the characters in the image exactly as they are displayed, without replacing them with common notation, thereby literally extracting the unique notation contained in the part region. The integration unit is characterized by including the step of integrating multiple partial information obtained from the multiple part areas to form structured information for the entire technical drawing.

[0011] The present invention's technical drawing information extraction program provides a computer with a procedure for receiving technical drawings as input, and for executing object detection processing on the technical drawings. While identifying the type of each part region through object detection processing, The aforementioned technical drawing is divided into multiple part areas. Along with this, a procedure for dividing a part area containing one or more of the following—title field, main shape, dimensions, material, paint, approver, and bill of materials—as an independent part area, based on the identified type, A procedure for performing information extraction processing using a multimodal large-scale language model for each of the aforementioned multiple part regions, and extracting partial information including contextual information from each part region, A procedure for literally extracting unique identifiers contained in a part region by applying customized extraction prompts according to the identified type, and for the independent part region, applying prompts that include instructions to extract the characters in the image exactly as they are displayed, without replacing them with common notations. This is intended to perform a procedure for integrating multiple pieces of information obtained from the aforementioned multiple parts regions to create structured information for the entire technical drawing. Other means will be described within the descriptions of embodiments for carrying out the invention. [Effects of the Invention]

[0012] According to the present invention, it is possible to extract information with high accuracy in consideration of the context from technical drawings.

Brief Description of the Drawings

[0013] [Figure 1] It is a logical configuration diagram of the technical drawing information extraction device according to the present embodiment. [Figure 2] It is a hardware configuration diagram of the technical drawing information extraction device. [Figure 3] It is a flowchart showing the technical drawing information extraction process. [Figure 4] It is a diagram for explaining the title prompt. [Figure 5] It is a diagram for explaining the main shape prompt. [Figure 6] It is a diagram for explaining the dimension prompt. [Figure 7] It is a diagram for explaining the approver prompt. [Figure 8] It is a diagram for explaining the parts list prompt. [Figure 9] It is a diagram showing an example of a technical drawing. [Figure 10] It is a diagram showing the correspondence of type identification obtained by dividing a technical drawing. [Figure 11] It is a conceptual diagram showing the operation of JSON integration processing. [Figure 12] It is a diagram showing the JSON of structured information. [[ID=4�]] [Figure 13] It is a diagram showing the JSON of dimension value information. [Figure 14] It is a diagram showing the JSON of approver information with corrected misrecognition of proper nouns.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the respective drawings. FIG. 1 is a logical configuration diagram of a technical drawing information extraction device 10 according to the present embodiment. As shown in Figure 1, the technical drawing information extraction device 10 according to this embodiment includes an input unit 11, a division unit 12, an extraction unit 13, and an integration unit 14.

[0015] The input unit 11 accepts technical drawings 20, such as CAD (Computer-Aided Design) drawings and design drawings, as electronic data. The format of the technical drawings 20 can be any image format, such as PNG, JPEG, or PDF.

[0016] The division unit 12 applies an object detection algorithm to the input technical drawing 20 using the object detection unit 121, dividing the drawing into semantic part regions. The object detection unit 121 uses YOLO (You Only Look Once), R-CNN (Region-Based Convolutional Neural Network), SSD (Single Shot MultiBox Detector), or any detection method with equivalent functionality as the object detection algorithm. The division criteria are set based on elements specific to the technical drawing (dimension lines, dimension values, part shapes, notes, symbols, etc.). For example, the object detection unit 121 divides an area containing a dimension value and its corresponding dimension line into a single part region. Furthermore, the division unit 12 uses a type identification unit 122 to identify the type of each divided part region, such as title, main shape, dimensions, material, paint, approver, bill of materials, etc.

[0017] The extraction unit 13 applies a multimodal large-scale language model 131 to each divided part region and performs information extraction on a part region basis. For example, the multimodal large-scale language model 131 can be GPT-5(registered trademark), Claude(registered trademark), Gemini(registered trademark), or any model with equivalent functionality. The extraction unit 13 includes a prompt application unit 132 that applies customized dedicated prompts according to the identified part region type. Using the multimodal large-scale language model 131, the extraction unit 13 extracts higher-order information based on semantic relationships between drawing elements. Here, higher-order information refers to semantic relationship information based on geometric constraints, standard symbols, and drawing notes, and is limited to what can be objectively derived from the drawing description.

[0018] The integration unit 14 uses the JSON generation unit 141 to integrate the partial information extracted from each part area into structured information 50 in JSON (JavaScript Object Notation) format or the like, according to a predefined schema. In the integration process, the integration unit 14 also considers the relationships between the part areas and generates structured information 50 that reflects the overall comprehensive information structure of the technical drawing 20.

[0019] Figure 1 shows the logical configuration of the technical drawing information extraction device 10, which includes an input unit 11, a division unit 12, an extraction unit 13, and an integration unit 14. The technical drawing information extraction device 10 divides the technical drawing 20 into units of part areas, performs multimodal LLM processing for each type, and integrates the results. This processing procedure allows for the generation of structured information 50 with higher accuracy than when the entire technical drawing 20 is processed at once, as attention is not dispersed.

[0020] Figure 2 is a hardware configuration diagram of the technical drawing information extraction device 10. The technical drawing information extraction device 10 includes a CPU 101, memory 102, secondary storage device 103, input / output interface 104, and network interface 106.

[0021] The CPU 101 is one or more central processing units that provide overall control for the technical drawing information extraction device 10. The memory 102 is a temporary storage device, such as a volatile or non-volatile storage device. The secondary storage device 103 is a large-capacity storage device such as a hard disk or an SSD (Solid State Drive).

[0022] The input / output interface 104 is an interface for input signals from input devices 108, such as a keyboard or mouse, and output signals to the display device 107. The network interface 106 is an interface for connecting to a network and sending and receiving information with other devices. In memory 102, each functional module of the technical drawing information extraction device 10 is loaded as a computer program and executed by the CPU 101.

[0023] Furthermore, the technical drawing information extraction program 105 is stored in the secondary storage device 103. The installer for the technical drawing information extraction program 105 is stored in the recording medium 100. The technical drawing information extraction program 105 is installed by loading the recording medium 100 into a recording medium reading unit (not shown) and executing the installer. The CPU 101 executes the technical drawing information extraction program 105, thereby realizing each of the functional units in Figure 1.

[0024] The technical drawing information extraction device 10 may be implemented as a virtual machine rather than a physical machine. The technical drawing information extraction device 10 connects to the equipment or the device that controls the equipment via a network. The technical drawing information extraction device 10 may also be implemented in a cloud environment. In that case, the technical drawing information extraction device 10 is connected to the environment in which the equipment is configured via a network.

[0025] With the hardware configuration shown in Figure 2, the technical drawing information extraction device 10 can be operated on the CPU 101, memory 102, secondary storage device 103, input / output interface 104, and network interface 106, enabling the stable and expandable implementation of each functional unit using general-purpose computing resources. The installation of the technical drawing information extraction program 105 from the recording medium 100 and its storage in the secondary storage device 103 ensures both ease of deployment and maintainability, resulting in rapid deployment to the field environment and continuous operation.

[0026] Figure 3 is a flowchart showing the process for extracting technical drawing information. First, the input unit 11 receives the input technical drawing 20 (step S10). Then, the division unit 12 applies an object detection algorithm to the input technical drawing 20 using the object detection unit 121 (step S11) and divides the technical drawing 20 into semantic part areas (step S12). The division unit 12 uses the type identification unit 122 to identify the type of each divided part area, such as title, dimensions, approval, main shape, material, paint, bill of materials, etc., and branches to process according to this type (step S13).

[0027] In step S13, if the type of part area is TITLE, the process proceeds to step S141. Here, the extraction unit 13 applies a prompt for the title field to each divided part area (step S141), applies the multimodal large-scale language model 131 (step S151), and extracts title information and part name information (step S161).

[0028] In step S13, if the type of part region is SHAPE (main shape), the process proceeds to step S142. Here, the extraction unit 13 applies a prompt for the main shape to each divided part region (step S142), applies the multimodal large-scale language model 131 (step S152), and extracts the main shape information (step S162).

[0029] In step S13, if the type of part area is DIMENSION, the process proceeds to step S143. Here, the extraction unit 13 applies a dimension prompt to each divided part area (step S143), applies the multimodal large-scale language model 131 (step S153), and extracts dimension values ​​and semantic information (step S163).

[0030] In step S13, if the type of part area is MATERIAL (material / paint), the process proceeds to step S144. Here, the extraction unit 13 applies a material / paint prompt to each divided part area (step S144), applies the multimodal large-scale language model 131 (step S154), and extracts material information and paint information (step S164).

[0031] In step S13, if the type of part area is APPROVAL for approver, the process proceeds to step S145. Here, the extraction unit 13 applies approver prompts to each divided part area (step S145), applies the multimodal large-scale language model 131 (step S155), and extracts approver information (step S165).

[0032] In step S13, if the type of part area is a bill of materials (TABLE), the process proceeds to step S146. Here, the extraction unit 13 applies a bill of materials prompt to each divided part area (step S146), applies the multimodal large-scale language model 131 (step S156), and extracts bill of materials information (step S166).

[0033] In step S17, the integration unit 14 integrates the partial information extracted from each part area into structured information 50 in JSON format or similar, according to a predefined schema. Then, when the integration unit 14 outputs the structured information 50 (step S18), the process shown in Figure 3 is completed.

[0034] Figure 3 shows the entire flow from input of the technical drawing 20 to object detection, part region division, type determination, application of prompts for each type, multimodal LLM processing, and JSON integration. In this flow, processing is branched in multiple directions depending on the type of part region. This allows for the comprehensive extraction of different information such as title, part name, dimensions, material, paint, approver, and bill of materials, and enables the stable and highly accurate acquisition of structured information with dedicated prompts corresponding to the extracted part region.

[0035] 《Extraction part 13》 The extraction unit 13 performs the following multi-layered information extraction on each divided part region using the multimodal large-scale language model 131:

[0036] (1) Basic information extraction The extraction unit 13 extracts numerical information and its meaning, such as height, width, depth, and radius. Furthermore, the extraction unit 13 extracts the name and type of the part, material information, surface treatment information, etc.

[0037] (2) Shape / structure information extraction The extraction unit 13 understands the three-dimensional shape of the parts, such as cylinders, prisms, and tapers, recognizes processing elements such as holes, grooves, and chamfers, and estimates the assembly relationships between the parts. In this specification, "estimating assembly relationships" is limited to relationships that can be mechanically derived based on the geometric constraints and fastening parts described in the technical drawing 20, and does not involve inferring uses or functional roles not explicitly stated in the technical drawing 20.

[0038] (3) Extraction of semantic related information The extraction unit 13 interprets the meaning of tolerance symbols and understands mating relationships such as H7 / g6. The extraction unit 13 performs a comprehensive understanding of screw standards. For example, the extraction unit 13 understands the string "M8×1.25-6H" as a metric thread, with a diameter of 8 mm, a pitch of 1.25, and a depth grade of 6H. The extraction unit 13 associates surface roughness symbols with machining methods.

[0039] (4) Structuring of compound expressions Conventional OCR recognizes "M8," "×," "1.25," and "depth 10" individually. However, the extraction unit 13 understands them as "M8×1.25 screw hole, depth 10mm" in an integrated manner.

[0040] Hallucination suppression mechanism The division section 12 optimizes the division process as follows from the viewpoint of suppressing hallucination. The division section 12 divides the title block, which contains the part name, part number, painting specifications, etc., into an independent part area. This part area has a unique notation area.

[0041] The division unit 12 adjusts the information density. Specifically, it limits the amount of information contained in a single part area to a range that the multimodal large-scale language model 131 can accurately process. The division unit 12 preserves contextual information by including named entities, their descriptions, and specifications within the same part area. Then, in order to extract information with high accuracy, the extraction unit 13 gives the following instructions to the multimodal large-scale language model 131 for each part region.

[0042] "Extract the text in the image exactly as it appears, without replacing it with a more common spelling." "Proper nouns, part numbers, and specification numbers should be read literally; no guesswork or supplementation is allowed."

[0043] Adaptive prompt application function The division unit 12 simultaneously identifies the type of each part area during object detection processing. The types of part areas identified include the title area containing the part name and drawing number, the dimension area, the material / specification area, the revision / approver area, the part shape area, and the notes / special notes area. Based on the identified part area types, the extraction unit 13 applies dedicated prompts as illustrated in Figures 4 to 8:

[0044] Figure 4 illustrates the title block prompt 41. The title block prompt 41 is reproduced below exactly as it was written.

[0045] This image is the title block of a technical drawing. Please extract the information keeping the following points in mind: 1. Part names may include company-specific naming conventions, so read them literally and accurately. 2. Extract the drawing number completely, including delimiters such as "-", "_", and "Rev". 3. Material symbols should be interpreted according to JIS standards. 4. Do not substitute with common notation.

[0046] Figure 4 shows the contents of the title block prompt 41, which provides instructions such as "read literally and accurately" and "do not replace with general notation" for unique notations such as part names, drawing numbers, and material symbols. This suppresses hallucination, in which the multimodal large-scale language model 131 arbitrarily standardizes and converts company-specific part numbers and names, and allows for the faithful extraction of information from the title block.

[0047] Figure 5 is a diagram illustrating the main shape prompt 42. The following is a direct transcription of the main shape prompt 42.

[0048] This image shows the area of ​​a technical drawing that displays the shape of a part. Please accurately extract only the visually verifiable information according to the following rules. 1. If the type of projection drawing (front view, side view, top view) is clearly indicated on the drawing, extract it as indicated. If it is not clearly indicated, record "Type of projection drawing: Not specified". 2. Describe the type and arrangement of lines indicating the shape (solid lines, dashed lines, dotted lines, etc.) exactly as they appear. Extract the name of the shape (cylindrical, prism, etc.) only if it is clearly written on the drawing, and do not try to guess it. 3. Regarding machining elements (holes, grooves, chamfers, etc.) - Extract only the names and specifications that are clearly written on the drawing. - Notations such as "φ20H7" and "M8×1.25" must be fully extracted, without omitting any symbols, numbers, or letters. - If the purpose or use of the processing is not specified, do not make assumptions and record it as "Use: Not specified". 4. If a hatching (diagonal line) pattern exists in the cross-sectional view, record it as "Hatching: Yes," but do not attempt to infer the material. Extract the material only if it is clearly stated in the notes. 5. Reference symbols (such as "Enlarged Section A," "Detailed Section B," etc.) should be extracted exactly as they are written. 6. If there are notes or explanations regarding the shape, extract them exactly as they are, word for word, and do not paraphrase, summarize, or supplement them. 7. If symmetry symbols, ellipsis symbols, etc. exist, record them as "Symmetry symbol: present," but do not write in a way that completes the omitted parts. 8. Dimensional values ​​should be fully extracted, including the numerical value and unit. However, the meaning of the dimensions (height, width, etc.) should be recorded along with the placement of the dimension lines, and no interpretation of meaning should be made based on speculation. 9. If there are characters or symbols that are visually illegible, record them as "illegible" and do not replace them with similar characters. 10. Industrial symbols, welding symbols, etc., should be accurately described in terms of their shape, and their meaning should only be interpreted if an explanation is clearly stated on the drawing.

[0049] Figure 5 shows dedicated prompts for the part shape region, meticulously defining rules for handling projection types, line type descriptions, machining elements and reference symbols, section hatching, symmetry symbols, and illegible characters. This allows the multimodal large-scale language model 131 to extract only the visual information clearly stated in the technical drawing 20 without arbitrarily inferring the application or material of the shape region, thereby structuring the shape information and machining information with high reliability.

[0050] 《Meaning information about shape and structure》 Three-dimensional shapes include cylinders, prisms, and tapers. Machining elements include holes, grooves, chamfers, and threads. Assembly relationships between parts include fitting, fastening, and bonding. Geometric constraints include parallelism, perpendicularity, concentricity, and symmetry.

[0051] FIG. 6 is a diagram for explaining the dimension prompt 43. The dimension prompt 43 is transcribed as it is below.

[0052] This image is an area containing dimension information. Please apply the following interpretation rules: 1. Determine the height, width, and depth from the positional relationship between the numerical value and the dimension line. 2. Interpret tolerance symbols (such as H7 / g6) as fitting grades. 3. Identify the "φ" symbol as the diameter and the "R" symbol as the radius. 4. Understand the thread notation "M8×1.25" integrally and structure it.

[0053] FIG. 6 shows the dimension prompt 43 and clarifies interpretation rules such as determining the height, width, and depth from the positional relationship between the numerical value and the dimension line, tolerance symbols, φ and R symbols, and the thread notation M8×1.25. As a result, it is possible to extract not only the dimension value as a simple character string but also its semantic information, including the dimension type, tolerance relationship, thread specification, etc., and generate high-level dimension information in JSON format by the multimodal large language model 131.

[0054] 《Semantic Information of Numerical Values》 The meaning of dimensions is height, width, depth, diameter, radius, length, thickness, etc. The meaning of tolerance symbols is the fitting relationship such as H7 / g6, etc. Surface roughness symbols are indicated by Ra0.8, ▽▽▽, etc.

[0055] FIG. 7 is a diagram for explaining the approver prompt 45. <00002 )The approver prompt 45 is transcribed as it is below.

[0056] This image includes approver information and change history: 1. Read the person's name accurately, paying attention to misconversions in on'yomi and kun'yomi readings. 2. Treat factory names and department names as proper nouns and do not perform speculative conversions. 3. Unify the date in the format of "YYYY / MM / DD". 4. Record the seal and signature as "Approved".

[0057] Figure 7 shows prompts for approver information and the change history area, instructing to treat person names, factory names, and department names as proper nouns, unify the date format, and record the seal as "Approved". This can accurately maintain the approval history, which is important for quality control and traceability, in a machine-readable form while preventing incorrect conversion of designer or factory names.

[0058] Figure 8 is a diagram explaining the prompt 46 for the parts list. The prompt 46 for the parts list is transcribed as follows.

[0059] This image is the parts list (BOM) of the technical drawing. Please accurately extract the information described in the table word by word according to the following rules. 1. Extract the column headings of the table (part number, product name, quantity, material, remarks, etc.) exactly as written. Do not standardize or unify the expressions of the headings. 2. For part numbers and item numbers, extract them exactly as written, including the presence or absence of uppercase and lowercase alphanumeric characters, symbols (-, _, / ,., etc.), and spaces. - Example: Do not convert "XYZ-2024A-Rev3" to "XYZ-2024A-REV3" or "XYZ2024A Rev3". 3. Extract the part names exactly as written, including abbreviations, general names, and company-specific names. - Do not replace them with common names or standard notations. - Example: Do not abbreviate "Engine Bracket_Ver2.1" to "Engine Bracket". 4. Extract the quantity and unit described in the table completely. - Accurately maintain the differences in notations such as "4 pieces", "4 items", "4 units", etc. 5. Extract the material symbols exactly as written. - Maintain the differences in notations such as "SUS304", "sus304", "SUS-304", etc. - Standard interpretations (JIS, ASTM, etc.) should only be extracted if they are clearly indicated on the drawing. 6. The procurement category (purchased goods, outsourced goods, in-house manufactured goods, etc.) should only be extracted if it is listed in the table; if it is not listed, write "Not listed". 7. The contents of the remarks column must be extracted exactly, word for word, and no summarization, paraphrasing, or interpretation is permitted. - Even if multiple pieces of information are included, maintain their accuracy, including the order in which they are listed. 8. Accurately maintain the correspondence between rows and columns in a table and integrate them as a single component of information. - If a cell is blank, please specify "blank" or "not entered". 9. If a hierarchical structure (indentation, numbering system, etc.) can be visually confirmed, record that structure, but do not attempt to infer parent-child relationships. 10. If there are illegible or faded characters, record them as "illegible" and do not replace them with similar characters. 11. If there are notes or legends outside the table, extract them completely, including those. 12. If there are conditional descriptions for alternative parts, options, etc., extract them accurately, including the conditional statements.

[0060] Figure 8 shows the contents of the bill of materials prompt 46, illustrating the rules for accurately extracting column headers, part numbers, part names, quantities, materials, remarks, procurement classifications, and hierarchical structures down to the character and cell level, while prohibiting standardization and summarization. This allows for the conversion of the table structure directly into structured data without altering the company's unique bill of materials representation, achieving the accuracy and completeness required for direct use in parts management and procurement systems.

[0061] 《Semantic Information of Proper Names》 Part names and drawing numbers follow company-specific naming conventions. By using a pre-configured RAG (Retrieval-Augmented Generation) database of approver and designer names, OCR errors and hallucinations can be avoided. The same applies to manufacturing sites and factory names.

[0062] 《Meaning information regarding material and surface treatment》 Materials include, for example, SUS304, A5052, and S45C. Painting specifications include acrylic resin baked coating (5Y7 / 1). Plating specifications include hot-dip galvanizing HDZ35 and electroless nickel plating (5-8 μm). Other surface treatments include anodizing, trivalent chromate, and shot peening.

[0063] Integrated semantic information of compound expressions The combined notation for screw specifications, "M8×1.25-6H," means "metric thread, 8mm diameter, 1.25mm pitch, internal thread grade 6H." The combined notation for hole processing specifications, "φ8.5 pilot hole, 10mm depth, 2 locations," means drilling two holes with a diameter of 8.5mm and a depth of 10mm. The combined notation for welding symbols, "fillet weld S6 all around," means performing a fillet weld with a leg length of 6mm all around the perimeter.

[0064] Processing of machine part drawings Figure 9 shows an example of a technical drawing 20. Figure 10 shows the correspondence of type identification when the technical drawing 20 is divided. An example of processing when a technical drawing 20 of a machine part, as shown in Figure 9, is used as input will be explained. Division process: The division section 12 divides the technical drawing 20 into the following part areas, and their types are identified. Part area 31 is a title field containing the part name, drawing number, etc., and is assigned the type "TITLE". Part regions 32a and 32b are the main part shape regions and are assigned the type SHAPE.

[0065] Part regions 33a to 33f are dimension regions and are assigned the type "DIMENSION". Part areas 34a and 34b are material and coating indication areas, and are assigned the type "MATERIAL".

[0066] Part area 35 is the approver field, and the type is assigned as APPROVAL. Parts area 36 is a bill of materials and is assigned the type TABLE.

[0067] The extraction unit 13 then applies a dedicated prompt corresponding to the type of each part area. The extraction unit 13 extracts the following information from each part area:

[0068] From part area 31, "Part name: Engine Bracket_Ver2.1" and "Drawing number: XYZ-2024A-Rev3" are extracted. From parts regions 32a and 32b, "rectangular-shaped parts" and the like are extracted.

[0069] From parts areas 33a to 33f, information such as "part height: 2000mm, main dimensions" is extracted. From parts areas 34a and 34b, information such as "Material: SUS304" and "Paint: Special heat-resistant TC-300" is extracted.

[0070] From part area 35, information such as "Designer: Taro Omika Factory Patent" and "Approved" is extracted. From part area 36, ​​information such as "Name: Glass window Specifications: 450 x 480 Material: Titanium frame Quantity: 1 Remarks: Front upper part" is extracted.

[0071] Figure 10 shows the correspondence between the division of the technical drawing 20 in Figure 9 and the assignment of categories such as title, main shape, dimensions, material, approver, and bill of materials. The technical drawing 20 is then divided into semantic part areas, and dedicated prompts are applied. This allows for appropriate control of information density while preserving the context contained in each area, enabling processing by the multimodal large-scale language model 131 and simultaneously achieving high-precision extraction and hallucination suppression.

[0072] Figure 11 is a conceptual diagram illustrating the operation of the JSON integration process. The integration unit 14 integrates the title information 51, main shape information 52, dimension value information 53, material and paint information 54, approver information 55, and bill of materials information 56. Then, the JSON generation unit 141 integrates this information to generate structured information 50 in JSON format.

[0073] Figure 11 illustrates the concept of how the integration unit 14 and the JSON generation unit 141 combine title information 51, main shape information 52, dimension value information 53, material and paint information 54, approver information 55, and parts list information 56 into structured information. The integration unit 14 integrates the partially extracted information, which is separated by type, into structured information 50. This allows the overall semantic structure of the technical drawing 20 to be treated as a consistent JSON, facilitating searching, aggregation, and integration with downstream applications.

[0074] Figure 12 shows a JSON image that contains a portion of the structured information 50. Below is a partial JSON representation of structured information 50.

[0075] { "part_info" : { "name": "Engine Bracket_Ver2.1", "drawing_number": "XYZ-2024A-Rev3", "shape" : { "type": "cuboid shape" }, "dimensions" : { "height" : {"value": 2000, "unit": "mm", "type": "Main Dimensions"} }, "material": "SUS304", "coating" : { "type" : "Special heat resistant", "Specification": "TC-300" }, "approval" : { "designer": "Tataku Taro", "factory": "Ōmika Factory", "status": "Approved" } } }

[0076] Comprehensive understanding of screw specifications This section shows an example of processing a part area containing the notation "M8×1.25-6H, depth 10, 2-φ8.5". In conventional methods, "M8", "×", "1.25", "6H", "depth 10", "2", and "φ8.5" are recognized individually, and their interrelationships are not understood.

[0077] According to the present invention, the information is understood as "M8 x 1.25 metric screw, internal thread grade 6H, depth 10 mm, φ8.5 pilot hole, 2 locations" and output in the JSON structure shown in Figure 13.

[0078] Figure 13 shows the JSON for dimension value information 53. The JSON for dimension value information 53 is transcribed below.

[0079] { "fastener" : { "type" : "internal_thread", "thread_standard" : "metric", "nominal_diameter" : {"value" : 8, "unit" : "mm"}, "pitch" : {"value" : 1.25, "unit" : "mm"}, "tolerance_class" : "6H", "depth" : {"value" : 10, "unit" : "mm"}, "pilot_hole_diameter" : {"value" : 8.5, "unit" : "mm"}, "quantity": 2, "application": "Not specified" } }

[0080] 《Prevention of misrecognition of proper nouns》 Consider the case where the approver section of the drawing states "Designer: Taro Tokatsu, Omika Factory". In conventional methods, if the input is "Designer: Taro Patent, Omika Factory," and hallucination causes a mistranslation, the output may become "Designer: Taro Mochimoto, Dalian Factory."

[0081] In this invention, when the splitting process and the approver prompt 45 are applied, the approver field, which is the input, is split into independent parts and accurately extracted by the approver prompt 45. As a result, "Designer: Omika Factory, Taro Patent" can be obtained as output. In this invention, in addition to the approver prompt 45, the designer and factory name may be stored in the RAG database, and the user may be allowed to select from the correct designer and factory names, but this is not limited to this invention.

[0082] Figure 14 shows the JSON of approver information 55 after correcting the misrecognition of proper nouns. The JSON of approver information 55 is transcribed below.

[0083] { "approval_info": { "designer": "Tataku Taro", "factory": "Ōmika Factory", "role": "Designer", "location_verified": true } }

[0084] 《Improved accuracy through adaptive prompts》 When using the conventional unified prompt method and applying the generic prompt "Read the text in the image" to all part areas of a drawing, there is a risk of insufficient understanding of the meaning of dimensions, misrecognition of proper nouns, and misinterpretation of specialized symbols.

[0085] As in the present invention, by identifying the part area type and applying a dedicated prompt, company-specific notation can be accurately extracted from the title block, the engineer's tacit knowledge can be applied to the extent that it can be mechanically derived from the dimension field using geometric constraints, etc., and names of people and places can be correctly recognized from the approver field without mistranslation. In the present invention, tacit knowledge is limited to assisting in validating the extracted values ​​and normalizing terminology, and is not used for estimating the intended use.

[0086] As an example of incorporating tacit knowledge, for instance, as a design convention, it would be good to input "In this industry, φ20H7 means bearing fit." As for quality control, it would be good to input "This approver's signature indicates that the final inspection is complete." As for manufacturing knowledge, it would be good to input "SUS304 + heat-resistant coating TC-300 is a component for high-temperature environments."

[0087] (modified version) The present invention is not limited to the embodiments described above, and includes various modifications. For example, the embodiments described above are described in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. It is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations.

[0088] Each of the above configurations, functions, processing units, and processing means may be implemented in part or in whole by hardware, such as an integrated circuit. Each of the above configurations and functions may also be implemented in software by a processor interpreting and executing a program that implements each function. Information such as programs, tables, and files that implement each function can be stored in a recording device such as memory, a hard disk, or an SSD (Solid State Drive), or on a recording medium such as a flash memory card or a DVD (Digital Versatile Disk).

[0089] In each embodiment, the control lines and information lines shown are those deemed necessary for explanation and do not necessarily represent all control lines and information lines in the actual product. In practice, it can be assumed that almost all components are interconnected. Examples of modified versions of the present invention include the following (a) and (b).

[0090] (a) Structured information is not limited to JSON format, but may be any structured data format such as XML format. (b) Multimodal large-scale language models are not limited to those running on cloud servers, but may also run on-premises. [Industrial applicability]

[0091] This invention can be widely used in the manufacturing industry for automated analysis of design drawings, quality control systems, parts procurement systems, and the like.

[0092] It is particularly effective in the following situations: In quality control, it clarifies responsibility through accurate records of designers and approvers. In traceability management, it allows for accurate retention of unique information such as manufacturing sites and inspectors. In parts procurement, it enables accurate identification of manufacturer-specific part numbers and specifications. In legal document management, it can accurately extract proper nouns contained in contracts and specifications. In design work support, it can search for similar parts from past drawings and extract design rules.

[0093] In manufacturing instruction generation, processing instructions can be generated from drawing information, and assembly procedures can be automatically generated. This significantly improves accuracy and efficiency compared to conventional methods, especially in situations requiring the processing of large volumes of technical drawings. [Explanation of symbols]

[0094] 10. Technical drawing information extraction device 11 Input section 12 Division 121 Object detection unit 122 Type Identification Unit 13 Extraction part 131 Multimodal Large-Scale Language Models 132 Prompt Application Section 14. Integration Department 141 JSON generation section 20 Technical Drawings 50 Structured information 101 CPU 102 memory 103 Secondary storage device 104 Input / Output Interfaces 106 Network Interfaces 108 Input device 107 Display device 105 Technical Drawing Information Extraction Program 100 recording media 31 Parts Area 32a Parts area 32b Parts area 33a Parts area 33b Parts area 33c Parts Area 33d Part area 33e Parts Area 33f Parts Area 34a Parts area 34b Parts area 35 parts area 36 parts area 41 Prompt for Title Block 42 Prompts for major shapes 43 Dimensional prompts 45. Prompt for Approvers 46 Prompts for bill of materials 51 Title Information 52 Main shape information 53 Dimensional Information 54. Material and Paint Information 55 Approver Information 56. Bill of Materials Information

Claims

1. An input unit that accepts technical drawings as input, The division unit analyzes the aforementioned technical drawing, identifies the type of each part region through object detection processing, and divides it into multiple part regions based on the type of drawing element. An extraction unit extracts partial information, including contextual information, from each of the divided part regions using a pre-stored multimodal large-scale language model. An integration unit that integrates multiple pieces of information obtained from the aforementioned multiple parts regions to form structured information for the entire technical drawing, Equipped with, The division section divides the part area into independent part areas based on the identified type, where one or more of the following are listed: title block, main shape, dimensions, material, paint, approver, and bill of materials. The extraction unit applies a customized extraction prompt according to the type identified by the division unit, and for the independent part region, applies a prompt that includes an instruction to extract the characters in the image exactly as they are displayed, without replacing them with common notation, thereby literally extracting the unique notation contained in the part region. A technical drawing information extraction device characterized by the following features.

2. The division section divides the technical drawing based on one or more of the following: dimension lines, dimension values, part shapes, notes, and symbols. The technical drawing information extraction device according to feature 1.

3. The extraction unit extracts partial information from each part area that includes one or more of the following: numerical meaning, part shape information, material information, geometric constraint information, and composite representation information that associates multiple drawing elements. The technical drawing information extraction device according to feature 1.

4. The integration unit integrates the plurality of partial information into a structured data format. The technical drawing information extraction device according to feature 1.

5. The extraction unit uses the multimodal large-scale language model to extract higher-order information based on the semantic relationships between drawing elements. The technical drawing information extraction device according to feature 1.

6. The aforementioned segmentation unit performs object detection processing using one or more of the following: YOLO (You Only Look Once), R-CNN (Region-Based Convolutional Neural Network), or SSD (Single Shot MultiBox Detector). The technical drawing information extraction device according to feature 1.

7. The input unit accepts technical drawings as input, The division unit performs object detection processing on the technical drawing, identifies the type of each part area through object detection processing, divides the technical drawing into multiple part areas, and, based on the identified type, divides the part areas in which one or more of the following are described as independent part areas: title block, main shape, dimensions, material, paint, approver, and bill of materials. The extraction unit performs information extraction processing for each of the multiple part regions using a multimodal large-scale language model, and extracts partial information including contextual information from each part region. The extraction unit applies a customized extraction prompt according to the type identified by the division unit, and for the independent part region, applies a prompt that includes an instruction to extract the characters in the image exactly as they are displayed, without replacing them with common notation, thereby literally extracting the unique notation contained in the part region. The integration unit integrates multiple partial information obtained from the multiple part areas to form structured information for the entire technical drawing. A method for extracting technical drawing information, characterized by including the following.

8. On the computer, Procedure for accepting technical drawings as input. A procedure to perform object detection processing on the aforementioned technical drawing, to identify the type of each part area through object detection processing, to divide the technical drawing into multiple part areas, and to divide the part areas in which one or more of the following are described, such as the title block, main shape, dimensions, material, paint, approver, and bill of materials, into independent part areas based on the identified type, A procedure for performing information extraction processing using a multimodal large-scale language model for each of the aforementioned multiple part regions, and extracting partial information including contextual information from each part region, A procedure for literally extracting unique identifiers contained in a part region by applying customized extraction prompts according to the identified type, and for the independent part region, applying prompts that include instructions to extract the characters in the image exactly as they are displayed, without replacing them with common notations. A procedure for integrating multiple partial information obtained from the aforementioned multiple part areas to form structured information for the entire technical drawing, A program for extracting technical drawing information to perform the necessary actions.

Images