Automating computer-aided graph drawing from tags using machine learning.

Abstract

This application discloses the automation of computer-aided drawing (CAD) from tags using machine learning. A method and system for automating computer-aided drafting (CAD) are provided. A CAD drawing in a first format native to a CAD application is obtained. Tags for the CAD drawing in a second, non-native format are acquired. A machine learning engine processes the tags using a machine learning model to identify tag types, identify tag attributes, associate tags with CAD objects required to update the CAD drawing, and determine CAD object attributes and CAD application attributes. CAD objects with CAD object attributes are provided in the CAD application using the CAD application attributes. User input is accepted and the CAD drawing is updated based on this input. The machine learning model is updated based on user input and used to process future tags.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of the following co-pending and co-assigned U.S. provisional patent applications, which are incorporated herein by reference, pursuant to Section 119(e) of 35 U.S.SC:

[0003] Provisional application serial number 63 / 114,952, entitled "Optical Character Recognition (OCR) for Automating Drawing from Markup Using Machine Learning-Assisted Drawing" filed on November 17, 2020, by inventors Kevin Cheung, Ravinder P. Krishnaswamy, and Damian Paul Stephen Willcox, Attorney General’s File No. 30566.0595USP1. Background of the Invention

[0004] 1. Field of Invention

[0005] This invention relates generally to computer-aided design (CAD) applications, and more specifically to a method, apparatus, system, and article of manufacture for simplifying, automating, saving time, and reducing errors in the drawing process.

[0006] 2. Description of related technologies

[0007] Once a CAD drawing is created, the parties involved in the design and construction process can provide editing / modification of the drawing. Drawing editing is typically prompted by feedback, usually in the form of annotations, which involve physical methods, such as using different colored pens / pencils on paper printouts, or digitally created by annotating digital files (such as PDFs (Portable Document Format)).

[0008] In both cases, the markings must be manually sent back to the drafter, who must then refer to the paper on their desktop or the PDF in another application on a separate monitor. The drafter then returns to the CAD application to make changes to the appropriate CAD application object using the correct criteria (e.g., layers, styles (sizes, text, etc.), colors, etc.). In other words, collaborators can add markings at low fidelity using pen / paper, along with instructions / instructions for what to add / change. The drafter then returns to the CAD application and issues manual commands and / or draws / repeats the collaborator's markings into the CAD application. Figure 1 illustrates a prior art process for the creation, marking, and merging of changes. More specifically, the CAD drawing / design 100 and the markings 104 for the requested changes (e.g., drawn by collaborator 106 on paper printout 102 or a PDF of drawing 100) are obtained by the drafter 108 (who possesses the necessary CAD drawing / drafting experience / skills). Then, the drafter 108 manually creates / updates / incorporates the requested changes into drawing 100 (i.e., using a CAD application) to produce the modified drawing 100R.

[0009] Given the above, drafter 108 must possess a knowledge base / skill set regarding how to use and create markers in CAD applications. However, while the prior art requires drafter 108 to have such a skill set, markers / feedback are typically created not by drafter 108 but by collaborator 106 (i.e., different people) who lack such a knowledge base / skill set. In the prior art, the need for manual updating of CAD drawing 100, coupled with collaborator 106's lack of experienced CAD application skills, necessitates drafter 108 and a slow manual process. This round-trip feedback creation and manual updating process is slow, error-prone, and lacks any connection between the feedback and the original design file.

[0010] Therefore, existing technology systems have not yet succeeded in bringing the marker data 104 back to the design file 100 across multiple applications (e.g., CAD applications, modeling applications, etc.). Furthermore, several prior art efforts have been undertaken, in which various tools have been created to create markers 104 in derived outputs of design files (DWF [Design Web Format], PDF, viewer, etc.). Processes such as PDF import functions or image import functions within CAD applications (e.g., in AutoCAD) are also employed. TM The application (available) allows users to bring in a PDF of the entire drawing along with the markup they are working on, but there is significant redundancy, and there is no distinction between the markup and the design drawing. In other words, the imported image may be a raster-based image, which does not separate the markup from the underlying drawing—it imports a single raster image file.

[0011] Furthermore, existing systems do not retrospectively interpret markings as CAD application objects. Additionally, various existing technologies have strived to make marking tools more intelligent, such as allowing gesture recognition to create circles or revClouds (revision clouds [cloud-shaped objects formed by polylines of continuous arcs]) or other objects. However, these efforts aim to provide cleaner markings for collaborators providing feedback. Drafters still need to ensure that CAD application objects are correct, on the correct layer, and using the correct layer styles, etc. Summary of the Invention

[0012] To address the problems of existing technologies, embodiments of the present invention construct CAD application tracking features (i.e., AutoCAD TRACE features). Tracking features provide a safe space within a CAD application, where markers can exist within the context of designing CAD drawings without interfering with them. In other words, embodiments of the present invention utilize safe features that allow storing and creating a safe space for marker feedback without affecting the drawing content / geometry, and further leverage machine learning (ML) to help drafters complete their work quickly and accurately, while still maintaining drafter review and intervention when needed.

[0013] Embodiments of this invention utilize machine learning (ML) to identify specific markers, regardless of how they were created, and associate them with the expected CAD application objects needed to update CAD drawing files. In short, ML will understand handwritten annotations, dimensions, revClouds, and instructions to modify CAD drawings, and assist drafters in updating their CAD drawings by providing the appropriate CAD application objects with the correct style in the right places.

[0014] Embodiments of the present invention also distinguish the coloring of the markings, because markings in the design phase of a project tend to follow standard colors (red means text addition—adding these annotations, dimensions, clouds as CAD application objects to the drawing, while blue is to teach "move these objects" rather than putting the text "move these objects" on the CAD drawing itself).

[0015] Therefore, embodiments of the present invention use ML to obtain a sense of text style, size style, usage pattern, etc., to ensure that geometry is created on the correct layer and has the correct attributes.

[0016] A unique capability of embodiments of the present invention includes attempting to replicate the work typically done by a drafter in a CAD application when the drafter receives a marker indicating that changes to the drawing are needed. In this respect, the person providing feedback often lacks knowledge of the drawing standards and other CAD-specific information the drafter relies on. Furthermore, while prior art systems may attempt to improve marker tools, they have never properly bridged the gap between markers and the drawing while still retaining the markers as a record of the requested changes, and CAD drawing editing itself often becomes ambiguous in these areas.

[0017] The advantages of embodiments of the present invention include significant improvements to the drafting process, reducing the time spent modifying drawings through the primary mechanism for feedback—marking. Less time spent on drafting results in fewer errors, which is attractive not only to drafters themselves but also to companies, where they can see the benefits of accuracy, efficiency, and output—allowing them to use this as a competitive advantage in projects to turn the tide and outbid their competitors. Attached Figure Description

[0018] Now refer to the accompanying drawings, in which the same reference numerals always denote corresponding parts:

[0019] Figure 1 illustrates the existing technology process for creating, tagging, and merging changes.

[0020] Figure 2 A workflow for automating assisted mapping using machine learning from markers is illustrated according to one or more embodiments of the present invention;

[0021] Figure 3 Additional views of an exemplary workflow for an automated drafting process according to one or more embodiments of the present invention are shown;

[0022] Figure 4 An advanced architecture for automating assisted mapping via mobile devices using machine learning is illustrated according to one or more embodiments of the present invention.

[0023] Figure 5 Further details of the logical flow and architecture for processing tags in machine learning to aid in automated mapping, according to one or more embodiments of the present invention, are shown;

[0024] Figure 6 A logical flow for CAD automation according to one or more embodiments of the present invention is shown.

[0025] Figure 7 This is an exemplary hardware and software environment for implementing one or more embodiments of the present invention; and

[0026] Figure 8A typical distributed / cloud-based computer system according to one or more embodiments of the present invention is illustrated schematically. Detailed Implementation

[0027] In the following description, reference is made to the accompanying drawings, which form a part thereof, and several embodiments of the invention are illustrated by way of illustration. It should be understood that other embodiments and structural modifications may be utilized without departing from the scope of the invention.

[0028] Machine learning (ML)-assisted automation of assisted mapping from labeling

[0029] Embodiments of the present invention utilize machine learning to identify marker / redline input from collaborators (i.e., in non-CAD applications). Machine learning processes the input and identifies both textual and instructional markers. Such identified markers can be used to: (i) serve as suggestions (for user acceptance), or (ii) automatically update CAD drawings within the CAD application using appropriate CAD application objects based on correct CAD application standards.

[0030] Figure 2 A workflow for automating assisted mapping using machine learning from markers is illustrated according to one or more embodiments of the present invention. An ML processing engine 200 receives marker feedback / input 204 (which may be scanned / photographed from paper markers or may be a PDF including markers). The ML processing engine 200 processes the marker feedback / input 204 and identifies different markers (e.g., text markers and / or instructional markers) (e.g., based on an ML model). In one or more embodiments, the recognition process may utilize ML model-based optical character recognition (OCR) or other image processing / recognition techniques, which may be iteratively updated via subsequent marker processing cycles. To distinguish between hand-drawn and printed entities, embodiments of the invention may utilize entity matching techniques, color, etc. Furthermore, once an entity / object has been scanned / recognized, the user may be prompted to acknowledge / confirm the scanned entity / identifier.

[0031] In embodiments of the invention, a QR code or other identifier in the document containing the mark / feedback 204 can be used to identify and associate the CAD drawing 202 as the subject of the mark 204. Alternatively, image mapping, barcodes, user input, etc., can be used to determine the appropriate CAD drawing 202. Subsequently, computer vision and / or other functions within the ML processing engine 200 can be used to orient, place, align, scale / sort by size, register, and overlay / position the mark 204 onto the drawing 202. Furthermore, the identifier can be used to map multiple different versions of the document (e.g., to a 3D solid modeling application, a CAD application, etc.).

[0032] The ML processing engine 200 can be used to provide autonomous / automated workflows where identified markers are processed into CAD application commands that can be used to execute the markers within the CAD application. Alternatively, such CAD application commands can be offered to the user as suggestions (e.g., one or more possible options) for acceptance / selection and / or can be executed autonomously. If offered as suggestions, once selected by the user, the ML processing engine 200 can update the ML model based on the user's input, thereby improving the accuracy of the ML model and marker suggestions over time. It can be noted that the original ML model can be generated using synthetic / simulated data (e.g., synthetic marker data from one or more CAD drawings). Once completed, an updated CAD drawing 202R is generated for further use / modification by the CAD application designer / drafter.

[0033] In addition to the above, in one or more embodiments of the present invention, the ML model can be run / executed / maintained locally on a client computer, or alternatively, it can be run / executed / maintained remotely on a server computer / cloud. In one or more embodiments, device and / or network capabilities can determine where the model is executed (e.g., if a local device is capable of executing the model, it can be maintained locally; if not, it can be maintained remotely on a server and / or in the cloud). Alternatively, to provide a more comprehensive model used across multiple client installations of a CAD application, the model can be maintained / executed remotely on a server / cloud accessed by all (or some) local client installations.

[0034] Figure 3 Additional views of an exemplary workflow for an automated drafting process according to one or more embodiments of the present invention are shown. Markings are captured at 302. For example, a photograph of a drawing with markings can be captured, or a PDF including markings can be acquired / captured. The paper / PDF with markings at 304 can be imported at 306 into an OVERLAY / TRACE application / feature / characteristic (e.g., a CAD application). This OVERLAY / TRACE application / feature / characteristic provides a safe space in the CAD application where markings can exist in the context of designing the CAD drawing without interfering with it. In other words, embodiments of the present invention utilize a safe feature that allows storing and creating a safe space for marking feedback without affecting the drawing content / geometry.

[0035] Marker 304 is processed by ML processing engine 308. Using ML, ML processing engine 308 identifies marker content 310 (e.g., dimensions, blocks, etc.) that can be used in CAD applications. Not all marker content 310 is intended to be placed literally on the drawing, as some are annotations for the draftsman rather than for containing CAD drawings. Furthermore, the color of the markers may have significance. Therefore, embodiments of the invention can automatically convert text annotations intended to be placed on the drawing while ignoring other unplaced annotations.

[0036] Display screen 312 shows the identification of different CAD application objects, such as revCloud 314 already placed on the desired layer (i.e., the CAD drawing layer) and the identification block 316 to be inserted into the drawing. At this time, the ML processing engine 308 can automatically perform CAD drawing modifications / edits, or alternatively, can provide the user with commands to perform editing as suggestions 318. When provided as suggestions 318, the drafting process can be automated 320 by allowing CAD application objects to be placed on the appropriate layer, style, etc. with one click based on tags (e.g., based on previous use and previous selections in the CAD drawing—both of which can be used to modify the model used by the ML processing engine 308). For example, a suggestion tag 318 can be provided, and the user can choose to accept / reject the suggestion with one click. If accepted, the suggestion tag 318 can be automatically placed according to the suggestion.

[0037] In one or more embodiments, within the OVERLAY / TRACE feature / function / application, the identified marker content 310 can be presented on an overlay that visually resembles tracing paper that has been aligned and placed on a CAD drawing. This overlay can have a hazy, semi-transparent texture (e.g., similar to actual physical tracing paper). In one or more embodiments, such an OVERLAY / TRACE capability can be an embedded CAD drawing within a parent CAD drawing (i.e., it can be an embedded DWG file, not just a layer within a DWG file). Once all marker suggestions 318 have been incorporated into the CAD drawing as needed by the user, the user can choose to turn off the overlay, thus no longer displaying the handwritten annotations / markers (i.e., leaving only the CAD drawing displaying the required merged markers). For example, the user might determine that the green annotation 322 on the drawing may not be needed on the CAD drawing and therefore can turn off the displayed overlay view.

[0038] Mobile device processing architecture

[0039] Embodiments of the present invention provide assisted mapping automation using machine learning. As described above, markers can be captured via a camera or other device with image capture capabilities (e.g., a mobile device). Once captured, various actions can be performed to provide mapping automation. Figure 4 A high-level architecture for automating assisted mapping using ML via a mobile device, according to one or more embodiments of the present invention, is illustrated. As shown, mobile device 402 captures an image containing markers (at 404) and applies a transformation at 406. The transformation converts the captured image into a form acceptable to use by ML processing engine 410. Therefore, the captured image is sent at 408 to a server or cloud for processing by ML processing engine 410. ML processing engine 410 extracts the bounding boxes of the text at 412 and extracts the bounding boxes of the revCloud at 414. These extractions at 412 and 414 are based on ML model 416.

[0040] The ML processing engine 410 sends the bounding boxes (at 418) to a server or cloud-based application for further processing. In one or more embodiments, the server / cloud application 420 is a CRX application (e.g., a browser extension). Within the server / cloud application 420, the bounding boxes are processed in steps 422-426. In step 422, an HSV (Hue, Saturation, Value) detector (or other type of detector) is used to detect the color of entities within the bounding box. For example, in addition to HSV and other color space-based segmentation, image filters and other computer vision processing techniques may be applied to more accurately distinguish the markers from the background. In other words, embodiments of the invention map / move from the RGB (Red, Green, Blue) space to the HSV space to segment the markers from the background of the drawing. In step 424, the bounding boxes (revCloud and text) are mapped to the CAD application paper space (e.g., the paper layout of the drawing within the CAD application). In step 426, the revCloud, text, and remaining markers (e.g., other marker images) are inserted as images into the CAD drawing as composite objects on a CAD layer.

[0041] ML processing details

[0042] Figure 5 Further details of the logical flow and architecture for processing markers using ML to aid in automated drafting, according to one or more embodiments of the present invention, are shown. Marker feedback 502 is input / received via paper 504, PDF 506, and / or third-party CAD / modeling / architectural software applications 508 (e.g., BIM360 applications or BIM [Building Information Modeling] viewer applications).

[0043] The primary CAD application 510 (e.g., a tracking feature within the CAD application) receives / acquires marker feedback 502. In one or more embodiments, such marker feedback 502 is imported 512 (e.g., from a tracking feature within the CAD application) into the CAD drawing itself (e.g., model space [MS] or paper space [PS]). In alternative embodiments, marker feedback 502 may also include house markers / feedback 514 created within the primary CAD application 510.

[0044] The main CAD application then requests / processes the marker feedback 502 from the ML processing engine 516. The ML processing engine 516 performs ML processing on the marker at 518. The first step is to identify the marker type at 520. The marker type can be an explicit / text drawing object 522 (e.g., a red marker used to determine the marker type using color) or a drawing instruction 524 (e.g., a blue marker used to determine the marker type using color). If the marker type is an explicit / text drawing object 522, the text of the drawing object can be determined at 524 using Optical Character Recognition (OCR), for example, via the OCR Application Programming Interface [API]. Based on the OCR processing, target CAD attributes (e.g., layers, styles, etc.) can be identified at 526. For example, the drawing object can be identified using the Object Extraction and Classification API.

[0045] At 528, appropriate drawing objects are created with suitable styles / standards. For example, drawing objects may consist of text, dimensions, blocks, revCloud, geometry, leaders, or any drawing object. In one or more embodiments, both ML and OCR are used to identify explicit / textual drawing objects 522.

[0046] If the mark type is a drafting instruction 524, then at 532 the appropriate command and object to be included in the action are identified. For example, the appropriate command and object could be a mark instruction specifying that the selection of an object be moved a set distance. At 534, the CAD application issues a command (e.g., issues a move command and selects the object specified at the appropriate distance). In this respect, using explicit / text mark 522, embodiments of the invention can initiate a CAD application command to create a marked drafting object in step 530 and place / insert it into the drawing. However, using a drafting instruction 524, the CAD application will interpret the instruction, which may include initiating a command in step 534 and / or following the instruction. Furthermore, for such drafting instructions, the selection set may assume other indicators of interest for selecting boundaries or entities.

[0047] In one or more embodiments, the markers are retained in the CAD application's "tracking" space, separate from the CAD drawing geometry. Further drafting edits can then be performed within the geometry of the CAD drawing itself.

[0048] Logical Flow

[0049] Figure 6 A logical flow for CAD automation according to one or more embodiments of the present invention is shown.

[0050] In step 602, the CAD drawing is obtained in the first format native to the CAD application.

[0051] In step 604, the markers for the CAD drawing are acquired. The markers are in a second format not native to the CAD application. In one or more embodiments, physical means are used to create the markers for transcription feedback on a paper printout of the CAD drawing. In such embodiments, the markers are acquired by acquiring an image of the transcription feedback on the paper printout (e.g., a scan / photograph of the image). In alternative embodiments, the markers can be created by digitally annotating a digital file that includes an image of the CAD drawing (e.g., a PDF).

[0052] In step 606, the markers are processed in the machine learning engine using a machine learning model. This processing identifies the marker type, identifies the marker attributes, associates the marker type with the CAD object required to update the CAD drawing (based on marker type), and determines (based on marker type and marker attributes) the CAD object attributes and the CAD application attributes of the CAD application.

[0053] In one or more embodiments, the marker type can be text markers. For text marker types, optical character recognition (OCR) can be used to determine CAD object attributes to recognize text, whereby the CAD object is provided based on the recognized text. Subsequently, the CAD drawing is updated by creating the CAD object. In such embodiments, CAD application attributes can include CAD standards, which include / for example, target layers and styles. Furthermore, CAD objects can be created in / based on such CAD standards.

[0054] In one or more embodiments, the marker type may be a drafting instruction. The drafting instruction identifies the CAD application commands and CAD objects necessary to execute the drafting instruction when a CAD object with CAD object attributes is provided. The CAD drawing is then updated by executing the identified CAD application commands using the identified CAD objects. In one or more embodiments, the drafting instruction includes moving the CAD object a predetermined distance. Based on the instruction, a group of one or more CAD objects (including the identified CAD object) is selected, and a move command is issued to the group of one or more CAD objects by the predetermined distance to update the CAD drawing.

[0055] In one or more embodiments of the present invention, the marker type is identified based on the marker color. For example, a first color may represent a text marker type, while a second color may represent a drawing instruction marker type.

[0056] To process the markers, a first bounding box of the text and a second bounding box of one or more geometric objects can be extracted. These bounding boxes can then be sent to a cloud-based application for further processing. Such further processing may include detecting the color of each of the entities in the first and second bounding boxes, determining the marker type based on the color, and mapping the first and second bounding boxes to the paper space of a CAD application. Subsequently, based on the marker type, the color of each of the entities in the first and second bounding boxes can be mapped to a representative CAD application color. Furthermore, the CAD drawing can be updated based on the text, one or more geometric objects, the marker type, and the representative CAD application color.

[0057] In step 608, CAD objects (with CAD object properties) are provided / presented in the CAD application using CAD application properties.

[0058] In step 610, in response to providing step 608 to accept user input.

[0059] In step 612, the CAD drawing is updated in the CAD application based on user input.

[0060] In step 614, the ML model is updated based on user input (e.g., feedback such as accepting and / or rejecting suggestions). This updated ML model is used to process future tagging.

[0061] Hardware environment

[0062] Figure 7An exemplary hardware and software environment 700 (referred to as a computer-implemented system and / or computer-implemented method) for implementing one or more embodiments of the present invention is provided. The hardware and software environment includes a computer 702 and may include peripheral devices. Computer 702 may be a user / client computer, a server computer, or a database computer. Computer 702 includes a hardware processor 704A and / or a dedicated hardware processor 704B (hereinafter alternatively collectively referred to as processor 704) and memory 706, such as random access memory (RAM). Computer 702 may be coupled to and / or integrated into other devices, including input / output (I / O) devices such as a keyboard 714, a cursor control device 716 (e.g., a mouse, pointing device, pen, tablet computer, touchscreen, multi-touch device, etc.), and a printer 728. In one or more embodiments, computer 702 may be coupled to or may include a portable or media viewing / listening device 732 (e.g., an MP3 player, iPod, NOOK, portable digital video player, cellular device, personal digital assistant, etc.). In yet another embodiment, computer 702 may include a multi-touch device, a mobile phone, a gaming system, an internet-enabled television, a set-top box, or other internet-enabled devices running on various platforms and operating systems.

[0063] In one embodiment, computer 702 operates under the control of operating system 708 by executing instructions defined by computer program 710 (e.g., a computer-aided design [CAD] application) via hardware processor 704A. Computer program 710 and / or operating system 708 may be stored in memory 706 and may interface with users and / or other devices to accept input and commands, and based on such input and commands, as well as the instructions 710 and operating system 708 defined by the computer program, to provide output and results.

[0064] Output / results can be presented on display 722 or provided to another device for presentation, further processing, or action. In one embodiment, display 722 includes a liquid crystal display (LCD) having a plurality of individually addressable liquid crystals. Alternatively, display 722 may include a light-emitting diode (LED) display having clusters of red, green, and blue diodes driven together to form full-color pixels. In response to data or information generated by processor 704 on input and commands according to instructions of computer program 710 and / or operating system 708, each liquid crystal or pixel of display 722 becomes opaque or translucent to form a portion of an image on the display. The image can be provided via graphical user interface (GUI) module 718. Although GUI module 718 is depicted as a separate module, the instructions for performing GUI functions may reside in or be distributed within operating system 708, computer program 710, or implemented using dedicated memory and processor.

[0065] In one or more embodiments, the display 722 is integrated with / integrated into the computer 702 and includes a multi-touch device having a touch-sensing surface (e.g., a trackpad or touchscreen) capable of recognizing the presence of two or more points of contact with the surface. Examples of multi-touch devices include mobile devices (e.g., iPhone, Nexus S, DROID devices, etc.), tablet computers (e.g., iPad, HP Touchpad, Surface devices, etc.), portable / handheld gaming / music / video player / console devices (e.g., iPod Touch, MP3 player, NINTENDO SWITCH, PLAYSTATION PORTABLE, etc.), touch tables and walls (e.g., images are projected through acrylic and / or glass and then illuminated with LED backlighting).

[0066] Some or all of the operations performed by computer 702 according to the instructions of computer program 710 can be implemented in dedicated processor 704B. In this embodiment, some or all of the instructions in computer program 710 can be implemented via firmware instructions stored in read-only memory (ROM), programmable read-only memory (PROM), or flash memory within dedicated processor 704B or memory 706. Dedicated processor 704B can also be hardwired by circuit design to perform some or all of the operations to implement the invention. Furthermore, dedicated processor 704B can be a hybrid processor, including dedicated circuitry for performing a subset of functions and other circuitry for performing more general functions such as responding to the instructions of computer program 710. In one embodiment, dedicated processor 704B is an application-specific integrated circuit (ASIC).

[0067] Computer 702 may also implement compiler 712, which allows application programs or computer programs 710 written in programming languages ​​such as C, C++, assembly, SQL, Python, PROLOG, MATLAB, Ruby, Rails, Haskell, or others to be translated into code readable by processor 704. Alternatively, compiler 712 may be an interpreter that directly executes instructions / source code, translates source code into an intermediate representation to be executed, or executes stored pre-compiled code. Such source code can be written in various programming languages, such as JAVA, JAVASCRIPT, PERL, BASIC, etc. Once completed, application programs or computer programs 710 use the relationships and logic generated by compiler 712 to access and manipulate data received from I / O devices and stored in memory 706 of computer 702.

[0068] Computer 702 may also optionally include external communication devices, such as modems, satellite links, Ethernet cards, or other devices for accepting input from other computers 702 and providing output to other computers 702.

[0069] In one embodiment, the instructions for implementing the operating system 708, computer program 710, and compiler 712 are tangibly embodied in a non-transitory computer-readable medium, such as a data storage device 720, which may include one or more fixed or removable data storage devices, such as a zip drive, floppy disk drive 724, hard disk drive, CD-ROM drive, magnetic tape drive, etc. Furthermore, the operating system 708 and computer program 710 consist of computer program 710 instructions that, when accessed, read, and executed by computer 702, cause computer 702 to perform steps necessary for implementing and / or using the present invention or for loading the instruction program into memory 706, thereby creating a dedicated data structure that enables computer 702 to operate as a specially programmed computer performing the method steps described herein. Computer program 710 and / or operating instructions may also be tangibly embodied in memory 706 and / or data communication device 730, thereby creating a computer program product or article of manufacture according to the present invention. Therefore, the terms “article of manufacture,” “program storage device,” and “computer program product” as used herein are intended to cover a computer program accessible from any computer-readable device or medium.

[0070] Of course, those skilled in the art will recognize that any combination of the above components, or any number of different components, peripherals and other devices, can be used with computer 702.

[0071] Figure 8A typical distributed / cloud-based computer system 800 is schematically illustrated, using network 804 to connect client computer 802 to server computer 806. A typical combination of resources may include network 804, which may include the Internet, LAN (Local Area Network), WAN (Wide Area Network), SNA (System Network Architecture) network, etc., as well as personal computers or workstations (such as…). Figure 7 The client 802 (described in the text), and as a personal computer, workstation, minicomputer or mainframe (such as...) Figure 7 The server 806 is described in the text. However, it can be noted that different networks, such as cellular networks (e.g., GSM [Global System for Mobile Communications] or others), satellite-based networks, or any other type of network, can be used to connect the client 802 and the server 806 according to embodiments of the invention.

[0072] A network 804, such as the Internet, connects client 802 to server computer 806. Network 804 can utilize Ethernet, coaxial cable, wireless communication, radio frequency (RF), etc., to connect and provide communication between client 802 and server 806. Furthermore, in a cloud-based computing system, resources (e.g., storage, processors, applications, memory, infrastructure, etc.) in client 802 and server computer 806 can be shared by client 802, server computer 806, and users across one or more networks. Resources can be shared by multiple users and can be dynamically reallocated as needed. In this respect, cloud computing can be described as a model for enabling access to a shared pool of configurable computing resources.

[0073] Client 802 can execute client applications or web browsers and communicate with server computer 806, which executes web server 810. This web browser is typically a program such as Microsoft Internet Explorer / Edge, Mozilla Firefox, Opera, Apple Savari, Google Chrome, etc. Furthermore, software running on client 802 can be downloaded from server computer 806 to client computer 802 and installed as a web browser plugin or ActiveX control. Therefore, client 802 can utilize ActiveX components / Component Object Model (COM) or Distributed COM (DCOM) components to provide a user interface on client 802's display. Web server 810 is typically a program such as Microsoft's Internet Information Server.

[0074] Web server 810 can host Active Server Pages (ASP) or Internet Server Application Programming Interface (ISAPI) applications 812 that may be executing scripts. Scripts invoke objects (called business objects) to perform business logic. The business objects then manipulate data in database 816 through a database management system (DBMS) 814. Alternatively, database 816 may be part of or directly connected to client 802, rather than transmitting / retrieving information from database 816 via network 804. When developers encapsulate business functionality into objects, the system may be referred to as a Component Object Model (COM) system. Therefore, scripts executed on web server 810 (and / or application 812) invoke COM objects that implement business logic. Furthermore, server 806 can utilize Microsoft's Streaming Server (MTS) to access desired data stored in database 816 via interfaces such as ADO (Active Data Objects), OLE DB (Object Linking and Embedded Database), or ODBC (Open Database Connectivity).

[0075] Typically, these components 800-816 include logic and / or data embodied in or retrieveable from a device, medium, signal, or carrier, such as a data storage device, a data communication device, a remote computer, or a device coupled to a computer via a network or another data communication device. Furthermore, when this logic and / or data is read, executed, and / or interpreted, steps necessary for implementing and / or using the present invention are performed.

[0076] Although the terms “user computer,” “client computer,” and / or “server computer” are used herein, it should be understood that such computers 802 and 806 may be interchangeable and may also include portable devices with limited or full processing capabilities, such as mobile phones, laptops, pocket computers, multi-touch devices, and / or any other device with appropriate processing, communication, and input / output capabilities.

[0077] Of course, those skilled in the art will recognize that any combination of the above-described components, or any number of different components, peripherals, and other devices, can be used with computers 802 and 806. Embodiments of the present invention are implemented as software / CAD applications on client 802 or server computer 806. Furthermore, as described above, client 802 or server computer 806 may include thin client devices or portable devices with multi-touch-based displays.

[0078] in conclusion

[0079] This concludes the description of the preferred embodiments of the invention. Some alternative embodiments for implementing the invention are described below. For example, any type of computer, such as a mainframe, minicomputer, or personal computer, or computer configuration, such as a time-sharing host, local area network, or standalone personal computer, can be used with the invention.

[0080] The foregoing description, which has presented preferred embodiments of the invention, is for illustrative and descriptive purposes. This description is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in accordance with the teachings above. The scope of the invention is intended to be limited not by the detailed description provided, but by the appended claims.

Claims

1. A computer-implemented method for computer-aided drafting (CAD) automation, comprising: (a) obtaining a CAD drawing, wherein the CAD drawing is in a first format native to a CAD application; (b) obtaining first and second markings on an image of the CAD drawing, wherein: (i) the first and second markings are in a second format that is not native to the CAD application; (ii) the first and second markings each have a marking type; and (iii) the marking type is either a text drafting object to be added to the CAD drawing or a drafting instruction to modify the CAD drawing; (c) processing, in a machine learning engine, the first and second markings using a machine learning model to: (i) identify the marking type of the first marking, wherein the marking type comprises the drafting instruction to modify the CAD drawing; (ii) identify the marking type of the second marking as the text drafting object; (iii) identify marking attributes of the first and second markings; (iv) based on the marking type, associate the first marking with a first CAD object needed to update the CAD drawing and associate the second marking with a second CAD object needed to update the CAD drawing; (v) based on the marking type and the marking attributes, determine CAD object attributes of the first and second CAD objects and CAD application attributes of the CAD application; and (d) providing, in the CAD application using the CAD application attributes, the first and second CAD objects with the CAD object attributes, wherein the providing comprises: (i) identifying one or more CAD application commands necessary to execute the drafting instruction; (ii) identifying the first CAD object necessary to execute the drafting instruction; and (iii) identifying the second CAD object to be added to the CAD drawing; (e) in response to the providing, accepting user input; (f) updating, in the CAD application, the CAD drawing based on the user input, wherein the updating the CAD drawing comprises: (i) executing the identified one or more CAD application commands using the identified CAD objects; and (ii) creating the second CAD object in the CAD drawing; and (g) updating the machine learning model based on the user input, wherein the machine learning model is used to process future markings.

2. The computer-implemented method of claim 1, wherein: the first and second markings are created using a physical method to transcribe feedback on a paper printout of the CAD drawing; and the obtaining comprises obtaining an image of the feedback transcribed on the paper printout.

3. The computer-implemented method of claim 1, wherein: The first mark and the second mark are created by digitally annotating a digital file comprising the image of the CAD drawing.

4. The computer-implemented method of claim 1, wherein: text is identified using optical character recognition (OCR) to determine the CAD object properties of the second CAD object; and the providing comprises providing the second CAD object based on the identified text.

5. The computer-implemented method of claim 4, wherein: the CAD application properties comprise one or more CAD standards comprising a target layer and a style; the second CAD object is created in the one or more CAD standards.

6. The computer-implemented method of claim 1, wherein: the drawing instruction comprises moving the first CAD object a set distance; the identifying the first CAD object comprises selecting a group of one or more CAD objects, wherein the group includes the identified first CAD object; and the updating the CAD drawing comprises issuing a move command to the group of one or more CAD objects using the set distance.

7. The computer-implemented method of claim 1, wherein: the identifying the mark type is based on a color of the mark; a first color represents the mark type comprising a text drawing object; and a second color represents the mark type comprising a drawing instruction.

8. The computer-implemented method of claim 1, wherein the processing the mark comprises: extracting a first bounding box of text; extracting a second bounding box of one or more geometric objects; sending the first bounding box and the second bounding box to a cloud-based application; the cloud-based application: detecting a color of each of one or more entities in the first bounding box and the second bounding box; determining the mark type based on the color; mapping the first bounding box and the second bounding box to a paper space of the CAD application; and updating the CAD drawing based on the text, the one or more geometric objects, the mark type, and a representative CAD application color.

9. A computer-implemented system for computer-aided drawing (CAD) automation, comprising: (a) a computer having a memory; (b) a processor executing on the computer; (c) the memory storing a set of instructions, wherein the set of instructions, when executed by the processor, cause the processor to perform operations comprising: (i) obtaining a CAD drawing, wherein the CAD drawing is in a first format native to a CAD application; (ii) obtaining a first mark and a second mark on an image of the CAD drawing, wherein: (1) the first mark and the second mark are in a second format that is not native to the CAD application; (2) the first mark and the second mark each have a mark type; (3) the mark type is either a text drawing object to be added to the CAD drawing or a drawing instruction to modify the CAD drawing. ​ ​ (iii) in a machine learning engine, processing the first mark and the second mark using a machine learning model to: (1) identify the mark type of the first mark, wherein the mark type includes modifying the drawing instruction of the CAD drawing; (2) identify the mark type of the second mark as the text drawing object; (3) identify mark attributes of the first mark and the second mark; (4) based on the mark type, associate the first mark with a first CAD object required to update the CAD drawing and associate the second mark with a second CAD object required to update the CAD drawing; (5) based on the mark type and the mark attributes, determine CAD object attributes of the first CAD object and the second CAD object and CAD application attributes of the CAD application; and (iv) in the CAD application using the CAD application attributes, provide the first CAD object and the second CAD object with the CAD object attributes, wherein the providing includes: (1) identifying one or more CAD application commands necessary to execute the drawing instruction; (2) identifying the first CAD object necessary to execute the drawing instruction; and (3) identifying the second CAD object to be added to the CAD drawing; (v) in response to the providing, accepting user input; (vi) in the CAD application, updating the CAD drawing based on the user input, wherein the updating the CAD drawing includes: (1) executing the identified one or more CAD application commands using the identified CAD object; and (2) creating the second CAD object in the CAD drawing; and (vii) updating the machine learning model based on the user input, wherein the machine learning model is used to process future marks.

10. The computer-implemented system of claim 9, wherein: the first mark and the second mark are created using a physical method for transcribing feedback on a paper printout of the CAD drawing; and the obtaining includes obtaining an image of the feedback transcribed on the paper printout.

11. The computer-implemented system of claim 9, wherein: the first mark and the second mark are created by digitally annotating a digital file including the image of the CAD drawing.

12. The computer-implemented system of claim 9, wherein: text is identified using optical character recognition (OCR) to determine the CAD object attributes of the second CAD object; and the providing includes providing the second CAD object based on the identified text.

13. The computer-implemented system of claim 12, wherein: the CAD application attributes include one or more CAD standards including a target layer and a style; the second CAD object is created in the one or more CAD standards.

14. The computer-implemented system of claim 9, wherein: the drawing instruction includes moving the first CAD object a set distance; the identifying the first CAD object includes selecting a group of one or more CAD objects, wherein the group includes the identified first CAD object; and the updating the CAD drawing includes issuing a move command to the group of one or more CAD objects using the set distance.

15. The computer-implemented system of claim 9, wherein: the identifying the mark type is based on a color of the mark; a first color representation includes the mark type of the text drawing object; and a second color representation includes the mark type of the drawing instruction.

16. The computer-implemented system of claim 9, wherein the processing the mark includes: extracting a first bounding box of text; extracting a second bounding box of one or more geometric objects; sending the first bounding box and the second bounding box to a cloud-based application; the cloud-based application: detecting a color of each of one or more entities in the first bounding box and the second bounding box; determining the mark type based on the color; mapping the first bounding box and the second bounding box to a paper space of the CAD application; and updating the CAD drawing based on the text, the one or more geometric objects, the mark type, and a representative CAD application color.

Images