Progress tracking with automatic symbol detection

Abstract

A method and system provide the ability to track progress of objects in a drawing. An object type is created and an activity type is assigned to the object type. The activity type represents progress of objects of the object type. A portable document format (PDF) drawing having a plurality of symbol instances is obtained. A graphical area containing a symbol instance is selected in the drawing. A marker is created on the drawing based on the selected graphical area. The plurality of symbol instances is autonomously detected based on the selected graphical area. A progress tracking marker instance of the marker is autonomously created for the detected symbol instance and linked to the object type. The progress of the object instance is visually tracked using a graphical user interface (GUI) visualization that provides a visual representation of the progress via the marker.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of the following jointly pending and jointly assigned U.S. provisional patent application, which is incorporated herein by reference: 35 USC119(e)

[0003] Provisional application serial number 63 / 114,933, filed on November 17, 2020, with inventors Xin Xu, Graham Garland, James Wang, Cory Wolnewitz, Christine Laffitte, Alexander Huang, Nikita Shalimov, Nicholas Moores, Brian Suwan Soe, Anand Rajagopal, Arjun Nayini, Sanjay Penumetsa Raju, Jeffrey Lin, Joseph Michael Bryan, and Paulo Rodrigues EspeschiteArantes, entitled "Progress Tracking With Automatic Symbol Detection," and agent file number 30566.0594USP1.

[0004] This application relates to the following jointly pending and jointly assigned patent applications, which are incorporated herein by reference:

[0005] U.S. Patent Application Serial No. 63 / 114,952, filed on November 17, 2020, by Kevin Cheung, Ravnidar P. Krishnaswamy, and Damian Paul Stephen Wilcox, entitled "Optical Character Recognition (OCR) for Drafting Using Machine Learning: Assisted Drafting Automation from Markups Using Machine Learning", Agent File No. 30566.0595USP1. Background Technology 1. Technical Field

[0007] This invention relates generally to Building Information Modeling (BIM), and more specifically to methods, apparatus, systems, and articles for automatically detecting symbols in site progress markers of BIM.

[0008] 2. Description of related technologies

[0009] In the construction industry, architects and designers can use computer-aided design (CAD) applications to generate accurate two-dimensional (2D) and three-dimensional (3D) CAD drawings, which will be used throughout the design project, from conceptual design to construction or assembly. In other words, CAD applications are essentially drawing tools that use computer systems to create lines and arcs to represent architectural designs. Based on CAD drawings, Building Information Modeling (BIM) applications typically provide context and tools to further manipulate and use the CAD design (e.g., labeling a line or set of lines as a wall element). However, the use of both CAD and BIM applications can be complex. Therefore, it is often useful to provide simple drawings (e.g., raster-based or vector-based drawings) that can be used by non-CAD / non-BIM designers (or other users who do not have access to, have not installed, and / or are unfamiliar with CAD / BIM applications, such as site workers). For example, such raster / vector-based drawings can be drawn by a physical printer or printed / drawn in a vector / raster-based format for on-site use. These vector / raster-based drawings do not have the associated attributes / parameters of individual CAD / BIM objects / elements within the drawing. Furthermore, once converted to this type of vector / raster-based drawing, information based on the CAD / BIM context is no longer available for or obtained from the vector / raster-based drawing.

[0010] Construction teams often seek to leverage vector / raster-based drawings across various workflows and environments. Site progress marking is one solution that allows construction teams to track the status of objects they are installing at a project site. In such workflows, real-world objects (e.g., equipment, assets, modules, panels, supports, etc.) are already graphically represented on drawings (e.g., via graphic symbols / icons). However, site workers may wish to represent / mark these objects at different construction phases (e.g., when building / constructing these objects). Therefore, site workers can manually draw shapes (called site markers) on top of the graphic symbols / icons (i.e., the real-world objects represented on the drawings) to track progress. This type of workflow can be part of a larger labor tracking concept and allows teams to see if they are installing objects and materials on time and within budget.

[0011] Marker creation is currently a manual process, driven by users drawing marks themselves on drawings (e.g., via direct annotation of PDF images) or importing them directly from BIM as pre-existing data points. However, BIM objects are often deeply coupled with other metadata and are not typically represented with the markup fidelity required by users. Other existing technology products utilize 3D modeling systems, which are complex and difficult for field workers to use and understand (e.g., compared to simple 2D raster / vector-based drawings). Furthermore, for large drawings consisting of dozens or hundreds of objects, it is difficult, if not impossible, to manually draw marks in an efficient and economical manner. For example, each digital floor plan may have dozens or hundreds of marks that need to be made on multiple floor plans representing all floors of a building. The manual markup placement process is slow and inefficient, where each mark needs to be drawn in a specific shape within a specific area of ​​the floor plan.

[0012] In view of the above, what is needed is the ability to automatically and autonomously generate site markings on vector / raster-based drawings / plans in an efficient and accurate manner. Summary of the Invention

[0013] Embodiments of this invention enable the use of on-site progress markers. On-site progress markers with automatic symbol detection allow office teams to automatically identify specific areas on drawings that they wish to make traceable markers, and these specific areas are associated with the objects they are installing / constructing.

[0014] More specifically, site progress markers allow construction teams to visually track on-site production progress. Each trackable object is represented by a marker placed on a digital construction plan (viewable via tablet or web experience), and the object is updated through a series of color-coordinated activities. For example, a drywall object might go through the installation phases of “reservation,” “framing,” and “closure,” each activity represented by a unique color code.

[0015] Automatic symbol detection allows creators to automatically identify areas or numerical symbols to be marked, and digital floor plans can be instantly drawn with markings that match the characteristics of the areas. What used to take hours with existing technology can now be completed in less than a minute.

[0016] Embodiments of the present invention further provide the use of heuristic or ML models. The heuristic or ML model is created from pre-existing digital drawings with similar or trained symbols. When the model is invoked, the user applies it to areas in which they wish to create similar markings. The user can then add or delete markings that do not meet the requirements. When the user corrects the markings, the model captures these changes and learns to improve its symbol detection over time. Attached Figure Description

[0017] Now refer to the accompanying drawings, in which the same reference numerals always denote corresponding parts.

[0018] Figure 1 A workflow for progress tracking according to one or more embodiments of the present invention is shown;

[0019] Figure 2A Screenshots are shown of a graphical user interface for creating object types according to one or more embodiments of the present invention;

[0020] Figure 2B A dialog box for creating object types according to one or more embodiments of the present invention is shown;

[0021] Figure 2C An add activity dialog window is shown according to one or more embodiments of the present invention;

[0022] Figure 3A Exemplary vector / raster-based drawings obtained according to one or more embodiments of the present invention are shown;

[0023] Figure 3B An exemplary dialog window for selecting an object type is shown according to one or more embodiments of the present invention;

[0024] Figure 3C A graphical user interface for identifying graphical areas from which a user expects to create markers is shown according to one or more embodiments of the present invention;

[0025] Figure 3D The creation of a circular marker according to one or more embodiments of the present invention is illustrated;

[0026] Figure 3E An exemplary dialog window for confirming object type and marker shape selection and initiating symbol detection processing according to one or more embodiments of the present invention is shown;

[0027] Figure 3F Drawings showing multiple progress tracking markers generated / created according to one or more embodiments of the present invention;

[0028] Figure 3G Drawings showing final progress tracking markers are illustrated according to one or more embodiments of the present invention;

[0029] Figure 4A A dashboard visualization for real-time snapshots is shown according to one or more embodiments of the present invention;

[0030] Figure 4BA dashboard visualization of the past six (6) weeks is shown according to one or more embodiments of the present invention;

[0031] Figure 4C A dashboard visualization showing specific drawings according to one or more embodiments of the present invention;

[0032] Figure 5 The diagram illustrates a logical flow for tracking the progress of objects in the current drawing according to one or more embodiments of the present invention;

[0033] Figure 6 These are exemplary hardware and software environments for implementing one or more embodiments of the present invention; and

[0034] Figure 7 This illustration schematically depicts a typical distributed / cloud-based computer system that uses a network to connect a client computer to a server computer according to one or more embodiments of the present invention. Detailed Implementation

[0035] In the following description, reference is made to the accompanying drawings, which form a part of the description and illustrate several embodiments of the invention by way of illustration. It should be understood that other embodiments and structural changes may be utilized without departing from the scope of the invention.

[0036] Progress tracking

[0037] Figure 1 A workflow for progress tracking according to one or more embodiments of the present invention is shown.

[0038] During Phase 1.102, object types and activities were created. The first step of Phase 1 was to specify / create object type 104. Object type 104 is the high-level object that the user expects to track. For example, a dry wall connector can track “wall, firewall, etc.”, and an electrical connector can track “socket or junction box.” Figure 2A A screenshot of the graphical user interface 200 is shown, in which the user can select the "Create Object Type" button 202 to create the object type that the user expects to track. Figure 2B Showing the options Figure 2A The Create Object Type dialog box 204 is displayed when the Create Object Type button 202 is pressed. As shown, the user can name the object type in area 206. In this respect, Figure 2B An entry with the exemplary object type name "Basic Wall" is shown. When ready to move to the next step, the user can select the "Next" button 208.

[0039] Back Figure 1For each object type 104, a collection of one or more activity types 106 can be created. Activity type 106 is the specific activity whose progress the user wants to track. For example, for activity types, the "wall" object type could have "framework, insulation, drywall, etc." (i.e., the wall object type can be configured through different activities). In another example, for activity types, the "outlet" object type could have "reserved, ready to be powered, powered, etc." Figure 2C It shows what happens once the user has finished creating the application. Figure 2B The object type in (e.g., by selecting) Figure 2B The "Next" button 208 displays the Add Activity dialog box 210. In the Add Activity dialog box 210, the user can assign activities to the created objects. In an exemplary implementation, the user can specify / assign an optional color for each activity in column 212 and specify / assign a name for each activity in column 214. To complete object type creation and activity assignment, the user can select the "Create" button 216.

[0040] Back to Figure 1 After creating object type 104 and assigning activity 106 in stage 1 102, the embodiment of the invention transitions to stage 2 108 of symbol detection 110, which can be selected... Figure 2A The "Mark on Drawing" button 218 is used to initiate the process. During symbol detection 110, the user can select symbols on drawing 112 to automatically create progress tracking markers 114 linked to a specific object type 104. The application programming interface (API) for symbol detection can be based on a heuristic machine learning (ML) model. Markers 114 can be linked to traceable objects 116 to provide a visual tracking experience. In this respect, traceable objects 116 are the link between object type 104 and markers 114 for the visual tracking experience.

[0041] To begin phase 2 108, design drawings 112 are obtained. More specifically, vector / raster (e.g., PDF) based drawings can be obtained. Figure 3A An exemplary vector / raster (e.g., PDF) based drawing 302 is shown. The next step in stage 2108 is to pre-select the object type to which symbol detection will be performed (e.g., drywall object type). Figure 3B An exemplary dialog window 304 is shown for selecting an object type (e.g., via a drop-down selection box 306).

[0042] When selecting the object type (e.g., via...), Figure 3B After the dialog window returns a 304 error, Figure 3CThe interactive selection screen can be used to identify the graphical area from which the user expects to create markers. Specifically, the user can draw a bounding box around the symbol on which the model is to be run (e.g., it can be displayed with different lines / color patterns / shadings, etc.). Figure 3C In the process, the user has drawn a bounding box 308 around a specific symbol instance 310.

[0043] Once the bounding box 308 is drawn, the user creates a marker of the desired shape and size, which will be created for each detected symbol. Figure 3D The creation of circular marker 312 is shown. Different shapes (e.g., circle, X-shape, square, line, arrow line, etc.) can be selected from selection area 314 to be used as marker 312.

[0044] After creating marker 312, the user can be prompted to confirm the selection in order to run symbol detection. Figure 3E An exemplary dialog window 316 is shown for confirming the selection (i.e., the selection of object type and marker shape) and initiating the symbol detection process (e.g., via selecting the "Run Detection" button 318). Alternatively, the user can select the "Redo Region Selection" button 320 to redo the bounding box / marker.

[0045] Once the user selects to continue, the system processes drawing 302 to identify any other symbols in drawing 302 that are similar to the symbols within bounding box 308. Additionally, the system generates progress tracking markers on such identified symbols. Various image recognition techniques, including template matching, computer vision techniques, machine learning models, and vector data queries (e.g., search / comparison based on similar combinations of vectors), can be used to perform the process of searching the drawing within bounding box 308 and identifying symbols based on images. Furthermore, embodiments of the invention can also consider the orientation and scaling of symbols and / or the type of drawing / symbol being analyzed. For example, mechanical drawings / symbols or electrical drawings / symbols can utilize search techniques that differentiate them from each other and / or from other drawing / symbol types. Moreover, embodiments of the invention are not limited to identifying and searching symbols, but can also identify and search for different patterns, rooms, and / or locations (e.g., identifying room boundaries, room types, etc.). Furthermore, image patterns in drawings can represent different types of materials (e.g., different types of firewalls, hardwood floors with different patterns compared to carpets, etc.). Therefore, embodiments of the invention can distinguish / identify different types of elements / objects within a drawing.

[0046] Then, a graphical user interface with the created tags can be presented to the user. Figure 3FDrawing 302 is shown with multiple progress tracking markers 322 that have been generated / created (i.e., based on other symbols in the drawing that match symbol 310). Furthermore, at this stage, the user can select an option to remove a specific marker (e.g., marker 324) (e.g., by selecting marker 324 and clicking the "Remove" button 326). For example, if a specific marker 324 is created incorrectly or out of order, it can be deleted. As described above, in one or more embodiments, the symbol identification and marker creation process can be based on a heuristic or machine learning model. In this regard, when marker 324 is removed, the model used to identify / search drawing 302 can be updated so that the model detects / identifies / recognizes symbols more accurately over time. More specifically, the symbol detection process iteratively detects symbols based on such an ML model through multiple detection cycles, thereby improving the model over time.

[0047] Once the user has completed the created markers, the system can display the drawing with appropriate progress tracking markers. Figure 3G A drawing is shown that displays all final progress tracking markers 322 (e.g., distinguishable colors / patterns as described in the attributes / configurations above) according to one or more embodiments of the invention. This process can be repeated for each drawing (e.g., one drawing for each floor of a building). For example, if a building has fifty (50) floors and one hundred (100) markers are defined for each floor, the operation of manually identifying each marker for all fifty (50) floors, which previously required several hours, can now be performed almost instantly in an automated / autonomous manner.

[0048] Once complete, you can add / track the activity sequence of individual progress tracking markers 322 (or subsets / sets of such markers 322). In this regard, you can select a set of one or more progress tracking markers 322 (or select all markers 322 as a group), and update such a set of selected markers 322 by updating the activity of the selected markers 322. For example, you can select two or three markers 322 (or all markers 322) for the "Basic Wall" symbol, and set the activity to "Insulation" for all selected markers 322.

[0049] Use Case Overview

[0050] As described above, the core workflow is the progress tracking workflow. In the progress tracking (also known as "symbol detection") workflow, the user identifies the type of progress object to be created based on the previous progress tracking configuration. Once the symbol to be found on the drawing is identified, a separate progress tracking object instance (marker) is automatically created on the drawing. The user can then track the progress created against the created progress tracking marks, which will provide progress data in the dashboard (see description below) and in the exported data. To provide this type of workflow, the user can upload a PDF (Portable Document Format) document to the application, which converts the PDF into a "drawing". In the application, the user can then navigate to the given drawing and access, such as... Figure 1 , Figures 2A to 2C and Figures 3A to 3G The described "symbol detection" or "progress tracking" workflow. Users identify symbols on drawings (e.g., such as...). Figure 3C The depicted symbol represents the type of object / location (e.g., wall, column, junction box, bathroom, etc.). Figure 3D The depicted marker). The user confirms the selection (e.g., as shown). Figure 3E The tool, as depicted, identifies the symbols being detected and runs a detection tool. The tool in an embodiment of the invention (e.g., via an ML model) identifies the symbols being detected and marks any additional such symbols present on the "drawing". The tool then marks these symbols throughout the drawing as categories of previously defined objects / locations. The user can delete / add marked objects (e.g., such as...). Figure 3F (As described), these deletions / additions can be fed back into the system to improve the ML model.

[0051] An alternative workflow can be called a material and labor estimation workflow. In this type of workflow, during the symbol detection workflow, the user identifies the "estimate type" object to be created (based on a list of estimate types in the estimate package). Once the symbol to be found on the drawing is identified, a separate estimate object instance (marker) is created on the drawing. Users can then use this estimate data to support their cost estimations for materials and labor.

[0052] The additional workflow can be termed the location detection workflow. Building upon and during the symbol detection workflow, users identify that they are attempting to automatically / autonomously define room and location boundaries. The system automatically / autonomously locates room boundaries and identifies specific rooms and locations within each bounded location using text, numbers, and other symbols within different “rooms” (i.e., based on a “location decomposition structure” that includes a comprehensive hierarchical decomposition of all locations in the project). Once location boundaries are identified and confirmed, these bounded locations can automatically / autonomously provide “location” data to other products utilizing the drawing viewer. For example, issues, Requests for Information (RFIs), progress markers, etc., created on the drawings can automatically “inherit” the locations defined in this process.

[0053] Dashboard

[0054] Once a traceable object is defined on the drawing, it may be desirable to track the object / element's progress over time. (Reference) Figure 1 Dashboard 118 provides users with the opportunity to view their current progress and their position in time via one or more visualizations (e.g., which may be “easy to scan”).

[0055] Figure 4A A dashboard visualization according to one or more embodiments of the present invention is shown. The user can view the dashboard in area 402 (this area may have, as described above, i.e., regarding...). Figure 2B In the configuration of the same object type, select "Object Type". Users can also select the drawing whose progress they wish to track in area 404, and whether to track progress as a percentage 406A or a count 406B. In this regard, the data is represented as a percentage 406A or a count 406B of the markers for a given object type 402. Alternative implementations may also provide the ability to calculate progress based on measurement data, which can be based on the user's "calibration" of their drawings, which can be used to derive measurement data from the markers. For example, a wall using linear markers might provide "linear feet" in the progress calculation.

[0056] Once the parameters are selected / defined in zones 402-406, the results of progress tracking can be dynamically displayed in real time in charts 408A and 408B. If no specific drawing is selected (e.g., in zone 404), charts 408A and 408B represent a collection of all individual progress tracking object instances (markers) across the entire project (see “252”, as the denominator of each column in the basic wall breakdown bar chart 408A). Each item in chart 408 comes from an “Activity” defined in the configuration. Each progress tracking object instance has a list of activities marked as completed. Chart 408 reflects the sum of activities marked as completed for each object type. The left-hand chart 408A shows a snapshot of completion at a specific point in time (e.g., today on a specific date in the past) (as indicated by the date in zone 41), while the right-hand chart 408B shows progress over time (e.g., completion status over the past 7 days) (as indicated in zone 412). Figure 4A In the example selected in Zone 410, Chart 408A shows a snapshot as of March 8, 2021, where 249 of the 252 basic walls have completed layout activities, 248 of the 252 basic walls have completed top track activities, 215 of the 252 basic walls have completed framing activities, 133 of the 252 basic walls have completed dry-hanging activities, and 102 of the 252 basic walls have completed mortar sealing activities. Chart 408B shows the progress of the basic walls selected in Zone 412 over the past 7 days.

[0057] Figure 4B The dashboard visualization is shown, and its display is consistent with... Figure 4A The same visualization is used, but the basic wall progress chart 408B utilizes the past 6 weeks selected in area 412. In this respect, users are able to select different date ranges.

[0058] Figure 4C A dashboard visualization is shown that is filtered by selecting a specific drawing (e.g., one (1) selected drawing) in area 404. Such individual drawing selection allows users to filter the view to a specific drawing in order to understand the current progress status or rate of progress at a given location. In an alternative implementation, users may also have the ability to filter by a defined “location” (such as “second floor” or “room 201”).

[0059] Logical Flow

[0060] Figure 5 The diagram illustrates a logical flow for tracking the progress of objects in the current drawing according to one or more embodiments of the present invention.

[0061] In step 502, an object type is created in the computer application.

[0062] At step 504, two or more activity types are assigned to the object type. These two or more activity types represent the progress of objects of the object type.

[0063] At step 506, the current drawing is obtained. The current drawing is a portable document format (PDF) document, which includes multiple symbol instances of symbols. Each of these symbol instances represents an object instance of an object.

[0064] At step 508, select a graphic area in the current drawing. This graphic area contains one of multiple symbol instances. To select the graphic area, the user can draw a bounding box around the symbol.

[0065] In step 510, a mark is created on the current drawing based on the selected graphic area. To create the mark, the user can specify its shape and size.

[0066] At step 512, multiple symbol instances are detected autonomously (e.g., automatically, dynamically, and in real-time, without additional user input) based on the selected graphic region. Such detection can be performed using a maintained ML or heuristic model. This ML model models symbols previously detected on other / previously processed drawings. Furthermore, the ML model is updated based on user input correcting progress tracking marker instances (described below in step 514). In this respect, the ML model is applied based on the selected graphic region to detect multiple symbol instances in the current drawing.

[0067] At step 514, progress tracking tag instances are autonomously (e.g., automatically, dynamically, in real-time, without additional user input) created for multiple symbol instances. Each progress tracking tag instance is linked to an object type. Once autonomously created, user input can be accepted / received to correct (e.g., add / remove) one or more progress tracking tag instances.

[0068] At step 516, a graphical user interface (GUI) visualization is used to visually track the progress of object instances. This GUI visualization provides a visual representation of progress via progress tracking marker instances. The GUI visualization may graphically distinguish progress tracking marker instances based on the current activity type associated with each progress tracking marker instance. In one or more embodiments, the GUI visualization may include a dashboard visualizing progress over time. Alternatively (or additionally), the GUI visualization may include a dashboard visualizing the sum of activity types completed for each object type. Furthermore, the GUI visualization may provide a unique color for each activity type.

[0069] Hardware environment

[0070] Figure 6 This is an exemplary hardware and software environment 600 (referred to as a computer-implemented system and / or computer-implemented method) for implementing one or more embodiments of the present invention. The hardware and software environment includes a computer 602 and may include peripheral devices. Computer 602 may be a user / client computer, a server computer, or a database computer. Computer 602 includes a hardware processor 604A and / or a dedicated hardware processor 604B (hereinafter alternatively collectively referred to as processor 604) and memory 606, such as random access memory (RAM). Computer 602 may be coupled to and / or integrated with other devices, including input / output (I / O) devices such as a keyboard 614, a cursor control device 616 (e.g., a mouse, pointing device, pen and graphics tablet, touchscreen, multi-touch device, etc.), and a printer 628. In one or more embodiments, computer 602 may be coupled to or may include a portable or media viewing / listening device 632 (e.g., an MP3 player, iPod, NOOK, portable digital video player, cellular device, personal digital assistant, etc.). In yet another embodiment, computer 602 may include a multi-touch device, a mobile phone, a gaming system, an internet-enabled television, a set-top box, or other internet-enabled devices running on various platforms and operating systems.

[0071] In one embodiment, computer 602 is operated by hardware processor 604A under the control of operating system 608, executing instructions defined by computer program 610 (e.g., computer-aided design [CAD] application, building information modeling (BIM) application, etc.). Computer program 610 and / or operating system 608 may be stored in memory 606 and may interact with users and / or other devices to accept input and commands, and provide output and results based on such input and commands and instructions defined by computer program 610 and operating system 608.

[0072] The output / result may be presented on display 622 or provided to another device for presentation, further processing, or action. In one embodiment, display 622 includes a liquid crystal display (LCD) having a plurality of individually addressable liquid crystals. Alternatively, display 622 may include a light-emitting diode (LED) display having clusters of red, green, and blue diodes driven together to form full-color pixels. In response to data or information generated by input and commands applied by processor 604 from computer program 610 and / or operating system 608, each liquid crystal or pixel of display 622 changes to an opaque or translucent state to form a portion of an image on the display. The image may be provided via graphical user interface (GUI) module 618. Although GUI module 618 is depicted as a separate module, the instructions for performing GUI functions may reside or be distributed within operating system 608, computer program 610, or implemented using dedicated memory and processor.

[0073] In one or more embodiments, display 622 is integrated with / integrated into computer 602 and includes a multi-touch device with a touch-sensing surface (e.g., a tracking box or touchscreen) capable of recognizing the presence of two or more contact points with the surface. Examples of multi-touch devices include mobile devices (e.g., iPhone, Nexus S, Droid devices, etc.), tablet computers (e.g., iPad, HP Touchpad, Surface devices, etc.), portable / handheld gaming / music / video player / console devices (e.g., iPod Touch, MP3 player, NINTENDO SWITCH, PLAYSTATION PORTABLE, etc.), touchpads, and walls (e.g., where images are projected through acrylic and / or glass and then illuminated with LED backlighting).

[0074] Some or all of the operations performed by computer 602 according to the instructions of computer program 610 can be implemented in dedicated processor 604B. In this embodiment, some or all of the instructions in computer program 610 can be implemented via firmware instructions stored in read-only memory (ROM), programmable read-only memory (PROM), or flash memory within dedicated processor 604B or memory 606. Dedicated processor 604B can also be hardwired by circuit design to perform some or all of the operations implementing the present invention. Furthermore, dedicated processor 604B can be a hybrid processor, including dedicated circuitry for performing a subset of functions, and other circuitry for performing more general functions such as responding to the instructions of computer program 610. In one embodiment, dedicated processor 604B is an application-specific integrated circuit (ASIC).

[0075] Computer 602 may also implement compiler 612, which allows application programs or computer programs 610 written in programming languages ​​such as C, C++, assembly, SQL, Python, PROLOG, MATLAB, Ruby, Rails, Haskell, or others to be translated into code readable by processor 604. Alternatively, compiler 612 may be an interpreter that directly executes instructions / source code, translates source code into an intermediate representation to be executed, or executes stored pre-compiled code. Such source code can be written in various programming languages ​​such as JAVA, JAVASCRIPT, PERL, BASIC, etc. Once completed, application programs or computer programs 610 use the relationships and logic generated by compiler 612 to access and manipulate data received from I / O devices and stored in memory 606 of computer 602.

[0076] Computer 602 may also optionally include external communication devices, such as modems, satellite links, Ethernet cards, or other devices for accepting input from other computers 602 and providing output to other computers.

[0077] In one embodiment, the instructions implementing the operating system 608, computer program 610, and compiler 612 are tangibly embodied in a non-transitory computer-readable medium (e.g., data storage device 620), which may include one or more fixed or removable data storage devices, such as a zip drive, floppy disk drive 624, hard disk drive, CD-ROM drive, magnetic tape drive, etc. Furthermore, the operating system 608 and computer program 610 include computer program 610 instructions that, when accessed, read, and executed by computer 602, cause computer 602 to perform steps necessary for implementing and / or using the present invention, or to load the instruction program into memory 606, thus creating a dedicated data structure that enables computer 602 to operate as a specially programmed computer performing the method steps described herein. Computer program 610 and / or operating instructions may also be tangibly embodied in memory 606 and / or data communication device 630, thereby forming a computer program product or article of manufacture according to the present invention. Thus, the terms “article of manufacture,” “program storage device,” and “computer program product” as used herein are intended to include computer programs accessible from any computer-readable device or medium.

[0078] Of course, those skilled in the art will recognize that computer 602 may use any combination of the above-described components, or any number of different components, peripherals and other devices.

[0079] Figure 7A typical distributed / cloud-based computer system 700 is illustrated schematically, connecting a client computer 702 to a server computer 706 using a network 704. A typical combination of resources may include the network 704, client 702, and server 706, such as the Internet, LAN (Local Area Network), WAN (Wide Area Network), SNA (System Network Architecture) network, etc., with the client being a personal computer or workstation (e.g., ...). Figure 6 As shown), the server is a personal computer, workstation, minicomputer, or mainframe (e.g., Figure 6 (As shown). However, it can be noted that, according to embodiments of the invention, different networks such as cellular networks (e.g., GSM [Global System for Mobile Communications] or others), satellite-based networks, or any other type of network can be used to connect client 702 and server 706.

[0080] Network 704 (such as the Internet) connects client 702 to server computer 706. Network 704 can utilize Ethernet, coaxial cable, wireless communication, radio frequency (RF), etc., to connect and provide communication between client 702 and server 706. Furthermore, in a cloud-based computing system, resources (e.g., storage devices, processors, applications, memory, infrastructure, etc.) in client 702 and server computer 706 can be shared by client 702, server computer 706, and users across one or more networks. Resources can be shared by multiple users and can be dynamically reallocated as needed. In this respect, cloud computing can be described as a model for enabling access to a shared pool of configurable computing resources.

[0081] Client 702 may execute client applications or web browsers and communicate with server computer 706, which executes web server 710. Such web browsers are typically programs such as Microsoft Internet Explorer / Edge, Mozilla Firefox, Opera, Apple Savari, and Google Chrome. Furthermore, software executing on client 702 may be downloaded from server computer 706 to client computer 702 and installed as a web browser plugin or ActiveX control. Therefore, client 702 may utilize ActiveX components / Component Object Model (COM) or Distributed COM (DCOM) components to provide a user interface on client 702's display. Web server 710 is typically a program such as Microsoft Internet Information Server.

[0082] Web server 710 may host Dynamic Server Pages (ASP) or Internet Server Application Programming Interface (ISAPI) application 712, which may be executing scripts. The scripts invoke objects (referred to as business objects) to perform business logic. The business objects then manipulate data in database 716 via database management system (DBMS) 714. Alternatively, database 716 may be part of client 702 or directly connected to client 702, rather than transmitting / retrieving information from database 716 via network 704. When developers encapsulate business functionality into objects, this system may be referred to as a Component Object Model (COM) system. Therefore, scripts executed on web server 710 (and / or application 712) invoke COM objects that implement business logic. Furthermore, server 706 may utilize Microsoft's Transmission Server (MTS) to access desired data stored in database 716 via interfaces such as ADO (Active Data Objects), OLE DB (Object Linking and Embedded Database), or ODBC (Open Database Connectivity).

[0083] Generally, these components 700-716 include logic and / or data embodied in or retrieved from a device, medium, signal, or carrier (e.g., a data storage device, a data communication device, a remote computer, or a device coupled to a computer via a network or another data communication device). Furthermore, when read, executed, and / or interpreted, this logic and / or data results in the execution of steps necessary to implement and / or use the present invention.

[0084] Although the terms “user computer,” “client computer,” and / or “server computer” are used herein, it should be understood that such computers 702 and 706 may be interchangeable and may also include thin client devices with limited or full processing capabilities, portable devices (such as cellular phones, laptops, pocket computers, multi-touch devices), and / or any other device with appropriate processing, communication, and input / output capabilities.

[0085] Of course, those skilled in the art will recognize that any combination of the above-described components, or any number of different components, peripherals, and other devices, can be used with computers 702 and 706. Embodiments of the present invention are implemented as software / CAD applications on client 702 or server computer 706. Furthermore, as described above, client 702 or server computer 706 may include a thin client device or a portable device with a multi-touch-based display.

[0086] in conclusion

[0087] The description of the preferred embodiments of the present invention concludes here. Some alternative embodiments for implementing the present invention are described below. For example, any type of computer, such as a mainframe, minicomputer, or personal computer, or computer configuration such as a time-sharing mainframe, local area network, or standalone personal computer, can be used with the present invention.

[0088] For purposes of illustration and description, the preferred embodiments of the invention have been described above. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible based on the foregoing teachings. The scope of the invention is not limited by this detailed description, but rather by the appended claims.

Claims

1. A computer-implemented method for tracking progress of objects in a current drawing, the computer-implemented method comprising: (a) creating an object type in a computer application; (b) assigning two or more activity types to the object type in the computer application, wherein the two or more activity types represent progress of objects of the object type; (c) obtaining the current drawing in the computer application, wherein: (i) the current drawing comprises a portable document format (PDF) document without a computer-aided design (CAD) or building information modeling (BIM) context; (ii) the current drawing comprises a plurality of symbol instances of a symbol, wherein the plurality of symbol instances comprises graphical symbols on the current drawing; (iii) the plurality of symbol instances each represents an object instance of the object; (d) selecting a graphical region in the current drawing, wherein the graphical region contains one of the plurality of symbol instances; (e) creating a marker on the current drawing based on the selected graphical region; (f) autonomously detecting the plurality of symbol instances based on the selected graphical region; (g) autonomously creating progress tracking marker instances of the marker for the detected plurality of symbol instances, wherein each progress tracking marker instance is linked to the object type; and (h) visually tracking the progress of the object instances intuitively using a graphical user interface (GUI) visualization, wherein the GUI visualization provides a visual representation of the progress via the progress tracking marker instances.

2. The computer-implemented method of claim 1, wherein the selecting the graphical region comprises drawing a bounding box around the symbol.

3. The computer-implemented method of claim 1, wherein the creating the marker comprises specifying a shape and size of the marker.

4. The computer-implemented method of claim 1, further comprising: accepting user input that corrects the progress tracking marker instances.

5. The computer-implemented method of claim 4, wherein the autonomously detecting comprises: maintaining a machine learning (ML) model, wherein: the ML model models symbols previously detected on other drawings; and the ML model is updated based on the user input that corrects the progress tracking marker instances; and applying the ML model to detect the plurality of symbol instances in the current drawing based on the selected graphical region.

6. The computer-implemented method of claim 4, wherein the correcting removes one or more of the progress tracking marker instances.

7. The computer-implemented method of claim 1, wherein the GUI visualization graphically distinguishes the progress tracking marker instances based on a current activity type associated with each progress tracking marker instance.

8. The computer-implemented method of claim 1, wherein the GUI visualization comprises a dashboard that visualizes the progress over time.

9. The computer-implemented method of claim 1, wherein the GUI visualization comprises a dashboard that visualizes a sum of the activity types that have been completed for each object type.

10. The computer-implemented method of claim 1, wherein the GUI visualization provides a unique color for each activity type.

11. A computer-implemented system for tracking object progress in a current drawing, the computer-implemented system comprising: (a) a computer having a memory; (b) a processor executing on the computer; (c) the memory storing a set of instructions, wherein the set of instructions, when executed by the processor, cause the processor to perform operations comprising: (i) creating an object type in a computer application; (ii) assigning two or more activity types to the object type in the computer application, wherein the two or more activity types represent progress of objects of the object type; (iii) obtaining the current drawing in the computer application, wherein: (1) the current drawing comprises a portable document format (PDF) document without a computer-aided design (CAD) or building information modeling (BIM) context; (2) the current drawing comprises a plurality of symbol instances of a symbol, wherein the plurality of symbol instances comprises graphical symbols on the current drawing; (3) the plurality of symbol instances each represent an object instance of the object; (iv) selecting a graphical region in the current drawing, wherein the graphical region contains one of the plurality of symbol instances; (v) creating a marker on the current drawing based on the selected graphical region; (vi) autonomously detecting the plurality of symbol instances based on the selected graphical region; (vii) autonomously creating progress tracking marker instances of the marker for the detected plurality of symbol instances, wherein each progress tracking marker instance is linked to the object type; and (viii) visually tracking the progress of the object instances using a graphical user interface (GUI) visualization, wherein the GUI visualization provides a visual representation of the progress via the progress tracking marker instances.

12. The computer-implemented system of claim 11, wherein the selecting the graphical region comprises drawing a bounding box around the symbol.

13. The computer-implemented system of claim 11, wherein the creating the marker comprises specifying a shape and size of the marker.

14. The computer-implemented system of claim 11, further comprising: accepting user input that corrects the progress tracking marker instances.

15. The computer-implemented system of claim 14, wherein the autonomously detecting comprises: maintaining a machine learning (ML) model, wherein: the ML model models symbols previously detected on other drawings; and the ML model is updated based on the user input that corrects the progress tracking marker instances; and applying the ML model based on the selected graphical region to detect the plurality of symbol instances in the current drawing sheet.

16. The computer-implemented system of claim 14, wherein the correction removes one or more of the progress tracking marker instances.

17. The computer-implemented system of claim 11, wherein the GUI visualization graphically distinguishes the progress tracking marker instances based on a current activity type associated with each progress tracking marker instance.

18. The computer-implemented system of claim 11, wherein the GUI visualization comprises a dashboard that visualizes the progress over time.

19. The computer-implemented system of claim 11, wherein the GUI visualization comprises a dashboard that visualizes a sum of the activity types that have been completed for each object type.

20. The computer-implemented system of claim 11, wherein the GUI visualization provides a unique color for each activity type.

Images