Hand-drawn sketch recognition using visual arrow relationship detection

Abstract

Computer-readable media, methods, and systems for converting an image into a digital image format are disclosed. The image is received as an image file, and multiple recognition levels generate a final image based on predicted information from one or more recognition levels. The multiple recognition levels include a shape detection level for detecting multiple shapes and at least one arrow detection level for detecting arrows as relationships between shape pairs. The final image is generated based on the predicted information and converted into a digital image format compatible with an image modeling language.

Description

Technical Field

[0001] This disclosure relates to the recognition of hand-drawn illustrations using visual arrow relationship detection. Background Technology

[0002] Diagrams can be used to represent various concepts, such as business processes, algorithms, and software architectures. Diagrams can include various symbols, lines, and arrows representing concepts and the relationships between them. In many cases, such diagrams can be created using hand-drawn techniques, where it may be difficult to convert hand-drawn diagrams into digital modeling formats.

[0003] Hand-drawn illustrations can be manually recreated using modeling tools or automatically recreated using illustration recognition technology. Manually recreating illustrations is tedious and often requires users to invest significant time in remodeling within the software, thus introducing the possibility of user errors. Automatic illustration recognition technology has been used to create digital representations of hand-drawn illustrations. However, current illustration recognition technologies strive to correctly identify and recreate many illustration features, such as arrows within the initial illustration, and typically require additional user input to correct errors during the recognition process. Summary of the Invention

[0004] Embodiments of this disclosure address the aforementioned problems by providing systems, methods, and computer-readable media for automatic image recognition to convert an initial image of an image into a digital format. In some embodiments, visual relationship detection techniques are used to detect a relationship between one or more arrows as a pair of detected shapes within a drawn image.

[0005] The first embodiment relates to one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by a processor, perform a method for converting an image into a digital model. The method includes: receiving image data associated with the image; identifying a plurality of shapes within the image data using computer vision techniques; defining bounding boxes for each of the plurality of shapes; using a shape-prediction neural network; predicting one or more shape degrees for each of the plurality of shapes corresponding to the number of arrows from a plurality of arrows within the image data; generating a plurality of edge candidates corresponding to the plurality of shapes; predicting the probability of a pair of shape connections among the plurality of shapes using an edge prediction neural network; identifying arrows among the plurality of arrows using an edge prediction neural network; classifying arrows into arrow types using an edge prediction neural network; predicting arrow paths for a sequence of arrows including keypoints; determining a final image based on the predicted probabilities and predicted shape degrees; and converting the final image into a digital image format.

[0006] The second embodiment relates to a method for converting an image into a digital model, the method comprising: receiving image data associated with the image; identifying a plurality of shapes within the image data using computer vision techniques; defining bounding boxes for each of the plurality of shapes; using a shape prediction neural network; predicting one or more shape degrees for each of the plurality of shapes corresponding to the number of arrows from a plurality of arrows within the image data; generating a plurality of edge candidates corresponding to the plurality of shapes; predicting the probability of a pair of shape connections among the plurality of shapes using an edge prediction neural network; identifying arrows among the plurality of arrows using an edge prediction neural network; classifying arrows into arrow types using an edge prediction neural network; predicting arrow paths of arrows in a sequence including keypoints; determining a final image based on the predicted probabilities and predicted shape degrees; and converting the final image into a digital image format.

[0007] The third embodiment relates to a recognition system, comprising: a shape detection level associated with a shape detection neural network, a shape degree prediction level associated with a shape degree prediction neural network, an edge candidate level, an edge connection prediction level associated with an edge connection prediction neural network, an edge optimization level, and at least one processor programmed to perform a method for converting an image into a digital model, the method comprising: receiving image data associated with the image; identifying a plurality of shapes within the image data using a shape detection neural network at the shape detection level; defining bounding boxes for each of the plurality of shapes; predicting one or more shape degrees for each of the plurality of shapes using a shape degree prediction neural network corresponding to the number of arrows from a plurality of arrows within the image data at the edge candidate level; generating a plurality of edge candidates corresponding to the plurality of shapes at the edge candidate level; predicting the probability of a pair of shape connections among the plurality of shapes using an edge connection prediction neural network at the edge connection prediction level; identifying arrows among the plurality of arrows using an edge prediction neural network; classifying arrows into arrow types using an edge prediction neural network; predicting arrow paths of arrows including a sequence of keypoints; determining a final image based on the predicted probabilities and predicted shape degrees; and converting the final image into a digital image format.

[0008] The additional embodiments pertain to a method for automatically recognizing symbols, lines, and arrows within a hand-drawn illustration to generate a final illustration in a digital modeling format.

[0009] Another embodiment relates to a method for comprehensively increasing the size and effectiveness of the training dataset for an artificial neural network by copying images from within the training dataset and applying various image augmentations to the copied images. In some such embodiments, image augmentations can be applied to simulate natural image differences associated with photography and human error.

[0010] This overview is provided to introduce, in a simplified form, the selection of concepts further described in the detailed description below. This overview is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Other aspects and advantages of this disclosure will become apparent from the following detailed description and accompanying drawings of embodiments. Attached Figure Description

[0011] The embodiments of this disclosure are described in detail below with reference to the accompanying drawings, in which:

[0012] Figure 1 Exemplary hand-drawn illustrations related to some embodiments are depicted;

[0013] Figure 2 An identification system related to some embodiments is described;

[0014] Figure 3 Exemplary process flows related to some embodiments are described;

[0015] Figure 4 An exemplary comparison of arrow relationship bounding box generation techniques related to some embodiments is depicted;

[0016] Figures 5A-5B Exemplary methods for converting diagrams into digital models, related to some embodiments, are described; and

[0017] Figure 6 Exemplary hardware platforms associated with some embodiments are described.

[0018] The accompanying drawings are not intended to limit this disclosure to the specific embodiments disclosed and described herein. The drawings are not necessarily drawn to scale, but rather to clearly illustrate the principles of this disclosure. Detailed Implementation

[0019] The following detailed description is taken with reference to the accompanying drawings, which illustrate specific embodiments in which the present disclosure may be practiced. The embodiments are intended to describe aspects of the present disclosure in sufficient detail to enable those skilled in the art to practice the teachings. Other embodiments may be utilized and changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description should not be interpreted in a limiting sense. The scope of the present disclosure is defined only by the full scope of the appended claims together with the equivalents claimed by such claims.

[0020] In this description, references to "one embodiment," "embodiment," or "multiple embodiments" mean that one or more features referenced are included in at least one embodiment of the present technology. Separate references to "one embodiment," "embodiment," or "multiple embodiments" in this description do not necessarily refer to the same embodiment and are not mutually exclusive unless stated otherwise, and / or unless it will be readily apparent to those skilled in the art from the description. For example, features, structures, actions, etc., described in one embodiment may also be included in other embodiments, but are not required to be included. Therefore, the present technology can include various combinations and / or integrations of the embodiments described herein.

[0021] First go to Figure 1 The illustration depicts an exemplary hand-drawn diagram 100 in relation to some embodiments. The hand-drawn diagram 100 may be a diagram or flowchart representing a concept or combination of concepts, such as, for example, a manufacturing process, a business process, computer processing technology, computer hardware architecture, or computer software architecture. In some embodiments, the hand-drawn diagram 100 may be hand-drawn by a user using a pen, pencil, marker, or some other drawing medium. Alternatively, in some embodiments, the hand-drawn diagram 100 may be created using computer-aided drawing techniques such as using a touchscreen, virtual whiteboard, or interactive whiteboard. In some embodiments, the hand-drawn diagram 100 includes one or more defects that present recognition challenges when converting the hand-drawn diagram 100 to a digital diagram format, as described below.

[0022] In some embodiments, the hand-drawn illustration 100 includes partially drawn shapes 102, as shown. Therefore, accurately recognizing and identifying these shapes as illustration objects can be difficult. In some embodiments, the hand-drawn illustration 100 includes one or more visible bleed-through elements 104, which are unintentionally visible from the back of the paper or other drawing surfaces. For example, the hand-drawn illustration 100 can be drawn on a used sheet of paper using a pen, such that the bleed-through elements 104 on the back of the paper are visible in the hand-drawn illustration 100. Additionally, in some embodiments, the hand-drawn illustration 100 may include more than one sheet of paper, as shown, such that the edges 106 between the sheets are visible within the hand-drawn illustration 100.

[0023] In some embodiments, one or more crossed-out elements 108 may be included in the sketch diagram 100. For example, if a user uses a pen to create the sketch diagram 100, it may be difficult to remove unintentional markings and errors that cause the user to cross out the markings. Furthermore, in some embodiments, as shown, the sketch diagram 100 may include one or more interrupted lines 110. For example, as shown, an interrupted line 110 may be created when two sheets of paper are joined at edge 106. Additionally, in some embodiments, one or more missing arrows 112 may be present. For example, a user may forget to include arrows indicating the direction of a process flow within the sketch diagram 100. Further still, in some embodiments, one or more intersecting lines 114 may be present, where one line intersects another, creating some ambiguity regarding the direction of the lines.

[0024] In some embodiments, additional defects and challenges may include paper warping, reflections, shadows, varying line thickness, varying line color, and motion blur. In some embodiments, any combination of the above defects may be present in the hand-drawn illustration 100. Alternatively, in some embodiments, additional defects not described herein may be present. Furthermore, embodiments without defects are contemplated. In some embodiments, it may be desirable to overcome the above-described recognition challenges to generate a digital illustration that is an accurate representation of the initial hand-drawn illustration 100. In some embodiments, machine learning algorithms such as artificial neural networks may be used to overcome the various recognition challenges described herein. Thus, neural networks can be trained to correctly recognize various illustration elements, regardless of the recognition challenges that may exist in the hand-drawn illustration.

[0025] Turn now Figure 2 The text describes an identification system 200 associated with some embodiments. In such embodiments, an identification pipeline 202 is included for image recognition and transformation. In some embodiments, the identification pipeline 202 includes a shape detection stage 204, a shape degree prediction stage 206, an edge candidate stage 208, an edge prediction stage 210, and an edge optimization stage 212. In some embodiments, one or more of the shape detection stage 204, shape degree prediction stage 206, edge candidate stage 208, edge prediction stage 210, and edge optimization stage 212 include a machine learning model. Therefore, a training data storage device 214 can be communicatively coupled to the identification pipeline 202 to supply training data for the machine learning model. In some embodiments, multiple separate training data storage devices may be included.

[0026] In some embodiments, an image capture device 218 may be used to capture the initial hand-drawn illustration 216. The image capture device 218 may be a camera such as a digital camera or a smartphone camera, or an image scanning device. For example, in some embodiments, a scanning device such as a computer scanner or a scanning application on a mobile phone or tablet may be used to scan the image. Thus, the image file 220 may be supplied from the image capture device 218 to the recognition pipeline 202. Alternatively, in some embodiments, the image file 220 may be stored first and then transmitted to the recognition pipeline 202. In some embodiments, the image file 220 may be stored as any of Portable Document Format (PDF), Portable Web Graphics (PNG), or Joint Image Experts Group (JPEG) image files, and any other suitable form of image file type. The recognition pipeline 202 receives the image file 220 and identifies one or more illustration objects through various recognition levels, as will be described in further detail below.

[0027] After image recognition is performed by recognition pipeline 202, recognition pipeline 202 generates a digital image file 222 based on the received image file 220. In some embodiments, digital image file 222 may be an Extensible Markup Language (XML) file or a file in another digital format. In some embodiments, digital image file 222 may be stored in data storage device 224, which may be communicatively coupled to recognition pipeline 202. In some embodiments, data storage device 224 may be communicatively coupled to a user device, making digital image file 222 accessible to the user. For example, digital image file 222 may be sent to a user's smartphone, which the user uses to capture the image file 220 of the initial hand-drawn image 216. Furthermore, it is envisioned that data storage device 224 is an embodiment of local storage on the user device on which the image was initially captured. Thus, in some embodiments, the entire recognition system 200 is included on a user device such as a smartphone, tablet, or personal computer. Alternatively, it is envisioned that some components of the recognition system 200 are remote embodiments. For example, in some embodiments, the data storage device 224 and the image capture device 218 are parts of the user equipment that communicate with a remote server including the identification pipeline 202. Thus, in such an example, image file 220 can be sent from the user equipment to the remote server, and digital image file 222 can be returned from the remote server after the stages of the identification pipeline 202 are executed.

[0028] Turn now Figure 3This describes an exemplary process flow 300 of the operation of the recognition pipeline 202 in relation to some embodiments. In some embodiments, an image file 220 including a representation of an initial hand-drawn illustration 216 is received by the recognition pipeline 202. A shape detection stage 204 identifies and classifies multiple shapes within the image file 220. In some embodiments, a shape is an illustration object that may include text or may not include text. For example, a shape may be an illustration symbol, such as a box or decision block within an illustration. Furthermore, in some embodiments, a shape may include various types of illustration objects, such as symbols, lines, or arrows. In some embodiments, the shape detection stage 204 utilizes a shape detection machine learning model, which may be a shape detection neural network trained to identify one or more shapes or other illustration objects. In some embodiments, the shape detection neural network may be trained using training data from a training data storage device 214. In some embodiments, the shape detection stage 204 uses computer vision techniques to identify and classify shapes. Therefore, multiple shapes 302 can be detected and identified at the shape detection stage 204.

[0029] In some embodiments, shape prediction level 206 generates multiple degree predictions 304 for each of a plurality of shapes 302. In some embodiments, degree prediction 304 predicts the number of leading and trailing arrows in a given direction for each of the plurality of shapes 302. In some embodiments, degree prediction 304 may be made for each of four directions, including up, down, left, and right. In some embodiments, degree prediction 304 is made using a shape degree neural network, for example, by training the shape degree neural network using training data from training data storage device 214 to predict the number of leading and trailing arrows of the shape.

[0030] In some embodiments, edge candidate level 208 generates an edge candidate graph 306 comprising multiple edge candidates, based at least in part on syntactic rules from the graph modeling language and degree prediction 304. In such embodiments, each edge candidate is associated with a shape pair comprising two shapes from a plurality of shapes 302 and indicates a possible connection between the two shapes. In some embodiments, edge prediction level 210 predicts multiple edge probabilities 308 indicating the probability that a given shape pair is connected by a specific arrow type. In some embodiments, edge prediction level 210 also predicts multiple arrow paths 310 as sequences of keypoints. In some embodiments, edge prediction level 210 uses an edge prediction neural network to predict the probabilities and arrow paths.

[0031] In some embodiments, edge optimization stage 212 determines the final plot 312 based on the predicted edge probability 308 and the degree prediction 304. In some embodiments, edge optimization stage 212 optimizes the final plot 312 by removing redundant components such as additional arrows. Here, edge optimization stage 212 can determine the degree penalty term by comparing the edge probability 308 and the shape degree prediction 304, and removing the arrows if the total penalty term is greater than a predetermined threshold penalty value. Furthermore, in some embodiments, edge optimization stage 212 converts the final plot 312 into a digital plot format, such as Business Process Modeling Annotation (BPMN) XML or Unified Modeling Language (UML), to produce a digital plot file 222.

[0032] Turn now Figure 4 This section describes an exemplary comparison of arrow relationship bounding box generation techniques related to some embodiments. In some embodiments, arrow relationship bounding boxes can be generated at edge prediction level 210. In some embodiments, a simple method 402 can be used to form a union bounding box 404, which is generated as the minimum possible union of bounding boxes for a pair of shapes. For example, the union bounding box 404 can be formed as a union of a first shape bounding box of a first shape 406 and a second shape bounding box of a second shape 408, as shown. However, in some cases, the simple method 402 may produce a union bounding box 404 that is missing a portion of the drawn arrow 410 extending from either the first or second shape. Therefore, in some embodiments, it may be desirable to use a direction-based method 412, which allows for closer capture of the arrow, as described below.

[0033] In some embodiments, the orientation-based method 412 can be used to generate an orientation-based bounding box 414 at least in part based on predicted shape degrees. Thus, in some embodiments, the joint bounding box 404 can be initially generated for the shape pair (408, 406) and then transformed into an orientation-based bounding box 414 based on one or more of the predicted introductory and outtroductory shape degrees. For example, shape 408 has an outtroductory arrow 410 in the bottom direction. Given the predicted outtroductory shape degree of shape 408 in the bottom direction, the initial joint bounding box 404 of the shape pair can be transformed by padding the bounding box on the bottom side to generate the orientation-based bounding box 414. The orientation-based bounding box 414 is more likely to include the entire arrow 410. In some embodiments, the initial joint bounding box 404 can be padded with a predetermined pixel value. For example, in one embodiment, the joint bounding box 404 is padded with a fraction of the height of the shape bounding box. Furthermore, in some embodiments, the amount of padding can be determined relative to the size of the entire freehand drawing.

[0034] Turn now Figure 5AThis document describes an exemplary method 500 for converting an illustration into a digital model, relevant to some embodiments. In some embodiments, a processor associated with the identification system 200 may be used to perform one or more steps described with respect to method 400. For example, in some embodiments, an identification pipeline 202 may be used to perform at least a portion of the steps of method 400. Furthermore, embodiments are envisioned in which a first portion of the steps described herein is performed using the identification pipeline 202, while a second portion of the steps is performed using a processor or user equipment on another computer or server.

[0035] At step 502, one or more neural networks of the recognition pipeline 202 are trained using a set of training data from training data storage device 214. In some embodiments, the neural networks include any combination of shape detection neural networks, degree prediction neural networks, edge candidate neural networks, edge prediction neural networks, and edge optimization processes. Furthermore, in some such embodiments, the one or more neural networks may be deep convolutional neural networks including convolutional kernels. In some embodiments, the set of training data includes multiple image data sets. In some embodiments, it may be desirable to use image augmentation to increase the size of the set of training data, for example by randomly altering the image data by adjusting image parameters such as saturation, value, and contrast. Thus, a given image can be copied and adjusted to increase the number of images in the set of training data. In some embodiments, image augmentation can simulate image effects from natural photography to create a more robust network for the training data. Furthermore, in some embodiments, image data can be augmented by adding one or more additional text images and shapes to the training image plot. Additionally, image data can be augmented by shifting, scaling, rotating, and flipping images (any of which can be applied with randomly varying magnitudes). For example, a random number generation algorithm can be used to determine a random value between 0 and 360, which can then be used as the angle value for rotating the image.

[0036] In some embodiments, image augmentation of the training data can be applied to additionally or alternatively improve the accuracy of the artificial neural network components of recognition pipeline 202. For example, random text images including words can be randomly added to the training data to improve the training of the neural network to distinguish between text and arrows. In one example, a handwritten letter "I" might be confused with an arrow. Therefore, it would be desirable to insert text including the letter "I" into the training images so that the neural network can be trained to more accurately distinguish between the letter "I" and the illustrated arrow. Furthermore, in some embodiments, elastic distortion augmentation can be used to augment the training data to simulate the natural, uncontrolled vibrations of hand muscles when a person draws an illustration. Here, random distortion can be applied to objects and lines within the training data images so that the neural network is adapted to graphically represent features caused by the natural vibrations of hand muscles in a hand-drawn illustration.

[0037] At step 504, image data is received into identification pipeline 202. In some embodiments, the image data may be received by a user uploading or sending a captured image from a smartphone or other mobile device. Furthermore, in some embodiments, the image data may include a scanned image from a scanning device. Additionally, embodiments are envisioned where the image data may be stored on a user device such as a user's mobile phone or computer, and identification pipeline 202 is executed on the user device to allow direct access without requiring uploading or sending the image data. In some embodiments, the received image data includes one or more image files, such as image file 220 corresponding to the hand-drawn illustration 216. At step 506, one or more preprocessing operations may be performed on the received image data. In some embodiments, preprocessing includes resizing the image file to a fixed size by scaling the image so that the longest side of the image matches a fixed pixel value. For example, in some embodiments, it may be desirable to scale the image until the longest side is 1333 pixels long. Therefore, the aspect ratio of the image can be maintained such that both the length and height of the image are scaled by an equal amount.

[0038] At step 508, shape detection level 204 is used to identify one or more shapes within the image data. In some embodiments, shape detection level 204 may employ various computer vision techniques to identify and classify shapes from the image data. In some embodiments, one or more shapes are detected as one or more object nodes. In some embodiments, a probability may be determined for each shape. For example, the probability may be determined as the likelihood of a given shape corresponding to a given classification. In some embodiments, this probability may be determined by comparing drawn shapes from an initial hand-drawn illustration with expected symbols within a modeling language. Thus, for example, a drawn shape that perfectly matches the expected symbol will receive a probability of 1.0 or 100%. In some embodiments, only shapes with probabilities exceeding a minimum threshold are used. For example, only shapes with probabilities of 0.7 (70%) or greater are retained in the illustration.

[0039] In some embodiments, shape detection level 204 determines one or more regions of interest (ROIs) within image data. In such embodiments, each ROI indicates a potential object within the illustration, such as a symbol, line, or arrow. However, in some such embodiments, the ROIs do not classify objects. Instead, objects may be classified after one or more ROIs within the illustration have been identified. Furthermore, in some embodiments, the ROIs may be updated based on later determinations and classifications during the process. In some embodiments, shape detection level 204 assigns an object score to each identified ROI. The object score may indicate the probability that the ROI contains an illustration object such as a symbol, line, or arrow.

[0040] In some embodiments, shape detection level 204 can classify shapes according to a specific set of modeling rules for a desired modeling format. For example, if the BPMN modeling format is intended for use in the final diagram, the shape can be classified into the corresponding BPMN category. Thus, for example, an identified shape including a circle with letter icons can be classified as a BPMN message event diagram object. In some embodiments, a variety of different diagram object classifications are envisioned. For example, classifications may include activity objects such as task objects and subprocess objects, event objects such as typeless objects, message objects, and timer objects, gateway objects such as exclusive gateway objects, parallel gateway objects, and event-based gateway objects, and any one of data element objects such as data objects and data storage objects.

[0041] At step 510, a shape bounding box is defined for each of the identified shapes. In some embodiments, the shape bounding box may be defined based on a determined classification for the shape. In some embodiments, the shape bounding box may be defined by determining a set of corner points representing the outermost edge of the shape. Additionally, in some embodiments, the shape bounding box may be padded so that the entire shape fits within the bounding box. For example, in some embodiments, the bounding box may be padded and stretched to include an exit arrow associated with the shape. In some embodiments, a joint bounding box and / or a direction-based joint bounding box may be defined, such as... Figure 4 As shown.

[0042] In some embodiments, it may be desirable to identify and remove one or more repeating and / or overlapping bounding boxes. For example, if a shape has repeating edges, multiple repeating bounding boxes may be defined for each shape. Therefore, it may be desirable to identify repeating bounding boxes, for example, by testing whether the bounding boxes overlap or whether one bounding box is concentric with another. In some embodiments, the largest repeating bounding box may be retained and smaller repeatings may be removed. In some embodiments, the object scores of each bounding box may be compared and the bounding box with the higher object score may be retained because it is determined to be more likely to contain the drawing object.

[0043] At step 512, one or more shape degrees are predicted for each of the identified shapes. In some embodiments, the predicted shape degrees include one or more out-degrees and one or more in-degrees for the shape in a given direction. Here, shape degree prediction level 206 may predict the number of leading and trailing arrows corresponding to each edge of the shape (including the top edge, bottom edge, left edge, and right edge). For example, leading shape degree 2.2 may predict approximately two leading arrows for a given shape at a given edge. In some embodiments, binary masks associated with the shape bounding box may be cascaded and used as input to shape degree prediction level 206. In some embodiments, a sum may be calculated for each of the leading and trailing shape degree predictions in all directions to estimate the total number of leading and trailing arrows.

[0044] In some embodiments, regression analysis can be used to generate a degree prediction network that predicts the degree of each shape based on visual features in the initial sketch, such as drawn arrows within shape bounding boxes. In some cases, drawn arrows may be difficult to identify. For example, in some cases, there may be a distance between the drawn arrow and the intended target object or shape. Therefore, in some embodiments, it may be desirable to fill each shape bounding box with a predetermined number of pixels. For example, in some embodiments, the shape bounding box may be filled with 50 pixels in each direction. Thus, even drawn arrows not connected to shapes can be identified, and shape degree prediction becomes more accurate.

[0045] At step 514, edge candidates are generated for the shape using edge candidate level 208. In some embodiments, the number of edge candidates corresponds to the total number of arrows with predicted shape degrees from all shapes. In some embodiments, edge candidates may be generated at least in part based on one or more rules of a modeling language. For example, in some embodiments, the modeling language includes syntactic rules governing how shape elements and other objects can be combined. Furthermore, in some embodiments, edge candidates may be generated by considering only a subset of the syntactic rules. In one example, the modeling software may include syntactic rules that a gateway object should not be connected to a data element object. Therefore, edge candidates between the gateway object and the data element object can be removed. Further, in some embodiments, edge candidates may be removed based on the shape degree predicted at step 512. For example, if the degree is less than a predetermined threshold, edge candidates may be pruned (e.g., removed). In some embodiments, a predetermined threshold of 0.05 may be used such that edge candidates corresponding to shape degree predictions less than 0.05 are automatically removed. In some embodiments, it may be desirable to remove some edge candidates to optimize the processing of the recognition system 200 so that processing power is not wasted on identifying useless edge candidates.

[0046] At step 516, edge prediction level 210 is used to predict edge connections. In some embodiments, edge connections can be predicted as the probability that a given pair of shapes is connected by a specific arrow type. In some embodiments, edge prediction level 210 classifies the edge candidates generated at step 514 based on the original hand-drawn image and the predicted shape degree from step 512.

[0047] Turn now Figure 5B Method 500 continues to step 518. At step 518, arrows are identified using edge prediction level 210 based on one or more of the initial hand-drawn image, predicted shape degree, and edge candidates. In some embodiments, an arrow bounding box indicating the relevant arrow region for a given shape pair is defined for each identified arrow. At step 520, the arrow type is classified using edge prediction level 210. In some embodiments, the arrow classification may be determined based on one or more visual features of the arrow. Furthermore, in some embodiments, the arrows may be classified based on the context associated with the modeling language. For example, in some embodiments, a specific arrow type may be expected for a given pair of shape types. At step 522, arrow paths are predicted using keypoints within the illustration. In some embodiments, the arrow path is predicted as a sequence of equidistant points by analyzing the initial hand-drawn illustration 216. In some embodiments, arrow paths including dashed lines may be difficult to identify. Therefore, arrow paths can be predicted for difficult arrows with dashed lines by associating the dashed lines with the sequence of equidistant points in the initial hand-drawn illustration.

[0048] At step 524, the final plot is determined using edge optimization level 212. In some embodiments, the final plot is determined by comparing predicted edge connectivity with predicted shapeness. In some such embodiments, a set of penalty terms is determined by comparing the predicted ingress and egress shapeness at a given edge of the shape with the predicted edge connectivity of that edge. The penalty terms can then be summed and compared with a predetermined threshold penalty value, wherein if the sum of the penalty terms exceeds the threshold, one or more predicted edge connections can be removed.

[0049] At step 526, the final plot is converted into a digital plot modeling format corresponding to a given modeling language. In some embodiments, the digital plot modeling format is compatible with the plot modeling language, allowing the final plot to be accessed within the modeling software. For example, in some embodiments, the final plot may be converted to BPMN format and stored as an XML file. In some embodiments, converting the final plot to a digital modeling format allows a user to edit and copy features from the plot using modeling tools such as plot modeling software. Therefore, in some embodiments, the user may be able to edit aspects of the final plot after it has been generated. In some embodiments, the recognition system 200 may monitor changes made by the user to improve the training of the machine learning components. For example, if the classification is deemed incorrect based on user-corrected classifications, the neural network may be updated to reflect the changes, thereby improving the neural network for subsequent use.

[0050] In some embodiments, the identification process can be performed within a short time frame. For example, an embodiment in which method 500 can be performed within 100 milliseconds is envisioned. Therefore, method 500 can be used to generate a digitally formatted illustration from a hand-drawn illustration in environments where a fast response time is desired.

[0051] In some embodiments, a text recognition process can be used to recognize text within the illustration. Embodiments are envisioned in which text is recognized within the final illustration after it has been generated. Alternatively, in some embodiments, text can be recognized beforehand or simultaneously during the illustration recognition stage. Furthermore, in some embodiments, text can be identified and removed, for example, during shape detection stage 204, and then added back to the final illustration after the text has been recognized and converted to digital text format.

[0052] Turn now Figure 6This describes an exemplary hardware platform for certain embodiments. Computer 602 may be a desktop computer, laptop computer, server computer, mobile device such as a smartphone or tablet, or any other general-purpose or special-purpose computing device of any form factor. For illustrative purposes, several components are depicted with respect to computer 602. In some embodiments, some components may be arranged differently or may not be present. Additional components may also be present. Computer 602 includes a system bus 604, whereby other components of computer 602 can communicate with each other. In some embodiments, there may be multiple buses or components that can communicate directly with each other. Central processing unit (CPU) 606 is connected to system bus 604. One or more random access memory (RAM) modules 608 are also attached to system bus 604, and modules 608 may store, in particular, non-transitory form, computer-executable instructions. Graphics card 610 is also attached to system bus 604. In some embodiments, graphics card 610 may not be a physically separate card, but may be integrated into the motherboard or CPU 606. In some embodiments, graphics card 610 has a separate graphics processing unit (GPU) 612, which may be used for graphics processing or general-purpose computing (GPGPU). The graphics card 610 also includes GPU memory 614. A display 616 is connected (directly or indirectly) to the graphics card 610 for user interaction. In some embodiments, a display is not present, while in other embodiments, the display is integrated into the computer 602. Similarly, peripheral devices such as a keyboard 618 and a mouse 620 are connected to the system bus 604. Like the display 616, these peripheral devices may be integrated into the computer 602 or may not be present. Local storage 622 is also connected to the system bus 604. Local storage 622 can be any form of computer-readable medium and may be internally installed in the computer 602 or externally and removably attached.

[0053] Computer-readable media includes volatile and non-volatile media, removable and non-removable media, and envisioned as media that are readable by databases. For example, computer-readable media includes (but is not limited to) RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD), holographic media or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage, and other magnetic storage devices. These technologies can store data temporarily or permanently. However, unless explicitly stated otherwise, the term "computer-readable media" should not be construed as including forms of physical but temporary signal transmission, such as radio broadcasts, electrical signals over wires, or light pulses over fiber optic cables. Examples of stored information include computer-usable instructions, data structures, program modules, and other data representations.

[0054] Finally, a network interface card (NIC) 624 is also attached to the system bus 604 and allows computer 602 to communicate over a network such as network 626. NIC 624 can be any form of network interface known in the art, such as Ethernet, ATM, fiber optic, Bluetooth, or Wi-Fi (i.e., the IEEE 802.11 family of standards). NIC 624 connects computer 602 to local network 626, which may also include one or more other computers, such as computer 628, and network storage such as data storage device 630. Typically, a data storage device such as data storage device 630 can be any library that can store and retrieve information as needed. Examples of data storage devices include relational or object-oriented databases, spreadsheets, file systems, flat files, directory services (such as LDAP and Active Directory), or email storage systems. Data storage devices can be accessed via complex APIs (such as, for example, Structured Query Language), simple APIs that only provide read, write, and seek operations, or any level of complexity in between. Some data storage devices may also provide management functions for the datasets stored therein, such as backup or version control. The data storage device may be local to a single computer (such as computer 628), accessible on a local network (such as local network 626), or remotely accessible via the Internet 632. Local network 626, in turn, is connected to the Internet 632, which connects to numerous networks, such as local network 626, remote network 634, or directly attached computers, such as computer 636. In some embodiments, computer 602 itself may be directly connected to the Internet 632.

[0055] Although the teachings have been described with reference to embodiments shown in the accompanying drawings, it should be noted that equivalents and substitutions may be used herein without departing from the scope of the disclosure as set forth in the claims.

[0056] Various embodiments of this disclosure have been described thus, and the new and patent-protected content claimed includes the appended content.

Claims

1. One or more non-transitory computer-readable media storing computer-executable instructions, which, when executed by a processor, perform a method for converting a diagram into a digital model, the method comprising: Receive image data associated with the display; Using computer vision technology to identify multiple shapes within image data; Define a bounding box for each of the plurality of shapes; A shape-degree prediction neural network is used to predict one or more shape degrees for each of the plurality of shapes, each shape degree representing the number of arrows associated with a shape from a plurality of arrows within the image data; Based at least in part on syntactic rules and shape degree prediction from a plotting modeling language, a plurality of edge candidates representing possible connections between the plurality of shapes are generated, the syntactic rules including a syntactic rule that removes an edge candidate if the shape degree is less than a first predetermined threshold; The edge prediction neural network is used to predict the probability that a pair of shapes among the plurality of shapes are connected by a specific arrow type; An edge prediction neural network is used to identify the arrow among the plurality of arrows based on one or more of the plot, predicted shapeness, and edge candidates; Use an edge prediction neural network to classify arrows into arrow types; Predict arrow paths of arrows in a sequence of key points, wherein the arrow paths are predicted as a sequence of equidistant points by analyzing a plot. The final diagram is determined by: for each of the plurality of shapes, comparing the predicted probability with the predicted shape degree to define one or more penalty terms; summing the one or more penalty terms for each corresponding shape; determining whether the sum of the penalty terms exceeds a second predetermined threshold; and if the sum of the penalty terms exceeds the second predetermined threshold, removing one or more arrows associated with the one or more penalty terms; and Convert the final image to a digital image format.

2. The medium of claim 1, wherein, The image data is associated with a hand-drawn illustration received from the user.

3. The medium of claim 2, wherein, The image data is generated by scanning a hand-drawn illustration using a scanning device.

4. The medium as described in claim 2, wherein The image data is generated by taking hand-drawn illustrations using the camera on a mobile phone, and The processor is a processor for a mobile phone.

5. The medium as described in claim 1, further comprising: Preprocess image data to produce an image with a fixed size.

6. The medium as described in claim 1, further comprising: A joint bounding box is defined between the two or more shapes based on the corresponding bounding boxes of each of the plurality of shapes and the predicted arrow paths.

7. A method for converting a diagram into a digital model, the method comprising: Receive image data associated with the display; Using computer vision technology to identify multiple shapes within image data; Define a bounding box for each of the plurality of shapes; A shape-degree prediction neural network is used to predict one or more shape degrees for each of the plurality of shapes, each shape degree representing the number of arrows associated with a shape from a plurality of arrows within the image data; Based at least in part on syntactic rules and shape degree prediction from a plotting modeling language, a plurality of edge candidates representing possible connections between the plurality of shapes are generated, the syntactic rules including a syntactic rule that removes an edge candidate if the shape degree is less than a first predetermined threshold; The edge prediction neural network is used to predict the probability that a pair of shapes among the plurality of shapes are connected by a specific arrow type; An edge prediction neural network is used to identify the arrow among the plurality of arrows based on one or more of the plot, predicted shapeness, and edge candidates; Use an edge prediction neural network to classify arrows into arrow types; Predict arrow paths of arrows in a sequence of key points, wherein the arrow paths are predicted as a sequence of equidistant points by analyzing a plot. The final diagram is determined by: for each of the plurality of shapes, comparing the predicted probability with the predicted shape degree to define one or more penalty terms; summing the one or more penalty terms for each corresponding shape; determining whether the sum of the penalty terms exceeds a second predetermined threshold; and if the sum of the penalty terms exceeds the second predetermined threshold, removing one or more arrows associated with the one or more penalty terms; and Convert the final image to a digital image format.

8. The method of claim 7, wherein, The final digital image format is compatible with the image modeling language.

9. The method of claim 8, wherein, The digital visualization format is XML and the visualization modeling language is BPMN.

10. The method of claim 7, further comprising: The image data is preprocessed to produce an image of a fixed size, such that the longest side of the image is scaled to a value of 1333 pixels.

11. The method of claim 10, wherein, Fill the bounding box for each of the plurality of shapes with 50 pixels added to each side of the image.

12. The method of claim 7, further comprising classifying each of the plurality of shapes based on one or more predetermined syntax rules of a diagram modeling language, wherein, The digital plotting format is compatible with the plotting modeling language.

13. An identification system, comprising: Shape detection level associated with the shape detection neural network; Shape prediction level associated with the shape prediction neural network; Edge candidate level; Edge connection prediction level associated with the edge connection prediction neural network; Edge optimization level; as well as At least one processor is programmed to perform a method for converting a diagram into a digital model, the method comprising: Receive image data associated with the display; At the shape detection level, a shape detection neural network is used to identify multiple shapes within the image data; Define a bounding box for each of the plurality of shapes; At the shape prediction level, a shape prediction neural network is used to predict one or more shape degrees for each of the plurality of shapes, each shape degree representing the number of arrows associated with a shape from a plurality of arrows within the image data; At the edge candidate level, multiple edge candidates representing possible connections between the multiple shapes are generated, at least in part based on syntactic rules and shape degree predictions from the plotting modeling language, the syntactic rules including a syntactic rule that removes an edge candidate if the shape degree is less than a first predetermined threshold. At the edge connection prediction level, an edge prediction neural network is used to predict the probability that a pair of shapes among the plurality of shapes are connected by a specific arrow type; An edge prediction neural network is used to identify the arrow among the plurality of arrows based on one or more of the plot, predicted shapeness, and edge candidates; Use an edge prediction neural network to classify arrows into arrow types; Predict arrow paths of arrows in a sequence of key points, wherein the arrow paths are predicted as a sequence of equidistant points by analyzing a plot. The final diagram is determined by: for each of the plurality of shapes, comparing the predicted probability with the predicted shape degree to define one or more penalty terms; summing the one or more penalty terms for each corresponding shape; determining whether the sum of the penalty terms exceeds a second predetermined threshold; and if the sum of the penalty terms exceeds the second predetermined threshold, removing one or more arrows associated with the one or more penalty terms; and Convert the final image to a digital image format.

14. The system of claim 13, further comprising a training data storage device storing training data including training image data, wherein, One or more of the shape detection neural network, shape prediction neural network, and edge connection prediction neural network are trained using training data from a training data storage device.

15. The system of claim 14, wherein, The training image data in the training data storage device includes simulated training image data generated using image augmentation techniques to increase the amount of training data.

16. The system of claim 15, wherein, The image augmentation techniques include one or more of the following: adjusting saturation, adjusting brightness, and adjusting contrast.

17. The system of claim 13, wherein, The method also includes removing at least one overlapping duplicate bounding box.

18. The system of claim 13, wherein, Fill the bounding box of each of the plurality of shapes by a predetermined amount to identify one or more arrows connected to the shape.

Images