Drawing instruction generation system and method thereof

IN595278BActive Publication Date: 2026-07-14SAMSUNG ELECTRONICS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
IN202211061884
Authority / Receiving Office
IN · IN
Patent Type
Patents
Current Assignee / Owner
Filing Date
2022-10-31
Publication Date
2026-07-14
Estimated Expiration
2042-10-31

AI Technical Summary

Technical Problem

Existing drawing applications lack assistance features, making it difficult for users, especially children and adults learning to draw, to complete complex shapes and designs without external help, as they rely on manual guidance which is not always available.

Method used

A drawing instruction generation system that includes an object processing unit, edge image generation unit, and processing unit to extract objects from input images, detect edges, divide them into grids, fit curves, and generate sequential drawing instructions, providing visual and audio guidance through a virtual assistant.

Benefits of technology

Enables users to draw complex objects by following numbered dots or curves, improving the drawing experience by providing step-by-step instructions, thus reducing the need for external assistance and enhancing learning and creativity.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The present invention provides a drawing instruction generation system (208) that receives an image from the user, identifies an object in the image, generates and processes an edge image of the object, fits curves or lines to generate drawing instructions that are displayed serially to the user. The drawing instruction generation system (208) facilitates a user to easily and quickly draw the object by joining serially numbered edges.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present invention relates generally to electronic systems, and, particularly but not exclusively, to a system and a method for generating drawing instructions. BACKGROUND

[0002] Increasing use of tablets and touchscreen devices with input means such as stylus and digital pencil has resulted in development of interactive applications. Most common applications include writing or drawing using stylus or digital pencil. The challenge involved in such applications is that drawing on a tablet is not as intuitive as drawing on a paper. Moreover, complexity of shapes in a drawing add to the difficulty of the drawing.

[0003] Recently, drawing applications use Artificial Intelligence (AI) and Machine Learning (ML) techniques to create artworks that are beyond traditional definitions of artworks. AI and ML has also been used to remaster old art forms. Looking at such artforms may want one to draw or to learn to draw these artforms.

[0004] One of the existing prior-arts provides an interactive hub, "Klimt vs Klimt - The Man of Contradictions" launched by Google Arts & Culture, in which visitors are offered an insight into Gustav Klimt's biography, artistic inspiration, legacy, and more.

[0005] In another prior art, "DeepMind", AI techniques are used to generate paintings using a neural visual grammar system evaluated using a text conditioned dual encoder.

[0006] In another prior art, "Pen and paper go digital", illustrators use digital tablets like Cintiq to give them the freedom and flexibility they need to do their job. On such tablets, buttons and stylus have very specific settings that can be customized.

[0007] In another prior art, "Moleskine Smart Notebook" by Adobe, a drawing device is provided, that looks and feels like a notebook and allows users to convert drawings done by hand into a digital file.

[0008] In US patent no. US10713821B1, techniques for generating images based on text inputs are discussed. In TW patent no. TWI553513B, a device for drawing an image according to the position coordinates output by the touch device and the color set by the setting module is provided. The device is used for practicing calligraphy strokes or painting.

[0009] However, all the drawing devices and applications of prior art have limited functionality. Some of the prior art drawing applications can only display the drawings or provide blank canvas for making the drawing. Some prior art drawing applications can only process multiple pictures and provide a near sketch for the pictures.

[0010] If users require any assistance while drawing, such assistance cannot be provided by prior art drawings applications. For assistance, the users must rely on help of others, for instance, kids learning to draw generally rely on assistance provided by their parents or teachers, as shown in FIG. 1. When such assistance is no longer available, kids fail to complete the drawings. Such assistance may be required by kids and adults alike.

[0011] Therefore, there is a need for a drawing assistance system to assist users in drawing and to overcome at least one of the problems faced by the prior art drawings systems.SUMMARY

[0012] This summary is provided to introduce concepts related to drawing instruction generation system and drawing instruction generation method. This summary is neither intended to identify essential features of the present invention nor is it intended for use in determining or limiting the scope of the present invention.

[0013] In an embodiment of the present invention, a drawing instruction generation system is provided. The drawings instruction generation system includes an object processing unit, an edge image generation unit, and a processing unit. The object processing unit is configured to extract at least one object from an input image. The edge image generation unit is configured to generate an edge image based on detection of a plurality of edges of the extracted object. The plurality of edges are numbered serially. The processing unit divides the edge image into a plurality of serially numbered grids. The processing unit generates a plurality of serially numbered fitted edge sections corresponding to the plurality of serially numbered edges and grids thereof. The processing unit generates a set of drawing instructions based on the serially numbered fitted edge sections and corresponding grids. The processing unit displays the set of drawing instructions on a display sequentially.

[0014] In another embodiment of the present invention, a drawing instruction generation method is provided. The drawing instruction generation method includes extracting at least one object from an input image by an object processing unit. The drawing instruction generation method includes generating an edge image based on detection of a plurality of edges of the extracted object by an edge image generation unit. The plurality of edges are numbered serially. The drawing instruction generation method includes dividing the edge image into a plurality of serially numbered grids. The drawing instruction generation method includes generating a plurality of serially numbered fitted edge sections corresponding to the plurality of serially numbered edges and grids thereof. The drawing instruction generation method includes generating a set of drawing instructions based on the serially numbered fitted edge sections and corresponding grids. The drawing instruction generation method includes displaying the set of drawing instructions on a display sequentially.

[0015] In an exemplary embodiment, the processing unit includes a curve fitting unit. The curve fitting unit is configured to fit, in every serially numbered grid, at least one of: a straight line and a Bezier curve on one or more edges within the aforesaid grid to generate the corresponding serially numbered fitted edge section.

[0016] In another exemplary embodiment, the drawing instruction generation system includes a fit test unit and a recursive unit. The fit test unit measures a distance metric for the one or more fitted Bezier curves of a fitted edge section. The fit test unit compares the measured distance metric with a predetermined distance metric to determine an error indicative of a quality of the fitted Bezier curve for the fitted edge section.

[0017] In another exemplary embodiment, the drawing instruction generation system includes a recursive unit. The recursive unit compares the error of the fitted edge section with a predefined threshold. The recursive unit divides the grid corresponding to the fitted edge section into a plurality of sub-grids when the error is above the predefined threshold.

[0018] In another exemplary embodiment, the curve fitting unit is configured to perform curve fitting on the plurality of sub-grids to generate a plurality of fitted edge sections corresponding to the plurality of sub-grids.

[0019] In another exemplary embodiment, the object processing unit includes an object recognition unit, an object detection unit, an object segmentation unit, and an instance segmentation unit. The object recognition unit is configured to identify presence of the object in the input image. The object detection unit is configured to extract the object and to generate a bounding box around the object, thereby localizing the object in the input image. The object segmentation unit is configured to separate the object from a plurality of other objects in the input image. The instance segmentation unit is configured to separate a plurality of instances of a same object in the input image.

[0020] In another exemplary embodiment, curve fitting unit includes a training unit configured to train a Convolutional Neural Network (CNN) to classify one or more edges into one or more classes.

[0021] In another exemplary embodiment, the training unit performs regression on one or more anchor boxes to determine a rotation for the corresponding edge. A result of the regression is indicative of a probability of one or more angles of rotation. The training unit performs a classification of the edge. A result of the classification is indicative of a probability of the one or more classes.

[0022] In another exemplary embodiment, the input unit receives a voice command from a user. The input unit fetches the input image from one or more databases based on the received voice command.BRIEF DESCRIPTION OF ACCOMPANYING DRAWINGS

[0023] The detailed description is described with reference to the accompanying figures. The same numbers are used throughout the drawings to reference like features and modules.

[0024] FIG. 1 is a pictorial representation of limitations of conventional drawing systems.

[0025] FIG. 2 shows a schematic block diagram of a drawing instruction generation system in accordance with an embodiment of the present invention.

[0026] FIGS. 3-5 are a flowchart illustrating a method of generating drawing instructions in accordance with another embodiment of the present invention.

[0027] FIGS. 6A-6B are a schematic flow diagram illustrating a drawing instruction generation technique in accordance with another embodiment of the present invention.

[0028] FIG. 7A is a pictorial representation of object identification and separation in accordance with another embodiment of the present invention.

[0029] FIG. 7B is a matrix equation used in edge detection in accordance with another embodiment of the present invention.

[0030] FIGS. 8A-8B are a schematic flow diagram illustrating a feature extraction technique used in Decomposition Curve Fitting (DCF) in accordance with another embodiment of the present invention.

[0031] FIG. 8C shows a sub-divided image grid in accordance with another embodiment of the present invention.

[0032] FIG. 9 is a pictorial representation of a use case showing different modes of inputs to a drawing instruction generation system in accordance with another embodiment of the present invention.

[0033] FIG. 10 is a pictorial representation of a use case showing a drawing instruction generation system implemented in a television in accordance with another embodiment of the present invention.

[0034] FIGS. 11-12 show pictorial comparisons of a drawing instruction generation system in accordance with another embodiment of the present invention with a conventional drawing system.

[0035] FIG. 13 is a pictorial representation of a use case showing a drawing instruction generation system implemented in a television in accordance with another embodiment of the present invention.

[0036] FIG. 14A is a pictorial representation of a use case showing a drawing instruction generation system using a voice input in accordance with another embodiment of the present invention.

[0037] FIG. 14B is a pictorial representation of a use case showing a drawing instruction generation implemented in Augmented Reality (AR) / Virtual Reality (VR) headset in accordance with another embodiment of the present invention.

[0038] FIG. 15 is a pictorial representation of a use case showing a drawing instruction generation system implemented in a television in accordance with another embodiment of the present invention.

[0039] FIGS. 16-17 are a pictorial representation of a use case showing a drawing instruction generation implemented in Augmented Reality (AR) / Virtual Reality (VR) headset in accordance with another embodiment of the present invention.

[0040] FIG. 18 is a pictorial representation of a use case showing a drawing instruction generation implemented in Augmented Reality (AR) / Virtual Reality (VR) glasses and a smartphone in accordance with another embodiment of the present invention.

[0041] FIG. 19 is a pictorial representation of a use case showing a drawing instruction generation system implemented in a television in accordance with another embodiment of the present invention.

[0042] FIG. 20 is a pictorial representation of a use case showing display of drawing instructions on an electronic device in accordance with another embodiment of the present invention.

[0043] FIG. 21 is a pictorial representation of a use case showing display of drawing instructions on an electronic device in accordance with another embodiment of the present invention.

[0044] It should be appreciated by those skilled in the art that any block diagrams herein represent conceptual views of illustrative systems embodying the principles of the present invention.

[0045] Similarly, it will be appreciated that any flowcharts, flow diagrams, and the like represent various processes which may be substantially represented in computer readable medium and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.DETAILED DESCRIPTION

[0046] In the following description, for the purpose of explanation, specific details are set forth in order to provide an understanding of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced without these details. One skilled in the art will recognize that embodiments of the present invention, some of which are described below, may be incorporated into a number of systems.

[0047] Furthermore, connections between components and / or modules within the figures are not intended to be limited to direct connections. Rather, these components and modules may be modified, re-formatted or otherwise changed by intermediary components and modules.

[0048] References in the present invention to "one embodiment" or "an embodiment" mean that a particular feature, structure, characteristic, or function described in connection with the embodiment is included in at least one embodiment of the invention. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

[0049] The various embodiments of the present invention provide a drawing instruction generation system and a method for drawing instruction generation.

[0050] The present invention provides a drawing instruction generation system in an embodiment. The drawing instruction generation system receives an image from a user. The drawing instruction generation system identifies a portion of the image that includes an object which the user desires to draw. If the image includes multiple objects, the drawing instruction generation system identifies the desired object using object recognition techniques. The drawing instruction generation system converts the image into a grayscale image and detects edges of the desired object using edge detection techniques. The drawing instruction generation system divides the edges of the object into multiple square grids. The drawing instruction generation system fits a straight line or a Bezier curve on the edges in every grid. The drawing instruction generation system checks if the straight line or the Bezier curve fit the edge. If the straight line or the Bezier curve do not fit the edge, the drawing instruction generation system subdivides the grids into sub-grids and fits the straight line or the Bezier curve on the edges in the sub-grid. The drawing instruction generation system identifies serial connections between the fitted curves and / or lines and generates drawing instructions based on serially connected edges. In an example, the drawing instructions are numbered dots which can be traced to complete drawing the object.

[0051] Referring now to FIG. 2, a schematic block diagram of a drawing instruction generation system (200) is shown in accordance with an embodiment of the present invention. The drawing instruction generation system (200) includes an Input / Output (I / O) unit (202), a compute unit (204), a plurality of databases (206), a drawing instruction generation system (208), an output unit (210), and a Virtual Assistance (VA) unit (212).

[0052] The I / O unit (202) is in communication with the databases (206) and the drawing instruction generation system (208). The I / O unit (202) includes a settings unit (214) having one or more configuration files (216) and an I / O unit (218). The I / O unit (202) further includes I / O devices (220) such as, but not limited to, a camera, microphone, speaker, a touchscreen display etc. The I / O unit (202) further includes an image and / or object input unit (222), a voice input unit (224), and a gesture and / or sensors input unit (226).

[0053] The I / O unit (202) receives input data from a user. The I / O unit (202) acts as interface for capturing user inputs like image input, voice input, gesture input (Hand, Finger, Face, and Eyes etc.) or any sensor input etc. The received input data is provided to the drawing instruction generation system (208) as input to perform further analysis.

[0054] The I / O unit (202) also facilitates selection of user preferences by the user. The user can manually give input and other settings the settings unit (214). The user data and the preferences settings are saved in the user database (250) in the databases (206).

[0055] The compute unit (204) is in communication with the drawing instruction generation system (208). The compute unit (204) includes a cloud compute unit (228), a device compute and acceleration unit (230), and a memory (232) for storing one or more processing modules (234).

[0056] The compute unit (204) performs analysis for cloud, device etc. The compute unit (204) provides software library support for the drawing instruction generation system (208). The data computed by the compute unit (204) is used by the processing modules (234) executed by the cloud compute unit (228) and / or the device compute and acceleration unit (230).

[0057] The databases (206) are in communication with the drawing instruction generation system (208). The databases (206) include a knowledge base and rules database (236), an object database (238), an image database (240), a sensor database (242), a context database (244), a threshold database (246), an analytics database (248), a user database (250), a common drawings database (252), a curve fit dictionary (254), a Machine Learning (ML) models database (256), and a training and testing database (258).

[0058] The databases (206) stores information about user data, analytics, learnt data, common drawings, settings etc. or any other data useful for smooth functioning of the drawing instruction generation system (208). Type of data stored in the databases (206) may vary based on the user's preferences. The databases (206) store the inputted information from the I / O unit (202) and the data processed by each processing module (234) for future usage.

[0059] The VA unit (212) is in communication with the drawing instruction generation system (208). The VA unit (212) includes a user profile integration unit (2008), a dialog manager unit (2010), a VA manager unit (2012), a VA server unit (2014), and a response unit (2016).

[0060] The VA unit (212) executes technical features of a virtual assistant. The VA unit (212) provides a responsive user interface for the user's device. The VA unit (212) provides voice guide to the user for an image.

[0061] The drawing instruction generation system (208) includes an object processing unit (260), an edge image generation unit (262), and a processing unit (264). The object processing unit (260) includes an object recognition unit (266), an object detection unit (268), an object segmentation unit (270), and an instance segmentation unit (272).

[0062] The object processing unit (260) extracts an object of interest from the image. If the image has multiple objects, the object processing unit (260) identifies and labels the objects to determine an object of interest.

[0063] The edge image generation unit (262) includes a cloud compute unit (274) and a device compute and acceleration unit (276).

[0064] The edge image generation unit (262) creates an edge image. The edge image includes edges of the object of interest, and hence, is appropriate for making a sketch of the object by drawing over the edges. The edge image generation unit (262) uses a combination of edge detection and smoothing to generate the edge image having a continuous outline of the object to be drawn. The edges of the object in the image are serially numbered.

[0065] The processing unit (264) includes a subdivision unit (278), a curve fitting unit (280), a fit test unit (288), a recursive unit (290), an iteration unit (292), and a presentation unit (294). The curve fitting unit (280) includes a curve check unit (282), a 1st / 2nd degree fit check unit (284), and a training unit (286).

[0066] The processing unit (264) is configured to perform decomposition and sketch instruction generation for the image. The processing unit (264) receives the edge image of the object and decomposes the edge image into a sequence of lines or simple curves with one or more instructions that the user can use as a guide to draw the complete object, thereby assisting the user in the drawing.

[0067] The output unit (210) is in communication with the databases (206) and the drawing instruction generation system (208). The output unit (210) includes I / O unit (296), an Augmented Reality (AR) / Virtual reality (VR) / Mixed reality (MR) unit (298), a camera (2000) and a memory (2002). The memory (2002) stores an Operating System (OS) (2004) and one or more applications (2006).

[0068] The output unit (210) displays the drawing instructions generated by the drawing instruction generation system (208). The output unit (210) provides the drawing instructions in audio and visual format to the user. The drawing instructions may be output in form of both: image and voice with corresponding instruction. The users may use the output as per their requirements. The output unit (210) may also provide the output in form of AR / VR.

[0069] In operation, the drawing instruction generation system (208) receives the image from a user. The drawing instruction generation system (208) may receive the image by way of various input means, such as, but not limited to, the camera (220), the image input unit (222), and the common drawings database (252). The image may have one or more objects. If the image has multiple objects, the drawing instruction generation system (208) identifies an object of interest out of the multiple objects. The drawing instruction generation system (208) may identify the desired object based on one or more labels received from the user. If the user does not provide any label, the drawing instruction generation system (208) identifies all objects in the image as objects of interests, for which the drawing instructions are generated.

[0070] The object processing unit (260) extracts the object of interest from the input image using trained Deep Neural Nets (DNN). The object processing unit (260) may also use various object recognition techniques such as, but not limited to, bounding box based, segmentation based or instance segmentation based object recognition techniques, based on complexity of scene in the image. The object processing unit (260) may use pre-trained models such as, but not limited to, SSD Mobile Net and U-Net, optimized for embedded systems.

[0071] The edge image generation unit (262) receives the image from the object processing unit (260) with information about the object of interest. The edge image generation unit (262) uses one or more edge detection techniques to generate the edge image. The edge image generation unit (262) detects the edges, lines and boundaries of the object in the image using the edge detection techniques. The edge image generation unit (262) fills up gaps or missing sections in the detected edges using interpolation techniques such as, but not limited to, RANSAC and Hough Transform. The edge image generation unit (262) outputs the edge image indicative of the edges of the object of interest in the image.

[0072] The processing unit (264) receives the edge image from the edge image generation unit (262). The subdivision unit (278) subdivides the edge image into M x N square grids (M vertically and N horizontally) to simplify the edge detection. The grids are serially numbered. By subdivision, only a portion of the complete edge is captured in each grid which reduces complexity of curve fitting and subsequent reproduction by the user.

[0073] For each grid with an edge section, the curve fitting unit (280) fits a straight line or a second degree Bezier curve on the edges to generate fitted edge sections. The fitted edge sections are serially numbered. In an example, second degree Bezier curves are used for curve-fitting on the decomposed image. Second degree Bezier curves are preferred over higher degree Bezier curves owing to relative simplicity of the second degree Bezier curves. The second degree Bezier curves are simpler, and hence, easier to sketch for children. Also, a family of second degree Bezier curves can cover / closely fit a broad variety of edge sections as detected by the edge detection and passed on to curve fitting.

[0074] The fit test unit (288) determines a quality of the curve fit. The fit test unit (288) measures the quality of fit using a suitable distance metric from the original edge to the fitted curve. Based on a preset threshold distance, the fit test will either pass or fail. If the distance between the original curve and the fitted curve is below the preset threshold distance, the quality of fit is satisfactory, and the fit test unit (288) marks the grid as visited and moves to check the curve fit of the edges in the next grid.

[0075] If the distance between the original curve and the fitted curve is above the preset threshold distance, the quality of the curve fit is not satisfactory, and therefore, the recursive unit (290) recursively sub-divides the grid into one or more sub-grids. The iteration unit (292) fits straight lines and / or second degree Bezier curves on the edges in each of the sub-grids. In an example, the recursive unit (290) divides the grid into four equal sub-grids. In an example default recursive levels of subdivision of the grid are set to three to have good run-time performance. In an example, the iteration unit (292) starts from the top left grid and then covers each row of grids from left to right in a top-down sequence.

[0076] The default recursive levels of subdivision may be increased by the user for capturing more details on complex images. The recursive unit (290) will automatically stop subdivision if the subdivided grid size falls below a preset threshold, as there would be no perceptible gain in precision at that resolution.

[0077] The presentation unit (294) tracks edge sections that can be drawn in sequence to complete the drawing. The presentation unit (294) sets a static numbering convention to identify the grids and relative location after subdivision - Each grid is numbered as [row number]- [column number]- [first subdivision id]- [second subdivision id]- [third subdivision id]. So, a grid in the first row and 3rd column is numbered as 0-2, while a first level sub-grid in the 3rd row and 3rd column is numbered 2-2- [0-3]. The presentation unit (294) defines a grid object to contain the grid number / id, start and end coordinates of the edge in the grid.

[0078] The presentation unit (294) after successful curve fitting, collect grid specifics in 2 separate data types: (1) list<Grid> - The list is meant to hold the sequence of all the grids containing curve fitted edge portions. It includes a mix of grids and sub-grids and is a proper subset of the initial m x n grid. This data structure is used during the instruction generation to iterate over all the edges. (2) map<pair,list<Grid>> - The map includes all grid objects which have either their start or end coordinates match the key (pair / coordinate). This data structure is used during the instruction generation to perform DFS iteration for discovering connected components.

[0079] The presentation unit (294) creates a list V, to hold ids of grids to visit (in order) so that user has a smooth drawing experience; he / she can complete a connected section of the drawing before moving onto the next connected section. The presentation unit (294) iterates over the vector of grids. The presentation unit (294) performs DFS to collect all the connected sections in the vector of grids. The presentation unit (294) pushes grid / sub-grid ids into V as they are visited (do mark girds as visited to prevent cycles).

[0080] Thereafter, the processing unit (264) generates a set of drawing instructions based on the serially numbered fitted edge sections and corresponding grids.

[0081] Referring now to FIGS. 3-5, a flowchart illustrating a method of generating drawing instructions is shown in accordance with another embodiment of the present invention.

[0082] In the method of generating drawing instructions, the drawing instruction generation system (208) acquires the image and the label from the user. The object processing unit (260) identifies and extracts the object of interest from the image. The edge image generation unit (262) detects the edges of the object of interest. The edge image generation unit (262) smoothens the edges to generate the edge image. The subdivision unit (278) divides the image into multiple grids. The curve fitting unit (280) fits straight lines or second degree Bezier curves over the edges in each grid. The fit test unit (288) determines a curve fit error indicative of a quality of curve fit for the grid. The fit test unit (288) compares the determined error with a threshold. If the curve fit error is not less than the threshold, the recursive unit (290) divides the grid into one or more sub-grids. If the curve fit error is less than the threshold, the presentation unit (294) populates one or more data structures with grid specifics. the iteration unit (292) determines if the curve fitting is completed. If the curve fitting is completed, the presentation unit (294) presents the drawing instructions to the user by way of the output unit (210). If the curve fitting is not completed, the iteration unit (292) visits an unvisited sub-grid to perform curve fitting on the unvisited sub-grid. The presentation unit (294) assigns static numbering to the fitted curves or straight lines. The presentation unit (294) defines the data structure used for storing the objects and the grids thereof. The presentation unit (294) populates the data structures corresponding to the straight lines or the Bezier curves filled on the edges in each grid. The iteration unit (292) determines the unvisited grids and performs curve fitting on the unvisited grids. The iteration unit (292) determines whether the fitted curves in the grids are connected. If the grids are not connected, the iteration unit (292) determines the unvisited grids and performs curve fitting on the unvisited grids. If the grids are connected, the presentation unit (294) stores the connected edges and the grids thereof in form of a list. The drawing instruction generation system (208) receives a voice input through the I / O unit (202). The drawing instruction generation system (208) converts the voice input into text. The drawing instruction generation system (208) searches in the common drawings database (252) to find the image that matches the text. The drawing instruction generation system (208) checks in the image is found. If the image is found, the object processing unit (260) identifies and extracts the object of interest from the image. If the image is not found, the drawing instruction generation system (208) searches for a matching image on the internet.

[0083] Referring now to FIGS. 6A-6B, a schematic flow diagram illustrating a drawing instruction generation technique is shown in accordance with another embodiment of the present invention.

[0084] The drawing instruction generation system (208) receives the image. The image is a two-dimensional array of color values. The drawing instruction generation system (208) also receives the label from the user. The object processing unit (260) extracts the object of interest from the image using trained DNN for object identification and separation. The edge image generation unit (262) detects the edges of the object of interest in the image. The edge image generation unit (262) smoothens the edges, reduces noise or textures in the detected edges, and retains sharp edges to generate the edge image. The subdivision unit (278) subdivides the edge image into the grids. In an example, the image is subdivided into M x N grids to reduce the complexity of the image. The curve fitting unit (280) fits the straight lines or the second order Bezier curves onto the edges in the grids. The fit test unit (288) determines the fit quality for the fitted curves. If the quality of curve fit is bad, the grid is subdivided into sub-grids by the recursive unit (290). If the quality of curve fit is not bad, the grid is marked as visited. The steps 6-7 shown in FIG. 6 are repeated for all unvisited grids and sub-grids until all the grids and sub-grids are visited. Thereafter, the line image is displayed to the users sequentially along with the drawing instructions.

[0085] In the method of displaying the drawing instructions to the users, the presentation unit (294) sets the static numbering for the grids and sub-grids to identify relative locations of the grids and sub-grids after subdivision. The presentation unit (294) determines a grid number and start / end coordinates of the edges in the grids and creates list V to hold the grid numbers to be visited in sequence for smooth drawing. The presentation unit (294) stores the grid specifics in two separate data structures, viz, list and map data structures. The list data structure includes the grids with the fitted edges. The map data structure includes the neighbor edges. The iteration unit (292) causes to perform curve fitting recursively on the unvisited grids and sub-grids. The connected edges are sequenced and are stored as connected portions in list V. The presentation (294) causes to display the drawing instructions to the user to complete the drawing.

[0086] FIG. 7A is a pictorial representation of object identification and separation in accordance with another embodiment of the present invention.

[0087] The object recognition unit (266) detects the presence of the objects in the image. DNN may be used to implement the object recognition.

[0088] The object detection unit (268) recognizes the object and localizes it in the image using a bounding box. The object detection unit (268) extracts the object along with the bounding box from an uncluttered scene. This can be used to detect a single object or a group of multiple objects.

[0089] The object segmentation unit (270) improves upon detection and localizes the object using the actual boundary of the object. This results in pixel level classification of the scene. The object segmentation unit (270) may use U-Net architecture to extract out the object from the scene with multiple object classes (cluttered scene). The object segmentation unit (270) also separates out one object or multiple objects in a group, depending on user preferences.

[0090] The instance segmentation unit (272) segments out multiple occurrences of the same object in the image. The instance segmentation unit (272) may use Mask R-CNN architecture to extract out an object from a scene with multiple occurrences of instances from the same object class (along with other object classes)

[0091] FIG. 7B is a matrix equation used in edge detection in accordance with another embodiment of the present invention.

[0092] For edge detection, techniques such as, but not limited to, can mention Sobel, Prewitt and Canny edge detection are used. While Sobel and Prewitt edge detectors are easy to implement, Canny is more involved and much more accurate. In an example, Canny edge detection is used for effective edge detection.

[0093] In the Canny Edge Detection, in first step spurious pixels are removed using Gaussian smoothing. Thereafter, magnitude and direction of gradient is determined using Sobel mask. It is based on a set of masks aligned in the horizontal and vertical direction. If we define A as the source image and Mx and My are masks in the horizontal and vertical direction respectively, then Gx (Mx * A) and Gy (My * A) are the 2 intensity images approximating the derivatives in the horizontal and vertical directions. Here, * is the convolution operator. After finding the intensity gradients, using non-maximum suppression technique, non-maximum edge pixels are removed in local neighborhood which will result in edge thinning. Thereafter, hysteresis thresholding is performed to thin edges further by removing edges not connected to the main edge pixels.

[0094] For edge continuity, the edge detection techniques usually suffer from gaps / missing sections in the edges detected. To interpolate between the holes and improve edge continuity, either the RANSAC algorithm or the Hough Transform algorithm is used.

[0095] In an example, RANSAC algorithm consists of the following steps: (a) Randomly selecting a subset of the data set (edge pixels); (b) Fitting a model to the selected subset; (c) Determining the number of outliers; (d) Repeating steps 1-3 for a prescribed number of iterations; and (e) Choose the model which gives the best fit.

[0096] In an example, Hough transform follows a similar majority based approach: (a) Given edge pixels (using Canny / Sobel), drawing curves through them and accumulate their parameters; and (b) Retaining the curves which occur the maximum number of times in the accumulator.

[0097] FIGS. 8A-8B are a schematic flow diagram illustrating a feature extraction technique used in Decomposition Curve Fitting (DCF) in accordance with another embodiment of the present invention.

[0098] In trained DNN based curve fitting, target classification includes 20 distinct 2-degree Bezier Curves to deal with the issue of rotated classes.

[0099] In Type A, CNN model trains with 60 separate orientations for each class. So, there are 20*60 = 1200 output classes. In an example, a fully trained Mobile Net removes the fully connected head, puts the FC head and retrains for the 1200 possible classes (softmax classifier).

[0100] In Type B, the target 20 classes are retained, but regressed by rotating the conventional object detection box. So, the usual object detection output modified as follows: {class_probability, rotation}. Here, the usual box center coordinates and width, height are redundant because it is an aim to classify the curve within the grid - so the grid is already the bounding box / region. In an example, a fully trained Mobile Net removes the fully connected head and put our own FC head with 2 branches: (a) Rotation / Orientation branch - regresses anchor boxes to find the rotation (sigmoid output; 0-3600 is mapped between 0-1), (b) Classification branch - to output class probability of the 20 possible 2-degree curves (softmax classifier).

[0101] FIG. 8C shows a sub-divided image grid in accordance with another embodiment of the present invention.

[0102] The image grid subdivision representation technique is inspired by the technique for finding connected components. Each top level DFS run (commencing at the generate_sketch_sequence function level) discovers one connected component in the grid image (in the undirected sense). Here, the edges from each connected component are equivalent to the edges drawn by the user. The edges of the connected component are sequenced using DFS using adjacent edges sharing start / end points / coordinates as neighbors. This guarantees that the drawing sequence is captured in a way so as to afford a smooth, natural and continuous drawing experience for the user.

[0103] In contrast, the drawing sequence could have been generated in the order in which the m x n grid space was visited during curve fitting. This would have caused a disconnected grid after grid drawing experience for the user, which is not at all ideal either in a pedagogical or enjoyment / usability sense. The whole sequence generation has been based on how one usually draws - draw a complete connected section in the drawing before we move on to the next connected portion.

[0104] While creating the neighbor map, each grid is inserted by both of its containing edge's start and end coordinates. This is done so as to allow DFS to absorb it from either direction. When the user chooses to sketch multiple object scenes, the decomposition step collects separate edge list and neighbor edge map pairs for each object (following the segmented object edges for each object as passed on from the object segmentation step) and the instruction generation algorithm runs on each pair (of edge list and neighbor maps to generate drawing sequence / steps for each object.

[0105] FIG. 9 is a pictorial representation of a use case showing different modes of inputs to a drawing instruction generation system in accordance with another embodiment of the present invention.

[0106] In this use case, it is shown that the image input may be provided by various means, such as, but not limited to, (1) providing the image with single object, (2) providing the image with multiple objects without the label, (3) providing the image with multiple objects with the label for the object of interest, (4) providing 3D object, and (5) providing voice input.

[0107] FIG. 10 is a pictorial representation of a use case showing a drawing instruction generation system implemented in a television in accordance with another embodiment of the present invention.

[0108] In this use case, the user shows a 3D object to the television. The television executes the drawing instruction generation method of the present invention to display the drawing instructions for drawing the elephant.

[0109] FIGS. 11-12 show pictorial comparisons of a drawing instruction generation system in accordance with another embodiment of the present invention with a conventional drawing system.

[0110] The present invention facilitates the users to draw difficult objects with ease, using the drawing instructions displayed on the television.

[0111] FIG. 13 is a pictorial representation of a use case showing a drawing instruction generation system implemented in a television in accordance with another embodiment of the present invention.

[0112] In this use case, the user shows a picture to an electronic device, such as, but not limited to, television, computer, and tablet. The electronic device executes the drawing instruction generation method of the present invention to display the drawing instructions for drawing the picture.

[0113] FIG. 14A is a pictorial representation of a use case showing a drawing instruction generation system using a voice input in accordance with another embodiment of the present invention.

[0114] In this use case, the user provides voice input to the television. The television executes the drawing instruction generation method of the present invention to display the drawing instructions.

[0115] FIG. 14B is a pictorial representation of a use case showing a drawing instruction generation implemented in Augmented Reality (AR) / Virtual Reality (VR) headset in accordance with another embodiment of the present invention.

[0116] In this use case, the drawing instructions are provided to the user by way of the AR / VR headset.

[0117] FIG. 15 is a pictorial representation of a use case showing a drawing instruction generation system implemented in a television in accordance with another embodiment of the present invention.

[0118] In this use case, the user shows a 3D object to the television. The television executes the drawing instruction generation method of the present invention to display the drawing instructions for drawing the peacock.

[0119] FIGS. 16-17 are a pictorial representation of a use case showing a drawing instruction generation implemented in Augmented Reality (AR) / Virtual Reality (VR) headset in accordance with another embodiment of the present invention.

[0120] In this use case, the drawing instructions are provided to the user by way of the AR / VR headset.

[0121] FIG. 18 is a pictorial representation of a use case showing a drawing instruction generation implemented in Augmented Reality (AR) / Virtual Reality (VR) glasses and a smartphone in accordance with another embodiment of the present invention.

[0122] In this use case, the drawing instructions are provided to the user by way of the AR / VR headset.

[0123] FIG. 19 is a pictorial representation of a use case showing a drawing instruction generation system implemented in a television in accordance with another embodiment of the present invention.

[0124] In this use case, two or more users can use the drawing instruction generation technique of the present invention to draw simultaneously in metaverse.

[0125] FIG. 20 is a pictorial representation of a use case showing display of drawing instructions on an electronic device in accordance with another embodiment of the present invention.

[0126] In this use case, the drawing instructions are provided to the user in the form of a video.

[0127] FIG. 21 is a pictorial representation of a use case showing display of drawing instructions on an electronic device in accordance with another embodiment of the present invention.

[0128] In this use case, the drawing instructions are provided to the user in interactive way using a television.

[0129] It should be noted that the description merely illustrates the principles of the present invention. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described herein, embody the principles of the present invention.

[0130] Furthermore, all examples recited herein are principally intended expressly to be only for explanatory purposes to help the reader in understanding the principles of the invention and the concepts contributed by the inventor(s) to furthering the art and are to be construed as being without limitation to such specifically recited examples and conditions.

[0131] Moreover, all statements herein reciting principles, aspects, and embodiments of the invention, as well as specific examples thereof, are intended to encompass equivalents thereof.

Claims

1. A drawing instruction generation system (208), comprising: an object processing unit (260) configured to extract at least one object from an input image; an edge image generation unit (262) configured to generate an edge image based on detection of a plurality of edges of the extracted object, wherein the plurality of edges are numbered serially; and a processing unit (264) configured to: divide the edge image into a plurality of serially numbered grids, generate a plurality of serially numbered fitted edge sections corresponding to the plurality of serially numbered edges and grids thereof, generate a set of drawing instructions based on the serially numbered fitted edge sections and corresponding grids, and display the set of drawing instructions on a display (220) sequentially.

2. The drawing instruction generation system (208) as claimed in claim 1, wherein the processing unit (264) comprises a curve fitting unit (280) configured to fit, in every serially numbered grid, at least one of: a straight line and a Bezier curve on one or more edges within the aforesaid grid to generate the corresponding serially numbered fitted edge section.

3. The drawing instruction generation system (208) as claimed in claim 2, comprising: a fit test unit (288) configured to: measure a distance metric for the one or more fitted Bezier curves of a fitted edge section, and compare the measured distance metric with a predetermined distance metric to determine an error indicative of a quality of the fitted Bezier curve for the fitted edge section; and a recursive unit (290) configured to: compare the error of the fitted edge section with a predefined threshold, and divide the grid corresponding to the fitted edge section into a plurality of sub-grids when the error is above the predefined threshold, wherein the curve fitting unit (280) is configured to perform curve fitting on the plurality of sub-grids to generate a plurality of fitted edge sections corresponding to the plurality of sub-grids.

4. The drawing instruction generation system (208) as claimed in claim 1, wherein the object processing unit (260) comprises: an object recognition unit (266) configured to identify presence of the object in the input image; an object detection unit (268) configured to extract the object and to generate a bounding box around the object, thereby localizing the object in the input image; an object segmentation unit (270) configured to separate the object from a plurality of other objects in the input image; and an instance segmentation unit (272) configured to separate a plurality of instances of a same object in the input image.

5. The drawing instruction generation system (208) as claimed in claim 2, wherein the curve fitting unit (280) includes a training unit (286) configured to: train a Convolutional Neural Network (CNN) to classify one or more edges into one or more classes.

6. The drawing instruction generation system (208) as claimed in claim 5, wherein the training unit (286) is configured to: perform regression on one or more anchor boxes to determine a rotation for the corresponding edge, wherein a result of the regression is indicative of a probability of one or more angles of rotation, and perform a classification of the edge, wherein a result of the classification is indicative of a probability of the one or more classes.

7. The drawing instruction generation system (208) as claimed in claim 1, wherein the input unit (202) is configured to: receive a voice command from a user, and fetch the input image from one or more databases (206) based on the received voice command.

8. A drawing instruction generation method (300), comprising: extracting, by an object processing unit (260), at least one object from an input image; generating, by an edge image generation unit (262), an edge image based on detection of a plurality of edges of the extracted object, wherein the plurality of edges are numbered serially; dividing, by a processing unit (264), the edge image into a plurality of serially numbered grids; generating, by the processing unit (264), a plurality of serially numbered fitted edge sections corresponding to the plurality of serially numbered edges and grids thereof; generating, by the processing unit (264), a set of drawing instructions based on the serially numbered fitted edge sections and corresponding grids; and displaying, by the processing unit (264), the set of drawing instructions on a display (220) sequentially.

9. The drawing instruction generation method (300) as claimed in claim 8, comprising fitting, by a curve fitting unit (280) in the processing unit (264), in every serially numbered grid, at least one of: a straight line and a Bezier curve on one or more edges within the aforesaid grid to generate the corresponding serially numbered fitted edge section.

10. The drawing instruction generation method (300) as claimed in claim 9, comprising: measuring, by a fit test unit (288), a distance metric for the one or more fitted Bezier curves of a fitted edge section; comparing, by the fit test unit (288), the measured distance metric with a predetermined distance metric for determining an error indicative of a quality of the fitted Bezier curve for the fitted edge section; comparing, by a recursive unit (290), the error of the fitted edge section with a predefined threshold; dividing, by the recursive unit (290), the grid corresponding to the fitted edge section into a plurality of sub-grids when the error is above the predefined threshold; and performing, by the curve fitting unit (280), curve fitting on the plurality of sub-grids for generating a plurality of fitted edge sections corresponding to the plurality of sub-grids.

11. The drawing instruction generation method (300) as claimed in claim 8, comprising: identifying, by an object recognition unit (266), presence of the object in the input image; extracting the object, by an object detection unit (268), for generating a bounding box around the object, thereby localizing the object in the input image; separating the object from a plurality of other objects in the input image by an object segmentation unit (270); and separating a plurality of instances of a same object in the input image by an instance segmentation unit (272).

12. The drawing instruction generation method (300) as claimed in claim 11, comprising: training, by the training unit (286), a Convolutional Neural Network (CNN), to classify one or more edges into one or more classes.

13. The drawing instruction generation method (300) as claimed in claim 12, comprising,: performing, by the training unit (286), regression on one or more anchor boxes for determining a rotation for the corresponding edge , wherein a result of the regression is output indicative of a probability of one or more angles of rotation; and performing a classification, by the training unit (286), of the edge, wherein a result of the classification is indicative of a probability of the one or more classes.

14. The drawing instruction generation method (300) as claimed in claim 8, comprising: receiving, by the input unit (202), a voice command from a user; and fetching, by the input unit (202), the image from one or more databases (206) based on the received voice command.