Methods, systems, and apparatus for digitizing surface feature data

Abstract

Systems, methods, and computer-readable media are provided for digitizing data associated with features on a surface and, in some cases, maintaining dimensional accuracy of the features in the digitized representation of the features. Various approaches include capturing an image of a feature on a surface with an arrangement of markers also included in the captured image. Some approaches use positions of markers from the arrangement in the captured image and known information describing the arrangement of markers to generate a digital representation of the feature. Two or more generated digital representations of a feature may be combined to create an updated digital representation that captures a larger portion of the feature that spans a larger area.

Description

Methods, Systems, and Apparatus for Digitizing Surface Feature DataCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority to U.S. Application Serial Number 63 / 507,471, filed on June 11, 2023, and U.S. Application Serial Number 63 / 583,251, filed on September 16, 2023, each of which are incorporated by reference herein for all purposes.BACKGROU N D OF TH E INVENTION

[0002] The subject matter discussed in the background section should not be assumed to be prior art merely as a result of its mention in the background section. Similarly, a problem mentioned in the background section or associated with the subject matter of the background section should not be assumed to have been previously recognized in the prior art. The subject matter in the background section merely represents different approaches, which in and of themselves may also correspond to implementations of the claimed technology.

[0003] The disclosure relates generally to the field of drawing, designing, wood-working, machining (e.g., metal or plastic using a CNC), and material cutting (e.g., paper, fabric, leather, vinyl, foam) and, in particular, to methods and tools that allow a user to capture information from their workspace (e.g., a drawing on a wood piece) to digital form, e.g., for subsequent use in multiple tasks. In an exemplary use case, it may be quicker for a user to draw a design for cutting on a workpiece than creating the design digitally. This may create an issue if the user wants to use an automated tool (e.g., a tool that can follow a digital design programmatically) for cutting the design. In another exemplary use case, an artist that wants to create digital versions of their designs may find it easier to draw on paper than to draw using digital tools.

[0004] Capturing an image of a design drawn on a surface using a camera and manually converting the captured image to a digital design may lead to design capture quality issues or require many manual steps. For example, capturing an image of the design from a "top view" may lead to the image having shadows. Any shadows in the image may lead to creation of artifacts that may be treated as being part of the design being captured if the shadows are not removed from the image before the digital design is created from the image. In another example, if the image of the design is captured from an angle relative to the top of the design (e.g., to avoid shadows), the perspective of the design captured in the image may be skewed sothat actual dimensions for features in the design are no longer accurate or no longer to scale compared to other features in the design. Additionally, many camera lenses distort images, and this lens distortion can cause the captured image to deviate from the true scale and shape of the design. Any skew or deformation of the design in the captured image requires correction by the user to maintain accuracy of the design. In another example, once an image of a design is captured in an image, it may be difficult to determine the physical size of the design. If the design scaling is known, it may be necessary to apply the scaling correction to the digital design as a separate step in the digital design creation process.SU MMARY OF TH E I NVENTION

[0005] Embodiments of the disclosure are directed to methods, systems, apparatus, and computer readable media that may be used to allow a user to capture information from their workspace to digital form. Some embodiments provide a seamless experience to a user who is provided an application to generate an accurate digital design by capturing an image of a graphic drawn on a surface without requiring the user to capture the image from a specific camera pose, calibrate a camera of the mobile device (e.g., to account for a focal length of a camera lens or a distortion characteristic of the camera lens), or to manually interact with the captured image.

[0006] In one implementation, a method includes placing an arrangement of one or more markers relative to a feature (e.g., drawing) on a surface. An image of the feature and at least a portion of the arrangement is captured using a camera. A digital representation of the feature (e.g., vector graphics data, bitmap graphics data) is generated using computer vision ("CV") techniques on the captured image. In some embodiments, the CV techniques rely on known properties of the arrangement of the one or more markers to rectify the image (e.g., correcting for camera pose, camera lens focal length and distortion) before generation of the digital representation. In some embodiments, the digital representation preserves the dimensional scale of the feature to an accuracy better than 10%, 8%, 5%, 3%, 2%, 1%, 0.5%, 0.3%, 0.2% or 0.1%.

[0007] In another implementation, a system includes one or more memories storing instructions that execute on one or more processors of a mobile device. In some embodiments, an application executing on a mobile device uses a camera on the mobile device to capture an image of an arrangement of one or more markers relative to a feature (e.g., drawing) on asurface. The application generates a digital representation of the feature (e.g., vector graphics data, bitmap graphics data) using CV techniques on the captured image. In some embodiments, the application uses CV techniques that rely on known properties of the arrangement of the one or more markers to rectify the image (e.g., correcting for camera pose, camera lens magnification) before generating the digital representation. In some embodiments, the digital representation generated using the application preserves the dimensional scale of the feature to an accuracy better than 10%, 8%, 5%, 3%, 2%, 1%, 0.5%, 0.3%, 0.2% or 0.1%.BRI EF DESCRI PTION OF DRAWINGS

[0008] Figures 1 and 2 illustrate an exemplary drawing to be generated into a digital representation, which may be used to fabricate a part. Note: Rectangular dashed box element shown in some figures represents a video playback III element (e.g., including playback, volume, time shifting controls; not shown) in screen capture of data shown in each respective figure.

[0009] Figure 3 shows an exemplary jig which may be used to digitize a design from a surface.

[0010] Figure 4 shows an exemplary code that may be present on a jig to provide information associated with the jig.

[0011] Figure 5 illustrates an embodiment that allows a user to access a progressive web application using information included in an exemplary code provided on a jig.

[0012] Figures 6 and 7 illustrate functionality of an application used to capture data using a jig.

[0013] Figure 8 illustrates an arrangement of a jig around a drawing to allow digitizing of the drawing.

[0014] Figure 9 illustrates a user interface of an application to capture data using a jig placed near a drawing.

[0015] Figure 10 illustrates a user interface of an application while captured data is being processed to generate a digital representation of a drawing.

[0016] Figures 11, 12, and 16 illustrate a user interface of an application displaying a first view of a digital representation of a drawing generated from an image captured by a camera.

[0017] Figures 13-15 illustrate a user interface of an application displaying a second view of a digital representation of a drawing generated from an image captured by a camera.

[0018] Figures 17 and 18 illustrate an application functionality to reset a view of a digital representation of a drawing.

[0019] Figures 19-23 illustrate an application workflow for a user logging into a system that permits digital design management.

[0020] Figures 24-29 illustrate an application workflow for a user storing a digital representation of a drawing.

[0021] Figures 30-37 illustrate an application workflow for a user selecting a digital representation and scaling the digital representation for use in fabricating a part.

[0022] Figures 38-40 illustrate an application workflow on a tool with a user selecting a digital representation and positioning the digital representation relative to a workpiece for fabrication.

[0023] Figures 41-45 illustrate fabrication of a part with a pattern from a digital representation using a tool.

[0024] Figures 46 and 47 illustrate views of a finished part having a pattern from a digital representation.

[0025] Figure 48 shows a plot of the positions of markers in an exemplary arrangement of markers on a jig.

[0026] Figure 49 illustrates a jig positioned to capture an image of a signature.

[0027] Figure 50 shows a schematic of an exemplary computer system used to implement a disclosed embodiment.

[0028] Figures 51-53 show a flow chart describing an exemplary workflow for generating a digital representation from a captured image.

[0029] Figure 54 illustrates an exemplary drawing to be generated into a digital representation.

[0030] Figure 55 illustrates a user interface of an application showing a jig aligned with a first portion of the exemplary drawing.

[0031] Figure 56 illustrates a user interface of an application showing a captured first digital representation of a first portion of the exemplary drawing.

[0032] Figure 57 illustrates a user interface of an application showing the jig aligned with a second portion of the exemplary drawing.

[0033] Figure 58 illustrates a user interface of an application showing a captured second digital representation of a second portion of the exemplary drawing.

[0034] Figure 59 illustrates a combined digital representation based on stitching together the captured first and second digital representations.

[0035] Figures 60-63 illustrate user interface views showing jig alignment and captured digital representations of an exemplary drawing as the drawing is modified.DETAI LED DESCRI PTION OF TH E I NVENTION

[0036] The present description is made with reference to the accompanying drawings, in which various example embodiments are shown. However, many different example embodiments may be used, and thus the description should not be construed as limited to the example embodiments set forth herein. Rather, these example embodiments are provided so that this disclosure will be thorough and complete. Various modifications to the exemplary embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the disclosure. Thus, this disclosure is not intended to be limited to the disclosed embodiments, but is to be accorded the widest scope consistent with the claims and the principles and features disclosed herein.

[0037] Some embodiments described herein relate to generating digital information associated with a drawing on a surface. The techniques may also be used to capture digital information related to other features on a surface. For example, features related to wood grain on a piece of wood or pattern of brick work on the ground or a wall may be digitized.

[0038] Figures 1 to 47 illustrate various steps of exemplary workflows for generating a digital representation of a drawing and fabricating a part using a version of the generated digital representation. Figure 1 shows a user drawing an exemplary design (line drawing of mountains) on a piece of paper. Figure 2 shows the completed drawing. Figure 3 shows an exemplary jig in the shape of a rectangle (frame) with an opening in the middle (surrounding a drawing of an outline of a chisel). The outer dimensions of the jig shown in Fig. 3 are approximately 340 mm by 250 mm, and the opening in the jig is approximately 275 mm by 185 mm (with the frame being 32.5 mm wide on all sides). The thickness of the jig is approximately 5 mm. The jig includes an arrangement of markers (white dots) around the edge of the frame. As shown in Fig. 4, the jig may include a QR code. In some embodiments, the QR code may include information describing the arrangement of markers. In other embodiments, a software application, described below, may be pre-programmed with information describing one or more arrangement of markers and can use the imaged marker pattern to match the jig with the appropriate marker arrangement information. In some embodiments, the QR code may provide a URL to access information related to the jig (e.g., information describing the arrangement of markers on the jig). In some embodiments, a unique code associated with the jig (e.g., QR code shown in Fig. 4) may be used to authenticate a user accessing an application. In some embodiments, the user authenticationmay be required only the first time the user accesses the application. This process may be used to prevent users that do not have a licensed jig from accessing the application functionality.

[0039] In some embodiments, as shown in Fig. 5, the QR code may provide a URL to access a progressive web application (PWA) that may be executed in a browser of a mobile device to generate a digital representation of a drawn design. Any steps or functionality discussed herein with respect to a PWA may be performed alternatively using any application (e.g., a native application downloaded on a mobile device, web application, or the like). In some embodiments, as shown in Fig. 6, a PWA executing on a browser of a mobile device requests permission to access a camera on the mobile device. In some embodiments, as shown in Fig. 7, a PWA executing on a browser of a mobile device provides instructions in a GUI on how to configure the camera to capture an image of the drawing to a user of the mobile device.

[0040] Figure 8 shows how an exemplary jig may be placed relative to a drawn design before capturing an image of the design. For a PWA executing in a browser of a mobile device, Figure 9 shows an exemplary GUI for capturing an image of the drawing with the image including at least a portion of the arrangement of markers. Once an image is captured, a PWA executing in a browser of a mobile device may show the GUI shown in Fig. 10 while digital information associated with the drawing is being generated. Figure 11 shows an exemplary GUI of the PWA showing the generated digital information associated with the drawing. In some embodiments, the generated digital information includes dimensions associated with the drawing (e.g., length, width). In some embodiments, the generated digital information includes vector graphics data (digital line drawing image) representing the drawing captured by the PWA. In some embodiments, the digital representation of the drawing shows an outline of the drawing on the surface - as shown in Fig. 11 with the "OUTLINE" selection activated. In some embodiments, the digital representation of the drawing shows a centerline of the drawing on the surface - for example, if "CENTERLINE" selection is activated, see Fig. 13 below. In some embodiments, as shown in Fig. 11, a GUI of the PWA may provide the user an option to "RETAKE" the image (to regenerate the digital information associated with the drawing). In some embodiments, as shown partially obscured in Figs. 11, 12, a GUI of the PWA may provide the user an option to "MODIFY" the generated digital information, see Fig. 13 below.

[0041] Embodiments of the invention may compute the centerline vector graphics data from outline vector graphics data or from the rectified bitmap graphics data directly. Embodiments of the invention may compute the centerline vector graphics data utilizing algorithms andtechniques including a medial axis transformation, a scale axis transformation, a Voronoi transformation, a Delaunay triangulation, a straight skeleton transformation, a grassfire transformation, and analyzing gradients of bitmap graphics data. Outline vector graphics data and / or bitmap graphics data may be filtered to reduce noise and / or shape complexity prior to computing centerline vector graphics data. An initial centerline vector graphics data may be further processed to improve the output quality, for example by applying Bezier curve simplification to reduce the number of curve nodes in the centerline vector graphics data, topological simplification by pruning redundant branches and / or detecting and removing graph cycles of the centerline vector graphics data, applying heuristics to centerline intersection regions to modify the vector graphics data to be more aesthetically pleasing or match the presumed artistic intent, and / or modifying the curvature or control points of portions of the centerline vector graphics data.

[0042] Additionally, embodiments of the centerline vector graphics data may include distance values for at least a portion of the points in the centerline vector graphics data. These distance values indicate or approximate the distance from a given centerline point to the nearest point or points of the outline expressed as vector or bitmap graphics. These values may also correspond with a radius or diameter of a circle inscribed within the boundaries of the drawn lines or filled areas and centered at a given point of the centerline vector graphics data. The distance values may be utilized to generate the centerline vector graphics data, further process the centerline vector graphics data (such as described above) and may optionally be output in conjunction with the centerline vector graphics data to assist with user fabrication of the design.

[0043] An example of the latter use is V-carving. V-carving is a three-dimensional cutting technique that uses a varying depth value proportional to the actual or interpolated distance values of the centerline vector graphics data. In this example, centerline vector graphics data, for example in the form of two-dimensional points and / or control points (e.g. (x, y) coordinates) describing lines and curves may optionally be augmented with a third dimension representing distance values (e.g. (x, y, z), where z is a function of the centerline distance value at location (x, y)).

[0044] Figure 12 shows another view of the PWA GUI shown in Fig. 11. Figure 13 shows a revised view of the generated digital information in the PWA GUI with the "CENTERLINE" mode of the digital representation of the drawing selected. Figure 14 shows an updated view of the digital representation in the PWA GUI after a pinch-to-zoom input is received from a user on thescreen of the mobile device. Figure 15 shows an updated view of the digital representation in the PWA GUI after a touch-and-drag to pan input is received from the user on the screen of the mobile device. Figure 16 shows an updated view of the digital representation in the PWA GUI after the "OUTLINE" selection is activated. In some embodiments, the PWA GUI may include functionality to allow a user to reset the view of the digital representation of the drawing - selecting the square icon near the user's fingertip in Fig. 17. After a user's input to reset the view is received, the PWA GUI shows a revised view of the digital representation of the drawing as shown in Fig. 18.

[0045] If a user of the PWA is satisfied with the generated digital information associated with the drawing, the user can select the checkbox icon in the GUI shown in Fig. 18. In some embodiments, the PWA may allow a user to log into an account to access additional functionality within the PWA - Figs. 19-24 show exemplary GUIs showing the login and authentication process. In some embodiments, the PWA may allow a user to export the digital information in one or more formats (e.g., SVG file) via one or more communication modes / channel as shown in Fig. 25. Figures 26-29 show an exemplary workflow for storing the digital information in a storage workspace (Shaper Studio) associated with the user's account.

[0046] In some embodiments, the PWA may allow a user to modify the generated digital representation of the drawing. As shown in Fig. 30, a user can select a version of the generated digital representation for "placement" on a cutting tool (for example, Shaper Origin), see Figs. 38-40 below. Once a digital design is selected, a user may be offered options to modify the design as shown in Fig. 31 - including: adjusting the size (left most icon near the tip of the user's thumb), rotating the design, translating the design, or placing the design on a tool (right most icon). Figures 31-37 show an exemplary workflow for a user selecting a portion of the design, scaling the selected portion to a designed dimension (3.4" (86.4 mm) in this example), and storing the revised design in the user's storage workspace. Figures 38-40 show an exemplary workflow for a user to select and place the revised design in the workspace of a cutting tool (Shaper Origin). Figures 41-45 show an exemplary workflow for cutting a workpiece using the placed revised design using a cutting tool (Shaper Origin). Figures 46-47 show views of the part fabricated using the revised design. Figure 49 shows a use of the jig to generate a digital representation of a signature on a paper and using the generated digital representation to carve the signature in various wood products. Other features that may be digitized include: calligraphy, artwork (e.g., logos), inlays, tool outlines (e.g., for shadow foam), or the like.

[0047] Figures 51-53 illustrate an exemplary process flow that details some of the steps for generating and using a digital design based on a drawing ("artwork"). The exemplary process includes a step for creating the drawing on a surface and setting up a jig with an arrangement of markers relative to the drawing. In the exemplary process, an application ("Reality Capture App") is used to capture image data associated with the drawing. The application may be used to convert the image data to a digital design (e.g., vector data) that may be used during fabrication. The application may output the digital design (e.g., captured image, rectified image, vector data) to be stored locally or added to a remote store (e.g., Shaper Studio). The output data may be modified and used for fabrication (e.g., cutting, drawing) of the digital design on a workpiece. The application may be used to modify or edit the digital design based on the captured image or any subsequently generated digital information.

[0048] In some embodiments, information describing the arrangement of markers may be used to rectify an image that includes at least a portion of the arrangement of markers using CV techniques (e.g., using least-squares solver to identify parameters that minimize reprojection error). In some embodiments, rectifying an image includes one or more of correcting for: a pose of the camera when the image was captured, a focal length of a lens of the camera, or a distortion characteristic of the lens of the camera. Further embodiments of the invention may compute or estimate a homography from the captured image to a space having a desired camera pose and minimized lens distortion. Additional embodiments may utilize iterative techniques to rectify an image by determining a first estimated camera pose and / or lens parameters, determining an error estimate from this, and using this error estimate to determine a better second estimated camera pose and / or lens parameters. This iterative process may be repeated multiple times until the error estimates converge below a desired threshold value. Numerical equation system solving techniques, including non-linear least-squares solvers, can be utilized to assist with these computations.

[0049] In some embodiments, the arrangement of markers is described by a list of (x, y) coordinates for the centers of the markers. Figure 48 shown the center locations for an arrangement of markers (open circles) with an outer edge of a jig (solid line), with the dimensions shown in meters. In some embodiments, information describing the arrangement of markers may also include a size (e.g., radius, diameter) for each marker. In some embodiments, a partial pattern of the arrangement of markers (e.g., 3 markers near the corner of the frame with 3 markers along the long edge and 2 markers along the short edge - corner at the top ofFig. 3) may be used to define an orientation of the jig (e.g., defining the top left corner of the digitized representation) and may be used to define an orientation of a digital representation of a drawing. In some embodiments, a digital representation of the drawing (or a bounding box including the digital representation) may be aligned with a feature associated with the arrangement of markers (e.g., along a long or short edge of the frame). For example, the digital representation of the chisel drawing may be rotated to align the long axis of the chisel along the long edge of the frame in Fig. 3. In some embodiments, a digital representation of the drawing may be orientated based on an orientation of a camera (e.g., portrait or landscape mode) when an image of the drawing is captured.

[0050] In some embodiments, the information describing the arrangement of markers (e.g., positions of some of the markers included in the arrangement) may be used to determine a pose of a camera used to capture an image that includes at least a portion of the arrangement of markers using CV techniques. In some embodiments, the information describing the arrangement of markers may be used to determine a focal length of a lens of a camera used to capture an image that includes at least a portion of the arrangement of markers using CV techniques. In some embodiments, the information describing the arrangement of markers may be used to determine a distortion characteristic of a lens of a camera used to capture an image that includes at least a portion of the arrangement of markers using CV techniques. In some embodiments, a focal length of a lens of the camera or a distortion characteristic of the lens of the camera may be calculated separately using the information describing the arrangement of markers. In some embodiments, the distortion characteristic of the lens of the camera or the focal length of the lens of the camera may be calculated next using the information describing the arrangement of markers. The results of these two calculations may be used as fixed parameters when a pose of the camera when the image was captured is calculated. In some embodiments, parameters associated with camera pose and lens-based parameters are allowed to vary when calculating a pose of the camera when the image was captured, a focal length of a lens of the camera, and a distortion characteristic of the lens of the camera simultaneously.

[0051] In some embodiments, the PWA may query the operating system of the mobile device for one or more of the focal length of the lens on the camera or a distortion characteristic of the lens of the camera. If the mobile device has more than one camera, the PWA may query the operating system to provide information related to the camera or lens being used by the PWA. In some embodiments, a PWA may receive an image of a drawing including at least a portion ofan arrangement of markers from a remote device. The PWA may generate digital information associated with the drawing by analyzing the portion of the arrangement of markers in the received image. In some embodiments, the PWA may analyze an image (e.g., live preview image, captured image, received image) to determine if a portion of one of a known set of arrangement of markers is present in the image. In some embodiments, if a portion of a known arrangement of markers is detected, the PWA may notify the user that the image contains sufficient information to generate digital information associated with a drawing in the image.

[0052] In some embodiments, after the image is rectified based on the information describing the arrangement of markers, the rectified image may be used as a digital representation of the drawing. In some embodiments, the portion of the rectified image including the drawing data is analyzed to generate a digital representation of the drawing (e.g., vector graphics data). In some embodiments, image data associated with the arrangement of markers is cropped out of the rectified image so that the digital information is based only on information associated with the drawing. In some embodiments, one or more image processing techniques (e.g., image filtering, fixed or adaptive image thresholding, shadow removal or minimization, morphological operations such shrinking or expanding features, edge detection, image masking and per-pixel Boolean operation, contour detection, image region operations) may be applied to the captured image or rectified image to remove features (e.g., shadows, stray marks) that may need to be removed from the captured image. In some embodiments, the digital representation of the drawing comprises points or line segments defining the edges of the digitized drawing. In some embodiments, if the height of the markers is not the same as the height of the drawing (e.g., if the jig has a non-negligible thickness), the jig thickness may be included in the analysis for generating the digital representation of the drawing (e.g., to correct for differences introduced by camera pose or lens parameters). In some embodiments, the thickness of the jig (e.g., distance between markers and drawing surface) may vary (for example, if the top surface and bottom surface of the jig are not parallel); however, this thickness variation may be included in the analysis to accurately generate the digital representation of the drawing.

[0053] In some embodiments, digital information is generated based on the captured image within 120 seconds, 90 second, 60 seconds, 45 seconds, 30 seconds, 20 seconds, 10 seconds, 5 seconds, 3 seconds, 1 second, 500 milliseconds, 300 milliseconds, or 100 milliseconds of capturing the image of the drawing. A digital representation of a drawing is generated in less than 1 second after capturing an image using a PWA in Safari on an iPhone 14 Pro running iOS16. In some embodiments, the time required to generate the digital information is based on the processing capabilities of the mobile device on which the browser with the PWA is executing. In some embodiments, a digital representation of the drawing is generated without any additional input from a user of the mobile device - e.g., to crop the captured image to remove a portion of the arrangement of markers, to remove shadows or stray marks from the captured image, etc.

[0054] In some embodiments, the digital representation of the drawing may provide dimensionally accurate data associated with the drawing. The digital representation may include values represented in physical units, such as inches, cm, or mm, rather than digital units such as pixels, percentages, or unitless numbers. For example, the digital representation may specify a bounding rectangle of the drawing in terms of physical units (e.g., width="2in" height="4in") and / or include physical units on the values of the vector data, such as the values of the Bezier curve control points (e.g., a point at 2.5in, 3.75in.). Additional embodiments of the invention may include graphical indicators of the bounding dimensions of the digital representation of the drawing. For example, if the drawing includes a rectangle that is 130 mm by 180 mm, then a digital representation of the drawing (generated from a first image including an arrangement of markers) may indicate that the short edge is 129 mm and the long edge is 179 mm - achieving an accuracy of better than 1% relative to the dimensions of the drawn rectangle. In some embodiments, one short edge of the rectangle may be described in the digital representation as a line extending from (0,0) to (0,129) - the digital representation describing the rectangle in millimeters. In some embodiments, one short edge of the rectangle may be described in the digital representation as a line extending from (0,0) to (0,12.9) with a scaling factor of 10. If another image of the drawing including the arrangement of markers is taken, a digital representation of the drawing generated from the second image may indicate that the short edge is 128 mm and the long edge is 178 mm - achieving an accuracy of better than 2% relative to the dimensions of the drawn rectangle. In some embodiments, the inclusion of at least a portion of an arrangement of markers with known dimensional characteristics in an image of a drawing provides data that may be used to generate a dimensionally accurate digital representation of the drawing without requiring additional information such as camera pose when the image was captured, focal length of lens on the camera, or information related to distortion characteristics of the lens on the camera.

[0055] In some embodiments, calculation of a camera pose or a parameter associated with a lens of the camera using a first image (including a drawing and a portion of the arrangement ofmarkers) may fail due to artifacts or singularities that appear in the calculations. In such instances, a second image including a portion of the arrangement of markers (e.g., from a different camera pose) may be used to determine the second camera pose or the parameter associated with the lens of the camera. Data from the first image or data from the second image may be used to generate digital information associated with the drawing.

[0056] In some embodiments, the arrangement of markers may be on a jig shaped like a "T", "L" or a "U" instead of having a frame shape as shown in Fig. 3. In some embodiments, the arrangement of markers may be on one or more parts (e.g., pucks, tiles) that may be placed relative to a drawing prior to capturing an image. In some embodiments, the arrangement of markers may be on a jig with a region through which the drawing is visible (e.g., the region including a transparent plastic or glass).

[0057] In some embodiments, an exemplary method may include "stitching" or merging adjacent data-captures together into an accurately-aligned combined scene or composite map. In some embodiments, one or more unique fiducial markers are placed into a field of view to be included in the capture of artwork (or workpiece) with an imaging device such as camera or mobile device (phone, tablet). Without moving the location of the unique fiducial marker(s), an overlapping or adjacent area of the workpiece is captured such that at least one of the same markers is still visible within the new field of view. Optionally, a jig (a fiducial marker frame, e.g., Shaper Frame) is utilized to accurately preserve scale and to flatten or undistort the captured images. A software application (e.g., Shaper Trace app) exactly aligns the unique marker(s) and subsequently stitches the image elements together.

[0058] In some embodiments, unique fiducial(s) in multiple fields of view may be used to combine data from the multiple fields of view into a one combined scene. In some embodiments, multiple unique fiducials may aid accuracy of image alignment. In some embodiments, other unique features or characters captured within a particular field of view can also be utilized to help align overlapping image areas. In some embodiments, fiducial marker(s) may have unambiguously identified orientation to permit inferring a marker's rotational orientation unambiguously. In some embodiments, additional fiducial (s) may be used to further extend the aggregated map. In some embodiments, images including unique marker(s) may not be captured at the same time, e.g., permitting asynchronous data capture. In some embodiments, when combined with a jig, a single unique fiducial may be used to define an orientation relative to a prior jig location. In some embodiments, when scanned images areconverted into an SVG and then combined together, paths from different images may be combined, i.e., paths combined into continuous paths across merged frames, instead of fragments paths. In some embodiments, a software application may subtract out (or remove) fid ucial (s) from the combined scene.

[0059] Figure 54 illustrates an exemplary drawing to be generated into a digital representation. Figures 55 and 56, respectively, illustrate the placement of a jig and captured first digital representation for a first portion of the exemplary drawing. Figures 57 and 58, respectively, illustrate the placement of the jig and captured second representation of a second portion of the exemplary drawing. Figure 59 illustrates a combined digital representation based on stitching together the first and second digital representations, e.g., using the "x" in the circle in the first and second captured images for alignment and stitching.

[0060] Figures 60 and 61, respectively, illustrate the placement of a jig and captured third digital representation for a third portion of an update of the exemplary drawing - with "GER" added to "BIG" to revise the drawing to read "BIGGER PICTURE." Figures 62 and 63, respectively, illustrate the placement of a jig and captured fourth digital representation for a fourth portion of a further update of the exemplary drawing - with the word "EVEN" added above "BIGGER PICTURE." Portions of the third and fourth digital representations may be aligned and merged with portions of the combined digital representation. Existing fiducial(s), existing patterns, or new f id ucial(s) may be used to permit alignment and stitching of captured digital representations. In some embodiments, applying fiducials, e.g., stickers with patterns - "Dots" on a roll, may be used permit alignment and stitching of captured digital representations. The patterns withing a set of "Dots" may be selected to avoid two patterns matching once pattern rotation or translation is taken into account (e.g., avoiding rotational or translational symmetry).

[0061] Figure 50 illustrates an example of a computer system 800 (e.g., a mobile device) that may be used to execute program code stored in a non-transitory computer readable medium (e.g., memory) in accordance with embodiments of the disclosure. The computer system includes an input / output subsystem 802, which may be used to interface with human users and / or other computer systems depending upon the application. The I / O subsystem 802 may include, e.g., a keyboard, mouse, graphical user interface, touchscreen, or other interfaces for input, and, e.g., an LED or other flat screen display, or other interfaces for output, including application program interfaces (APIs).

[0062] Program code may be stored in non-transitory computer-readable media such as persistent storage in secondary memory 810 or main memory 808 or both. Main memory 808 may include volatile memory such as random-access memory (RAM) or non-volatile memory such as read only memory (ROM), as well as different levels of cache memory for faster access to instructions and data. Secondary memory 810 may include persistent storage such as solid- state drives, hard disk drives or optical disks. One or more processors 804 reads program code from one or more non-transitory media and executes the code to enable the computer system to accomplish the methods performed by the embodiments herein. Those skilled in the art will understand that the processor(s) may ingest source code and interpret or compile the source code into machine code that is understandable at the hardware gate level of the processor(s) 804. The processor(s) 804 may include dedicated processors such as microcontrollers running firmware. The processor(s) 804 may include specialized processing units (e.g., GPUs) for handling computationally intensive tasks.

[0063] The processor(s) 804 may communicate with external networks via one or more communications interfaces 807, such as a network interface card, WiFi transceiver, etc. A bus 805 communicatively couples the I / O subsystem 802, the processor(s) 804, peripheral devices 806, communications interfaces 807, main memory 808, and secondary memory 810. Embodiments of the disclosure are not limited to this representative architecture. Alternative embodiments may employ different arrangements and types of components, e.g., separate buses for input-output components and memory subsystems. Elements of embodiments of the disclosure, such as one or more servers (e.g., in the cloud) communicating with computer system 800, may be implemented with at least some of the components (e.g., processor 804, memory 808, communication interfaces 807) of a computer system like that of computer system 800.

[0064] Those skilled in the art will understand that some or all of the elements of embodiments of the disclosure, and their accompanying operations, may be implemented wholly or partially by one or more computer systems including one or more processors and one or more memory systems like those of computer system 800. Some elements and functionality may be implemented locally and others may be implemented in a distributed fashion over a network through different servers, e.g., in client-server fashion, for example.

[0065] In some embodiments, an application may be an application executing in a mobile operating system (e.g., iOS from Apple, Android from Google). In some embodiments, anapplication may be a desktop application designed to run in an operating system such as macOS from Apple, Windows 10 or 11 from Microsoft, ChromeOS from Google, etc. In some embodiments, a browser may be a browser such as Safari from Apple, Chrome from Google, Edge from Microsoft, Firefox from Mozilla, etc. or a browser extension designed to run on such a browser in an operating system (e.g., macOS, Windows 10, ChromeOS, iOS, Android, etc.). Any of these executables may run on a computer system such as computer system 800. In some embodiments, a system may comprise one or more mobile applications executing on respective mobile devices and one or more server applications executing on one or more servers (e.g., in the cloud— Microsoft Azure, Amazon AWS, Google Cloud Platform, etc.).

[0066] Additional system embodiments include, at least and without limitation, the system described in embodiment 52 together with any combination of the method embodiments 2-51 converted to a corresponding system form. Additional computer-readable media embodiments include, at least and without limitation, the computer-readable media described in embodiment 53 together with any combination of the method embodiments 2-51 converted to a corresponding computer-readable media form.

[0067] Although the disclosure may not expressly disclose that some embodiments or features described herein may be combined with other embodiments or features described herein, this disclosure should be read to describe any such combinations that would be practicable by one of ordinary skill in the art. Unless otherwise indicated herein, the term "include" shall mean "include, without limitation," and the term "or" shall mean nonexclusive "or" in the manner of "and / or."

[0068] All references cited herein, including, without limitation, articles, publications, patents, patent publications, and patent applications, are incorporated by reference in their entireties for all purposes, except that any portion of any such reference is not incorporated by reference to the extent that it: (1) is inconsistent with embodiments of the disclosure expressly described herein; (2) limits the scope of any embodiments described herein; or (3) limits the scope of any terms of any embodiments recited herein. Mention of any reference cited herein is not an acknowledgment that it constitutes valid prior art, or that it discloses essential matter.

[0069] No portion of the Appendix shall be construed to: (1) limit the scope of any embodiments or inventions described in the disclosure excluding the Appendix, or (2) limit the scope of any embodiments recited in the disclosure excluding the Appendix.Embodiments:1. A method comprising: placing an arrangement of one or more markers relative to a feature on a surface, wherein each marker of the one or more markers is fixed relative to the remaining one or more markers in the arrangement when the arrangement is placed on the surface, and the arrangement is movable relative to the feature; capturing an image of the feature on a surface using a camera, wherein the image comprises information associated with at least a portion of the arrangement; and generating digital information associated with the feature based on the image, wherein the digital information comprises a digital representation of the feature.2. The method of embodiment 1, further comprising: prior to generating the digital information, solving for one or more of: a pose of the camera relative to the feature when the image is captured, a focal length of a lens of the camera, or a distortion characteristic of the lens of the camera.3. The method of embodiment 2, wherein two or more of the pose, the focal length, or the distortion characteristic of the lens are solved simultaneously.4. The method of embodiment 3, wherein the pose, the focal length, and the distortion characteristic of the lens are all solved simultaneously.5. The method of any one of embodiments 1-4, further comprising: creating a rectified image representing the feature on the surface based at least in part upon the image.6. The method of embodiment 5, wherein the rectified image illustrates the feature from a desired camera pose.7. The method of embodiment 5 or 6, wherein the rectified image compensates for a difference in height between a plane containing the arrangement and a plane defined by the surface.8. The method of any one of embodiments 5-7, wherein the rectified image does not include image data associated with the arrangement.9. The method of any one of embodiments 5-8, wherein the rectified image has an edge that is parallel to a reference line associated with the arrangement.10. The method of any one of embodiments 5-9, wherein an orientation of the feature represented in the rectified image is based on an orientation of the camera when the image is captured.The method of any one of embodiments 5-10, wherein the digital information is generated based at least in part upon the rectified image. The method of any one of embodiments 5-11, wherein the digital representation comprises the rectified image. The method of any one of embodiments 1-12, wherein the digital information comprises vector graphics data, and the vector graphics data encodes the digital representation of the feature. The method of embodiment 13, wherein the vector graphics data relates to an outline of a portion of the feature. The method of embodiment 13, wherein the vector graphics data relates to a centerline of a portion of the feature. The method of any one of embodiments 1-15, wherein the digital information comprises bitmap graphics data, and the bitmap graphics data encodes the digital representation of the feature. The method of any one of embodiments 1-16, wherein the one or more markers form a pattern, and the digital representation is based at least in part upon a calculation analyzing at least a portion of the pattern in the image. The method of any one of embodiments 1-17, wherein a first marker of the one or more markers comprises a machine-readable pattern, and the machine-readable pattern includes an identifier unique to the first marker. The method of any one of embodiments 1-18, wherein the arrangement is on a first surface of a jig- The method of embodiment 19, wherein the jig has an outer shape of a rectangle. The method of embodiment 20, wherein the jig has the outer shape of a flattened "U" with one of the rectangle's four sides missing. The method of any one of embodiments 19-21, wherein the jig has a region through which the feature is visible, and the arrangement of the one or more markers at least partially surrounds the region. The method of embodiment 22, wherein the region is a cut-out in the jig. The method of any one of embodiments 19-23, wherein the jig includes a second surface, the jig rests on the surface with the second surface contacting the surface, the first surface and the second surface are parallel and separated by a jig thickness, and the digital representation of the feature is based at least in part upon the jig thickness.The method of any one of embodiments 1-24, wherein the digital information does not include a second digital representation of the arrangement. The method of any one of embodiments 1-25, wherein the feature includes an element having a first length, the digital representation associated with the feature includes a digital element corresponding to the element, and the digital element has a dimensional attribute that matches the first length within plus or minus 10%, 8%, 5%, 3%, 2%, 1%, 0.5%, 0.3%, 0.2% or 0.1%. The method of embodiment 26, wherein the feature includes a second element having a second length, the first length is measured along a first direction, the second length is measured along a second direction, the first direction and the second direction are separated by more than 45 degrees, the digital representation associated with the feature includes a second digital element corresponding to the second element, and the second digital element has a second dimensional attribute that matches the second length within plus or minus 10%, 8%, 5%, 3%, 2%, 1%, 0.5%, 0.3%, 0.2% or 0.1%. The method of embodiment 26 or 1 , further comprising: capturing a second image of the feature on the surface using the camera, wherein the second image comprises information associated with at least a second portion of the arrangement, the arrangement is not moved relative to the feature when the image and the second image are captured, the camera is located at a first position relative to the arrangement when the image is captured, the first position is a first distance from a center of the arrangement, the camera is located at a second position relative to the arrangement when the second image is captured, the second position is a second distance from the center of the arrangement, and the second distance minus the first distance is at least 50%, 30%, 20%, or 10% of the first distance; and generating second digital information associated with the feature, wherein the second digital information comprises a second digital representation of the feature, the second digital representation associated with the feature includes a second digital element corresponding to the element, and the second digital element has a dimensional attribute that matches the first length within plus or minus 10%, 8%, 5%, 3%, 2%, 1%, 0.5%, 0.3%, 0.2% or 0.1%. The method of any one of embodiments 1-28, wherein the digital information is generated within 120 seconds, 90 second, 60 seconds, 45 seconds, 30 seconds, 20 seconds, 10 seconds, 5 seconds, 3 seconds, 1 second, 500 milliseconds, 300 milliseconds, or 100 milliseconds of capturing the image.The method of embodiment 29, wherein the digital information is generated without any input from a user after the image is captured. The method of any one of embodiments 1-30, wherein the capturing of the image and the generating of the digital information is completed by a progressive web app executing in a browser of a mobile device, and the mobile device includes the camera. The method of any one of embodiments 1-31, further comprising: capturing a third image of an identifier associated with the arrangement; and identifying a physical attribute associated with the arrangement based on the identifier. The method of embodiment 32, wherein the identifier is a QR code. The method of embodiment 32 or 33, wherein the physical attribute relates to a position or size of a first marker of the one or more markers. The method of embodiment 34, wherein the digital information is based at least in part upon the physical attribute. The method of any one of embodiments 1-35, wherein the feature lies in a first plane, the camera includes an image sensor, a first direction is defined by a normal direction relative to the image sensor, and the first direction is separated from a normal direction relative to the first plane by at least 10 degrees, 20 degrees, 30 degrees, or 40 degrees when the image is captured. The method of any one of embodiments 1-36, wherein the feature is a drawing made using a writing or drawing instrument, and the surface is a paper or wood-product surface. The method of any one of embodiments 1-37, wherein the digital representation of the feature comprises an orientation determined by the arrangement. The method of any one of embodiments 1-38, wherein the surface is a workpiece surface where the workpiece is made from wood, and the feature is associated with grains in the wood. The method of any one of embodiments 1-39, further comprising: capturing a fourth image of at least a third portion of the arrangement, wherein the digital information is generated based on the fourth image. The method of embodiment 40, wherein the digital information represents a refinement of first digital information based on the fourth image, and the first digital information is generated based on the first image. The method of embodiment 40 or 41, wherein the fourth image is used to solve for one or more of: a pose of the camera relative to the feature when the image is captured, a focal length of a lens of the camera, or a distortion characteristic of the lens of the camera.The method of any one of embodiments 1-42, wherein the digital information comprises an indication of a physical dimension of at least a portion of the feature. The method of embodiment 43, wherein the indication is a length of the portion of the feature. The method of any one embodiments of embodiments 1-44, further comprising: moving the arrangement to a different location relative to the feature on the surface, wherein the different location is different from the location at which the arrangement was located when the image was captured; capturing another image of the feature on the surface using the camera, wherein the other image comprises information associated with at least another portion of the arrangement, the other image comprises information associated with the feature that is not in the image, and the other image is captured with the arrangement located at the different location; generating another digital information associated with the feature based on the other image, wherein the other digital information comprises another digital representation of the feature; and combining the digital information and the other digital information to generate a combined digital representation of the feature, wherein the combined digital representation of the feature includes information associated with a portion of the feature included in the image and includes information associated with a portion of the feature included in the other image but not in the image. The method of embodiment 45, wherein the combined digital representation is based at least in part upon aligning a fiducial that appears in the image and the other image. The method of embodiment 46, wherein the fiducial comprises a portion of the feature. The method of embodiment 46, wherein the fiducial is not part of the feature. The method of embodiment 48, wherein the fiducial comprises a pattern on the surface. The method of embodiment 49, wherein the fiducial comprises an object placed on the surface, wherein the object is not moved relative to the surface between the capturing of the image and capturing of the other image. The method of embodiment 50, wherein the fiducial comprises a sticker applied to the surface. A system for generating a digital representation of a feature on a surface, the system comprising: one or more processors;a sensor operably coupled to at least one of the one or more processors, wherein a camera comprises the sensor; and one or more memories each operably coupled to a respective at least one of the one or more processors, wherein at least one of the one or more memories comprise instructions that, when executed by at least one of the one or more processors, cause the system to: capture an image of the feature using the sensor of the camera, wherein the image comprises information associated with at least a portion of an arrangement of markers placed relative to the feature, each marker of the one or more markers is fixed relative to the remaining one or more markers in the arrangement when the arrangement is placed on the surface, and the arrangement is movable relative to the feature; and generate digital information associated with the feature based on the image, wherein the digital information comprises a digital representation of the feature. One or more non-transitory computer-readable media storing instructions for generating a digital representation of a feature on a surface, wherein the instructions, when executed by one or more computing devices of a system, cause the system to: capture an image of the feature using a camera, wherein the image comprises information associated with at least a portion of an arrangement of markers placed relative to the feature, each marker of the one or more markers is fixed relative to the remaining one or more markers in the arrangement when the arrangement is placed on the surface, and the arrangement is movable relative to the feature; and generate digital information associated with the feature based on the image, wherein the digital information comprises a digital representation of the feature.

Claims

Claims1. A method comprising: placing an arrangement of one or more markers relative to a feature on a surface, wherein each marker of the one or more markers is fixed relative to the remaining one or more markers in the arrangement when the arrangement is placed on the surface, and the arrangement is movable relative to the feature; capturing an image of the feature on a surface using a camera, wherein the image comprises information associated with at least a portion of the arrangement; and generating digital information associated with the feature based on the image, wherein the digital information comprises a digital representation of the feature.

2. The method of claim 1, further comprising: prior to generating the digital information, solving for one or more of: a pose of the camera relative to the feature when the image is captured, a focal length of a lens of the camera, or a distortion characteristic of the lens of the camera.

3. The method of claim 1, further comprising: creating a rectified image representing the feature on the surface based at least in part upon the image.

4. The method of claim 3, wherein the rectified image compensates for a difference in height between a plane containing the arrangement and a plane defined by the surface.

5. The method of claim 3, wherein the rectified image does not include image data associated with the arrangement.

6. The method of claim 3, wherein the digital information is generated based at least in part upon the rectified image.

7. The method of claim 3, wherein the digital representation comprises the rectified image.

8. The method of claim 1, wherein the digital information comprises vector graphics data, and the vector graphics data encodes the digital representation of the feature.

9. The method of claim 1, wherein the arrangement is on a first surface of a jig.

10. The method of claim 9, wherein the jig has an outer shape of a rectangle.

11. The method of claim 9, wherein the jig has a region through which the feature is visible, and the arrangement of the one or more markers at least partially surrounds the region.

12. The method of claim 9, wherein the jig includes a second surface, the jig rests on the surface with the second surface contacting the surface, the first surface and the second surface areparallel and separated by a jig thickness, and the digital representation of the feature is based at least in part upon the jig thickness.

13. The method of claim 1, wherein the digital information does not include a second digital representation of the arrangement.

14. The method of claim 1, wherein the feature includes an element having a first length, the digital representation associated with the feature includes a digital element corresponding to the element, and the digital element has a dimensional attribute that matches the first length within plus or minus 10%, 8%, 5%, 3%, 2%, 1%, 0.5%, 0.3%, 0.2% or 0.1%.

15. The method of claim 14, wherein the feature includes a second element having a second length, the first length is measured along a first direction, the second length is measured along a second direction, the first direction and the second direction are separated by more than 45 degrees, the digital representation associated with the feature includes a second digital element corresponding to the second element, and the second digital element has a second dimensional attribute that matches the second length within plus or minus 10%, 8%, 5%, 3%, 2%, 1%, 0.5%, 0.3%, 0.2% or 0.1%.

16. The method of claim 1, wherein the digital information is generated within 120 seconds, 90 second, 60 seconds, 45 seconds, 30 seconds, 20 seconds, 10 seconds, 5 seconds, 3 seconds, 1 second, 500 milliseconds, 300 milliseconds, or 100 milliseconds of capturing the image.

17. The method of claim 1, wherein the feature lies in a first plane, the camera includes an image sensor, a first direction is defined by a normal direction relative to the image sensor, and the first direction is separated from a normal direction relative to the first plane by at least 10 degrees, 20 degrees, 30 degrees, or 40 degrees when the image is captured.

18. The method of claim 1, wherein the digital information comprises an indication of a physical dimension of at least a portion of the feature.

19. A system for generating a digital representation of a feature on a surface, the system comprising: one or more processors; a sensor operably coupled to at least one of the one or more processors, wherein a camera comprises the sensor; and one or more memories each operably coupled to a respective at least one of the one or more processors, wherein at least one of the one or more memories comprise instructions that, when executed by at least one of the one or more processors, cause the system to:capture an image of the feature using the sensor of the camera, wherein the image comprises information associated with at least a portion of an arrangement of markers placed relative to the feature, each marker of the one or more markers is fixed relative to the remaining one or more markers in the arrangement when the arrangement is placed on the surface, and the arrangement is movable relative to the feature; and generate digital information associated with the feature based on the image, wherein the digital information comprises a digital representation of the feature.

20. One or more non-transitory computer-readable media storing instructions for generating a digital representation of a feature on a surface, wherein the instructions, when executed by one or more computing devices of a system, cause the system to: capture an image of the feature using a camera, wherein the image comprises information associated with at least a portion of an arrangement of markers placed relative to the feature, each marker of the one or more markers is fixed relative to the remaining one or more markers in the arrangement when the arrangement is placed on the surface, and the arrangement is movable relative to the feature; and generate digital information associated with the feature based on the image, wherein the digital information comprises a digital representation of the feature.

Images