Dynamic 3D Surface Sketching

Abstract

A method and system for three-dimensional (3D) surface sketching are provided. A 3D scanner scans an exterior surface of a physical object and outputs data representing the exterior surface. A processor generates a 3D model of the object based on the received data and outputs a 3D rendering of the object. A display displays the 3D rendering of the object. An input device physically traces a portion of the exterior surface of the object, and a tracking device tracks positioning of the input device as the input device physically traces the portion of the exterior surface of the object. The processor receives data representing at least one spatial position of the input device, augments the 3D rendering of the object based at least in part on the data, and outputs the augmented 3D rendering to the display.

Description

Technical Field

[0001] This application is directed to generating a three-dimensional (3D) rendering of a physical object, physically tracing an input device on the physical object to annotate the 3D rendering, and refining the 3D rendering. This application is also directed to distance measurement of curves traced by an input device.

Background Art

[0002] In many industries, including the automotive industry, physical models such as clay models are used to model automotive designs and physically illustrate the design features of automobiles. Refining and expanding physical models is an important task in designing cars as well as other industrial or consumer products. During the industrial design process, designers and 3D modelers shape physical models with tools and tape-mark changes to the physical models. However, physically shaping a physical model takes time and it is often not easy to undo changes made to the model because it may be necessary to patch the physical model to undo the changes.

[0003] Accordingly, there is a need for methods and apparatus for rendering a 3D model of a physical object and expanding the 3D model by sketching on the 3D model to digitally or virtually display the expanded 3D model.

Summary of the Invention

Means for Solving the Problems

[0004] In one embodiment, the system includes a three-dimensional (3D) scanner configured to scan an exterior surface of a physical object and output data representative of the exterior surface of the object. In one embodiment, the system includes a processor configured to receive the data representative of the exterior surface of the object, generate a 3D model of the object based on the received data, and output a 3D rendering of the object based on the generated 3D model. In one embodiment, the system includes a display configured to receive the 3D rendering of the object and display the 3D rendering of the object. The system includes an input device operable to physically trace a portion of the exterior surface of the object, and a tracking device configured to track positioning of the input device as the input device physically traces at least a portion of the exterior surface of the object, and output data representative of at least one spatial position of the input device as the input device traces the object. The processor is configured to receive data representative of the at least one spatial position of the input device, augment the 3D rendering of the object based at least in part on the data representative of the at least one spatial position of the input device, and output the augmented 3D rendering of the object to the display in response to augmenting the 3D rendering of the object. In one embodiment, the display is configured to display the augmented 3D rendering of the object.

[0005] In one embodiment, the processor is configured to augment the 3D rendering of the object by at least identifying one or more curves having one or more respective spatial positions relative to an exterior surface of the object based on data representing at least one spatial position of the input device, and superimposing the one or more curves on the 3D rendering of the object at one or more rendering positions corresponding to the one or more respective spatial positions relative to the exterior surface of the object.

[0006] In one embodiment, the input device is pressure sensitive and configured to sense pressure applied to the input device as the input device physically traces at least a portion of the exterior surface of the object and to output data representative of the pressure. The processor is configured to determine one or more widths for each of the one or more curves based at least in part on the pressure applied to the input device as the input device physically traces at least a portion of the exterior surface of the object to form the one or more curves and to superimpose the one or more curves, each having the one or more widths, on a 3D rendering of the object.

[0007] In one embodiment, the input device includes a pressure-sensitive tip operable to sense pressure applied to the input device as the input device physically traces at least a portion of an exterior surface of the object. In one embodiment, the input device includes a first control input operable to receive one or more respective width indications of the one or more curves. The input device is configured to output data representative of the one or more respective width indications to a processor, the processor configured to receive the data representative of the one or more respective width indications, determine one or more widths for each of the one or more curves based on the data representative of the one or more respective width indications, and superimpose the one or more curves each having the one or more widths on a 3D rendering of the object. In one embodiment, the display is a head-mounted display configured to display a 3D rendering of the object superimposed on a physical object otherwise visually viewable through the head-mounted display.

[0008] In one embodiment, the system includes a three-dimensional (3D) scanner configured to scan an exterior surface of a physical object and output data representative of the exterior surface of the object. The system includes a processor configured to receive the data representative of the exterior surface of the object, generate a 3D model of the object based on the received data, and output a 3D rendering of the object based on the generated 3D model. In one embodiment, the system includes a display configured to receive the 3D rendering of the object and display the 3D rendering of the object, and an input device operable to physically trace at least a portion of the exterior surface of the object. The system includes a tracking device configured to track positioning of the input device as the input device traces at least a portion of the exterior surface of the object, and output data representative of at least one position of the input device in 3D space as the input device traces the exterior surface of the object. The processor is configured to receive the data representative of the at least one position of the input device, modify the 3D model of the object based at least in part on the data representative of the at least one position of the input device, generate an updated 3D rendering of the object based on the modified 3D model, and output the updated 3D rendering of the object to the display in response to generating the updated 3D rendering of the object. In one embodiment, the display is configured to display an updated 3D rendering of the object.

[0009] In one embodiment, the processor is configured to generate a 3D model of the object by generating a polygon mesh including a plurality of vertices and a plurality of edges. In one embodiment, the processor is configured to modify the 3D model of the object by changing at least one vertex of the plurality of vertices or one edge of the plurality of edges to correspond to at least one position of the input device in 3D space. In one embodiment, the processor is configured to modify the 3D model of the object by adding at least a first vertex to the plurality of vertices having an in-space location corresponding to at least one position of the input device in 3D space. In one embodiment, the processor is configured to modify the 3D model of the object by removing at least a second vertex from the plurality of vertices having a location in 3D space closest to the location of the first vertex. In one embodiment, the display is a head mounted display configured to display a 3D rendering of the object superimposed on a physical object otherwise visually viewable through the head mounted display, and further configured to display an updated 3D rendering of the object superimposed on the physical object otherwise visually viewable through the head mounted display.

[0010] In one embodiment, the system includes a three-dimensional (3D) scanner configured to scan an exterior surface of a physical object and output data representative of the exterior surface of the object. In one embodiment, the system includes a processor configured to receive data representative of the exterior surface of the object, generate a 3D model of the object based on the received data, and output a 3D rendering of the object based on the generated 3D model. In one embodiment, the system includes a display configured to receive the 3D rendering of the object and display the 3D rendering of the object. The system includes an input device operable to physically trace at least a portion of the exterior surface of the object, and a tracking device configured to track positioning of the input device as the input device traces at least a portion of the exterior surface of the object and output data representative of at least two positions of the input device as the input device traces the object. The processor is configured to receive the data representative of the at least two positions, determine a distance between the at least two positions, and output data representative of the distance.

[0011] The processor is configured to identify a curve and determine a distance between the at least two positions along the identified curve based on data representing a position of the input device between the at least two positions. The display is configured to receive data representing the distance and display the distance on the display. The input device includes a control input operable to receive a selection of a first operating mode of a plurality of operating modes of the input device and to output data indicative of the first operating mode.

[0012] In one embodiment, the processor is configured to receive data indicative of a first mode of operation and, in response to receiving the data indicative of the first mode of operation, determine a distance between at least two positions and output data indicative of the distance. In one embodiment, the input device receives a selection of a second mode of operation of the input device via the control input and outputs data indicative of the second mode of operation. The processor is configured to receive data indicative of the second mode of operation and, in response to receiving the data indicative of the second mode of operation, augment a 3D rendering of the object based on positioning information received from a tracking device tracking the input device as the input device traces at least a portion of an outer surface of the object. The processor is configured to receive data indicative of the second mode of operation and, in response to receiving the data indicative of the second mode of operation, modify a 3D model of the object based on positioning information received from a tracking device tracking the input device as the input device traces at least a portion of an outer surface of the object, and generate an updated 3D rendering of the object based on the modified 3D model. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 shows a three-dimensional (3D) scanner scanning a physical object. [Diagram 2] FIG. 2 shows a 3D rendering system. [Diagram 3] FIG. 3 illustrates an input device according to one embodiment of the present disclosure. [Figure 4] FIG. 4 illustrates a flow diagram of a method for enhancing 3D rendering of an object. [Diagram 5] FIG. 5 illustrates a flow diagram of a method for modifying a 3D rendering of an object based on the position of an input device. [Figure 6] FIG. 6 illustrates a flow diagram of a method for distance measurement based on the position of an input device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] FIG. 1 shows a three-dimensional (3D) scanner 102 scanning a physical object 101. The 3D scanner 102 may be any device configured to scan the physical object 101 or the outer surface of the physical object 101 to generate a three-dimensional model of the physical object 101. The 3D scanner 102 may be a non-contact scanner or a contact scanner. Furthermore, the 3D scanner 102 may be an active scanner or a non-active scanner. The 3D scanner 102 may use any technique for scanning an object, such as time-of-flight (ToF) or triangulation.

[0015] The 3D scanner 102 may be a ToF 3D laser scanner. The 3D scanner 102 may be an active scanner that uses laser light to probe the physical object 101. The 3D scanner 102 may be a volumetric scanner. The 3D scanner 102 may include a ToF laser range finder. The laser range finder may determine a distance between the 3D scanner 102 and the surface of the physical object 101 based on a timing of a round trip time of a pulse of light emitted by the 3D scanner 102. The 3D scanner 102 emits a laser pulse, detects a reflection of the laser pulse reflected by the surface of the physical object 101, and determines the time taken between the time the laser pulse is emitted and the time the reflection of the laser pulse is detected (round trip time). The 3D scanner 102 determines the distance between the 3D scanner 102 and the surface of the physical object 101 based on the determined time and the speed of light.

[0016] The 3D scanner 102 may emit laser pulses in a direction to scan the physical object 101. The 3D scanner 102 accordingly scans the physical object 101 from multiple viewpoints. The ToF laser range finder may scan the entire field of view one point at a time, or may change the viewing direction of the ToF laser range finder to scan different points on the outer surface of the object 101. The viewing direction may be changed either by rotating the range finder or by using a rotating mirror system, among others.

[0017] 2 illustrates a 3D rendering system 106. The system 106 includes a 3D scanner 102, a 3D rendering device 108 (shown in block diagram form), a display 110 (shown pictorially, e.g., as a head mounted display), an input device 112, and a tracking device 113 for the input device 112. The 3D rendering device 108 includes a processor 114, a memory 116, and one or more communication devices 118. The memory 116 and the one or more communication devices 118 are communicatively coupled to the processor 114. The 3D rendering device 108 is communicatively coupled to the 3D scanner 102, the display 110, the input device 112, and the tracking device 113.

[0018] The processor 114 may be any type of computing device configured to perform the operations described herein. The processor 114 may be a graphics processing unit (GPU) or a central processing unit (CPU), among others. The processor 114 may also be a controller, a microcontroller, or a microprocessor, among others. The memory 116 may be any type of storage device configured to store data. The data may be graphical data (such as a 3D rendering of the surface of the physical object 101), or the data may be executable instructions that, when executed by the processor 114, cause the processor to perform the operations described herein.

[0019] The one or more communication devices 118 may be any type of communication device configured to exchange or communicate data with other communication devices. The communication device 118 may be a wireless communication device or a wired device, and may be a modem or a transceiver, among others. The communication device 118 may receive data from or transmit data to another communication device. Although not shown in FIG. 2, the other communication device may be part of the 3D scanner 102, the display 110, the input device 112, and / or the tracking device 113. The one or more communication devices, which may include one or more communication devices, may communicate using any type of protocol associated with each communication device. The protocol may be cellular communication, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 protocol, Bluetooth protocol, Universal Serial Bus (USB) protocol, or 3rd Generation Partnership Project (3GPP®) Long Term Evolution (LTE) protocol, among others.

[0020] It should be noted that the 3D rendering device 108 may be a computer, a tablet, or a smartphone, among others. The 3D rendering device 108 may be separate from the display 110 or the tracking device 102. However, in alternative embodiments, the 3D rendering device 108 may be part of the display 110 or the tracking device 102, and operations performed by the 3D rendering device 108 may instead be performed by the display 110 and a processor, memory, or one or more communication devices of the 3D rendering device 108.

[0021] The 3D rendering device 108 receives, via one or more communication devices 118, signals carrying data representing the scanned physical object 101. The signals may be modulated and encoded according to the respective modulation and encoding of a communication protocol used by the one or more communication devices 118.

[0022] The one or more communication devices 118 demodulate and encode the signal and output data representing the scanned physical object 101 to the processor 114. The processor 114 evaluates the data representing the scanned physical object 101. The processor 114 generates a 3D model of the physical object 101 based on the data representing the physical object 101. The 3D model of the physical object 101 may include a polygon mesh including vertices and edges. The polygon mesh may also include surfaces. Each surface may be between three or more respective edges of the edges. A vertex of the vertices has a spatial location that corresponds to a spatial location of a point on an exterior surface of the physical object 101. The vertices, edges, and surfaces virtually (and digitally) represent the scanned physical object 101. The processor 114 stores the 3D model of the physical object 101 in the memory 116. The processor 114 outputs the 3D model of the physical object 101 to the display 110 via one or more communication devices 118.

[0023] The display 110 may be a head mounted display (HMD). As a head mounted display, the display 110 may be a virtual reality display or an augmented reality display. As an augmented reality display, the display 110 may be transmissive or semi-transmissive. Thus, a viewer viewing the physical object 101 through the display 110 sees the physical object 101 due to the transmissive properties of the display 110. Because the physical object 101 is transmissively viewable through the display 110, the display 110 may overlay a 3D rendering of the physical object 101 on the physical object 101 using a 3D model of the object. Thus, in such an embodiment, the viewer sees a 3D rendering of the physical object 101 overlaid on the physical object 101.

[0024] A viewer or user may use the input device 112 to annotate, enhance, refine, or modify (collectively, "enhance") the 3D rendering of a physical object. A user may use the input device 112 to enhance the 3D rendering of a physical object, typically by drawing one or more curves, or any other shape, on the 3D rendering. In this regard, the user may trace the input device or a tip of the input device in three-dimensional space on at least a portion of the physical object 101. The tracking device 113 tracks the position of the input device 112 in three-dimensional space and outputs data representative of the position to the 3D rendering device 108. The 3D rendering device 108 receives the data representative of the position of the input device 112 and generates an augmented 3D rendering of the physical object based on the data representative of the tracked position of the input device 112. As will be appreciated from the description herein, the augmented 3D rendering of a physical object may include designs and features that appear virtually on or relative to a surface of the physical object, but do not otherwise appear in the actual three-dimensional space of the physical object.

[0025] FIG. 3 illustrates an example of an input device 112 according to one embodiment. The input device 112 includes a housing 119, a tip 120, a marker 122, and a number of control inputs 124a, 124b, 124c. The tip 120 may be pressure sensitive. The marker 122 may be located on the tip 120 of the input device 112. In other embodiments, the marker 122 may be located elsewhere on the input device 112. The marker 122 may be a passive marker or an active marker used to track and determine the position of the tip 120. For example, the marker 122 may have a reflective coating that reflects light. Alternatively, or in addition, the marker 122 may be a light emitting diode (LED) that actively illuminates a light for tracking the tip 120 of the input device 112. In various embodiments, the marker 122 may be a strobe light that emits light having a specified wavelength or signature. In various embodiments, the input device 112 may be markerless, such that the position of the tip 120 or other portion of the input device may be tracked based on the shape or other characteristics of the input device 112 .

[0026] 2, the tracking device 113 tracks the spatial positions of the input device 112 or markers 122 of the input device 112 as the input device 112 moves through three-dimensional space. The tracking device 113 determines the spatial positions of the markers 122 and outputs data representing the positions to the 3D rendering device 108. In at least one embodiment, the tracking device 113 may include one or more cameras, such as motion capture cameras, that capture images of the markers 122 and may determine the positions of the markers, and thus the tip 120 and input device 112, based on the captured images.

[0027] The tracking device 113 may include a communication device (not shown). The tracking device 113 sends a signal including data representing a spatial position of the input device 112 via the communication device. The 3D rendering device 108 receives the signal via one or more communication devices 118 and outputs the data representing the spatial position to the processor 114. The processor 114 determines a position of the input device 112 or the marker 122 based on the received position data. The processor 114 then augments a 3D rendering of the physical object based on the received position data.

[0028] For example, a user may physically trace the outer surface of the physical object 101 with the input device 114 or the tip 120 of the input device 114 to draw a line, or generally a curve. In this way, the input device 114 may be used to sketch (or map) a 3D rendering of the physical object. As the user traces the outer surface of the physical object 101, the tracking device 113 tracks the spatial position of the tip 122 and outputs data representing the position to the 3D rendering device 108. The 3D rendering device 108 augments the 3D rendering of the physical object by adding a corresponding curve to the 3D rendering of the physical object. The curve may be a collection of points that are connected to each other and have a position in space that corresponds to the position of the tip detected by the tracking device 113. The 3D rendering device 108 superimposes the curve onto the 3D rendering of the physical object. The 3D rendering device 108 then generates an augmented 3D rendering of the physical object. The augmented 3D rendering includes a 3D rendering of the (previously generated) physical object with the curve superimposed.

[0029] The 3D rendering device 108 outputs data representing an augmented 3D rendering of the physical object to the display 110. The display 110 displays the augmented 3D rendering of the physical object. Note that detecting the spatial position of the input device 112, generating the augmented 3D rendering, and outputting the data representing the augmented 3D rendering to the display 110 may be performed in real time. Thus, a user viewing the display 110 sees a curve in the augmented 3D rendering in real time and as the user "draws" using the input device 112 (or as the user uses the input device 112 to trace the outer surface of the physical object 101). Note that the term "curve" is used herein to represent any general shape drawn by the user using the input device 112. The curve may be, for example, a straight line or any other shape.

[0030] In one embodiment, the tip 120 of the input device 112 may be pressure sensitive. The input device 112 may sense pressure applied to the tip by the user as the user manipulates the input device 112. The pressure may be used to determine the thickness of the curve drawn by the user. The input device 112 may output data representative of the pressure applied to the tip 120. The input device 112 may output the pressure data to the 3D rendering device 108. As described herein, the input device 112 may include a communication device (not shown) operable to communicate with one or more communication devices 118 of the 3D rendering device 108 and operable to output a signal including data representative of the pressure applied to the tip 120. The one or more communication devices 118 of the 3D rendering device 108 may receive the signal and output data representative of the pressure to the processor 114. The processor 114 determines the pressure based on the received pressure data. The processor 114 renders the curve with a line thickness corresponding to the determined pressure. The relationship between pressure and thickness may be proportional, such that a greater amount of pressure applied by the user results in rendering a thicker curve.

[0031] The processor 114 may evaluate the determined pressure along with the position of the tip 120. The processor 114 generates a curve based on both the pressure data and the position data to be superimposed on a 3D rendering of the physical object, with the thickness of the curve at a location in space corresponding to the determined pressure applied to the tip 120 at that location in space.

[0032] A number of control inputs 124a-c of the input device 112 may be used to control attributes of the curve. For example, a first control input 124a may be used to select between operational modes of the input device 112. A first operational mode may be an extension of the 3D rendering as described herein, in which one or more additional curves are superimposed on the 3D rendering. A second operational mode may be a modification of the 3D rendering, and a third operational mode may be a distance measurement as described herein. A user may operate the first control input 124a, which may be a multi-pole or multi-directional switch, to select an operational mode from various available operational modes.

[0033] Similarly, the second and third control inputs 124b, 124c may be used to select attributes of the curve, such as the color, type, or thickness of the lines that make up the curve. In one embodiment, the second control input 124b may be used to select the color of the curve, such as red, green, or blue, among others, and / or the type of curve, such as solid or dashed, among others. In one embodiment, the third control input 124c may be used to select a static or constant thickness of the curve. The thickness selected using the third control input 124c may override or take precedence over the thickness determined based on the pressure applied to the tip 120. In one embodiment, the control inputs may be user configurable. For example, a user may specify a control input function associated with each of the control inputs 124a-c that is different from the default control input function of the input device 112.

[0034] It should be noted that the input device 112 of FIG. 3 is exemplary and non-limiting. In various embodiments, any other type of input device 112 may be used. The input device 112 may have a different form factor than that illustrated in FIG. 3. In one embodiment, the input device may be a joystick, a touchpad, a pressure sensitive pad, or a wheel, among others. Additionally, the input device 112 may have more or less control inputs than those illustrated in FIG. 3.

[0035] The input device 112 outputs data representing the selected motion mode and / or attributes of the curve to the 3D rendering device 108. The 3D rendering device 108 receives the data representing the selected motion mode and / or attributes of the curve and uses this data, along with the data representing the position of the tip 120, to generate an augmented 3D rendering of the physical object. For example, the 3D rendering device 108 may color the curve or render the curve to have a thickness according to the received attributes.

[0036] In addition to or instead of enhancing the 3D rendering of the physics object 101, the 3D rendering device 108 may refine or modify the 3D rendering of the physics object 101 based on user input provided using the input device 112. To refine or modify the 3D rendering of the physics object (and improve the accuracy of the 3D rendering), a user may use the input device to trace the outer surface of the physics object 101. For example, the user may trace the physics object 101 to provide a precise location of the tip 120 at or near the outer surface of the physics object 101. The location of the tip 120 is then used to modify the 3D rendering of the physics object 101 to improve the accuracy of the 3D rendering of the physics object 101.

[0037] As the user utilizes the input device 112 to trace the exterior surface of the physical object 101, the tracking device 113 tracks the position of the tip. The tracking device 113 outputs data representing the spatial position of the tip 120 to the 3D rendering device 108. The position may be a position in space represented in a Cartesian coordinate system of a three-dimensional space as three coordinates (e.g., (x, y, z)) or a polar coordinate system as three coordinates relative to a reference point (or origin) (e.g., radial distance, polar angle, and azimuth angle). The position tracking of the input device 112 may have a more precise spatial resolution than the 3D scanner 102 that is otherwise used to generate a three-dimensional model of the physical object, as described above with respect to Figures 1 and 2. The 3D rendering device 108 receives the data representing the tracked position of the tip 120 of the input device 112 and uses the tracked position data to adjust or modify the 3D model that provides a 3D rendering of the physical object.

[0038] As described herein, the 3D rendering of the physical object may include a plurality of vertices, with each pair of vertices connected by one of the edges. The 3D rendering device 108 may set the position of the tip 120 received from the tracking device 113 as one of the vertices. Thus, the 3D rendering of the physical object is adjusted based on the data position received from the tracking device 113. Additionally, the 3D rendering device 108 may remove an existing vertex of the 3D rendering and replace the removed vertex with a vertex of the received position of the input device 112. The removed vertex may be a vertex having a position in Euclidean space that is closest to the received position of the input device 112. The 3D rendering device 108 may remove the vertex and replace it with a new vertex having a position that corresponds to (or is identical to) the spatial position of the tip 120 received from the tracking device 113. In this manner, the 3D rendering device 108 uses the tracked position data of the input device 112 to iteratively improve the 3D rendering of the physics object as the input device 112 traces portions of the surface of the physics object. Based on the adjustments made to the 3D model of the physics object, the 3D rendering device 108 generates an updated 3D rendering of the physics object 101 and outputs data representing the updated 3D rendering to the display 110.

[0039] Thus, the 3D rendering device 108 first generates a 3D model of the physical object 101 based on data representing the scanned physical object 101 output by the 3D scanner 102. The 3D rendering device 108 then refines the 3D model based on data representing the position of the input device 112, or the tip 120 of the input device 112, as the input device 112 traces portions of the surface of the physical object. Thus, the 3D rendering device 108 progressively improves the 3D rendering of the physical object.

[0040] In one embodiment, the system 106 may be used to measure a distance in space. The distance, which may be a Euclidean distance, may be anywhere in the space. The distance may be, for example, between two points on the exterior surface of the physical object 101. To measure the distance, a user may place the tip 120 of the input device 112 in a first location and move the tip 120 along the surface of the physical object to a second point different from the first point.

[0041] When the tip 120 is at the first point, the tracking device 113 determines a first spatial position of the tip and outputs the first position data to the 3D rendering device 108. The 3D rendering device 108 stores the first position data in the memory 116. The user then moves the tip 120 of the input device 112 along the surface of the physical object to a second point. The tracking device 113 determines a second position associated with the second spatial position. The tracking device 103 outputs the second position to the 3D rendering device 108. Upon receiving the first and second positions, the 3D rendering device 108 determines the Euclidean distance between the first and second positions. The 3D rendering device 108 then outputs data indicative of the distance to the display 110 for display to the user or to any other device that outputs the distance to the user.

[0042] It should be noted that in various embodiments, the distance may be a linear distance between two points, such as a first and second point. Additionally or alternatively, the distance may be the length of an arc or curve traced by the tip 120 of the input device 112. As the user traces the curve, the tracking device 113 determines the spatial position of the tip 120 in real time and outputs data representing the position to the 3D rendering device 108. It is recognized that it may be advantageous for the user to slowly trace the curve or arc, allowing the tracking device 113 to determine various positions of the tip 120 at small distance increments relative to each other and with greater granularity. Identifying the displacement of the tip 120 at smaller increments leads to improved accuracy in determining the length of the curve.

[0043] It should be noted that in various embodiments, the tracking device 113 may be part of the 3D scanner 102, or the tracking device 113 may not be required and the 3D scanner 102 may perform the tracking functions performed by the tracking device 113. Thus, the 3D scanner 102 may track the spatial position of the input device 112 and output data representative of the tracked position to the 3D rendering device 108. The tracking device 113 may be an outside-in tracking device, where a camera or other sensor at a fixed location and pointed towards the input device 112 tracks the movement of the input device as it moves within the viewing range of the camera or other sensor. Furthermore, the tracking device 113 may be part of or included in the head mounted display or the 3D rendering device 108. Alternatively or in addition, the display 110 may include inside-out tracking, whereby the display 110 may include a camera that "watches" or observes the external surrounding environment or space to determine the position of the display 110 or the input device 112 relative to the environment or space.

[0044] Figure 4 shows a flow diagram of a method 400 for enhancing 3D rendering of an object. In the method 400, at 402, a 3D scanner, such as the 3D scanner 102 described with reference to Figure 1, scans an exterior surface of a physical object. At 404, a 3D rendering device, such as the 3D rendering device 108 described with reference to Figure 2, generates a 3D model of the object based on data resulting from scanning the exterior surface at 402. The 3D model of the object may include vertices, edges, and surfaces determined from the 3D scan of the exterior surface of the object.

[0045] At 406, a display, such as the display 110 described with reference to FIG. 2, displays a 3D rendering of the physical object based on the generated 3D model. The display may be a virtual reality (VR) or augmented reality (AR) display. The physical object may be transparently viewable through the display. The display may superimpose the 3D rendering on the physical object, which is otherwise viewable through the display. At 408, a tracking device, such as the tracking device 113 described with reference to FIG. 2, tracks the positioning of the input device as the input device physically traces at least a portion of the outer surface of the object. At 410, the tracking device determines at least one spatial position of the input device as the input device traces the outer surface of the object.

[0046] At 412, a 3D rendering device augments a 3D rendering of the object based at least in part on one or more tracked positions of the input device. A user may physically trace the input device over a portion of an exterior surface of a physical object to draw a curve or any shape. The tracking device tracks the input device as the user physically traces the input device over the exterior surface of the physical object. Data representing the spatial position of the input device is provided to a 3D rendering device, which uses this data to determine the shape of the curve as well as the position of the curve relative to the 3D rendering of the object. The 3D rendering device augments the 3D rendering to include a rendering of the curve. At 414, a display displays the augmented 3D rendering of the object.

[0047] FIG. 5 shows a flowchart of a method 500 for modifying a 3D rendering of an object based on a tracked position of an input device. Steps 502, 504, 506, 508, and 510 of method 500 are similar to steps 402, 404, 406, 408, and 410 of method 400 described with reference to FIG. 4. Method 500 includes, at 502, scanning an outer surface of a physical object, at 504, generating a 3D model of the object based on the scan of the outer surface, and at 506, displaying a 3D rendering of the object based on the generated 3D model. Method 500 also includes, at 508, tracking a pose of the input device as the input device physically traces at least a portion of the outer surface of the object, and at 510, identifying at least one spatial position of the input device as the input device traces the outer surface of the object.

[0048] To provide more accurate physical coordinates of the surface of the physical object, the user may physically trace the input device on the surface of the physical object. By tracing the input device on the surface of the physical object or orienting the input device on this surface while tracking the input device, the user can efficiently provide the pose (or coordinates) of the surface. Using more accurate data reflecting the pose of the surface, the 3D rendering of the physical object can be modified and improved (e.g., if the 3D scan of the object is inaccurate).

[0049] Thus, in contrast to augmenting the 3D rendering, the method 500 proceeds to modify, at 512, by the 3D rendering device, a 3D model of the object based at least in part on one or more tracked positions of the input device. The tracked spatial position of the input device is used to refine or improve the accuracy of the 3D model of the object, rather than augmenting or appending to the 3D rendering. As described herein, the position of the input device is included as a vertex in the modified 3D model of the object. After modifying the 3D model of the object, the display displays, at 514, an updated 3D rendering of the object based on the modified 3D model of the object.

[0050] 6 shows a flow diagram of a method 600 of distance measurement based on one or more tracked positions of an input device. In the method 600, a tracking device tracks a spatial positioning of an input device as the input device traces at least a portion of an outer surface of a physical object at 602. A user may trace the outer surface of the physical object to measure a distance between two points or positions along the outer surface of the physical object. At 604, the tracking device determines at least two positions of the input device as the input device traces the object.

[0051] At 606, the 3D rendering device determines a distance between the at least two locations. The distance may be a Euclidean distance between the at least two locations. The distance may be a linear distance along a straight line or a curve traversed by the input device. The curve traversed by the input device may be approximated by a number of short line segments extending between multiple sensed locations of the input device as the input device traverses the curve. The distance along the curve may be determined by summing the individual distances of the short line segments. At 608, the 3D rendering device outputs data representing the distances, which may be displayed on a display.

[0052] The various embodiments described above can be combined to provide further embodiments. These and other changes can be made to the embodiments in light of the above detailed description. In general, in the following claims, the terms used should not be construed to limit the claims to the specific embodiments disclosed in the specification and claims, but should be construed to include all possible embodiments with the full scope of equivalents to which such claims are entitled. Thus, the claims are not limited by this disclosure.

Claims

1. 1. A system comprising: a three-dimensional (3D) scanner configured to scan an exterior surface of a physical object and to output data representative of the exterior surface of the object; a processor configured to receive the data representing the exterior surface of the object, generate a 3D model of the object based on the received data, and output a 3D rendering of the object based on the generated 3D model; a display configured to receive the 3D rendering of the object and to display the 3D rendering of the object; an input device operable to physically trace at least a portion of the exterior surface of the object; a tracking device configured to track positioning of the input device as the input device physically traces the at least a portion of the outer surface of the object, and to output data representative of at least one spatial position of the input device as the input device traces the object; the processor is configured to receive the data representing the at least one spatial position of the input device, augment the 3D rendering of the object based at least in part on the data representing the at least one spatial position of the input device, and in response to augmenting the 3D rendering of the object, output the augmented 3D rendering of the object to the display; the display is configured to display the augmented 3D rendering of the object. system.

2. The processor includes at least identifying one or more curves having one or more respective spatial positions relative to the exterior surface of the object based on the data representing the at least one spatial position of the input device; superimposing the one or more curves onto the 3D rendering of the object at one or more rendering positions each corresponding to the one or more spatial positions relative to the exterior surface of the object; and configured to augment the 3D rendering of the object by The system of claim 1 .

3. the input device is pressure sensitive and configured to sense pressure applied to the input device as the input device physically traces the at least a portion of the exterior surface of the object, and to output data representative of the pressure; The processor, determining one or more widths of each of the one or more curves based at least in part on pressure applied to the input device as the input device physically traces the at least a portion of the exterior surface of the object to form the one or more curves; and configured to superimpose the one or more curves, each having the one or more widths, onto the 3D rendering of the object. The system of claim 2.

4. the input device includes a pressure sensitive tip operable to sense the pressure applied to the input device as the input device physically traces the at least a portion of the exterior surface of the object. The system of claim 3.

5. the input device includes a first control input operable to receive a width indication for one or more of the respective ones of the one or more curves; the input device is configured to output data representative of one or more respective width indications to the processor; The processor, receiving data representative of each of the one or more width indications; determining a width of each of the one or more curves based on data representing each of the one or more width indications; and configured to superimpose the one or more curves having the determined widths onto the 3D rendering of the object. The system of claim 2.

6. the display is a head-mounted display, the head mounted display is configured to display the 3D rendering of the object superimposed on a physical entity of the object, while when the 3D rendering of the object is not displayed, the physical entity of the object is visually visible through the head mounted display. The system of claim 1 .

7. 1. A system comprising: a three-dimensional (3D) scanner configured to scan an exterior surface of a physical object and to output data representative of the exterior surface of the object; a processor configured to receive the data representing the exterior surface of the object, generate a 3D model of the object based on the received data, and output a 3D rendering of the object based on the generated 3D model; a display configured to receive the 3D rendering of the object and to display the 3D rendering of the object; an input device operable to physically trace at least a portion of the exterior surface of the object; a tracking device configured to track positioning of the input device as the input device traces the at least a portion of the outer surface of the object, and to output data representative of at least a position of the input device in 3D space as the input device traces the outer surface of the object; the processor is configured to receive the data representing the at least one position of the input device, modify the 3D model of the object based at least in part on the data representing the at least one position of the input device, generate an updated 3D rendering of the object based on the modified 3D model, and in response to generating the updated 3D rendering of the object, output the updated 3D rendering of the object to the display; the display is configured to display the updated 3D rendering of the object. system.

8. the processor is configured to generate the 3D model of the object by generating a polygon mesh including a plurality of vertices and a plurality of edges. The system of claim 7.

9. the processor is configured to modify the 3D model of the object by changing at least one vertex of the plurality of vertices or one edge of the plurality of edges to correspond to the at least one position of the input device in 3D space. The system of claim 8.

10. the processor is configured to modify the 3D model of the object by adding to the plurality of vertices at least a first vertex having an in-space location corresponding to the at least one location of the input device in 3D space. The system of claim 8.

11. the processor is configured to modify the 3D model of the object by removing from the plurality of vertices at least a second vertex having a position in 3D space closest to the position of the first vertex. The system of claim 10.

12. the display is a head-mounted display, the head mounted display is configured to display the 3D rendering of the object superimposed on a real object of the object, while in a case where the 3D rendering of the object is not displayed, the real object of the object is visually visible through the head mounted display, and further configured to display the updated 3D rendering of the object superimposed on the real object of the object, while in a case where the updated 3D rendering of the object is not displayed, the real object of the object is visually visible through the head mounted display. The system of claim 7.

13. 1. A system comprising: a three-dimensional (3D) scanner configured to scan an exterior surface of a physical object and to output data representative of the exterior surface of the object; a processor configured to receive the data representing the exterior surface of the object, generate a 3D model of the object based on the received data, and output a 3D rendering of the object based on the generated 3D model; a display configured to receive the 3D rendering of the object and to display the 3D rendering of the object; an input device operable to physically trace at least a portion of the exterior surface of the object; a tracking device configured to track positioning of the input device as the input device traces the at least a portion of the outer surface of the object, and to output data representative of at least two positions of the input device as the input device traces the object; the processor is configured to receive the data representative of the at least two locations, determine a distance between the at least two locations, and output data representative of the distance. system.

14. the processor is configured to identify a curve based on data representative of a position of the input device between the at least two positions, and to determine the distance between the at least two positions along the identified curve. The system of claim 13.

15. The display comprises: receiving the data representative of the distance; and configured to display the distance on the display; The system of claim 13.

16. the input device including a control input operable to receive a selection of a first operational mode of a plurality of operational modes of the input device and to output data indicative of the first operational mode; The system of claim 13.

17. The processor, receiving data indicative of the first mode of operation; and configured to, in response to receiving the data indicative of the first operational mode, determine the distance between the at least two locations and output the data representative of the distance.

17. The system of claim 16.

18. the input device receiving, via the control input, a selection of a second operational mode of the plurality of operational modes of the input device and outputting data indicative of the second operational mode.

17. The system of claim 16.

19. The processor, receiving the data indicative of the second mode of operation; and configured, in response to receiving the data indicative of the second operational mode, to augment the 3D rendering of the object based on positioning information received from the tracking device tracking the input device as the input device traces at least a portion of the exterior surface of the object.

20. The system of claim 18.

20. The processor, receiving the data indicative of the second mode of operation; and in response to receiving the data indicative of the second operational mode, modifying the 3D model of the object based on positioning information received from the tracking device tracking the input device as the input device traces at least a portion of the exterior surface of the object, and generating an updated 3D rendering of the object based on the modified 3D model.

20. The system of claim 18.

Images