System and method for dynamic sketching using exaggerated content

Abstract

The system receives a signal indicating a position of the position pointer and indicating a surface of the physical object. The system obtains a description of a portion of the surface of the physical object based on the position of the position pointer and the signal indicating the surface of the physical object. The system also determines whether the position pointer is on or above a portion of the surface of the physical object based on the signal indicating the position of the position pointer. In response to determining that the position pointer is on or above a portion of the surface of the physical object, the system obtains and stores coordinates corresponding to an input gesture based on the signal indicating the position of the position pointer. Thus, the position pointer can be used as an input device while being placed on or above any physical surface.

Description

Technical Field

[0001] The present disclosure relates to specifying dimensions of a multi-dimensional object represented by digital data, and more particularly, to a system and method for dynamically sketching the shape of such a multi-dimensional object using an input surface.

Background Art

[0002] Software applications have enabled a user of a tablet computer to sketch a multi-dimensional object represented by digital data or otherwise specify dimensions of the multi-dimensional object, for example, by performing input operations on a touch screen device of the tablet computer. However, it can be difficult to sketch an object that is larger than the input surface of the touch screen device.

[0003] Therefore, it is desirable to provide a system and method for exaggerating or emphasizing input gestures to enable a user to specify the shape, orientation, dimensions, etc. of a relatively large object represented by digital data. Further, it is desirable to provide a system and method that enables any physical surface of any size to be used as an input surface for specifying the shape, orientation, dimensions, etc. of a multi-dimensional object represented by digital data.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

[0005] This disclosure teaches a system and method that enable a user to specify the shape, orientation, dimensions, etc. of a multi-dimensional object represented by digital data using any physical surface having any size. Further, this disclosure teaches a system and method that enable a user to specify the shape, orientation, dimensions, etc. of a relatively large multi-dimensional object represented by digital data using exaggerated user input gestures. MEANS FOR SOLVING THE PROBLEMS

[0006] A method according to a first embodiment of the present disclosure includes receiving one or more signals indicating a plurality of spatial positions of a position indicator in a three-dimensional space, receiving one or more signals indicating a surface of a physical object in the three-dimensional space, and based on the one or more signals indicating the plurality of spatial positions of the position indicator and the one or more signals indicating the surface of the physical object, obtaining a description of a portion of the surface of the physical object, and based on the one or more signals indicating the plurality of spatial positions of the position indicator, determining whether the position indicator is on or above the portion of the surface of the physical object, and in response to determining that the position indicator is on or above the portion of the surface of the physical object, obtaining coordinates corresponding to an input gesture based on the one or more signals indicating the plurality of spatial positions of the position indicator, and storing the coordinates corresponding to the input gesture.

[0007] The method may further include displaying a virtual representation of the position indicator together with a virtual representation of the portion of the surface of the physical object.

[0008] The method may further include receiving one or more signals indicating a plurality of positions of the switch of the position indicator, and determining whether the switch of the position indicator is in a first position based on the one or more signals indicating the plurality of positions of the switch of the position indicator. Obtaining the coordinates corresponding to the input gesture may be in response to determining that the position indicator is on or above the portion of the surface of the physical object, and may also be in response to determining that the switch of the position indicator is in the first position.

[0009] Further included may be converting coordinates corresponding to the portion of the surface of the physical object from a first coordinate system to a second coordinate system, where the first coordinate system is different from the second coordinate system.

[0010] The position indicator may include a plurality of reference tags, and the one or more signals indicating the plurality of spatial positions of the position indicator indicate the plurality of positions of the reference tags. Each of the reference tags may include a visually distinguishable pattern formed on the reference tag, and the one or more signals indicating the plurality of spatial positions of the position indicator may include image data corresponding to the plurality of images of the reference tags. Each of the reference tags may emit light, and the one or more signals indicating the plurality of spatial positions of the position indicator may include image data corresponding to the plurality of images of the reference tags.

[0011] The method according to the first embodiment of the present disclosure may be outlined as including receiving one or more signals indicating a plurality of spatial positions of a position indicator in a three-dimensional space, obtaining one or more signals indicating a magnification, obtaining coordinates corresponding to an input gesture in the three-dimensional space based on the one or more signals indicating the plurality of spatial positions of the position indicator, scaling the coordinates corresponding to the input gesture based on the one or more signals indicating the magnification, and displaying a virtual representation of the input gesture based on the scaling of the coordinates corresponding to the input gesture.

[0012] The method may further include displaying the magnification.

[0013] The method may further include receiving a signal indicating a pressure applied to a part of the position indicator, wherein the magnification is based on the signal indicating the pressure applied to the part of the position indicator.

[0014] The method may further include receiving a signal indicating an acceleration of the position indicator, wherein the magnification is based on the signal indicating the acceleration of the position indicator.

[0015] The method may further include receiving one or more signals indicating a plurality of positions of a switch of the position indicator, and determining whether the switch of the position indicator is in a first position based on the one or more signals indicating the plurality of positions of the switch of the position indicator, and obtaining the coordinates corresponding to the input gesture is in response to determining that the switch of the position indicator is in the first position.

[0016] It may further include determining whether the switch of the position indicator is in a second position based on the one or more signals indicating the plurality of positions of the switch of the position indicator, and obtaining the coordinates corresponding to the input gesture ends in response to determining that the switch of the position indicator is in the second position.

[0017] The position indicator may include a plurality of reference tags, and the one or more signals indicating the plurality of spatial positions of the position indicator indicate the plurality of positions of the reference tags. Each of the reference tags may include a visually distinguishable pattern formed on the reference tag, and the one or more signals indicating the plurality of spatial positions of the position indicator may include image data corresponding to the plurality of images of the reference tags. Each of the reference tags may emit light, and the one or more signals indicating the plurality of spatial positions of the position indicator may include image data corresponding to the plurality of images of the reference tags.

[0018] The system according to the third embodiment of the present disclosure, during operation, receives one or more signals indicating a plurality of spatial positions of a position indicator in a three-dimensional space and one or more signals indicating a surface of a physical object in the three-dimensional space, and includes one or more receivers, one or more processors coupled to the one or more receivers, and one or more memory devices coupled to the one or more processors. The one or more memory devices store instructions that, when executed by the one or more processors, cause the system to obtain a description of a portion of the surface of the physical object based on the one or more signals indicating the plurality of spatial positions of the position indicator and the one or more signals indicating the surface of the physical object, determine whether the position indicator is on or above the portion of the surface of the physical object based on the one or more signals indicating the plurality of spatial positions of the position indicator, and in response to determining that the position indicator is on or above the portion of the surface of the physical object, obtain coordinates corresponding to an input gesture based on the one or more signals indicating the plurality of spatial positions of the position indicator and store the coordinates corresponding to the input gesture. This may be outlined as such.

[0019] The one or more memory devices may store instructions that, when executed by the one or more processors, cause the system to display a virtual representation of the position indicator together with a virtual representation of the portion of the surface of the physical object.

[0020] The one or more memory devices may store instructions that, when executed by the one or more processors, cause the system to obtain an indication of magnification and obtain coordinates corresponding to a scaled input gesture based on the magnification and the coordinates corresponding to the input gesture. The one or more memory devices may store instructions that, when executed by the one or more processors, cause the system to display a virtual representation of the scaled input gesture.

Brief Description of the Drawings

[0021]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 6A

Figure 6B

Figure 6C

Figure 6D

Figure 7

Figure 8A

Figure 8B

[0022] FIG. 1 shows a block diagram of a visualization system 100 according to one or more embodiments of the present disclosure. The visualization system 100 includes a position indicator 102, a processing device 104, a plurality of tracking devices 106a and 106b, a visualization device 108, and a sensor 109. In the illustrated embodiment, the position indicator 102 includes a hollow housing 110 having an opening 112 formed at one end, but the housing of the position indicator 102 may have other different forms. In one or more embodiments, the housing 110 generally has a cylindrical shape. The housing 110 may have other shapes without departing from the scope of the present disclosure. The tip of the core 114 protrudes from the housing 110 through the opening 112. In one or more embodiments, the core 114 is a rod-shaped member that transmits pressure corresponding to the pressure applied to a part of the position indicator (for example, the tip of the core 114) to the pressure detector 118, and the pressure detector 118 will be described below with reference to FIG. 2. In one or more embodiments, the core 114 is formed of a conductive material. In one or more embodiments, the core 114 is non-conductive and formed of resin.

[0023] Alternatively, or in combination therewith, in one or more embodiments, the opening 112 is formed on the side surface of the housing 110, and by extending the core 114 through the opening 112, a user's finger can apply pressure to the core to provide an input to the processing device 104. As will be described below with reference to FIG. 2, the position indicator 102 transmits a signal indicating the amount of pressure applied to the tip of the core 114 to the processing device 104. The position indicator 102 can be used as an input device of the processing device 104.

[0024] The processing device 104 includes an input surface 116 formed of a transparent material such as glass. In one or more embodiments, the processing device 104 is a tablet computer. As will be described below with reference to FIG. 3, below the input surface 116, a sensor 140 that tracks the current position of the position indicator 102 and a display device 138 may be provided. The processing device 104 generates visualization data based on the operation of the position indicator 102 by the user, transmits the visualization data to the visualization device 108, and the visualization device 108 displays an image based on the visualization data. Additionally, or alternatively, the display device 138 of the processing device 104 may display an image based on the visualization data.

[0025] In one or more embodiments, the visualization device 108 and the display device 138 each process a portion of the visualization data generated by the processing device 104 and display images simultaneously. In one or more embodiments, the visualization device 108 and the display device 138 operate at different screen refresh rates. Thus, it may be desirable to offload the processing of a device operating at a higher screen refresh rate to a device operating at a lower screen refresh rate. For example, the visualization device 108 can operate at a screen refresh rate of 90 Hz, and the display device 138 can operate at a screen refresh rate of 60 Hz, and in such a case, it may be desirable to offload some or all of the processing of the visualization data by the visualization device 108 to the display device 138. In this way, the processing device 104 may divide the visualization data such that the processing load of the visualization device 108 is offloaded to the display device 138.

[0026] In one or more embodiments, the processing device 104 receives, from the visualization device 108, a signal indicating the current processing load of the visualization device 108, and the processing device 104 dynamically adjusts the amount of visualization data transmitted to the visualization device 108 and the display device 138 based on the current processing load. In one or more embodiments, the processing device 104 estimates the current processing load of the visualization device 108 and dynamically adjusts the amount of visualization data transmitted to the visualization device 108 and the display device 138 based on the estimated current processing load. For example, if the current processing load, shown or estimated, of the visualization device 108 is above a predetermined threshold, the processing device 104 decreases the amount of visualization data transmitted to the visualization device 108 and increases the amount of visualization data transmitted to the display device 138. Additionally, or alternatively, the processing device 104 may offload processing from the display device 138 to the visualization device 108 in a similar manner.

[0027] The tracking devices 106a and 106b track the position and / or orientation of the position indicator 102, particularly, in some embodiments, the tip of the core 114 of the position indicator 102. The tracking devices 106a and 106b are collectively referred to as the tracking device 106. The embodiment shown in FIG. 1 includes two tracking devices 106, but the visualization system 100 may include a different number of tracking devices 106 without departing from the scope of the present disclosure. For example, the visualization system 100 may include three, four, or more tracking devices 106 according to the present disclosure. In one or more embodiments, the visualization system 100 does not include any tracking devices 106, and the position of the tip of the core 114 of the position indicator 102 is tracked using only the sensor 140 of the processing device 104.

[0028] In one or more embodiments, the tracking device 106 employs known optical motion tracking techniques to track the position and / or orientation of the tip of the core 114 of the position indicator 102. In one or more embodiments, the position indicator 102 has a reference tag in the form of an optical marker attached to the outer surface of the housing 110, and the optical marker is a passive device on which an optically detectable, unique, visually distinguishable color or pattern is formed on each. Each of the tracking devices 106 may include a camera that acquires an image of one or more of the optical markers and transmits the corresponding image data to the processing device 104. The processing device 104 stores data indicating the spatial relationship between each of the optical markers and the tip of the core 114 of the position indicator 102, and determines the current position and / or orientation of the tip of the core 114 of the position indicator 102 by processing the image data according to known techniques. In one or more embodiments, the optical marker is an active device each having a light emitting device (e.g., a light emitting diode) that emits light having different wavelengths. For example, the light emitted by such an optical marker may be ultraviolet light that is invisible to the human eye. In one or more embodiments, the tracking device 106 is a Constellation sensor that is part of an Oculus Rift system available from Oculus VR. In one or more embodiments, the tracking device 106 is a laser-based tracking device. For example, the tracking device 106 is a SteamVR base station 2.0 that is part of an HTC Vive system available from HTC Corporation.

[0029] The visualization device 108 processes visualization data generated by the processing device 104 and displays a corresponding image. In one or more embodiments, the visualization device 108 is a head-mounted display device. In one or more embodiments, the visualization device 108 is an HTC Vive Pro virtual reality headset that is part of an HTC Vive system available from HTC Corporation. In one or more embodiments, the visualization device 108 is an Oculus Rift virtual reality headset that is part of an Oculus Rift system available from Oculus VR. In one or more embodiments, the visualization device 108 is a HoloLens augmented reality headset available from Microsoft Corporation. Other types of headsets, particularly, for example, Magic Leap headsets and Meta headsets, may be used.

[0030] In one or more embodiments, the visualization device 108 includes a sensor 109 that is used to track the location of physical objects within the field of view of the sensor 109. For example, the visualization device 108 is a head-mounted display, and the sensor 109 includes a pair of cameras, each of which is located near one of the eyes of the user of the visualization device 108 and has substantially the same field of view as that eye. Additionally, the visualization device 108 includes a transmitter that transmits image data corresponding to the images captured by the cameras to the processing device 104, and the processing device 104 processes the image data and determines the coordinates of the objects captured by the cameras using, for example, conventional image processing techniques. For example, in one or more embodiments, the processing device 104 includes object recognition software configured in a manner similar to the object recognition engine described in Patent Document 1 (see, for example, paragraph 87). Patent Document 1 is hereby incorporated by reference in its entirety. Alternatively, the visualization device 108 includes a processor and a memory storing instructions that, when executed by the processor, cause the visualization device 108 to determine the coordinates of the objects captured by the cameras and transmit those coordinates to the processing device 104.

[0031] Having provided an overview of the visualization system 100, the position indicator 102 will now be described in more detail with reference to FIG. 2, which shows a block diagram of the position indicator 102 according to one or more embodiments of the present disclosure. The position indicator 102 includes a pressure detector 118 that, during operation, detects the pressure applied to the tip of the core 114 when, for example, the user presses the tip of the core 114 against the input surface 116 of the processing device 104. In one or more embodiments, the pressure detector 118 is configured in a manner similar to the pressure sensing component described in Patent Document 2 (see, for example, columns 13, line 49 to column 22, line 13). Patent Document 2 is hereby incorporated by reference in its entirety.

[0032] In one or more embodiments, the position indicator 102 includes a switch 120 that is in one of a plurality of positions during operation. A user can actuate the switch 120 to change its position in order to provide an input to the processing device 104. For example, the switch 120 is in a "closed" or "on" position while the user is pressing the switch 120, and in an "open" or "off" position while the user is not pressing the switch 120. In one or more embodiments, the switch 120 is configured in a manner similar to the side switch described in Patent Document 2 (see, for example, columns 11, lines 24 to 49). In one or more embodiments, the position indicator 102 includes two switches 120 that can be operated by the user to provide an input similar to the input provided by operating the left and right buttons of a computer mouse.

[0033] In one or more embodiments, the position indicator 102 includes an accelerometer 122 that outputs a signal indicating the acceleration of the position indicator 102 during operation. In one or more embodiments, the accelerometer 122 is configured as a microfabricated microelectromechanical system (MEMS).

[0034] The position indicator 102 also includes a transmitter 124 coupled to the pressure detector 118, and the transmitter 124 transmits, during operation, a signal indicating the pressure applied to the tip of the core body 114 detected by the pressure detector 118. In one or more embodiments, the transmitter 124 operates in accordance with one or more of the Bluetooth communication standards. In one or more embodiments, the transmitter 124 operates in accordance with one or more of the communication standards of the IEEE 802.11 family. In one or more embodiments, the transmitter 124 electromagnetically induces a signal via the tip of the core body 114 and the sensor 140 of the processing device 104. In one or more embodiments, the transmitter 124 is coupled to the switch 120, and the transmitter 124 transmits, during operation, a signal indicating the position of the switch 120. In one or more embodiments, the transmitter 124 is coupled to the accelerometer 122, and the transmitter 124 transmits, during operation, a signal indicating the acceleration of the position detection device 102 detected by the accelerometer 122.

[0035] In one or more embodiments, the position indicator 102 includes a plurality of reference tags 126a, 126b, and 126c. The reference tags 126a, 126b, and 126c are collectively referred to herein as reference tag 126. The reference tag 126 is tracked by the tracking device 106. In one or more embodiments, the reference tag 126 is a passive optical marker fixed to the outer surface of the housing 110 of the position indicator 102 as described above in connection with FIG. 1. Alternatively or in addition, in one or more embodiments, the reference tag 126 actively emits light or radio waves detected by the tracking device 106. The embodiment shown in FIG. 2 includes three reference tags 126, but the position indicator 102 may include a different number of reference tags 126. For example, the position indicator 102 may include four, five, six, or more reference tags 126 according to the present disclosure.

[0036] Having described the position indicator 102 in more detail, we will now describe the processing device 104 in more detail with reference to FIG. 3, which shows a block diagram of the processing device 104 according to one or more embodiments of the present disclosure. The processing device 104 includes a microprocessor 128 having a memory 130 and a central processing unit (CPU) 132, a memory 134, an input / output (I / O) circuit 136, a display device 138, a sensor 140, a transmitter 142, and a receiver 144.

[0037] The memory 134 stores processor-executable instructions that, when executed by the CPU 132, cause the processing device 104 to perform the operations of the processing device 104 described in connection with FIGS. 4, 5A, 5B, and 7. The CPU 132 uses the memory 130 as a working memory while executing instructions. In one or more embodiments, the memory 130 consists of one or more random access memory (RAM) modules and / or one or more non-volatile random access memory (NVRAM) modules such as, for example, an electrically erasable programmable read-only memory (EEPROM) or a Flash memory module.

[0038] In one or more embodiments, the I / O circuit 136 may include buttons, switches, dials, knobs, microphones, or other user interface elements for inputting commands to the processing device 104. The I / O circuit 136 may also include one or more speakers, one or more light-emitting devices, or other user interface elements for outputting information or instructions from the processing device 104.

[0039] The display device 138 graphically displays information to the operator. The microprocessor 128 controls the display device 138 to display information based on the visualization data generated by the processing device 104. In one or more embodiments, the display device 138 is a liquid crystal display (LCD) device. In one or more embodiments, the display device 138 simultaneously displays two images so that a user wearing appropriate eyewear can perceive a multi-dimensional image in a manner similar to viewing a three-dimensional (3D) image via, for example, a 3D-compatible television.

[0040] The sensor 140 detects the position indicator 102 and outputs a signal indicating the position of the position indicator 102 relative to the input surface of the sensor 140 (e.g., surface 116). In one or more embodiments, the microprocessor 128 processes the signal received from the sensor 140 and obtains the (X, Y) coordinates on the input surface of the sensor 140 corresponding to the position indicated by the position indicator 102. In one or more embodiments, the microprocessor 128 processes the signal received from the sensor 140 and obtains the (X, Y) coordinates on the input surface of the sensor 140 corresponding to the position indicated by the position indicator 102, in addition to the height (e.g., Z coordinate) above the input surface of the sensor 140 where the position indicator 102 is located. In one or more embodiments, the sensor 140 is an inductive type sensor configured in a manner similar to the position detection sensor described in Patent Document 3 (see, for example, column 7, lines 35 to column 10, line 27). Patent Document 3 is hereby incorporated by reference in its entirety. In one or more embodiments, the sensor 140 is a capacitive type sensor configured in a manner similar to the position detection sensor described in Patent Document 4 (see, for example, column 6, lines 5 to column 8, line 17). Patent Document 4 is hereby incorporated by reference in its entirety.

[0041] Transmitter 142 is coupled to microprocessor 128, and during operation, transmitter 142 transmits visualization data generated by microprocessor 128 to visualization device 108. For example, in one or more embodiments, transmitter 142 operates according to one or more of the Bluetooth and / or IEEE 802.11 family of communication standards. Receiver 144 is coupled to microprocessor 128, and during operation, receiver 144 receives signals from tracking device 106 and visualization device 108. For example, in one or more embodiments, receiver 144 operates according to one or more of the Bluetooth and / or IEEE 802.11 family of communication standards. In one or more embodiments, receiver 144 receives a signal from position indicator 102. In one or more embodiments, receiver 144 is included in sensor 140 and receives one or more signals from the tip of core 114 of position indicator 102 by electromagnetic induction.

[0042] Having described the structure of visualization system 100, next, an example of method 200 executed by visualization system 100 will be described in connection with FIG. 4, which shows a flowchart of method 200 according to one or more embodiments of the present disclosure. Method 200 begins at 202, for example, when power is applied to processing device 104.

[0043] At 202, one or more signals are received that indicate one or more spatial positions of position indicator 102 in three-dimensional space. For example, receiver 144 of processing device 104 receives one or more signals from tracking device 106. In addition to, or instead of, this, microprocessor 128 receives one or more signals from sensor 140 of processing device 104. The method 200 then proceeds to 204.

[0044] At 204, a signal indicating the position of switch 120 of position indicator 102 is received. For example, receiver 144 of processing device 104 receives a signal indicating the position of switch 120 from transmitter 124 of position indicator 102. Then, method 200 proceeds to 206.

[0045] Optionally, at 206, a signal indicating the acceleration of position indicator 102 is received. For example, receiver 144 of processing device 104 receives a signal indicating the acceleration of position indicator 102 from transmitter 124 of position indicator 102. Then, method 200 proceeds to 208.

[0046] At 208, a signal indicating the pressure applied to the tip of core 114 is received. For example, receiver 144 of processing device 104 receives a signal indicating the pressure applied to the tip of core 114 from transmitter 124 of position indicator 102. In addition to, or instead of, this, sensor 140 of processing device 104 receives, by electromagnetic induction, a signal indicating the pressure applied to the tip of core 114 from the tip of core 114 of position indicator 102. Then, method 200 proceeds to 210.

[0047] At 210, one or more signals indicating one or more physical objects located in the vicinity of the user of visualization system 100 are received. In one or more embodiments, receiver 144 of processing device 104 receives a signal indicating one or more physical objects located in the three-dimensional space in the vicinity of the user from sensor 109 of visualization device 108. For example, receiver 144 receives image data generated by a pair of cameras of sensor 109, and microprocessor 128 processes the image data to obtain a set of coordinates corresponding to the outer surfaces of the objects imaged by the cameras. Then, method 200 proceeds to 212.

[0048] At 212, the signals received at 202, 204, 206, 208, and 210 are processed. In one or more embodiments, the data transmitted by those signals is timestamped and stored in the memory 130 of the processing device 104, and the CPU 132 processes this data in chronological order based on the timestamps associated with the data. Processing corresponding to the flowcharts shown in FIGS. 5A, 5B, and 7 may be executed at 212 as described below. Then, method 200 proceeds to 214.

[0049] At 214, a determination is made as to whether an end processing instruction has been received. For example, the microprocessor 128 uses the position indicator 102 to determine whether a predetermined icon or object displayed by the display device 138 of the processing device 104 has been selected. As another example, at 214, the microprocessor 128 determines whether an audio command corresponding to an end operation has been received. If it is determined at 214 that an end operation has been received, method 200 ends. Otherwise, method 200 returns to 202.

[0050] FIGS. 5A and 5B show flowcharts of a method 300 that may be executed by the visualization system 100 at 212 of the above-described method 200 according to one or more embodiments of the present disclosure. Method 300 begins at 302 in response to the microprocessor 128 determining that an instruction for defining an input surface has been received. For example, the microprocessor 128 uses the position indicator 102 to determine that a predetermined icon or object displayed by the display device 138 of the processing device 104 has been selected. As another example, method 300 begins at 302 in response to the microprocessor 128 determining that an audio command corresponding to an instruction for defining an input surface has been received.

[0051] At 302, a description of the input surface is obtained. In one or more embodiments, the microprocessor 128 uses one or more signals indicating one or more spatial positions of the position indicator 102 received in 202 of the method 200 described above to determine a set of coordinates of the contour or boundary of the surface to be used as the input surface. For example, the microprocessor 128 uses one or more signals indicating one or more spatial positions of the position indicator 102 to obtain the contour of the region corresponding to the input surface in a "local" coordinate system with reference to a reference location (e.g., the origin of the coordinate system) used by the visualization device 108. Then, method 300 proceeds to 304.

[0052] At 304, the input surface is fixed as a virtual surface in the virtual environment. When the input surface is fixed as a virtual surface in the virtual environment, the virtual surface remains stationary with respect to the virtual environment even when the user wearing the visualization device 108 moves to a different physical location. In one or more embodiments, the visualization system 100 includes a position detection unit similar to that described in Patent Document 5 (see, for example, paragraph

[0074] ), and the processing device 104 displays the virtual surface by executing the method shown in FIG. 5 of Patent Document 5 and described in paragraphs

[0074] to

[0099] . Patent Document 5 is hereby incorporated by reference in its entirety.

[0053] In one or more embodiments, the microprocessor 128 uses one or more signals indicating one or more physical objects located in the vicinity of the user of the visualization system 100 received in 210 of the method 200 described above to construct a model of the physical object in a virtual environment. For example, the microprocessor 128 converts or otherwise changes a set of coordinates describing the input surface obtained in 302 of the method 300 described above from a "local" coordinate system based on a reference location used by the position of the visualization device 108 to a "global" coordinate system corresponding to the virtual environment using a virtual reference location corresponding to a physical location in the vicinity of the user of the visualization system 100, and uses the converted set of coordinates to divide or demarcate the physical surface in the vicinity of the user of the visualization system 100. In other words, the microprocessor 128 assigns the set of coordinates of the physical surface on and / or within the description (e.g., contour) of the input surface obtained in 302 to a virtual input surface corresponding to the demarcated physical surface. Then, the method 300 proceeds to 306.

[0054] In 306, the data describing the virtual input surface obtained in 304 is transmitted. In one or more embodiments, the microprocessor 128 of the processing device 104 causes the transmitter 142 to transmit the data describing the virtual input surface to the visualization device 108. In one or more embodiments, the microprocessor 128 transmits the data describing the virtual input surface to the display device 138 of the processing device 104. Then, the method 300 proceeds to 308.

[0055] In 308, the data describing the virtual input surface is rendered and the virtual input surface is displayed. In one or more embodiments, the visualization device 108 performs rendering of a two-dimensional image to obtain a three-dimensional (3D) representation of the virtual input surface. In one or more embodiments, the microprocessor 128 causes the display device 138 of the processing device 104 to render the visualization data and display the virtual input surface. Then, the method 300 proceeds to 310.

[0056] At 310, it is determined whether the position indicator 102 is located on or above the input surface. In one or more embodiments, the microprocessor 128 uses one or more signals indicating one or more spatial positions of the position indicator 102 received in 202 of the method 200 described above to determine whether the position indicator 102 is located on or above the input surface. If it is determined that the position indicator 102 is located on or above the input surface, the method 300 proceeds to 312. Otherwise, the method 300 returns to 308.

[0057] At 312, it is determined whether the switch of the position indicator 102 has been pressed. For example, the microprocessor 128 determines whether the switch 120 of the position indicator 102 is in the "on" position or the "closed" position based on the signal indicating the position of the switch 120 received in 204 of the method 200 described above. If it is determined that the switch 120 of the position indicator 102 is in the "on" position or the "closed" position, the method 300 proceeds to 314. Otherwise, the method 300 returns to 308.

[0058] At 314, a set of coordinates corresponding to the input gesture is acquired. In one or more embodiments, the microprocessor 128 uses one or more signals indicating one or more spatial positions of the position indicator 102 received in 202 of the method 200 described above with the position indicator 102 disposed on or above the input surface to acquire a set of coordinates corresponding to the input gesture. Then, the method 300 proceeds to 316.

[0059] In 316, in order to obtain a converted coordinate group corresponding to the input gesture, the coordinate group corresponding to the input gesture is converted. In one or more embodiments, the microprocessor 128 of the processing device 104 converts or otherwise changes the coordinate group describing the input gesture obtained in 314 from the "global" coordinate system corresponding to the virtual environment to the "local" coordinate system based on the reference position used by the visualization device 108. Then, method 300 proceeds to 318.

[0060] In 318, the coordinate group corresponding to the input gesture obtained in 314 or 316 is transmitted. In one or more embodiments, the microprocessor 128 of the processing device 104 causes the transmitter 142 to transmit the coordinate group corresponding to the input gesture obtained in 314 or 316 to the visualization device 108. In one or more embodiments, the microprocessor 128 transmits the coordinate group corresponding to the input gesture obtained in 314 or 316 to the display device 138 of the processing device 104. Then, method 300 proceeds to 320.

[0061] In 320, the input gesture is rendered and displayed. In one or more embodiments, the visualization device 108 performs the rendering of a two-dimensional image to obtain a three-dimensional (3D) representation of the input gesture. In one or more embodiments, the microprocessor 128 causes the display device 138 of the processing device 104 to render and display the input gesture. Then, method 300 proceeds to 322.

[0062] At 322, it is determined whether the switch of the position indicator has been released. For example, the microprocessor 128 determines whether the switch 120 of the position indicator 102 is in the "off" position or the "open" position based on the signal indicating the position of the switch 120 received at 204 of the method 200 described above. If it is determined that the switch 120 of the position indicator 102 is in the "off" position or the "open" position, the acquisition of the coordinate group corresponding to the input gesture is terminated, and the method 300 proceeds to 324. Otherwise, the method 300 returns to 314, and an additional coordinate group corresponding to the input gesture is acquired.

[0063] At 324, the coordinate group corresponding to the input gesture acquired at 314 or 316 is stored. In one or more embodiments, the microprocessor 128 of the processing device 104 stores the coordinate group corresponding to the input gesture acquired at 314 or 316 in the memory 130 and / or the memory 134. Then, the method 300 ends.

[0064] Figures 6A, 6B, 6C, and 6D are diagrams for explaining the operation of the visualization system 100 during the method 300 described above according to one or more embodiments of the present disclosure. As shown in Figure 6A, it is assumed that the user 144 is physically located in an environment including a table 146. The tracking devices 106a and 106b are also physically located in the vicinity of the user 144 in this environment. Further, the user 144 is wearing the visualization device 108.

[0065] As shown in FIG. 6B, user 144 designates portion 152 on the upper surface 150 of table 146 as an input surface by sketching pattern 148 on the upper surface 150 of table 146 using position indicator 102. While sketching pattern 148 in 302 of method 300 described above using position indicator 102, processing device 104 receives the coordinates of position indicator 102. Next, user 144 indicates to processing device 104 that portion 152 of the upper surface 150 of table 146 is to be used as an input surface, for example, by performing a "double click" operation using switch 120 of position indicator 102 or by issuing a corresponding voice command.

[0066] In response, processing device 104 fixes portion 152 of the upper surface 150 of table 146 as an input surface in 304 of method 300 described above. Next, processing device 104 transmits corresponding position data of portion 152 of the upper surface 150 of table 146 to visualization device 108 in 306 of method 300 described above. Visualization device 108 displays a virtual representation of portion 152 of the upper surface 150 of table 146 in 308 of method 300 described above. Portion 152 of the upper surface 150 of table 146 is hereinafter referred to as input surface 152. FIG. 6C shows an example of a virtual representation 102' of position indicator 102, a virtual representation 146' of table 146, and a virtual representation 152' of input surface 152 fixed to a virtual representation 150' of the upper surface 150 of table 146, which are displayed by visualization device 108.

[0067] In one or more embodiments, visualization device 108 displays the virtual representation 152' of input surface 152 in a visually distinguishable manner. For example, visualization device 108 displays the virtual representation 152' of input surface 152 in a distinguishable color or with a distinguishable brightness so that user 144 can easily identify the virtual representation 152' of input surface 152 while viewing the output of visualization device 108.

[0068] As shown in FIG. 6D, the user 144 can then move the position indicator 102 on or above the input surface 152 and use the input surface 152 in a manner similar to using the position indicator 102 on or above the input surface 116 of the sensor 140 of the processing device 104. For example, while using the position indicator 102 on or above the input surface 152, the user 144 may press the switch 120 of the position indicator 102 to indicate to the processing device 104 to store the coordinate group of the subsequent location group of the position indicator 102 as an input gesture. The processing device 104 determines, in each of 310 and 312 of the method 300 described above, that the position indicator 102 is located on or above the input surface 152 and that the user 144 has pressed the switch 120 of the position indicator 102.

[0069] Subsequently, the processing device 104 obtains, in 314 of the method 300 described above, the coordinate group corresponding to the input gesture, and these coordinates are in the "global" coordinate system corresponding to the virtual environment. The processing device 104 also, in 316 of the method 300 described above, converts or otherwise changes this coordinate group into the corresponding coordinate group in the "local" coordinate system of the visualization device 108. The processing device 104 transmits, in 318 of the method 300 described above, this coordinate group to the visualization device 108. The visualization device 108 displays the input gesture, for example, as a line segment connecting the coordinate groups corresponding to the input gesture. The user 144 may then release the switch 120 of the position indicator 102 to indicate to the processing device 104 to stop storing the coordinate group of the location group of the position indicator 102 as an input gesture. The processing device 104 determines, in 322 of the method 300 described above, that the user 144 has released the switch 120 of the position indicator 102. Then, the processing device 104 stores, in 324 of the method 300 described above, the coordinate group corresponding to the input gesture.

[0070] FIG. 7 shows a flowchart of a method 400 that may be executed by the visualization system 100 in 212 of the method 200 described above, according to one or more embodiments of the present disclosure. The method 400 begins, at 402, in response to, for example, the microprocessor 128 determining that an instruction for performing exaggerated input processing has been received. For example, the microprocessor 128 may determine that it has selected a predetermined icon or object displayed by the display device 138 of the processing device 104 using the position indicator 102. As another example, the method 400 begins, at 402, in response to the microprocessor 128 determining that an audio command corresponding to an instruction for performing exaggerated input processing has been received. As yet another example, the microprocessor 128 may evaluate the acceleration data of the position indicator 102 or evaluate the coordinate data corresponding to an input gesture made by the position indicator 102, and determine from the evaluated data that an instruction for performing exaggerated input processing has been received.

[0071] At 402, a determination is made as to whether the switch 120 of the position indicator 102 has been pressed. For example, the microprocessor 128 determines whether the switch 120 of the position indicator 102 is in the "on" position or the "closed" position based on a signal indicating the position of the switch 120 received at 204. If it is determined that the switch 120 of the position indicator 102 is in the "on" position or the "closed" position, the method 400 proceeds to 404. Otherwise, the method 400 returns to 402.

[0072] At 404, a set of coordinates corresponding to an input gesture executed using the position indicator 102 is obtained. In one or more embodiments, the microprocessor 128 of the processing device 104 obtains the set of coordinates corresponding to the input gesture based on a signal indicating the position of the position indicator 102 received at 202 of the method 200 described above. Next, the method 400 proceeds to 406.

[0073] At 406, it is determined whether the switch 120 of the position indicator 102 has been released. For example, the microprocessor 128 determines whether the switch 120 of the position indicator 102 is in the "off" position or the "on" position based on the signal indicating the position of the switch 120 received at 204 of method 200. If it is determined that the switch 120 of the position indicator 102 is in the "off" position or the "on" position, method 400 proceeds to 408. Otherwise, method 400 returns to 404.

[0074] At 408, the coordinate group corresponding to the input gesture obtained at 404 is scaled. In one or more embodiments, the microprocessor 128 of the processing device 104 scales the coordinate group corresponding to the input gesture using a predetermined magnification factor. For example, the microprocessor 128 may obtain one or more signals indicating the magnification factor in response to a predetermined icon or object displayed by the display device 138 of the processing device 104 being selected using the position indicator 102. Next, method 400 proceeds to 410.

[0075] For example, if the magnification factor is set to "10", the microprocessor 128 scales the coordinate group such that the actual input gesture is magnified by a factor of 10. In other words, if the input gesture corresponds to the user moving the position indicator 102 from the initial location in an arc having a length of 1 meter, the microprocessor 128 scales the coordinate group such that the scaled coordinate group defines an arc extending from the corresponding initial location to a length of 10 meters and having the same relative shape as the actual input gesture.

[0076] Similarly, for example, when the magnification is set to "-10" or "1 / 10", the microprocessor 128 scales the coordinate group so that the actual input gesture is reduced by a factor of 10. In other words, if the input gesture corresponds to the user moving the position indicator 102 from the initial location in an arc having a length of 1 meter, the microprocessor 128 scales the coordinate group so that the scaled coordinate group defines an arc extending from the corresponding initial location to a length of one-tenth of 1 meter and having the same relative shape as the actual input gesture. Thus, the magnification can be set to enable the user to more accurately sketch relatively small objects.

[0077] In one or more embodiments, the microprocessor 128 of the processing device 104 uses a magnification dynamically obtained based on the amount of pressure applied to the tip of the core 114 that may extend from an opening formed in the side surface of the housing 110 of the position indicator 102 to scale the coordinate group corresponding to the input gesture. For example, the microprocessor 128 dynamically obtains the magnification based on a signal indicating the pressure applied to the tip of the core 114 received in 208 of the method 200 described above. Thus, the user can indicate the magnification to the processing device 104 by applying pressure to the tip of the core 114. In one or more embodiments, the processing device 104 causes the visualization device 108 and / or the display device 138 to display the magnification. Thus, the user viewing the displayed magnification can determine whether to increase, decrease, or maintain the pressure applied to the tip of the core 114 to set the desired magnification.

[0078] In one or more embodiments, the magnification is directly proportional to the pressure applied to the tip of the core 114. For example, the magnification increases as the pressure applied by the user to the tip of the core 114 increases. As another example, the magnification decreases as the pressure applied by the user to the tip of the core 114 increases.

[0079] In one or more embodiments, when the user changes the amount of pressure applied to the tip of the core body 114 by more than a predetermined threshold amount between various segments of the input gesture, the microprocessor 128 dynamically adjusts the magnification. Thus, the microprocessor 128 may use various magnifications for various segments of the input gesture.

[0080] In one or more embodiments, the microprocessor 128 of the processing device 104 scales the coordinate group corresponding to the input gesture using a magnification dynamically obtained based on the acceleration of the position indicator 102. The microprocessor 128 may dynamically obtain the magnification based on the signal indicating the acceleration of the position indicator 102 received in 206 of the method 200 described above. For example, the user can indicate the magnification to the processing device 104 by accelerating the position indicator 102, and the greater the acceleration of the position indicator 102, the greater the magnification used by the processing device 104.

[0081] In 410, the coordinate group corresponding to the input gesture scaled in 408 is stored. In one or more embodiments, the microprocessor 128 of the processing device 104 stores the coordinate group corresponding to the input gesture scaled in 408 in the memory 130 and / or the memory 134. Then, the method 400 proceeds to 412.

[0082] In 412, the coordinate group corresponding to the input gesture stored in 410 is transmitted. In one or more embodiments, the microprocessor 128 of the processing device 104 causes the transmitter 142 to transmit the coordinate group corresponding to the input gesture scaled in 408 to the visualization device 108. In one or more embodiments, the microprocessor 128 transmits the coordinate group corresponding to the input gesture scaled in 408 to the display device 138 of the processing device 104. Then, the method 400 proceeds to 414.

[0083] At 414, a virtual representation of the input gesture is displayed. In one or more embodiments, the visualization device 108 executes a rendering of a two-dimensional image to obtain a three-dimensional (3D) representation of the input gesture. In one or more embodiments, the microprocessor 128 causes the display device 138 of the processing device 104 to display the virtual representation of the input gesture. The method 400 then ends.

[0084] Figures 8A and 8B are diagrams for explaining the operation of the visualization system 100 during the method 400 described above according to one or more embodiments of the present disclosure. While pressing the switch 120 of the position indicator 102, the user 144 moves the position indicator 102 from the initial position 154 to the final position 156 along an arc corresponding to the input gesture 158 as shown in FIG. 8A, and then releases the switch 120 of the position indicator 102. The processing device 104 determines that the switch 120 of the position indicator 102 has been pressed at 402 of the method 400 described above. In response, the processing device 104 obtains a set of coordinates corresponding to the input gesture 158 at 404 of the method 400 described above until it determines that the switch 120 of the position indicator 102 has been released at 406 of the method 400 described above. Next, the processing device 104 scales the set of coordinates corresponding to the input gesture 158 at 408 of the method 400 described above. Then, the processing device 104 stores the scaled set of coordinates corresponding to the input gesture 158 at 410 of the method 400 described above. The processing device 104 also transmits the scaled set of coordinates corresponding to the input gesture 158 at 412 of the method 400 described above.

[0085] Visualization device 108 displays a virtual representation of the scaled input gesture 160 in 414 of method 400 described above. FIG. 8B shows a virtual environment displayed by visualization device 108. The virtual environment includes a virtual representation 144' of user 144 according to scale and a virtual representation of the scaled input gesture 160. As can be seen by comparing FIG. 8A and FIG. 8B, the scaled input gesture 160 is many times larger than the actual input gesture 158. In 414 of method 400 described above, visualization device 108 may display a message 162 indicating the magnification factor used to create the scaled input gesture 160. Further, in 414 of method 400 described above, visualization device 108 may display a symbol 164 based on the magnification factor to visually show user 144 the scaled dimensions of the scaled input gesture 160. Thus, when method 400 is executed, user 144 can easily sketch a relatively large object through simple operation of position indicator 102.

[0086] Additional embodiments can be provided by combining the various embodiments described above. Where it is necessary to adopt the various patent concepts referred to herein, aspects of the embodiments can be modified and additional embodiments can be provided.

[0087] In light of the description detailed above, these and other modifications can be made to the embodiments. In general, in the following claims, the terms used are not to be construed as limiting the claims to the specific embodiments disclosed in the specification and claims, but rather are to be interpreted to include all possible embodiments along with the full scope equivalent to what is given in such claims. Accordingly, the claims are not limited by the present disclosure.

Claims

1. A method comprising: receiving, by a tracking device that tracks the spatial position of an object having a marker, one or more signals indicating a plurality of spatial positions of a position indicator having the marker in response to tracking the spatial position of the position indicator having the marker; acquiring a plurality of coordinates of the position indicator based on the one or more signals indicating the plurality of spatial positions of the position indicator received while the position indicator is moved to identify a portion of the surface of the physical object that is smaller than the surface of the physical object that does not have a sensor for detecting a position indicated by the position indicator below the surface; determining, based on the plurality of coordinates of the position indicator acquired while the position indicator is moved and the one or more signals indicating the plurality of spatial positions of the position indicator, whether the position indicator is on or above the portion of the surface of the physical object identified by moving the position indicator; acquiring, in response to determining that the position indicator is on or above the portion of the surface of the physical object, a coordinate group corresponding to an input gesture based on the one or more signals indicating the plurality of spatial positions of the position indicator; rendering the input gesture in a virtual environment based on the coordinate group corresponding to the input gesture; A method including the above steps.

2. displaying a virtual representation of the position indicator together with a virtual representation of the portion of the surface of the physical object; The method according to claim 1, further including the above step.

3. receiving one or more signals indicating a state of a switch of the position indicator; determining, based on the one or more signals indicating the state of the switch of the position indicator, whether the switch of the position indicator is in a first position; and further including the above steps, wherein acquiring the coordinate group corresponding to the input gesture is in response to determining that the position indicator is on or above the portion of the surface of the physical object and in response to determining that the switch of the position indicator is in the first position; The method according to claim 1.

4. Converting a coordinate group corresponding to the part of the surface of the physical object from a first coordinate system to a second coordinate system different from the first coordinate system. The method according to claim 1, further comprising.

5. The position indicator includes a plurality of reference tags. The method according to claim 1, wherein the one or more signals indicating the plurality of spatial positions of the position indicator indicate the positions of the respective reference tags.

6. Visually distinguishable patterns are formed on the surfaces of the respective reference tags. The one or more signals indicating the plurality of spatial positions of the position indicator include image data corresponding to the images of the respective reference tags. The method according to claim 5.

7. Each of the reference tags emits light. The method according to claim 5, wherein the one or more signals indicating the plurality of spatial positions of the position indicator include image data corresponding to the images of the respective reference tags. The method according to claim 5.

8. A method comprising: Receiving one or more signals indicating a plurality of spatial positions of a position indicator in a three-dimensional space; Obtaining one or more signals indicating a magnification; Obtaining a coordinate group corresponding to an input gesture in the three-dimensional space based on the one or more signals indicating the plurality of spatial positions of the position indicator; Scaling the coordinate group corresponding to the input gesture based on the one or more signals indicating the magnification; Displaying a virtual representation of the input gesture based on the scaling of the coordinate group corresponding to the input gesture; Including The one or more signals indicating the magnification are signals indicating the pressure applied to the tip of the core of the position indicator. The magnification is based on the pressure applied to the tip of the core of the position indicator. Method.

9. Displaying the magnification. The method according to claim 8, further comprising.

10. Further comprising receiving a signal indicating the acceleration of the position indicator, The magnification is based on the signal indicating the acceleration of the position indicator. The method according to claim 8.

11. Receiving one or more signals indicating the state of a switch of the position indicator; Further comprising determining whether the switch of the position indicator is in a first position based on the one or more signals indicating the state of the switch of the position indicator. Obtaining the coordinate group corresponding to the input gesture is in response to determining that the switch of the position indicator is in the first position. The method according to claim 8.

12. Further comprising determining whether the switch of the position indicator is in a second position based on the one or more signals indicating the state of the switch of the position indicator. Obtaining the coordinate group corresponding to the input gesture ends in response to determining that the switch of the position indicator is in the second position. The method according to claim 11.

13. The position indicator includes a plurality of reference tags. The one or more signals indicating the plurality of spatial positions of the position indicator indicate the positions of the respective plurality of reference tags. The method according to claim 8.

14. Visually distinguishable patterns are formed on the surfaces of the respective plurality of reference tags. The one or more signals indicating the plurality of spatial positions of the position indicator include image data corresponding to the images of the respective plurality of reference tags. The method according to claim 13.

15. Each of the reference tags emits light. The one or more signals indicating the plurality of spatial positions of the position indicator include image data corresponding to the images of the respective plurality of reference tags. The method according to claim 13.

16. A system comprising: One or more receivers that receive one or more signals indicating a plurality of spatial positions of a position indicator in a three-dimensional space in response to tracking the spatial position of the position indicator having a marker by a tracking device that tracks the spatial position of an object having the marker; One or more processors coupled to the one or more receivers; One or more memory devices coupled to the one or more processors, The one or more memory devices store instructions to be executed by the one or more processors, The instructions cause the system to Obtain a plurality of coordinates of the position indicator based on the one or more signals indicating the plurality of spatial positions of the position indicator received while the position indicator is moved so as to identify a portion of the surface of the physical object that is smaller than the surface of the physical object that does not have a sensor for detecting the position indicated by the position indicator under the surface and on the surface of the physical object in the three-dimensional space. Based on the plurality of coordinates of the position indicator obtained while the position indicator is being moved, and the one or more signals indicating the plurality of spatial positions of the position indicator, determine whether the position indicator is on or above the portion of the surface of the physical object identified by the movement of the position indicator. In response to determining that the position indicator is on or above the portion of the surface of the physical object, obtain a coordinate group corresponding to an input gesture based on the one or more signals indicating the plurality of spatial positions of the position indicator. Render the input gesture in a virtual environment based on the coordinate group corresponding to the input gesture. System.

17. The one or more memory devices store instructions for causing the system to display a virtual representation of the position indicator together with a virtual representation of the portion of the surface of the physical object. The system according to claim 16.

18. The one or more memory devices cause the system to obtain an indication of magnification, and obtain a coordinate group corresponding to a scaled input gesture based on the magnification and the coordinate group corresponding to the input gesture. store instructions. The system according to claim 16.

19. The one or more memory devices store instructions for causing the system to display a virtual representation of the scaled input gesture. The system according to claim 18.

Images