System and method for dynamic shape sketching

Abstract

A system includes a position indicator and a processing device that enable a user to intuitively and dynamically specify a size and shape of an object. The position indicator sends a signal indicative of a pressure applied to a portion of the position indicator. The processing device receives the signal indicative of the pressure applied to the portion of the position indicator and a signal indicative of a respective position of the position indicator. The processing device generates visualization data based on the signal indicative of the pressure applied to the portion of the position indicator and the signal indicative of the respective position of the position indicator. The visualization data defines an object extending from a predetermined position in a direction based on the signal indicative of the pressure applied to the portion of the position indicator.

Description

background Technical Field

[0002] This disclosure relates to specifying the dimensions of multidimensional objects represented by digital data, and more specifically, to systems and methods for dynamically sketching the shapes of such multidimensional objects based on intuitive user operations using a position indicator as an input device. Background Technology

[0004] Typically, users must utilize multiple input devices to manipulate a complex set of objects to specify the dimensions of a multidimensional object represented by digital data. For example, a conventional system that allows a user to specify the shape of a multidimensional object represented by digital data might require the user to operate one or more keys on a keyboard with one hand while simultaneously moving and manipulating computer mouse buttons with the other hand to specify shape, orientation, size, etc. Therefore, it is desirable to provide a system and method that allows users to intuitively specify the shape, orientation, size, etc., of a multidimensional object represented by digital data using a single input device. Summary of the Invention

[0005] This public teaching enables users to intuitively and dynamically specify the shape, orientation, size, etc. of multidimensional objects represented by digital data using a single input device.

[0006] A system according to a first embodiment of the present disclosure can be summarized as including: a position indicator comprising: a housing having a plurality of reference tags disposed on an outer surface of the housing; a core disposed within the housing and having a tip extending from the housing through an opening in the housing; a pressure detector for detecting pressure applied to the tip of the core during operation; and a transmitter coupled to the pressure detector, wherein the transmitter transmits one or more signals indicating the pressure applied to the tip of the core during operation; and a processing device comprising: at least one receiver for receiving, during operation, one or more signals indicating the pressure applied to the tip of the core of the position indicator and one or more signals indicating one or more corresponding positions of one or more of the reference tags; at least one processor coupled to at least one receiver; and at least one memory device storing instructions that, when executed by the at least one processor, cause the processing device to generate visualization data based on one or more signals indicating the pressure applied to the tip of the core of the position indicator and one or more signals indicating one or more corresponding positions of one or more of the reference tags, wherein the visualization data describes an object extending from a predetermined position in a direction based on one or more signals indicating the pressure applied to the tip of the core of the position indicator, and wherein the visualization data is provided for display by a visualization device.

[0007] When one or more signals indicating pressure applied to the tip of the position indicator core indicate that the pressure is greater than a predetermined threshold, instructions stored in at least one memory device, when executed by at least one processor, can cause the processing device to generate visualization data such that the object extends from a predetermined position in a first predetermined direction when displayed by the visualization device.

[0008] When one or more signals indicating pressure applied to the tip of the position indicator core indicate that the pressure is less than a predetermined threshold, instructions stored in at least one memory device, when executed by at least one processor, can cause the processing device to generate visualization data such that an object, when displayed by the visualization device, extends from a predetermined position in a second predetermined direction, which is opposite to the first predetermined direction.

[0009] The processing device may include a sensor having an input surface, which can detect a position indicator during operation and output a signal indicating the position of the position indicator on the input surface when sensed by the sensor, and instructions stored by at least one memory device, when executed by at least one processor, can cause the processing device to generate visualization data based on one or more signals indicating pressure applied to the tip of the core of the position indicator, one or more signals indicating one or more corresponding positions of one or more reference tags, and a signal indicating the position of the position indicator on the input surface when sensed by the sensor.

[0010] The position indicator may include a switch that is in one of a plurality of positions during operation; one or more signals transmitted by a transmitter may indicate the pressure applied to the tip of the core and the position of the switch; and instructions stored by at least one memory device, when executed by at least one processor, may enable a processing device to generate visualization data based on one or more signals indicating the pressure applied to the tip of the core of the position indicator and the position of the switch, and one or more signals indicating the corresponding positions of one or more reference tags.

[0011] The position indicator may include an accelerometer that outputs a signal indicating the acceleration of the position indicator during operation; one or more signals transmitted by a transmitter may indicate the pressure applied to the tip of the core and the acceleration of the position indicator; and instructions stored by at least one memory device, when executed by at least one processor, may enable a processing device to generate visualization data based on one or more signals indicating the pressure applied to the tip of the core of the position indicator and the acceleration of the processing device, and one or more signals indicating the corresponding positions of one or more reference tags.

[0012] The system according to the second embodiment of this disclosure can be summarized as including: a position indicator comprising: a housing having a core disposed within the housing and having a tip extending from the housing through an opening therethrough; a pressure detector for detecting pressure applied to the tip of the core during operation; and a transmitter coupled to the pressure detector, wherein the transmitter transmits one or more signals during operation indicating the pressure applied to the tip of the core detected by the pressure detector; and a processing device comprising: a sensor having an input surface, wherein the sensor detects the position indicator during operation and outputs a signal indicating the position of the tip of the core relative to the input surface of the sensor; and at least one receiver for operation. The device receives one or more signals indicating pressure applied to the tip of a position indicator core; at least one processor coupled to a sensor and at least one receiver; at least one memory device storing instructions that, when executed by the at least one processor, cause the processing device to generate visualization data based on signals indicating the position of the tip of the position indicator core on an input surface and one or more signals indicating pressure applied to the tip of the position indicator core, wherein the visualization data describes an object extending from a predetermined position in a direction based on one or more signals indicating pressure applied to the tip of the position indicator core, and wherein the visualization data is provided for display by a visualization device.

[0013] When one or more signals indicating pressure applied to the tip of the position indicator core indicate that the pressure is greater than a predetermined threshold, instructions stored in at least one memory device, when executed by at least one processor, can cause the processing device to generate visualization data such that the object extends from a predetermined position in a first predetermined direction when displayed by the visualization device.

[0014] When one or more signals indicating pressure applied to the tip of the position indicator core indicate that the pressure is less than a predetermined threshold, instructions stored in at least one memory device, when executed by at least one processor, can cause the processing device to generate visualization data such that an object, when displayed by the visualization device, extends from a predetermined position in a second predetermined direction, which is opposite to the first predetermined direction.

[0015] The processing device may include a switch that is in one of a plurality of positions during operation; one or more signals transmitted by a transmitter that may indicate the pressure applied to the tip of the core and the position of the switch; and instructions stored by at least one memory device that, when executed by at least one processor, may enable the processing device to generate visualization data based on signals indicating the position of the tip of the core relative to the input surface of a sensor, and one or more signals indicating the pressure applied to the tip of the core of a position indicator and the position of the switch.

[0016] The processing device may include an accelerometer that outputs a signal indicating the acceleration of the processing device during operation; one or more signals transmitted by a transmitter may indicate the pressure applied to the tip of the core and the acceleration of the processing device; and instructions stored in at least one memory device, when executed by at least one processor, may enable the processing device to generate visualization data based on a signal indicating the position of the tip of the core relative to the input surface of a sensor, and one or more signals indicating the pressure applied to the tip of the core to a position indicator and the acceleration of the processing device.

[0017] The method according to the third embodiment of this disclosure can be summarized as including: receiving one or more signals indicating one or more spatial positions of a position indicator in three-dimensional space relative to a surface of a sensor; receiving a signal indicating pressure applied to a tip of a core of the position indicator; generating visualization data based on the one or more signals indicating one or more positions of the position indicator and the signal indicating pressure applied to the tip of the core of the position indicator, wherein the visualization data describes an object that, when displayed, extends in a direction away from the surface of the sensor based on the signal indicating pressure applied to the tip of the core of the position indicator; and providing the visualization data for display.

[0018] When a signal indicating the pressure applied to the tip of the position indicator's core indicates that the pressure is greater than a predetermined threshold, the object can extend from the predetermined position in a first predetermined direction.

[0019] When a signal indicating the pressure applied to the tip of the position indicator's core indicates that the pressure is less than a predetermined threshold, the object can extend from the predetermined position in a second predetermined direction, which is opposite to the first predetermined direction.

[0020] The method may further include: receiving a signal indicating the position of a switch of a position indicator, wherein the generation of visualization data includes generating visualization data based on (i) a signal indicating pressure applied to the tip of the core of the position indicator, (ii) one or more signals indicating one or more spatial positions of the position indicator, and (iii) a signal indicating the position of the switch of the position indicator.

[0021] The method may further include: receiving a signal indicating the acceleration of a position indicator, wherein the generation of visualization data includes generating visualization data based on (i) a signal indicating pressure applied to the tip of the core of the position indicator, (ii) one or more signals indicating one or more spatial positions of the position indicator, and (iii) a signal indicating the acceleration of the indicator.

[0022] One or more signals indicating one or more spatial locations of a location indicator may include one or more signals indicating the corresponding locations of one or more of a plurality of reference labels set on the location indicator.

[0023] One or more signals indicating one or more spatial positions of a position indicator may include signals indicating the position of the tip of the position indicator relative to the surface of the sensor.

[0024] The method may further include: displaying a representation of the object based on a visual data display. The representation of the object may be displayed at least partially by a head-mounted display. Attached Figure Description

[0025] Figure 1 A block diagram of a visualization system according to one or more embodiments of the present disclosure is shown;

[0026] Figure 2 A block diagram is shown illustrating a position indicator used as an input device according to one or more embodiments of the present disclosure;

[0027] Figure 3 The illustration shows one or more embodiments of the present disclosure, via Figure 2 The block diagram shown is of a processing device that receives input from a position indicator.

[0028] Figure 4 This illustrates one or more embodiments that, according to this disclosure, can be derived from... Figure 1 The flowchart shows the method used by the visualization system.

[0029] Figure 5 This illustrates one or more embodiments that, according to this disclosure, can be derived from... Figure 1 The flowchart shows the method used by the visualization system.

[0030] Figure 6A It can be shown that it can be made by Figure 1 The visualization system shown displays a 3D view of the object.

[0031] Figure 6B One or more embodiments according to this disclosure are shown. Figure 6A Side view of the object shown;

[0032] Figure 7A It can be shown that it can be made by Figure 1 The visualization system shown displays a 3D view of the object; and

[0033] Figure 7B One or more embodiments according to this disclosure are shown. Figure 7A The side view of the object shown. Detailed Implementation

[0034] Figure 1 A block diagram of a visualization system 100 according to one or more embodiments of the present disclosure is shown. The visualization system 100 includes a position indicator 102, a processing device 104, a plurality of tracking devices 106a and 106b, a visualization device 108, and a sensor 109.

[0035] In the illustrated embodiment, the position indicator 102 includes a hollow, typically cylindrical housing 110 with an opening 112 at one end; however, the housing of the position indicator 102 may have other different forms. The tip of a core 114 extends from the housing 110 through the opening 112. In one or more embodiments, the core 114 is a rod-like member that sends pressure corresponding to the pressure applied to a portion of the position indicator (e.g., the tip of the core 114) to the following reference. Figure 2 The pressure detector 118 is described above. In one or more embodiments, the core 114 is formed of a conductive material. In one or more embodiments, the core 114 is non-conductive and formed of resin.

[0036] Alternatively or in combination, in one or more embodiments, an opening 112 is formed on the side of the housing 110, through which the core 114 extends, thereby allowing a user's finger to apply pressure to the core, thus providing input to the processing device 104. See below for reference. Figure 2 The position indicator 102 sends a signal to the processing device 104 indicating the amount of pressure applied to the tip of the core 114. The position indicator 102 can be used as an input device for the processing device 104.

[0037] The processing device 104 includes an input surface 116 formed of, for example, a transparent material such as glass. In one or more embodiments, the processing device 104 is a tablet computer. See below for reference. Figure 3 The sensor 140 and display device 138 can be positioned below the input surface 116, with the sensor 140 tracking the current position of the position indicator 102. The processing device 104 generates visualization data based on user actions on the position indicator 102 and sends the visualization data to the visualization device 108, which displays an image based on the visualization data. Alternatively or additionally, the display device 138 of the processing device 104 can display an image based on the visualization data.

[0038] In one or more embodiments, the visualization device 108 and the display device 138 process multiple portions of the visualization data generated by the processing device 104, respectively, and simultaneously display the image. In one or more embodiments, the visualization device 108 and the display device 138 operate at different screen refresh rates. Therefore, it may be necessary to transfer processing from the device operating at a higher screen refresh rate to the device operating at a lower screen refresh rate. For example, the visualization device 108 may operate at a 90 Hz screen refresh rate, and the display device 138 may operate at a 60 Hz screen refresh rate. In this case, it may be necessary to transfer some or all of the processing of the visualization data by the visualization device 108 to the display device 138. Therefore, the processing device 104 may segment the visualization data, thereby transferring the processing load of the visualization device 108 to the display device 138.

[0039] In one or more embodiments, the processing device 104 receives a signal from the visualization device 108 indicating the current processing load of the visualization device 108, and the processing device 104 dynamically adjusts the amount of visualization data sent to the visualization device 108 and the display device 138 based on the current processing load. In one or more embodiments, the processing device 104 estimates the current processing load of the visualization device 108 and dynamically adjusts the amount of visualization data sent to the visualization device 108 and the display device 138 based on the estimated current processing load. For example, if the indicated or estimated current processing load of the visualization device 108 is greater than or equal to a predetermined threshold, the processing device 104 reduces the amount of visualization data sent to the visualization device 108 and increases the amount of visualization data sent to the display device 138. Additionally or alternatively, the processing device 104 may transfer processing from the display device 138 to the visualization device 108 in a similar manner.

[0040] Tracking devices 106a and 106b track the position and / or orientation of position indicator 102, specifically, in some embodiments, the tip of the core 114 of position indicator 101. Tracking devices 106a and 106b are collectively referred to herein as tracking device 106. Although Figure 1 The illustrated embodiment includes two tracking devices 106, but the visualization system 100 may include a different number of tracking devices 106 without departing from the scope of this disclosure. For example, according to this disclosure, the visualization system 100 may include three, four or more tracking devices 106. In one or more embodiments, the visualization system 100 does not include any tracking devices 106 and uses only the sensor 140 of the processing device 104 to track the position of the tip of the core 114 of the position indicator 102.

[0041] In one or more embodiments, the tracking device 106 employs known optical motion tracking techniques to track the position and / or orientation of the tip of the core 114 of the position indicator 102. In one or more embodiments, the position indicator 102 has reference tags in the form of optical markers mounted on the outer surface of the housing 110, wherein the optical markers are passive devices, each marker having a unique, visually distinguishable color or pattern formed thereon that can be sensed optically. Each tracking device 106 may include a camera that acquires images of one or more optical markers and sends the corresponding image data to a processing device 104. The processing device 104 stores data indicating the spatial relationship between each optical marker and the tip of the core 114 of the position indicator 102, and determines the current position and / or orientation of the tip of the core 114 of the position indicator 102 by processing the image data according to known techniques. In one or more embodiments, the optical markers are active devices, each optical marker having a light-emitting device (e.g., a light-emitting diode) that emits light of a different wavelength. In one or more embodiments, the tracking device 106 is a constellation sensor, which is part of the Oculus Rift system available from Oculus VR. In one or more embodiments, the tracking device 106 is a laser-based tracking device. For example, the tracking device 106 is a SteamVR 2.0 base station, which is part of the HTC Vive system available from HTC Corporation.

[0042] The visualization device 108 processes the visualization data generated by the processing device 104 and displays the corresponding image. In one or more embodiments, the visualization device 108 is a head-mounted display device. In one or more embodiments, the visualization device 108 is an HTC Vive Pro virtual reality headset, which is part of the HTC Vive system available from HTC Corporation. In one or more embodiments, the visualization device 108 is an Oculus Rift virtual reality headset, which is part of the Oculus Rift system available from Oculus VR. In one or more embodiments, the visualization device 108 is a HoloLens augmented reality headset available from Microsoft Corporation.

[0043] In one or more embodiments, the visualization device 108 includes a sensor 109 for tracking the position of physical objects within its field of view. For example, the visualization device 108 is a head-mounted display and the sensor 109 includes a pair of cameras, each positioned near one eye of a user of the visualization device 108 and having substantially the same field of view as that eye. Furthermore, the visualization device 108 includes a transmitter that sends image data corresponding to images captured by the cameras to a processing device 104, which processes the image data, for example, using conventional image processing techniques, and determines the coordinates of the objects imaged by the cameras. For example, in one or more embodiments, the processing device 104 includes object recognition software configured in a manner similar to the object recognition engine described in U.S. Patent Application Publication No. 2012 / 0206452 (see, for example, paragraph 87, the entire contents of which are incorporated herein by reference). Alternatively, the visualization device 108 includes a processor and a memory storing instructions that, when executed by the processor, cause the visualization device 108 to determine the coordinates of objects imaged by the cameras and send these coordinates to the processing device 104.

[0044] An overview of the visualization system 100 has already been provided; please refer to the following: Figure 2 The position indicator 102 will be described in more detail. Figure 2 A block diagram of a position indicator 102 according to one or more embodiments of the present disclosure is shown. The position indicator 102 includes a pressure detector 118 that detects pressure applied to the tip of a core 114 during operation, such as when a user presses the tip of the core 114 against the input surface 116 of the processing device 104. In one or more embodiments, the pressure detector 118 is configured in a manner similar to that described in U.S. Patent No. 9,939,931 (see, for example, columns 13, lines 49 through 22, lines 13, the entire contents of which are incorporated herein by reference).

[0045] In one or more embodiments, the position indicator 102 includes a switch 120 that operates in one of a plurality of positions. To provide input to the processing device 104, a user can activate the switch 120 to change its position. For example, when the user presses the switch 120, the switch 120 is in a “closed” or “on” position, and when the user does not press the switch, the switch is in an “open” or “off” position. In one or more embodiments, the switch 120 is configured in a manner similar to a side switch as described in U.S. Patent No. 9939931 (see, for example, column 11, lines 24-49). In one or more embodiments, the position indicator 102 includes two switches 120 that a user can operate to provide input, similar to input provided by operating the left and right buttons of a computer mouse.

[0046] In one or more embodiments, the position indicator 102 includes an accelerometer 122 that outputs a signal indicating the acceleration of the position indicator 102 during operation. In one or more embodiments, the accelerometer 122 is configured as a micromechanical microelectromechanical system (MEMS).

[0047] Furthermore, the position indicator 102 includes a transmitter 124 coupled to a pressure detector 118, which transmits a signal during operation indicating pressure applied to the tip of the core 114 as detected by the pressure detector 118. In one or more embodiments, the transmitter 124 operates according to one or more Bluetooth communication standards. In one or more embodiments, the transmitter 124 operates according to one or more IEEE 802.11 family of communication standards. In one or more embodiments, the transmitter 124 electromagnetically senses signals via the tip of the core 114 and a sensor 140 of the processing device 104. In one or more embodiments, the transmitter 124 is coupled to a switch 120, and transmits a signal during operation indicating the position of the switch 120. In one or more embodiments, the transmitter 124 is coupled to an accelerometer 122, and transmits a signal during operation indicating the acceleration of the position indicator 102 detected by the accelerometer 122.

[0048] In one or more embodiments, the position indicator 102 includes a plurality of reference tags 126a, 126b, and 126c. Reference tags 126a, 126b, and 126c are collectively referred to herein as reference tag 126. The reference tag 126 is tracked by a tracking device 106. In one or more embodiments, the reference tag 126 is a passive optical marker fixed to the outer surface of the housing 110 of the position indicator 102, as described above. Figure 1 The aforementioned. Alternatively or additionally, in one or more embodiments, the reference tag 126 actively emits light or radio waves that are detected by the tracking device 106. Although Figure 2 The illustrated embodiment includes three reference tags 126, but the location indicator 102 may include a different number of reference tags 126. For example, according to this disclosure, the location indicator 102 may include four, five, six or more reference tags 126.

[0049] The position indicator 102 has been described in more detail, and reference is made below to one or more embodiments of this disclosure. Figure 3 The processing device 104 will be described in more detail. Figure 3A block diagram of the processing device 104 is shown. The processing device 104 includes a microprocessor 128 having a memory 130 and a central processing unit (CPU) 132, a memory 134, an input / output (I / O) circuit 136, a display device 138, a sensor 140, a transmitter 142, and a receiver 144.

[0050] The memory 134 stores processor-executable instructions that, when executed by the CPU 132, cause the processing device 104 to engage. Figure 4 , Figure 6A , Figure 6B The operation of the processing device 104 shown in FIG. 7. The CPU 132 uses the memory 130 as working memory when executing instructions. In one or more embodiments, the memory 130 includes one or more random access memory (RAM) modules and / or one or more non-volatile random access memory (NVRAM), such as electrically erasable programmable read-only memory (EEPROM) or flash memory modules.

[0051] In one or more embodiments, the I / O circuitry 136 may include buttons, switches, dial pads, knobs, microphones, or other user interface elements for inputting commands to the processing device 104. Furthermore, the I / O circuitry 136 may include one or more speakers, one or more light-emitting devices, or other user interface elements for outputting information or instructions from the processing device 104.

[0052] Display device 138 displays information to the operator graphically. Microprocessor 128 controls display device 138 to display information based on visualization data generated by processing device 104. In one or more embodiments, display device 138 is a liquid crystal display (LCD) device. In one or more embodiments, display device 138 displays two images simultaneously, enabling a user wearing appropriate glasses to perceive a multi-dimensional image, for example, in a manner similar to viewing a three-dimensional (3D) image via a 3D-enabled television.

[0053] Sensor 140 detects position indicator 102 and outputs a signal indicating the position of position indicator 102 relative to the input surface (e.g., surface 116) of sensor 140. In one or more embodiments, microprocessor 128 processes the signal received from sensor 140 and obtains the (X, Y) coordinates on the input surface of sensor 140 corresponding to the position indicated by position indicator 102. In one or more embodiments, microprocessor 128 processes the signal received from sensor 140 and obtains the (X, Y) coordinates on the input surface of sensor 140 corresponding to the position indicated by position indicator 102, as well as the height (e.g., Z coordinate) above the input surface of sensor 140 where position indicator 102 is located. In one or more embodiments, sensor 140 is an inductive sensor configured in a manner similar to that described in U.S. Patent No. 9,964,395 (see, for example, columns 7, lines 35 through 10, lines 27, the entire contents of which are incorporated herein by reference). In one or more embodiments, sensor 140 is a capacitive sensor configured in a manner similar to that of a position detection sensor as described in U.S. Patent No. 9600096 (see, for example, columns 6, lines 5 through 8, lines 17, the entire contents of which are incorporated herein by reference).

[0054] Transmitter 142 is coupled to microprocessor 128, and in operation, transmitter 142 transmits visualization data generated by microprocessor 128 to visualization device 108. For example, in one or more embodiments, transmitter 142 operates according to one or more of Bluetooth and / or IEEE 802.11 family of communication standards. Receiver 144 is coupled to microprocessor 128, and in operation, receiver 144 receives signals from tracking device 106 and visualization device 108. For example, in one or more embodiments, receiver 144 operates according to one or more of Bluetooth and / or IEEE 802.11 family of communication standards. In one or more embodiments, receiver 144 receives signals from position indicator 102. In one or more embodiments, receiver 144 is included in sensor 140 and receives one or more signals from the tip of core 114 of position indicator 102 via electromagnetic induction.

[0055] The structure of the visualization system 100 has already been described; the following section will combine... Figure 4 An example describing method 200 performed by visualization system 100, Figure 4 A flowchart of a method 200 according to one or more embodiments of the present disclosure is shown. For example, when the processing device 104 is powered on, method 200 begins at 202.

[0056] At 202, one or more signals indicating one or more positions of position indicator 102 are received. For example, receiver 144 of processing device 104 receives one or more signals from tracking device 106. Alternatively or additionally, microprocessor 128 receives one or more signals from sensor 140 of processing device 104. Method 200 then proceeds to 204.

[0057] At 204, a signal indicating the position of switch 120 of position indicator 102 is received. For example, receiver 144 of processing device 104 receives the signal indicating the position of switch 120 from transmitter 124 of position indicator 102. Then, method 200 proceeds to 206.

[0058] Optionally, at 206, a signal indicating the acceleration of the position indicator 102 is received. For example, the receiver 144 of the processing device 104 receives the signal indicating the acceleration of the position indicator 102 from the transmitter 124 of the position indicator 102. Then, method 200 proceeds to 208.

[0059] At 208, a signal indicating the pressure applied to the tip of the core 114 is received. For example, the receiver 144 of the processing device 104 receives the signal indicating the pressure applied to the tip of the core 114 from the transmitter 124 of the position indicator 102. Alternatively or additionally, the sensor 140 of the processing device 104 receives the signal indicating the pressure applied to the tip of the core 114 from the tip of the core 114 of the position indicator 102 via electromagnetic induction. Then, method 200 proceeds to 210.

[0060] At 210, one or more signals indicating one or more physical objects located near the user of the visualization system 100 are received. In one or more embodiments, the receiver 144 of the processing device 104 receives signals indicating one or more physical objects located near the user from the sensor 109 of the visualization device 108. For example, the receiver 144 receives image data generated by a pair of cameras of the sensor 109, the microprocessor 128 processes the image data and obtains coordinates corresponding to the outer surface of the object imaged by the camera. Then, method 200 proceeds to 212.

[0061] At 212, signals received at 202, 204, 206, 208, and 210 are processed. In one or more embodiments, data transmitted by these signals is timestamped and stored in the memory 130 of the processing device 104, and the CPU 132 processes the data chronologically based on the timestamps associated with the data. At 212, [further processing can be performed]. Figure 5 The corresponding processing for the flowchart shown is as follows. Then, method 200 proceeds to 214.

[0062] At 214, it is determined whether a termination instruction has been received. For example, microprocessor 128 determines whether position indicator 102 has been used to select a predetermined icon or object displayed by display device 138 of processing unit 104. As another example, microprocessor 128 determines whether a voice command corresponding to a termination operation has been received at 214. If it is determined that a termination operation has been received at 214, method 200 terminates. Otherwise, method 200 returns to 202.

[0063] Figure 5 A flowchart is shown of method 300, which may be performed by visualization system 100 at 212 of method 200 described above, according to one or more embodiments of this disclosure. As described below, method 300 provides a "squeeze" operation that results in a specific visual display. Method 300 begins at 302 in response to microprocessor 128 determining that an instruction to perform a squeeze operation has been received. For example, microprocessor 128 determines that position indicator 102 has been used to select a predetermined icon or object displayed by display device 138 of processing device 104. As another example, method 300 begins at 302 in response to microprocessor 128 determining that a voice command corresponding to the instruction to perform a squeeze operation has been received.

[0064] At 302, the pressure applied to the tip of core 114 is compared with a threshold pressure value. In the illustrated embodiment, it is determined whether the pressure applied to the tip of core 114 is greater than or equal to the threshold pressure value. For example, memory 134 stores a predetermined threshold pressure value, and microprocessor 128 determines whether the pressure applied to the tip of core 114, indicated by the signal received at 208 of method 200 described above, is greater than or equal to the threshold pressure value. If it is determined at 302 that the pressure applied to the tip of core 114 is greater than or equal to the threshold pressure value, then method 300 proceeds to 304. Otherwise, method 300 proceeds to 306.

[0065] In one or more embodiments, a user may instruct the processing device 104 to perform a squeezing operation by relatively slowly lifting the tip of the core 114 away from the input surface 116 of the processing device 104, rather than to perform another input operation at a different portion of the input surface 116, and relatively quickly lifting the position indicator 102 from the input surface 116. Therefore, at 302 of method 300, an additional determination can be made regarding whether the acceleration of the position indicator 102 is less than a threshold acceleration value. For example, memory 134 stores a predetermined threshold acceleration value, and microprocessor 128 determines whether the acceleration of the position indicator 102 indicated by the signal received at 206 of method 200 is greater than zero and less than or equal to the threshold acceleration value. If at 302 it is not determined that the acceleration of the position indicator 102 is greater than zero and less than or equal to the threshold acceleration value (and the pressure applied to the tip of the core 114 is determined to be greater than or equal to a threshold pressure value), then method 300 proceeds to 304. Otherwise, method 300 proceeds to 306.

[0066] At 304, visualization data describing an object extending from a predetermined position in a first direction is generated. For example, the predetermined position corresponds to a plane with a Z-coordinate of zero, such as the input surface 116 of the processing device 104, and the first direction corresponds to increasing a negative Z-coordinate value orthogonal to the plane of the input surface 116. Then, method 300 proceeds to 308.

[0067] At 306, visualization data describing an object extending from a predetermined position in a second direction is generated. For example, the predetermined position corresponds to a plane with a Z-coordinate of zero, such as the input surface 116 of the processing device 104, and the second direction corresponds to increasing the positive Z-coordinate value orthogonal to the plane orthogonal to the input surface 116. Then, method 300 proceeds to 308.

[0068] At 308, the visualization data generated at 304 or 306 is stored. For example, the microprocessor 128 of the processing device 104 stores the visualization data in memory 134. Then, method 300 proceeds to 310.

[0069] At 310, the visualization data generated at 304 or 306 is transmitted. In one or more embodiments, the microprocessor 128 of the processing device 104 causes the transmitter 142 to transmit the visualization data to the visualization device 108. In one or more embodiments, the microprocessor 128 transmits the visualization data to the display device 138 of the processing device 104. Then, method 300 proceeds to 312.

[0070] At 312, the visualization data is processed and the object is displayed based on the visualization data. In one or more embodiments, the visualization device 108 renders a two-dimensional image to obtain a three-dimensional (3D) representation of the object described by the visualization data. In one or more embodiments, the visualization device 108 renders a two-dimensional image to obtain a two-and-a-half-dimensional (2.5D) representation of the object described by the visualization data, wherein the 3D environment of an observer viewing the output of the visualization device 108 is projected onto a 2D plane of the observer's retina. In one or more embodiments, the microprocessor 128 causes the display device 138 of the processing device 104 to render the visualization data and display the object. Then, method 300 ends.

[0071] Figure 6A A perspective view of object 146 that may be displayed at 312 of method 300 according to one or more embodiments of the present disclosure is shown. Figure 6B Show Figure 6A The side view of object 146 shown.

[0072] In the illustrated example, suppose the user of the visualization system 100 holds the position indicator 102 with their hand 148 and traces the outline of the square 150 on the input surface 116 of the processing device 104 using the tip of the core 114. At 202 of the above method 200, the processing device 104 receives one or more signals indicating the corresponding position of the position indicator 102. Furthermore, suppose the user instructs the processing device 104 to perform a squeezing operation based on the square 150 by holding the position indicator 102 positioned above the outline of the square 150 while moving the switch 120 of the position indicator 102 to a closed or open position. At 204 of the above method 200, the processing device 104 receives a signal indicating the position of the switch 120. Furthermore, suppose the user indicates the direction of the squeezing operation by pressing the tip of the core 114 downwards onto the input surface 116 of the processing device 104. At 208 of the above method 200, the processing device 104 receives a signal indicating the pressure applied to the tip of the core 114.

[0073] Furthermore, assuming that because it is determined at 302 of method 300 that the pressure applied to the tip of core 114 is greater than or equal to a threshold, at 304 of method 300, processing device 104 generates visualization data, wherein object 146 extends downward from the plane corresponding to input surface 116 of processing device 104 in the direction away from the user. The extent to which object 146 extends downward is based on the magnitude of the pressure applied to the tip of core 114 and / or the amount of time the user applies pressure to the tip of core 114. That is, the greater the pressure applied to the tip of core 114 by the user, the greater the downward distance of object 146. Similarly, the longer the user applies pressure to the tip of core 114, the greater the downward distance of object 146.

[0074] For example, suppose the input surface 116 corresponds to a Z-coordinate of zero, the Z-coordinate increases with the distance above the input surface 116, and decreases with the distance below the input surface. Furthermore, suppose the microprocessor 128 generates coordinates for the object 146 such that the X and Y coordinates of the object 146 correspond to the corresponding X and Y coordinates of the shape 150, and the Z-coordinate of the object 146 ranges from 0 to a negative value corresponding to the magnitude of the pressure applied to the tip of the core 114. Therefore, when the visualization data generated by the processing device 104 at 304 of the above method 300 is displayed by the visualization device 108 at 312 of the above method 300, the object 146 is displayed as extending downwards from the surface of the input surface 116 corresponding to the processing device 104 in the direction away from the user, such as... Figure 6B As shown.

[0075] Figure 7A A perspective view of an object 152 that may be displayed at 312 in method 300 described above, according to one or more embodiments of the present disclosure. Figure 7B Show Figure 7A The side view of object 152 shown.

[0076] In the illustrated example, suppose the user of the visualization system 100 holds the position indicator 102 with their hand 148 and traces the outline of the square 154 on the input surface 116 of the processing device 104 using the tip of the core 114. At 202 of the method 200 described above, the processing device 104 receives one or more signals indicating the corresponding position of the position indicator 102. Furthermore, suppose the user instructs the processing device 104 to perform a squeezing operation based on the square 154 by holding the position indicator 102 positioned above the outline of the square 154 while moving the switch 120 of the position indicator 102 from an open or closed position to a closed or on position. At 204 of the method 200 described above, the processing device 104 receives a signal indicating the position of the switch 120.

[0077] Additionally or alternatively, suppose the user instructs the processing device 104 to perform a squeezing operation by relatively slowly moving the tip of the core 114 upward away from the input surface 116 of the processing device 104. At 206 of method 200 described above, the processing device 104 receives a signal indicating the acceleration of the position indicator 102. Furthermore, suppose that because it is determined at 302 of method 300 that the acceleration of the position indicator 102 is greater than zero and less than or equal to a threshold acceleration value, and it is not determined that the pressure applied to the tip of the core 114 is greater than or equal to a threshold pressure value, at 306 of method 300, the processing device 104 generates visualization data in which the object 152 extends upward from the plane corresponding to the input surface 116 of the processing device 104 in the direction toward the user. The extent to which the object 152 extends upward is based on the distance between the tip of the core 114 and the input surface 116 of the processing device 104. In other words, the greater the distance between the tip of the core 114 and the input surface 116 of the processing device 104, the greater the upward extension distance of the object 152.

[0078] For example, again assuming that input surface 116 corresponds to Z coordinate zero, the Z coordinate increases with the distance above input surface 116, and decreases with the distance below input surface 116. Furthermore, assuming that microprocessor 128 generates coordinates for object 152 such that the X and Y coordinates of object 152 correspond to the corresponding X and Y coordinates of square 154, and the Z coordinate of object 152 ranges from 0 to a positive value corresponding to the distance between the tip of core 114 and input surface 116 of processing device 104. Therefore, when the visualization data generated by processing device 104 at 306 of method 300 is displayed by visualization device 108 at 312 of method 300, object 152 is displayed as extending upwards from the surface corresponding to input surface 116 of processing device 104 in the direction toward the user, such as... Figure 7B As shown.

[0079] Through this invention, users can intuitively specify the shape, orientation, size, etc. of an object represented by digital data, and use the single input device to present the object in a multi-dimensional appearance above or below the plane of the input surface.

[0080] Further embodiments can be provided by combining the various embodiments described above. As necessary, various aspects of the embodiments can be modified to incorporate the concepts of the various patents referenced in this specification to provide further embodiments.

[0081] Based on the detailed description above, these and other changes can be made to the embodiments. Generally, the terminology used in the following claims should not be construed as limiting the claims to the specific embodiments disclosed in the specification and claims, but rather as encompassing all possible embodiments and the full scope of equivalents claimed by those claims. Therefore, the claims are not limited to this disclosure.

Claims

1. A system comprising: A position indicator, the position indicator comprising: A housing having a plurality of reference labels disposed on its outer surface; A core, the core being disposed within the housing and having a tip extending from the housing through an opening in the housing; A pressure detector that detects the pressure applied to the tip of the core during operation; and A transmitter coupled to the pressure detector, wherein the transmitter, in operation, transmits one or more signals indicating the pressure applied to the tip of the core; and Processing apparatus, the processing apparatus comprising: At least one receiver, wherein the at least one receiver receives, in operation, one or more signals indicating the pressure applied to the tip of the core of the position indicator and one or more signals indicating the corresponding positions of one or more of the reference tags; At least one processor, said at least one processor being coupled to said at least one receiver; At least one memory device stores instructions that, when executed by the at least one processor, cause the processing device to generate visualization data based on one or more signals indicating pressure applied to the tip of the core of the position indicator and one or more signals indicating one or more corresponding positions of one or more of the reference labels, wherein the visualization data describes an object extending from a predetermined position in a direction based on the one or more signals indicating pressure applied to the tip of the core of the position indicator, and wherein the visualization data is provided for display by a visualization device.

2. The system according to claim 1, wherein, When one or more signals indicating pressure applied to the tip of the core of the position indicator indicate that the pressure is greater than a predetermined threshold, the instructions stored in the at least one memory device, when executed by the at least one processor, cause the processing device to generate the visualization data such that the object extends from the predetermined position in a first predetermined direction when displayed by the visualization device.

3. The system according to claim 2, wherein, When one or more signals indicating pressure applied to the tip of the core of the position indicator indicate that the pressure is less than the predetermined threshold, the instructions stored in the at least one memory device, when executed by the at least one processor, cause the processing device to generate the visualization data such that the object, when displayed by the visualization device, extends from the predetermined position in a second predetermined direction, which is opposite to the first predetermined direction.

4. The system according to claim 1, wherein: The processing device includes a sensor with an input surface. The sensor detects the position indicator during operation and outputs a signal indicating the position of the position indicator on the input surface when sensed by the sensor. The instructions stored in the at least one memory device, when executed by the at least one processor, cause the processing device to generate the visualization data based on one or more signals indicating the pressure applied to the tip of the core of the position indicator, one or more signals indicating one or more corresponding positions of the reference tags, and a signal indicating the position of the position indicator on the input surface when sensed by the sensor.

5. The system according to claim 1, wherein: The position indicator includes a switch that is in one of a plurality of positions during operation; The one or more signals transmitted by the transmitter indicate the pressure applied to the tip of the core and the position of the switch; and The instructions stored in the at least one memory device, when executed by the at least one processor, cause the processing device to generate the visualization data based on one or more signals indicating the pressure applied to the tip of the core of the position indicator and the position of the switch, and one or more signals indicating one or more corresponding positions of one or more of the reference tags.

6. The system according to claim 1, wherein: The position indicator includes an accelerometer that outputs a signal indicating the acceleration of the position indicator during operation; The one or more signals transmitted by the transmitter indicate the pressure applied to the tip of the core and the acceleration of the position indicator; and When executed by the at least one processor, the instructions stored in the at least one memory device cause the processing device to generate the visualization data based on one or more signals indicating the pressure applied to the tip of the core of the position indicator and the acceleration of the processing device, and one or more signals indicating one or more corresponding positions of one or more of the reference tags.

7. A system comprising: A position indicator, the position indicator comprising: A housing having a core disposed within the housing and having a tip extending from the housing through an opening in the housing; A pressure detector that detects the pressure applied to the tip of the core during operation; and A transmitter coupled to the pressure detector, wherein the transmitter, in operation, transmits one or more signals indicative of the pressure applied to the tip of the core, as detected by the pressure detector; and Processing apparatus, the processing apparatus comprising: A sensor having an input surface, wherein the sensor detects the position indicator during operation and outputs a signal indicating the position of the tip of the core relative to the input surface of the sensor; At least one receiver, wherein the at least one receiver receives, in operation, one or more signals indicating the pressure applied to the tip of the core of the position indicator; At least one processor, said at least one processor being coupled to said sensor and said at least one receiver; At least one memory device stores instructions that, when executed by the at least one processor, cause the processing device to generate visualization data based on a signal indicating the position of the tip of the core of the position indicator on the input surface and one or more signals indicating the pressure applied to the tip of the core of the position indicator, wherein the visualization data describes an object extending from a predetermined position in a direction based on the one or more signals indicating the pressure applied to the tip of the core of the position indicator, and wherein the visualization data is provided for display by a visualization device.

8. The system according to claim 7, wherein, When one or more signals indicating pressure applied to the tip of the core of the position indicator indicate that the pressure is greater than a predetermined threshold, the instructions stored in the at least one memory device, when executed by the at least one processor, cause the processing device to generate the visualization data such that the object extends from the predetermined position in a first predetermined direction when displayed by the visualization device.

9. The system according to claim 8, wherein, When one or more signals indicating pressure applied to the tip of the core of the position indicator indicate that the pressure is less than the predetermined threshold, the instructions stored in the at least one memory device, when executed by the at least one processor, cause the processing device to generate the visualization data such that the object, when displayed by the visualization device, extends from the predetermined position in a second predetermined direction, which is opposite to the first predetermined direction.

10. The system according to claim 7, wherein: The processing device includes a switch, which is in one of a plurality of positions during operation; The one or more signals transmitted by the transmitter indicate the pressure applied to the tip of the core and the position of the switch; and When executed by the at least one processor, the instructions stored in the at least one memory device cause the processing device to generate the visualization data based on the signal indicating the position of the tip of the core relative to the input surface of the sensor, and one or more signals indicating the pressure applied to the tip of the core to the position indicator and the position of the switch.

11. The system according to claim 7, wherein: The processing device includes an accelerometer that outputs a signal indicating the acceleration of the processing device during operation; The one or more signals transmitted by the transmitter indicate the pressure applied to the tip of the core and the acceleration of the processing device; and The instructions stored in the at least one memory device, when executed by the at least one processor, cause the processing device to generate the visualization data based on one or more signals indicating the position of the tip of the core relative to the input surface of the sensor, the pressure applied to the tip of the core to the position indicator, and the acceleration of the processing device.

12. A method comprising: Receive one or more signals indicating one or more spatial positions of a position indicator relative to the surface of a sensor in three-dimensional space; Receive a signal indicating the pressure applied to the tip of the core of the position indicator; Visualization data is generated based on one or more signals indicating one or more positions of the position indicator and a signal indicating pressure applied to the tip of the core of the position indicator, wherein the visualization data describes an object that, when displayed, extends in a direction away from the surface of the sensor based on the signal indicating pressure applied to the tip of the core of the position indicator. as well as The visualization data is provided for display.

13. The method according to claim 12, wherein, When the signal indicating the pressure applied to the tip of the core of the position indicator indicates that the pressure is greater than a predetermined threshold, the object extends from the predetermined position in a first predetermined direction.

14. The method according to claim 13, wherein, When the signal indicating the pressure applied to the tip of the core of the position indicator indicates that the pressure is less than the predetermined threshold, the object extends from the predetermined position in a second predetermined direction, which is opposite to the first predetermined direction.

15. The method of claim 12, comprising: Receive a signal indicating the position of the switch of the position indicator. The generation of the visualization data includes generating the visualization data based on (i) a signal indicating the pressure applied to the tip of the core of the position indicator, (ii) one or more signals indicating one or more spatial positions of the position indicator, and (iii) a signal indicating the position of the switch of the position indicator.

16. The method of claim 12, comprising: Receive a signal indicating the acceleration of the position indicator. The generation of the visualization data includes generating the visualization data based on (i) a signal indicating the pressure applied to the tip of the core of the position indicator, (ii) one or more signals indicating one or more spatial positions of the position indicator, and (iii) a signal indicating the acceleration of the indicator.

17. The method according to claim 12, wherein, The one or more signals indicating one or more spatial locations of the location indicator include one or more signals indicating one or more corresponding locations of one or more of a plurality of reference labels disposed on the location indicator.

18. The method according to claim 12, wherein, The one or more signals indicating one or more spatial positions of the position indicator include signals indicating the position of the tip of the position indicator relative to the surface of the sensor.

19. The method of claim 12, further comprising: The object is represented based on the visualization data.

20. The method according to claim 19, wherein, The representation of the object is displayed at least partially by a head-mounted display.

Images