Input system and input method

Abstract

The present invention relates to an input system and an input method. One or more processors (56) with which an input system (10) is provided perform: a detection process for applying image processing for detecting a sensor device (30) to an image signal acquired through image capture by a camera (53), and acquiring a device state quantity indicating the position / orientation of the sensor device (30) in a display coordinate space (CS2); a calculation process for calculating a coordinate conversion rule for mapping from a sensor coordinate space (CS1) to the display coordinate space (CS2), using the acquired device state quantity; and a conversion process for converting a first pen state quantity that indicates the position / orientation of an electronic pen (20) in the sensor coordinate space (CS1) into a second pen state quantity that indicates the position / orientation of the electronic pen (20) in the display coordinate space (CS2), in accordance with the calculated coordinate conversion rule.

Description

Input system and input method

[0001] The present invention relates to an input system and an input method.

[0002] A digital ink system capable of generating digital ink describing a collection of strokes through handwriting operations using a pointing device, including an electronic pen, has been known. For example, a digital ink system may be used to display handwritten content in a location separate from the handwritten content, following the user's handwritten input.

[0003] Japanese Patent Application Laid-Open Publication No. 2013-125487 discloses a technology in which a two-dimensional virtual plane is set in a three-dimensional space, and then strokes handwritten on this virtual plane are instantly displayed on a display device that can be worn by the user.

[0004] JP 2013-125487 A

[0005] For example, by using a wearable display device such as AR (Augmented Reality) glasses to display various objects superimposed on the user's field of vision, the user can have a simulated experience of leaving handwriting in that space. However, in the technology disclosed in JP 2013-125487 A, since the virtual plane is set and then fixed, in some cases the handwritten content may be displayed in a location different from the user's intention, impairing the user's input operability.

[0006] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide an input system and input method that can improve input operability for users in an apparatus configuration in which a sensor device that detects the position and orientation of an electronic pen and a wearable display device that displays space are provided separately.

[0007] The input system of the present invention comprises an electronic pen, a sensor device including a planar pen sensor capable of detecting the position and orientation of the electronic pen, a wearable display device that can be worn over a user's eyes and is provided with a camera that captures an image of at least the space in front of the user, and one or more processors that perform display control on the wearable display device so as to display a virtual object within the display area of ​​the wearable display device, wherein the one or more processors perform image processing to detect the sensor device on an image signal acquired through imaging by the camera, and perform a detection process to acquire device state quantities that indicate the position and orientation of the sensor device in a display coordinate space defined for the display control, a calculation process to calculate a coordinate transformation rule for mapping from the sensor coordinate space defined by the sensor device to the display coordinate space using the device state quantities acquired through the detection process, and a conversion process to convert a first pen state quantity that indicates the position and orientation of the electronic pen in the sensor coordinate space into a second pen state quantity that indicates the position and orientation of the electronic pen in the display coordinate space in accordance with the calculated coordinate transformation rule.

[0008] The input method of the present invention is an input method using an input system including: an electronic pen; a sensor device including a planar pen sensor capable of detecting the position and orientation of the electronic pen; a wearable display device that can be worn over a user's eyes and is provided with a camera that captures an image of at least a space in front of the user; and one or more processors that perform display control on the wearable display device so as to display a virtual object within a display area of ​​the wearable display device, wherein the one or more processors perform image processing on an image signal acquired through imaging by the camera to detect the sensor device and acquire device state quantities that indicate the position and orientation of the sensor device in a display coordinate space defined for the display control; a calculation step that uses the device state quantities acquired through the detection step to calculate a coordinate transformation rule for mapping from a sensor coordinate space defined by the sensor device to the display coordinate space; and a conversion step that converts a first pen state quantity that indicates the position and orientation of the electronic pen in the sensor coordinate space into a second pen state quantity that indicates the position and orientation of the electronic pen in the display coordinate space in accordance with the calculated coordinate transformation rule.

[0009] According to the present invention, in an apparatus configuration in which a sensor device that detects the position and orientation of an electronic pen and a wearable display device that displays space are provided separately, it is possible to improve input operability for users.

[0010] 1 is an overall configuration diagram of an input system in one embodiment of the present invention. FIG. 2 is a block diagram showing an example of the configuration of the AR glasses in FIG. 1. FIG. 3 is a state transition diagram showing transitions of operation modes executed by the input system of FIG. 1. FIG. 4 is a diagram showing an example of a pick-up operation on a virtual object. FIG. 5 is a diagram showing an example of a drop-off operation on a virtual object. FIG. 6 is a diagram showing an example of the arrangement of a haptic device possessed by the electronic pen of FIG. 1. FIG. 7 is a diagram showing an example of the relationship between the movement of the electronic pen and the operation of the haptic device. FIG. 8 is a diagram showing the correspondence relationship between a sensor coordinate space and a display coordinate space. FIG. 9 is a flowchart related to the pen tracking operation by the AR glasses of FIG. 2. FIG. 10 is a diagram showing an example of a method for acquiring state quantities of each device on the sensor coordinate space or the display coordinate space. FIG. 11 is a diagram showing a second example of the correspondence relationship between the sensor coordinate space and the display coordinate space. FIG. 12 is a diagram showing an example of the display mode of a seamless object. FIG. 13 is a diagram showing an example of the result of an operation on a seamless object.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. To facilitate understanding of the description, the same components in each drawing will be denoted by the same reference numerals as much as possible, and duplicate explanations will be omitted. Furthermore, the term "part" may be replaced with other terms such as unit, module, device, or element.

[0012] [Configuration of Input System 10] Conventional digital handwriting systems have a problem in that the entire content cannot be seen when the content is enlarged due to the physical limitations of the display. Therefore, by expanding the display using AR (Augmented Reality) starting from the sensor surface, a new digital handwriting service can be provided that is not bound by the limitations of the display area of ​​physical displays. Furthermore, it is preferable to seamlessly track (or track) the electronic pen in order to operate this expanded display or to instruct other large display units. Therefore, we propose an input system 10 that can seamlessly track the position of the pen even in the blind spot of the AR glasses camera.

[0013] 1 is a diagram showing the overall configuration of an input system 10 according to one embodiment of the present invention. The input system 10 is a "digital ink system" that generates digital ink describing a collection of strokes through handwriting operations using an electronic pen 20 and stores the digital ink for use. Specifically, the input system 10 includes at least one electronic pen 20, at least one sensor device 30, and a server device 40.

[0014] The electronic pen 20 is a pointing device configured to be able to communicate unidirectionally or bidirectionally with the sensor device 30. The electronic pen 20 may be an active electrostatic coupling (AES) system or an electromagnetic induction (EMR) system. For example, the electronic pen 20 includes a pen pressure sensor 21, an inertial measurement unit (IMU) 22, a micro control unit (MCU) 23, and a communication chip 24. An operation switch 26 is provided on the side of a housing 25 of the electronic pen 20.

[0015] The writing pressure sensor 21 is, for example, a pressure sensor using a variable capacitance capacitor that detects a change in capacitance caused by pressure on the pen tip. The writing pressure sensor 21 can detect not only writing pressure but also pen events including pen-down and pen-up of the electronic pen 20.

[0016] The IMU 22 is a measurement unit that combines, for example, a triaxial gyro sensor and a triaxial acceleration sensor. This allows the IMU 22 to measure state quantities that indicate the state or time changes in the state of the device itself in a sensor coordinate space CS1 ( FIG. 8 ), which will be described later. These state quantities include various physical quantities used to identify the position and orientation, such as position, velocity, acceleration, jerk, angle, and angular velocity.

[0017] The MCU 23 is a control unit including a processor capable of controlling the operation of the electronic pen 20. For example, the MCU 23 performs various calculations related to the calculation of the position indicated by the electronic pen 20, as well as controls the transmission and reception of data including the results of these calculations. Furthermore, using the technology disclosed in Japanese Patent Application Laid-Open No. 2022-037377, the MCU 23 can convert the state quantities of the electronic pen 20 according to predetermined conversion rules and sequentially add the converted state quantities to determine the position and orientation in the sensor coordinate space CS1 ( FIG. 8 ).

[0018] The communication chip 24 is an integrated circuit for wireless communication with an external device in accordance with various wireless communication standards, including Bluetooth (registered trademark) or UWB (Ultra-Wide Band). This allows the electronic pen 20 to exchange various data with the sensor device 30 via the communication chip 24.

[0019] The sensor device 30 is a device that can detect the position indicated by the electronic pen 20, and may or may not have a display function. The user U can write pictures or characters within the display area by holding the electronic pen 20 in one hand and moving the pen tip while pressing it against the touch surface 31 of the sensor device 30.

[0020] The sensor device 30 is, for example, a computer owned by a user, such as a tablet terminal with or without a display function, a smartphone, or a personal computer. The sensor device 30 may also be a paper-like device specialized for pen detection function. Alternatively, the touch device 14 may be a device installed inside a building (e.g., home appliances, furniture, fixtures, etc.), or a device that constitutes part of a building (e.g., a wall, floor, window, pillar, etc.).

[0021] In the example of FIG. 1, the sensor device 30 includes a planar pen sensor 32 , a sensor IC (Integrated Circuit) 33 , a host processor 34 , and a communication chip 35 .

[0022] The pen sensor 32 is a planar sensor for detecting the position and orientation of the electronic pen 20. If the electronic pen 20 is an AES-type stylus, the pen sensor 32 is a capacitance-type touch sensor. If the electronic pen 20 is an EMR-type stylus, the pen sensor 32 is an EMR sensor having a plurality of detection coils arranged in a planar manner for detecting an alternating magnetic field.

[0023] The capacitance type touch sensor includes, for example, a plurality of X-line electrodes for detecting the position on the X axis of the sensor coordinate system and a plurality of Y-line electrodes for detecting the position on the Y axis. Instead of the mutual capacitance type sensor described above, the touch sensor may be a self-capacitance type sensor in which block-shaped electrodes are arranged in a two-dimensional lattice pattern.

[0024] The EMR sensor may be partially or entirely incorporated into other components constituting the sensor device 30. For example, the transmission coil (or power coil) of the EMR sensor may be integrally provided on the light source substrate of the display panel, and the detection coil of the EMR sensor may be integrally provided on the touch sensor.

[0025] The pen sensor 32 may be an "internal type" (further classified as an on-cell type or an in-cell type) sensor that is integrally formed with a display panel (not shown) such as a liquid crystal panel or an organic EL (Electro-Luminescence) panel, or an "external type" (or out-cell type) sensor that is attached to the display panel from the outside.

[0026] The sensor IC 33 is an integrated circuit that controls the driving of the pen sensor 32. The sensor IC 33 drives the pen sensor 32 based on a control signal supplied from the host processor 34. This causes the sensor IC 33 to perform a "pen detection function" that detects the state of the electronic pen 20. If the pen sensor 32 is a capacitive touch sensor, the sensor IC 33 also performs a "touch detection function" that detects the presence or absence and position of a touch by a passive pointer such as a user's finger.

[0027] The host processor 34 is configured by an arithmetic processing device including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an MPU (Micro-Processing Unit). The host processor 34 can execute various functions, including generating digital ink and controlling the transmission and reception of data, by reading and executing programs stored in a memory (not shown).

[0028] The communication chip 35 is an integrated circuit for wireless communication with an external device in accordance with various communication standards including Bluetooth (registered trademark) or UWB, similar to the communication chip 24. This allows the sensor device 30 to exchange various data with the electronic pen 20 or the server device 40 via the communication chip 35.

[0029] The server device 40 is a computer that performs overall control over the management of digital ink, and may be either a cloud-based or on-premise type. While the server device 40 is illustrated as a single computer, the server device 40 may instead be a group of computers that form a distributed system.

[0030] In addition to the electronic pen 20, sensor device 30, and server device 40 described above, the input system 10 further includes AR (Augmented Reality) glasses 50 (corresponding to a "wearable display device") that can be worn over the eyes of the user U. The AR glasses 50 are configured to be able to communicate with the electronic pen 20 or the sensor device 30 via wireless communication. The AR glasses 50 are also configured to be able to communicate with the server device 40 via a network NT. In this way, more advanced information input is performed through cooperation between the electronic pen system and the AR glasses 50.

[0031] <Block diagram of AR glasses 50> Figure 2 is a block diagram showing an example of the configuration of the AR glasses 50 in Figure 1. Specifically, the AR glasses 50 include a frame 51, a display panel 52, a pair of cameras 53, 53, an IMU 54, a communication chip 55, a control unit 56, and a storage unit 58.

[0032] The frame 51 is a housing portion excluding the lenses, and is configured to include, for example, rims, temples, etc. The type of the frame 51 may be a glasses type or a goggle type.

[0033] The display panel 52 is fitted into the lens portion of the frame 51 and displays an image or video superimposed on the space in front of the user's eyes. The display panel 52 may be, for example, a liquid crystal panel, a mini LED panel, a micro LED panel, an organic EL panel, or a quantum dot panel.

[0034] The pair of cameras 53 are imaging devices that generate signals for each frame by capturing images in a space and output them as image signals that indicate a time series of images. The pair of cameras 53 are fixed to the left and right rims of the frame 51, respectively, and capture images of at least the space in front of the user U when the user U is wearing the cameras. Each camera 53 is configured, for example, by a visible light camera, an infrared camera, a TOF (Time Of Flight) camera, or the like.

[0035] The IMU 54 is a measurement unit that combines, for example, a three-axis gyro sensor and a three-direction acceleration sensor, similar to the IMU 22 in Fig. 1. This allows the IMU 54 to measure a state quantity that indicates the state or time change of the state of the device itself in a display coordinate space CS2 (Fig. 8) described later.

[0036] 1, the communication chip 55 is an integrated circuit for wireless communication with an external device in accordance with various wireless communication standards including Bluetooth (registered trademark) or UWB. This allows the AR glasses 50 to exchange various data with the electronic pen 20, the sensor device 30, or the server device 40 via the communication chip 55.

[0037] The control unit 56 is configured with a processor including a CPU, a GPU, or an MPU. The control unit 56 reads and executes the programs and data stored in the storage unit 58, thereby functioning as a data acquisition unit 60, a tracking processing unit 62, an operation processing unit 64, a rendering processing unit 66, and an ink generation unit 68.

[0038] The data acquisition unit 60 performs an "acquisition process" to acquire various data necessary for inputting and outputting information. For example, the data acquisition unit 60 acquires image signals from the cameras 53 and measurement signals from the IMU 54. The data acquisition unit 60 also acquires data indicating the state of the electronic pen 20 (e.g., position, orientation, amount of writing pressure, on / off status of the switch 26, etc.) from the electronic pen 20 or the sensor device 30 via the communication chip 55. The data acquisition unit 60 also acquires digital ink 74 or object data 76 from the sensor device 30 or the server device 40 via the communication chip 55.

[0039] The tracking processing unit 62 performs a "tracking process" for tracking the electronic pen 20 or the sensor device 30 using various data acquired by the data acquisition unit 60. This tracking process is performed by sequentially repeating [1] a process for detecting the sensor device 30, [2] a process for calculating a coordinate transformation rule, and [3] a process for converting pen state quantities.

[0040] The above-mentioned "detection process" includes: [1] an "image processing operation" for acquiring the position of a characteristic part of the electronic pen 20 or the sensor device 30 on the image from an image signal acquired through imaging by the pair of cameras 53, 53; and [2] a "coordinate transformation operation" for converting the position on the image into a position and orientation in the display coordinate space CS2 using triangulation. The shape of the characteristic part is a point, a line, or a combination thereof. Examples of the characteristic part include: [1] a part of the external shape of the sensor device 30, including a contour or mark; [2] an external marker detachably attached to the sensor device 30; or [3] a mark displayed on the sensor device 30. Hereinafter, a state quantity related to the position and orientation of a characteristic part of the sensor device 30 will be referred to as a "device state quantity."

[0041] The above-mentioned "calculation process" includes [1] an "acquisition operation" for acquiring a pair of device state quantities, and [2] a "rule determination process" for determining a coordinate transformation rule from the pair of device state quantities.

[0042] With regard to the above-described acquisition calculation, a "pair of device state quantities" refers to a combination of a device state quantity on the sensor coordinate space CS1 ( FIG. 8 ) and a device state quantity on the display coordinate space CS2 ( FIG. 8 ). Here, the "sensor coordinate space" corresponds to a two-dimensional or three-dimensional Cartesian coordinate space defined inside the sensor device 30 and used for detection control of the electronic pen 20. Furthermore, the "display coordinate space" corresponds to a three-dimensional Cartesian coordinate space defined inside the AR glasses 50 and used for display control of the display panel 52.

[0043] Regarding the rule determination calculation described above, the "coordinate transformation rule" refers to a calculation rule for mapping from the sensor coordinate space CS1 to the display coordinate space CS2. This calculation rule is described, for example, by a three-dimensional affine transformation matrix that combines linear transformations (rotation, scaling, and shear) and translation. This transformation matrix can be found, for example, by solving simultaneous linear equations.

[0044] The above-mentioned "conversion processing" includes [1] a "conversion operation" that converts a pen state quantity in accordance with coordinate conversion rules, [2] a "selection operation" that selects one of a plurality of pen feature quantities, and [3] a "correction operation" that corrects the pen feature quantity.

[0045] With regard to the above-described conversion calculation, the "pen state quantity" refers to a state quantity that indicates the position and orientation of the electronic pen 20. Through this conversion calculation, a pen state quantity (hereinafter referred to as a second pen feature quantity) on the display coordinate space CS2 is obtained from a pen state quantity (hereinafter referred to as a first pen feature quantity) on the sensor coordinate space CS1.

[0046] The above-described selection calculation includes the selection of a first pen state quantity. The first pen state quantity is, for example, [1] a detection value detected by the pen sensor 32, or [2] an integrated value obtained by integrating the movement amount of the electronic pen 20 sequentially measured by the IMU 22 of the electronic pen 20. For example, when both the detection value and the integrated value can be acquired, the "detection value" with relatively high positional accuracy is selected as the first pen state quantity. Furthermore, when either the detection value or the integrated value cannot be acquired, the "other value" that can be acquired is selected as the first pen state quantity. Furthermore, when the current detection value cannot be acquired, the "previous detection value" that can be acquired is selected as the first pen state quantity.

[0047] The above-mentioned selection calculation includes the selection of a second pen state quantity. The second pen state quantity is, for example, [1] a converted value obtained through a conversion process, or [2] a detected value obtained through detection of the electronic pen 20 in a detection process. For example, when both a converted value and a detected value can be acquired, the "converted value" with relatively high positional accuracy is selected as the second pen state quantity. Furthermore, when either the converted value or the detected value cannot be acquired, the "other value" that can be acquired is selected as the second pen state quantity. Furthermore, when the current converted value cannot be acquired, the "previous converted value" that can be acquired is selected as the second pen state quantity.

[0048] The operation processing unit 64 analyzes the state or change in state over time of the electronic pen 20 and performs "operation processing" to determine the content of the pen operation. This operation processing includes [1] "selection processing" of the pen operation, [2] "decision processing" of the operation content, or [3] "command processing" for the electronic pen 20.

[0049] The above-mentioned "selection process" is specifically data processing for selecting whether the pen operation of the user U using the electronic pen 20 is an operation based on a first pen state quantity (hereinafter referred to as a first operation) or an operation based on a second pen state quantity (hereinafter referred to as a second operation). Whether or not this selection process is performed and what is selected differ depending on the configuration of the input system 10.

[0050] As a first example, the above-described selection process is omitted when the AR glasses 50 are responsible for both the input and output functions of the digital ink 74. In other words, of the first operation and the second operation, the second operation is always selected.

[0051] As a second example, when the sensor device 30 and the AR glasses 50 share the input / output function of the digital ink 74, a selection process is performed regarding the operation target. In this case, the selection process selects whether the operation is for the sensor device 30 (i.e., the first operation) or the AR glasses 50 (i.e., the second operation). The operation target is selected automatically or manually.

[0052] As a third example, when the sensor device 30 and the AR glasses 50 share the input / output function of digital ink 74, a selection process is performed regarding the coordinate space. In this case, the selection process selects whether the operation is for the sensor coordinate space CS1 (i.e., the first operation) or the operation is for the display coordinate space CS2 (i.e., the second operation). The coordinate space is selected automatically or manually.

[0053] Examples of determination conditions for automatic selection include [1] the presence or absence of writing pressure acting on the electronic pen 20 and the amount of writing pressure, [2] the reception strength of the electronic pen 20, or [3] the type of operation mode. For example, if writing pressure is detected, the first operation is selected, whereas if writing pressure is not detected, the second operation is selected. Furthermore, if the reception strength is equal to or less than a threshold, the first operation is selected, whereas if the reception strength is greater than the threshold, the second operation is selected. Furthermore, if an operation mode in which the operation target is limited to the sensor device 30 is being executed, the first operation is selected, whereas if an operation mode in which the operation target is limited to the AR glasses 50 is being executed, the second operation is selected. Furthermore, if the operation target is both the sensor device 30 and the AR glasses 50, the first operation or the second operation may be dynamically selected according to other determination conditions.

[0054] Examples of determination conditions for manual selection include [1] the operation state of the switch 26 of the electronic pen 20, [2] the operation state of the sensor device 30, or [3] a gesture using the electronic pen 20. For example, the first operation is normally selected, while the second operation is selected while the switch 26 is being pressed. Also, the first operation or the second operation is switched via a user control displayed on the sensor device 30. Also, when a shake operation of strongly shaking the electronic pen 20 is received, the first operation or the second operation is switched sequentially.

[0055] Specifically, the above-mentioned "determination process" is data processing for analyzing the state or time change of the state of the electronic pen 20 from the second pen state quantity supplied from the tracking processing unit 62, and for identifying the content of the operation corresponding to the obtained analysis result. The second operation includes various operations such as [1] a stroke operation for drawing handwritten content, or [2] an object operation for the user U to edit the virtual object 90 ( FIG. 5 ) in the space displayed on the AR glasses 50. The object operation includes, for example, a pick-up operation and a drop-off operation.

[0056] A "pickup operation" is an operation for moving a virtual object 90 located outside the sensor device 30 to the position of the sensor device 30. A "drop-off operation" is an operation for moving a virtual object 92 ( FIG. 6 ) located at the position of the sensor device 30 to the outside of the sensor device 30 after the pickup operation. The pick-up operation or the drop-off operation includes an operation of dragging and dropping the virtual object 90 through a change in the pointing position or pointing direction of the electronic pen 20. The drag-and-drop may be enabled, for example, while the switch 26 of the electronic pen 20 is being operated.

[0057] The above-mentioned "command processing" is, for example, data processing for outputting a command signal so that the haptic device 27 (FIGS. 6 and 7) performs a desired operation when the electronic pen 20 is equipped with the haptic device 27. Specific examples of command processing will be described later with reference to FIGS. 5 and 6.

[0058] The rendering processing unit 66 performs a "rendering process" to display the virtual object 90 in accordance with the content of the operation identified by the operation processing unit 64. This rendering process includes: [1] a "support display" related to the pen operation; [2] a rasterization process of the digital ink 74; [3] a layout process of the virtual object 90; [4] a correction process of the display coordinate space CS2; or [5] a "seamless display process" of the virtual object 90.

[0059] Examples of the above-mentioned "assistance display" include [1] display of an operation icon, [2] thumbnail display of the virtual object 90, [3] display related to the currently executing operation mode, or [4] display related to the current operation target. The above-mentioned "rasterization process" is specifically data processing for converting vector-format image data described by the digital ink 74 into raster-format image data. The above-mentioned "layout process" is specifically data processing for arranging the raster-format image data in a specified area. The above-mentioned "correction process" is data processing for correcting the second state quantity using measurement values ​​obtained from the IMU 54 so that the position and orientation of the AR glasses 50 coincide with the reference state in the display coordinate space CS2.

[0060] The above-mentioned "seamless display processing" is data processing for displaying a seamless object 110 (FIGS. 12 and 13). This "seamless object" is a virtual object 90 that includes handwritten content formed through a handwriting operation on the sensor device 30 and has no seams along the boundary line 102 (FIG. 11) of the device area 104 formed by the sensor device 30. This "seamless" does not necessarily mean a completely seamless state, but also includes a state in which there are gaps or misalignments to the extent that the content can be recognized as a single piece.

[0061] The above-mentioned "seamless display processing" is performed in different modes depending on whether or not the sensor device 30 displays an image. Specifically, when the sensor device 30 does not display a part of the seamless object 110 within the device region 104, an area including the device region 104 is specified as the position of the seamless object 110. On the other hand, when the sensor device 30 displays a part of the seamless object 110 within the device region 104, an area excluding the device region 104 is specified as the position of the seamless object 110.

[0062] During seamless display processing, a process may be performed on a portion of the seamless object 110 that is outside the device area 104 (hereinafter referred to as the outer object 114) to impart a visual effect that enhances its distinction from a portion that is inside the device area 104 (hereinafter referred to as the inner object 112). This process includes [1] a "blurring process" that converts the outer object 114 so that it is less sharp than the inner object 112, or [2] an "image deformation process" that arranges the outer object 114 in a deformed state.

[0063] Examples of "unsharpening processing" include [1] color processing that reduces brightness, [2] color processing that reduces saturation, [3] color processing that changes hue, or [4] masking (or filtering) that reduces spatial frequency characteristics. Examples of "image deformation processing" include [1] image reduction processing that reduces image size, [2] panorama processing that arranges the image in a curved state so as to surround the user U (so-called panoramic display), or [3] tracking processing that dynamically changes the shape or arrangement to follow changes in the position and orientation of the sensor device 30.

[0064] The ink generation unit 68 performs a "generation process" to generate ink data (hereinafter, "digital ink 74") describing handwritten content based on the stroke operation identified by the operation processing unit 64. When generating the digital ink 74, the ink generation unit 68 may generate either "first ink data" corresponding to the sensor coordinate space CS1 or "second ink data" corresponding to the display coordinate space CS2, or may generate both the first and second ink data simultaneously.

[0065] The storage unit 58 stores programs and data necessary for the control unit 56 to control each component. The storage unit 58 is composed of a non-transitory, computer-readable storage medium. Here, the computer-readable storage medium includes: [1] a storage device such as a hard disk drive (HDD) or solid state drive (SSD) built into the computer system; and [2] a portable medium such as a magneto-optical disk, a read-only memory (ROM), a compact disc (CD)-ROM, or a flash memory. In the example of FIG. 2, the storage unit 58 stores content data 70 and rendering information 72.

[0066] The content data 70 is data that describes the content to be displayed in the AR space. The content data 70 includes digital ink 74 and object data 76.

[0067] Digital ink 74 is composed of ink data representing handwritten content. Examples of "ink description languages" for describing digital ink include WILL (Wacom Ink Layer Language), InkML (Ink Markup Language), and ISF (Ink Serialized Format). The content may be, for example, a document, a painting, a calligraphy piece, an illustration, or artwork.

[0068] The object data 76 is image data representing various objects related to viewing and editing the virtual object 90. Examples of objects include [1] handwritten content reproduced using digital ink 74, [2] derived content including thumbnails and images for viewing, or [3] user controls such as operation icons and status columns.

[0069] The rendering information 72 includes various information related to the rendering process executed by the AR glasses 50. For example, the rendering information 72 includes conversion data 78, operation definition data 80, and a correction table 82.

[0070] The transformation data 78 includes mapping information for specifying a coordinate transformation rule between different coordinate spaces. Examples of the mapping information include matrix elements of a coordinate transformation matrix or state quantities for each feature point. Examples of the coordinate transformation include [1] transformation from the sensor coordinate space CS1 to the display coordinate space CS2, [2] transformation from the camera coordinate space to the display coordinate space CS2, or [3] transformation from the pen coordinate space to the sensor coordinate space CS1.

[0071] The operation definition data 80 includes definition information regarding the definition of an operation event by the electronic pen 20. Examples of the definition information include [1] the type of operation event, [2] the start and end conditions of the operation event, or [3] the relationship between the content of the operation and the result.

[0072] The correction table 82 is used to correct deviations in the first pen state quantity or the second pen state quantity. In the example of the first pen state quantity, the input quantity of the correction table 82 is an integrated value, and the output quantity of the correction table 82 is a correction quantity for offsetting deviations from the detected value. In the example of the second pen state quantity, the input quantity of the correction table 82 is a detected value, and the output quantity of the correction table 82 is a correction quantity for offsetting deviations from the converted value.

[0073] [First Operation of Input System 10] The input system 10 in this embodiment is configured as described above. First, the AR cooperative operation by the input system 10 will be described with reference to Figures 3 to 7. Here, "AR cooperative operation" refers to an operation for coordinating the electronic pen system and the AR glasses 50.

[0074] <Explanation of Operation Modes> Fig. 3 is a state transition diagram showing transitions of operation modes executed by the input system 10 of Fig. 1. In the example of Fig. 3, three operation modes are shown, specifically, [1] normal mode, [2] first cooperation mode, and [3] second cooperation mode. The normal mode corresponds to a mode in which the input system 10 operates without cooperation with the AR glasses 50. The first and second cooperation modes correspond to modes in which the input system 10 operates in cooperation with the AR glasses 50.

[0075] [T1] During execution of the normal mode, the input system 10 enables input operations using the electronic pen 20 for the sensor device 30, but disables them for the AR glasses 50. [T2] During execution of the normal mode, the input system 10 transitions from the normal mode to the first coordination mode when an operation to start an AR coordination operation is accepted. [T3] During execution of the first coordination mode, the input system 10 transitions from the first coordination mode to the normal mode when an operation to end the AR coordination operation is accepted.

[0076] [T4] During execution of the first cooperation mode, the input system 10 enables input operations using the electronic pen 20 for both the sensor device 30 and the AR glasses. [T5] During execution of the first cooperation mode, the input system 10 transitions from the first cooperation mode to the second cooperation mode when it receives a pick-up operation for the virtual object 90 ( FIG. 4 ) to be viewed. [T6] During execution of the second cooperation mode, the input system 10 receives a handwriting operation for the virtual object 92 ( FIG. 5 ) to be worked on. [T7] During execution of the second cooperation mode, the input system 10 transitions from the second cooperation mode to the first cooperation mode when it receives a drop-off operation for the virtual object 92.

[0077] 4 is a diagram showing an example of a pick-up operation for a virtual object 90 to be viewed. In the example of FIG. 4, three virtual objects 90 to be viewed are displayed side by side on the AR glasses 50. The user U starts a pen operation by pressing the switch 26 of the electronic pen 20 while holding the electronic pen 20 and with the pen tip facing in the pointing direction D1. As a result, the virtual object 90 in the pointing direction D1 of the electronic pen 20 is selected.

[0078] While pressing the switch 26, the user U performs a pen operation to move the tip of the electronic pen 20 along the movement direction M1. As a result, the selected virtual object 90 moves downward, following the movement direction M1 of the electronic pen 20. The start point P1 of the movement direction M1 corresponds to a position outside the area defined by the touch surface 31 of the sensor device 30. The end point P2 of the movement direction M1 corresponds to a position inside the area defined by the touch surface 31 of the sensor device 30.

[0079] The user U performs a pen operation to release the switch 26 while the pen tip of the electronic pen 20 is at the end point P2. Then, the selected virtual object 90 is placed on the touch surface 31 of the sensor device 30 and functions as a virtual object 92 to be worked on. In this way, a pickup operation on the virtual object 90 is performed.

[0080] 5 is a diagram showing an example of a drop-off operation on a virtual object 92 to be worked on. In the example of FIG. 5, two virtual objects 90 to be viewed and one virtual object 92 to be worked on are displayed on the AR glasses 50. While holding the electronic pen 20, the user U performs an operation of pressing the switch 26 provided on the electronic pen 20 with the pen tip facing the touch surface 31 of the sensor device 30. As a result, the virtual object 92 to be worked on that is located on the touch surface 31 is selected.

[0081] The user U, while holding down the switch 26, performs an operation to move the tip of the electronic pen 20 along the movement direction M2. As a result, the selected virtual object 92 moves upward, following the movement direction M2 of the electronic pen 20. The start point P3 of the movement direction M2 corresponds to a position within the area defined by the touch surface 31 of the sensor device 30. The end point P4 of the movement direction M2 corresponds to a position outside the area defined by the touch surface 31 of the sensor device 30.

[0082] The user U releases the switch 26 while the pen tip of the electronic pen 20 is at the end point P4. The selected virtual object 92 is then placed in the gap in the direction D2 indicated by the pen tip, and functions as the virtual object 90 to be viewed. In this manner, a drop-off operation is performed on the virtual object 92.

[0083] <Explanation of Haptic Operation> Next, the haptic operation by the input system 10 will be described with reference to Fig. 6 and Fig. 7. Here, the "haptic operation" refers to an operation that allows the user U operating the electronic pen 20 to experience an analog feeling of use through haptics (transmission of the sense of touch).

[0084] Fig. 6 is a diagram showing an example of the arrangement of the haptic device 27 included in the electronic pen 20 of Fig. 1. The electronic pen 20 is equipped with a haptic device 27 that imparts a tactile sensation to the contact part of the user U from multiple positions around the periphery of the pen. In the example of Fig. 6, the haptic device 27 is provided inside a housing 25 whose cross-sectional shape is approximately a regular hexagon.

[0085] The haptic device 27 includes one or more output elements that transmit a tactile sensation to a contact portion of the housing 25 by outputting vibration, force, movement, or the like to the outside. Examples of output elements include a piezoelectric element, a linear resonant actuator, an ultrasonic transducer, or an eccentric motor. In the example of FIG. 6 , the haptic device 27 includes six piezoelectric elements provided on each side of the housing 25. Note that the haptic device 27 is not limited to the configuration shown in FIG. 6 and may be, for example, a single module incorporating multiple output elements.

[0086] The MCU 23 of the electronic pen 20 receives, for example, a command signal from the AR glasses 50 via the communication chip 24, and determines the amount of operation of the haptic device 27 according to operation information included in the command signal. Examples of the operation information include the movement direction and movement speed of the electronic pen 20, and the types of virtual objects 90 and 92.

[0087] When simulating the weight of the electronic pen 20, the control unit 56 of the AR glasses 50 may set the amount of operation of the haptic device 27 so as to reflect forces (e.g., gravity, inertial force, etc.) that may act on the electronic pen 20. For example, the control unit 56 may set the operation position of the haptic device 27 so as to provide a tactile sensation in a direction opposite to the movement direction of the electronic pen 20 (i.e., toward the starting point of the movement direction). Furthermore, the control unit 56 may set the operation level of the haptic device 27 so that the tactile sensation changes depending on the movement speed or movement acceleration of the electronic pen 20. For example, the operation level is set lower as the movement speed or movement acceleration decreases, and is set higher as the movement speed of the electronic pen 20 increases.

[0088] When simulating the sense of weight of the virtual objects 90 and 92, the control unit 56 of the AR glasses 50 may set the amount of operation of the haptic device 27 so as to reflect forces (e.g., frictional forces, repulsive forces, etc.) that may act on the electronic pen 20. For example, the control unit 56 may set the operation level of the haptic device 27 so that the tactile sensation changes depending on the type of the virtual objects 90 and 92. For example, the operation level is set lower as the mass, size, and thickness of the virtual objects 90 and 92 become smaller, and is set higher as the mass, size, and thickness of the electronic pen 20 become larger.

[0089] 7 is a diagram showing an example of the relationship between the movement of the electronic pen 20 and the operation of the haptic device 27. Here, it is assumed that the electronic pen 20 moves from the top to the bottom of FIG. 7. In this case, an inertial force is generated in the electronic pen 20 in the direction opposite to the direction of movement (from bottom to top), so the electronic pen 20 operates only the upper haptic device 27 to assist the inertial force. This makes it possible to create a sense of weight similar to that of a wooden pencil.

[0090] <Summary of First Operation> As described above, the input system 10 in this embodiment includes the electronic pen 20, the sensor device 30 including the planar pen sensor 32 capable of detecting the position and orientation of the electronic pen 20, a wearable display device (here, AR glasses 50) that can be worn to cover the eyes of the user U, and one or more processors that perform display control on the AR glasses 50 so as to display virtual objects 90, 92 within the display area of ​​the AR glasses 50. The one or more processors are provided inside or outside the AR glasses 50, and correspond to the host processor 34 or the control unit 56 in this case.

[0091] The one or more processors execute a determination process to determine the content of the pen operation using the electronic pen 20 based on pen state quantities indicating the position and orientation of the electronic pen on the display coordinate space CS2 defined for display control.

[0092] The pen operation may also include a pick-up operation for moving the virtual object 90 located outside the sensor device 30 to the position of the sensor device 30, and, after the pick-up operation, a drop-off operation for moving the virtual object 92 located at the position of the sensor device 30 to the outside of the sensor device 30. This allows the user U to easily understand whether the virtual objects 90 and 92 are work targets.

[0093] Furthermore, the pick-up operation or drop-off operation may include an operation of dragging and dropping the virtual objects 90, 92 by changing the pointing position or pointing direction of the electronic pen 20. This allows the user U to easily switch the working state of the virtual objects 90, 92 through drag-and-drop.

[0094] Furthermore, drag and drop may be enabled while the switch 26 of the electronic pen 20 is being operated, which makes it easier to reflect the user's operational intentions.

[0095] Furthermore, the one or more processors may execute the following processes: [1] a selection process for selecting whether the pen operation of the user U using the electronic pen 20 is a first operation on the sensor device 30 or a second operation on the AR glasses 50; and [2a] if the pen operation of the user U is the first operation, a first pen state quantity on the sensor coordinate space CS1 defined by the sensor device 30 is used to determine the content of the operation on the sensor device 30, and [2b] if the pen operation of the user U is the second operation, a determination process for determining the content of the operation on the AR glasses 50 using a second pen state quantity on the display coordinate space CS2 defined for display control. By alternatively selecting the operation target by the electronic pen 20, the operation range of the electronic pen 20 can be expanded beyond the sensor device 30 to the space within the user U's field of view, thereby improving the input operability of the user U.

[0096] Furthermore, the one or more processors may execute the following processes: [1] a selection process for selecting whether a pen operation by the user U using the electronic pen 20 is a first operation on the sensor coordinate space CS1 or a second operation on the display coordinate space CS2; and [2a] if the pen operation by the user U is the first operation, a first pen state quantity on the sensor coordinate space CS1 is used to determine the content of the operation on the sensor coordinate space CS1, and [2b] if the pen operation by the user U is the second operation, a determination process for determining the content of the operation on the display coordinate space CS2 using a second pen state quantity on the display coordinate space CS2 defined for display control. By alternatively selecting the coordinate space in which the electronic pen 20 is located, the operation range of the electronic pen 20 can be expanded not only to the sensor device 30 but also to a space within the field of view of the user U, thereby improving the input operability of the user U.

[0097] Furthermore, when the electronic pen 20 includes a haptic device 27 that imparts tactile sensations to the contact portion of the user U from multiple positions around the pen, one or more processors (here, the MCU 23 or the control unit 56) may execute a command process to command the operation of the haptic device 27 so as to impart tactile sensations at positions corresponding to the direction in which an inertial force or a repulsive force is generated as the electronic pen 20 moves. This allows the user U holding the electronic pen 20 to experience a simulated sense of weight of the electronic pen 20.

[0098] Furthermore, if the electronic pen 20 includes a haptic device 27 that provides a tactile sensation to the contact portion of the user U, one or more processors may execute a command process to command the haptic device 27 to change the amount of movement depending on the movement speed of the electronic pen 20 or the type of the virtual object 90, 92. This allows the user U holding the electronic pen 20 to virtually experience the subtle differences in the sense of weight of the electronic pen 20 or the virtual object 90, 92.

[0099] [Second Operation of Input System 10] Second, the pen tracking operation by the input system 10 will be described with reference to Figures 8 to 10. Here, the "pen tracking operation" refers to an operation for tracking the electronic pen 20 in the display coordinate space CS2 using the sensor device 30 as a reference plane.

[0100] <Explanation of Pen Tracking Operation> For example, if the AR glasses 50 are equipped with a camera 53, the AR glasses 50 can directly detect the position and orientation of the electronic pen 20 using the camera 53, thereby allowing the electronic pen 20 to function as a pointing device. However, if the detection accuracy of the camera 53 is not high, precise drawing cannot be performed using the electronic pen 20. Therefore, the input system 10 performs a "pen tracking operation" that tracks the electronic pen 20 with high accuracy by indirectly detecting the position and orientation of the electronic pen 20 using a device area 94 ( FIG. 8 ) formed by the sensor device 30 as a reference plane.

[0101] FIG. 8 is a diagram showing the correspondence between the sensor coordinate space CS1 and the display coordinate space CS2. The sensor coordinate space CS1 is a three-dimensional Cartesian coordinate space defined within the sensor device 30 and used for detection control of the electronic pen 20. The origin Os of the sensor coordinate space CS1 is set, for example, at one vertex Pm1 on the device area 94 formed by the pen sensor 32. The sensor coordinate space CS1 is composed of three axes: [1] a "U axis" parallel to one side of the device area 94, [2] a "V axis" parallel to the other side of the device area 94, and [3] a "W axis" perpendicular to the device area 94. Note that (φu, φv, φw) are angles around the U axis, V axis, and W axis, respectively.

[0102] The display coordinate space CS2 is a three-dimensional Cartesian coordinate space defined inside the AR glasses 50 and used for display control of the display panel 52. The origin Od of the display coordinate space CS2 is set, for example, at the midpoint between the imaging centers Pc1 and Pc2 of the pair of cameras 53. The display coordinate space CS2 is composed of three axes: [1] the "X-axis," which is a first axis parallel to the horizontal plane; [2] the "Y-axis," which is a second axis parallel to the horizontal plane; and [3] the "Z-axis," which is parallel to the vertical direction. Note that (θx, θy, θx) are angles around the X-axis, Y-axis, and Z-axis, respectively.

[0103] Here, the device area 94 corresponds to a "reference plane" for the pen tracking operation by the tracking processing unit 62 of the AR glasses 50. The placement area 96 corresponds to a position where the virtual object 90 is placed. Feature points (in the example of FIG. 3 , diagonal vertices Pm1 and Pm2 in the device area 94) whose positional relationship with the sensor device 30 (or the pen sensor 32) is known are set in the sensor coordinate space CS1. By acquiring the state quantities of the vertices Pm1 and Pm2 in the display coordinate space CS2 and determining a coordinate transformation rule from the pair of device state quantities, it is possible to uniquely map from the sensor coordinate space CS1 to the display coordinate space CS2.

[0104] Fig. 9 is a flowchart relating to the pen tracking operation by the AR glasses 50 of Fig. 2. The AR glasses 50 periodically or irregularly repeatedly execute the flowchart of Fig. 9 to track the electronic pen 20 in the display coordinate space CS2.

[0105] In step SP10, the data acquisition unit 60 acquires, from the sensor device 30, state quantities (i.e., first pen state quantities) indicating the position and orientation of the electronic pen 20 in the sensor coordinate space CS1.

[0106] In step SP12, the data acquisition unit 60 acquires data corresponding to the first pen state quantity acquired in step SP10 and necessary for determining the operation content (hereinafter referred to as tracking data). The tracking data includes an image signal, rendering information 72, or judgment condition information.

[0107] In step SP14, the tracking processing unit 62 performs image processing on the image signal acquired in step SP12 to detect the position and orientation of the sensor device 30 at least in front of the user U. This allows the device state quantity in the display coordinate space CS2 to be obtained.

[0108] In step SP16, the tracking processing unit 62 calculates a coordinate transformation matrix for transforming from the sensor coordinate space CS1 to the display coordinate space CS2 using the transformation data 78 acquired in step SP12 and the device state quantity acquired in step SP14.

[0109] In step SP18, the tracking processing unit 62 uses the coordinate transformation matrix calculated in step SP16 to convert the first pen state quantity acquired in step SP10 into a state quantity indicating the position and posture of the electronic pen 20 in the display coordinate space CS2 (i.e., a second pen state quantity).

[0110] In step SP20, the operation processing unit 64 uses the determination condition information acquired in step SP12 to determine whether the operation target of the electronic pen 20 is the sensor device 30 or the AR glasses 50. If the operation processing unit 64 determines that the operation target is the sensor device 30 (step SP20: sensor device), it proceeds to step SP21, whereas if it determines that the operation target is the AR glasses 50 (step SP20: AR glasses), it proceeds to step SP22.

[0111] In step SP21, the operation processing unit 64 instructs the sensor device 30, which is the operation target, to perform an operation (i.e., a first operation) using the electronic pen 20, and then proceeds to step SP24, which will be described later.

[0112] In step SP22, the operation processing unit 64 uses the second pen state quantity converted in step SP18 to determine the content of the operation (i.e., the second operation) by the electronic pen 20 on the AR glasses 50, which are the operation target. After that, the operation processing unit 64 proceeds to the next step SP24.

[0113] In step SP24, the sensor device 30 or the AR glasses 50 executes an operation (i.e., the first operation or the second operation) using the electronic pen 20. In this manner, by repeatedly executing the flowchart of FIG. 9 , it is possible to continue tracking the electronic pen 20 with high accuracy.

[0114] 10 is a diagram showing an example of a method for acquiring the state quantities of each device in the sensor coordinate space CS1 or the display coordinate space CS2. More specifically, Fig. 10 shows a list of [1] the entity that acquires the state quantities, [2] the type of coordinate space, [3] the state quantities of the electronic pen 20, [4] the state quantities of the sensor device 30, and [5] the state quantities of the AR glasses 50.

[0115] [1] First state quantity related to electronic pen 20 (sensor coordinate space CS1) The electronic pen 20 measures the amount of movement of its position and orientation using its own IMU 22, and acquires the first state quantity related to the electronic pen 20 by cumulatively adding up the amount of movement. The sensor device 30 acquires the first state quantity related to the electronic pen 20 through communication with the electronic pen 20. The AR glasses 50 acquire the first state quantity related to the electronic pen 20 through communication with the electronic pen 20 or the sensor device 30.

[0116] [2] Second state quantity related to the electronic pen 20 (display coordinate space CS2) Because the electronic pen 20 and the sensor device 30 are not involved in the rendering process by the AR glasses 50, acquisition of the second state quantity related to the electronic pen 20 may be omitted. The AR glasses 50 acquire the second state quantity related to the electronic pen 20 through coordinate transformation from the sensor coordinate space CS1 to the display coordinate space CS2. In addition to or separately from this, the AR glasses 50 acquire the second state quantity related to the electronic pen 20 through detection processing using the pair of cameras 53, 53.

[0117] [3] First State Quantity Related to the Sensor Device 30 (Sensor Coordinate Space CS1) Because the electronic pen 20 is not involved in the handwriting process by the sensor device 30, acquisition of the first state quantity related to the sensor device 30 may be omitted. Because the sensor coordinate space CS1 is independently defined for the sensor device 30, the sensor device 30 holds the first state quantity related to the sensor device 30 as a fixed value. Alternatively, the sensor device 30 measures the amount of movement of its position and orientation using its own IMU (not shown) and cumulatively adds up the amount of movement to acquire the first state quantity indicating the relative position and orientation with respect to a reference state (fixed value). Because the definition of the sensor coordinate space CS1 is shared in advance with the AR glasses 50, the AR glasses 50 hold the first state quantity related to the sensor device 30 as a fixed value. In addition to or separately from this, the AR glasses 50 acquire the first state quantity related to the sensor device 30 via communication with the sensor device 30.

[0118] [4] Second state quantity related to the sensor device 30 (display coordinate space CS2) The electronic pen 20 and the sensor device 30 are not involved in the rendering process by the AR glasses 50, so acquisition of the second state quantity related to the sensor device 30 may be omitted. The AR glasses 50 acquire the second state quantity related to the sensor device 30 through a detection process using the pair of cameras 53, 53.

[0119] [5] First state quantity related to the AR glasses 50 (sensor coordinate space CS1) Since the electronic pen 20, the sensor device 30, and the AR glasses 50 do not require the first state quantity related to the AR glasses 50, acquisition of the first state quantity may be omitted.

[0120] [6] Second state quantity related to the AR glasses 50 (display coordinate space CS2) The electronic pen 20 and the sensor device 30 are not involved in the rendering process by the AR glasses 50, and therefore may omit acquisition of the second state quantity related to the AR glasses 50. The AR glasses 50 acquire the first state quantity related to the AR glasses 50 by measuring the amount of movement of the position and orientation using the IMU 54 of the AR glasses 50 itself and cumulatively adding up the amount of movement.

[0121] 10 , the input system 10 exchanges state quantities among the electronic pen 20, the sensor device 30, and the AR glasses 50. When the input system 10 can acquire pen feature quantities by multiple means, it can select and correct them to improve the accuracy of position and orientation detection.

[0122] <Summary of Second Operation> As described above, the input system 10 in this embodiment includes the electronic pen 20, the sensor device 30 including the planar pen sensor 32 capable of detecting the position and orientation of the electronic pen 20, a wearable display device (here, AR glasses 50) that is a display device that can be worn to cover the eyes of the user U and is provided with a camera 53 that captures images of at least the space in front of the user U, and one or more processors that perform display control on the AR glasses 50 so as to display virtual objects 90, 92 within the display area of ​​the AR glasses 50. The one or more processors are provided inside or outside the AR glasses 50 and correspond to the control unit 56 in this case.

[0123] The one or more processors perform the following operations: [1] a detection process that performs image processing to detect the sensor device 30 on the image signal acquired through imaging by the camera 53 and acquires device state quantities indicating the position and orientation of the sensor device 30 on the display coordinate space CS2 defined for display control; [2] a calculation process that uses the device state quantities acquired through the detection process to calculate a coordinate transformation rule for mapping from the sensor coordinate space CS1 defined by the sensor device 30 to the display coordinate space CS2; and [3] a conversion process that converts a first pen state quantity indicating the position and orientation of the electronic pen 20 on the sensor coordinate space CS1 into a second pen state quantity indicating the position and orientation of the electronic pen 20 on the display coordinate space CS2 according to the calculated coordinate transformation rule.

[0124] In this way, by detecting the position and orientation of the electronic pen 20 using the position of the pen sensor 32 (or the device area 94) as the reference plane, the second pen state quantity indicating the position and orientation of the electronic pen 20 on the display coordinate space CS2 can be determined with high accuracy.

[0125] Furthermore, the one or more processors may further perform a tracking process to track the electronic pen 20 in the display coordinate space CS2 by sequentially repeating the detection process, calculation process, and conversion process. This enables the electronic pen 20 to be tracked with high accuracy, and even if the camera 53 cannot detect the electronic pen 20, tracking of the electronic pen 20 can be continued through detection by the sensor device 30.

[0126] Furthermore, the first pen state quantity is [1] a detection value detected by the pen sensor 32, or [2] an integrated value obtained by integrating the movement amount of the electronic pen 20 sequentially measured by an inertial sensor (here, the IMU 22) included in the electronic pen 20. In this case, if the one or more processors are unable to acquire a detection value in the conversion process, they may convert the integrated value as the first pen state quantity into the second pen state quantity. Even if the pen sensor 32 is unable to detect the electronic pen 20, tracking of the electronic pen 20 can be continued through measurements by the electronic pen 20.

[0127] The one or more processors may further perform an acquisition process to acquire a correction amount for bringing the integrated value closer to the detected value, and in the conversion process, the integrated value may be corrected using the correction amount, and the corrected integrated value may be converted into the second pen-state quantity. In this way, even when the integrated value is used, a calculation result closer to the detected value may be obtained.

[0128] The second pen state quantity is a converted value obtained through the conversion process, or a detected value obtained through detection of the electronic pen 20 in the detection process. In this case, if the converted value cannot be obtained, the one or more processors may further execute a determination process to determine the content of the operation on the display coordinate space CS2 using the detected value as the second pen state quantity. Even if the conversion cannot be performed correctly, tracking of the electronic pen 20 can be continued through detection by the camera 53.

[0129] The one or more processors may further execute an acquisition process to acquire a correction amount for bringing the detected value closer to the converted value, and in the determination process, the detected value may be corrected using the correction amount, and the corrected detected value may be used to determine the content of the operation for the display coordinate space CS2. In this way, even when the detected value is used, a calculation result closer to the converted value can be obtained.

[0130] [Third Operation of Input System 10] Next, a seamless display operation by the input system 10 will be described with reference to Figures 11 to 13. Here, the "seamless display operation" refers to an operation for seamlessly displaying the virtual objects 90 and 92 (Figures 4 and 5).

[0131] <Explanation of Seamless Display Operation> Figure 11 is a diagram showing an example of a method for specifying a placement area 100 on the display coordinate space CS2. The definitions of the sensor coordinate space CS1 and the display coordinate space CS2 are the same as those in Figure 8, so their description will be omitted. The placement area 100 of the virtual object is set so as to encompass, for example, a device area 104 formed by a boundary line 102 of the sensor device 30. In this case, the placement area 100 is made up of the device area 104 corresponding to the inside of the boundary line 102 and an outer area 106 corresponding to the outside of the boundary line 102.

[0132] 12 is a diagram showing an example of a display mode of a seamless object 110. A seamless object 110, which is seamless at the boundary line 102 of the sensor device 30, is displayed on the display panel 52 of the AR glasses 50. The seamless object 110 includes handwritten content formed through a handwriting operation on the sensor device 30. The seamless object 110 is composed of an inner object 112 located inside the boundary line 102 and an outer object 114 located outside the boundary line 102. In the example of FIG. 12 , the outer object 114 is displayed in a lighter color (or lower brightness) than the inner object 112.

[0133] If the sensor device 30 does not have a display function or if the sensor device 30 does not display the seamless object 110, both the inner object 112 and the outer object 114 are displayed on the display panel 52. In other words, the placement area 100 in FIG. 11 is specified as the display position of the seamless object 110.

[0134] When the sensor device 30 has a display function and displays a part of the seamless object 110, only one side of the outer object 114 is displayed on the display panel 52. In other words, the outer area 106 in FIG. 11 is specified as the display position of the seamless object 110.

[0135] FIG. 13 is a diagram showing an example of the result of an operation on the seamless object 110. In the example of FIG. 13 , similar to the case of FIG. 12 , the seamless object 110 is displayed on the display panel 52 of the AR glasses 50. While holding the electronic pen 20 and pointing at a designated location 116 (here, below the sensor device 30) within the seamless object 110, the user U starts a pen operation by pressing the switch 26 of the electronic pen 20, and then performs a drag operation to move the pen tip from the designated location 116 to a location of interest 118. The AR glasses 50 then update the display content of the display panel 52 so that the seamless object 110 moves in a lower right direction. As a result, the location of interest 118 fits within the device area 104, allowing the user U to perform handwriting operations on the surrounding area including the location of interest 118.

[0136] <Summary of Third Operation> As described above, the input system 10 in this embodiment includes the electronic pen 20, the sensor device 30 including the planar pen sensor 32 capable of detecting the position and orientation of the electronic pen 20, a wearable display device (here, AR glasses 50) that can be worn to cover the eyes of the user U, and one or more processors that perform display control on the AR glasses 50 so as to display virtual objects 90, 92 within the display area of ​​the AR glasses 50. The one or more processors are provided inside or outside the AR glasses 50, and correspond to the control unit 56 in this case.

[0137] The one or more processors perform a rendering process to display a seamless object 110, which is a virtual object 90, 92 that includes handwritten content formed through handwriting operations on the sensor device 30 and has no seams at the boundary 102 of the device area 104 formed by the sensor device 30.

[0138] With this configuration, the working range using the electronic pen 20 can be expanded beyond the sensor device 30 to the outside of the boundary line 102, thereby improving the input operability for the user U.

[0139] Furthermore, if the sensor device 30 does not display a portion of the seamless object 110 within the device area 104 during the rendering process, the one or more processors may specify the placement area 100 that includes the device area 104 as the position of the seamless object 110. This makes it easier to maintain the continuity and connectivity of the handwritten content around the boundary line 102.

[0140] Furthermore, in the rendering process, when the sensor device 30 displays a part of the seamless object 110 within the device area 104, the one or more processors may specify the outer area 106 excluding the device area 104 as the position of the seamless object 110. This makes it possible to prevent image blurring due to overlapping display of handwritten content.

[0141] The rendering process may also include an application process for applying a visual effect to an outer object 114, which is a portion of the seamless object 110 that is outside the device area 104, to enhance its distinction from an inner object 112, which is a portion of the seamless object 110 that is inside the device area 104. This allows the user U to grasp at a glance the effective range of the handwriting operation through the visual contrast between the inner object 112 and the outer object 114.

[0142] [Modifications] The present invention is not limited to the above-described embodiment, and can be freely modified without departing from the spirit of the present invention. Alternatively, the respective configurations may be arbitrarily combined within the scope of no technical contradiction. Alternatively, the execution or execution order of each step constituting the flowchart may be changed within the scope of no technical contradiction.

[0143] In the above-described embodiment, an example was given in which the wearable display device displays an augmented reality space using augmented reality (AR) technology. However, instead of this, a space may be displayed using, for example, mixed reality (MR) technology or virtual reality (VR) technology.

[0144] In the above embodiment, the control unit 56 and the storage unit 58 are incorporated into the AR glasses 50 of Fig. 2, but the present invention is not limited to this device configuration. For example, the AR glasses 50 may be connected to a separate computer, and the separate computer may perform some or all of the functions executed by the control unit 56 (for example, the tracking processing unit 62, the operation processing unit 64, the rendering processing unit 66, or the ink generation unit 68).

[0145] In the above embodiment, an example has been described in which the AR glasses 50 in Fig. 2 communicate via the communication chip 55, but the AR glasses 50 may also include a receiving circuit for receiving an uplink signal from the sensor device 30, either in addition to the communication chip 55 or separately from the communication chip 55. This "uplink signal" is a signal that the sensor device 30 transmits to the electronic pen 20 when the electronic pen 20 is an AES-based stylus. This allows data to be exchanged while utilizing the AES system.

[0146] [Explanation of symbols] 10...input system, 20...electronic pen, 22...IMU (inertial sensor), 27...haptic device, 30...sensor device, 32...pen sensor, 34...host processor (processor), 50...AR glasses (wearable display device), 52...display panel, 53...camera, 56...control unit (processor), 58...storage unit, 60...data acquisition unit, 62...tracking processing unit, 64...operation processing unit, 66...rendering processing unit, 68...ink generation unit, 90, 92...virtual object, 94, 104...device area, 96, 100...placement area, 102...boundary line, 106...outer area, 110...seamless object, 112...inner object, 114...outer object, CS1...sensor coordinate space, CS2...display coordinate space

Claims

1. An input system comprising: an electronic pen; a sensor device including a planar pen sensor capable of detecting a position and orientation of the electronic pen; a wearable display device that is a display device that can be worn to cover a user's eyes and is provided with a camera that captures an image of at least a space in front of the user; and one or more processors that perform display control on the wearable display device to display a virtual object within a display area of ​​the wearable display device, wherein the one or more processors perform a detection process that performs image processing on an image signal acquired through imaging by the camera to detect the sensor device, and acquires device state quantities that indicate the position and orientation of the sensor device on a display coordinate space defined for the display control; a calculation process that uses the device state quantities acquired through the detection process to calculate a coordinate transformation rule for mapping from a sensor coordinate space defined by the sensor device to the display coordinate space; and a conversion process that converts a first pen state quantity that indicates the position and orientation of the electronic pen on the sensor coordinate space into a second pen state quantity that indicates the position and orientation of the electronic pen on the display coordinate space in accordance with the calculated coordinate transformation rule.

2. The input system according to claim 1, wherein the one or more processors further execute a determination process for determining the content of a pen operation using the electronic pen based on the second pen state quantity.

3. The input system of claim 2, wherein the pen operations include: a pick-up operation for moving the virtual object located outside the sensor device to a position of the sensor device; and a drop-off operation for moving the virtual object at the position of the sensor device outside the sensor device after the pick-up operation.

4. The input system according to claim 3, wherein the pick-up operation or the drop-off operation includes an operation of dragging and dropping the virtual object through a change in a pointing position or a pointing direction of the electronic pen.

5. The input system according to claim 4, wherein the drag-and-drop is enabled while a switch of the electronic pen is being operated.

6. The input system of claim 1, wherein the one or more processors further execute a selection process for selecting whether the user's pen operation using the electronic pen is a first operation for the sensor device or a second operation for the wearable display device, and a determination process for determining the content of the operation for the sensor device using the first pen state amount if the user's pen operation is the first operation, and for determining the content of the operation for the wearable display device using the second pen state amount if the user's pen operation is the second operation.

7. The input system of claim 1, wherein the one or more processors further execute a selection process for selecting whether the user's pen operation using the electronic pen is a first operation for the sensor coordinate space or a second operation for the display coordinate space, and a determination process for determining the content of the operation for the sensor coordinate space using the first pen state quantity if the user's pen operation is the first operation, and for determining the content of the operation for the display coordinate space using the second pen state quantity if the user's pen operation is the second operation.

8. The input system of claim 1, wherein the electronic pen is equipped with a haptic device that imparts tactile sensations to the part of the user's contact from multiple positions around the pen, and the one or more processors further execute command processing to command the operation of the haptic device so as to impart the tactile sensation at a position corresponding to a direction in which an inertial force or a repulsive force is generated as the electronic pen moves.

9. The input system of claim 1, wherein the electronic pen is equipped with a haptic device that imparts a tactile sensation to the part of the user that is in contact with the electronic pen, and the one or more processors further execute command processing to command the haptic device to change an amount of operation depending on a moving speed of the electronic pen or a type of the virtual object.

10. The input system according to claim 4, wherein the one or more processors further perform a rendering process for displaying handwritten content formed through the pen operation superimposed on the virtual object located at the position of the sensor device after the pick-up operation and before the drop-off operation.

11. The input system according to claim 1, wherein the one or more processors sequentially repeat the detection process, the calculation process, and the conversion process, and further perform a tracking process of tracking the electronic pen on the display coordinate space.

12. The input system described in claim 11, wherein the first pen state quantity is a detection value detected by the pen sensor, or an accumulated value obtained by accumulating the amount of movement of the electronic pen measured sequentially by an inertial sensor possessed by the electronic pen, and when the detection value cannot be obtained in the conversion process, the one or more processors convert the accumulated value as the first pen state quantity into the second pen state quantity.

13. The input system according to claim 12, wherein the one or more processors further execute an acquisition process to acquire a correction amount for bringing the accumulated value closer to the detection value, and in the conversion process, the accumulated value is corrected using the correction amount, and the obtained corrected accumulated value is converted into the second pen state quantity.

14. The input system of claim 11, wherein the second pen state quantity is a converted value obtained through the conversion process, or a detected value obtained through detection of the electronic pen in the detection process, and when the converted value cannot be obtained, the one or more processors further execute a determination process to determine the content of the pen operation on the display coordinate space using the detected value as the second pen state quantity.

15. The input system of claim 14, wherein the one or more processors further execute an acquisition process to acquire a correction amount for bringing the detection value closer to the conversion value, and in the determination process, the detection value is corrected using the correction amount, and the content of the pen operation in the display coordinate space is determined using the obtained corrected detection value.

16. The input system of claim 1, wherein the one or more processors further perform a rendering process for displaying a seamless object, which is the virtual object including handwritten content formed through a handwriting operation on the sensor device and has no seams at the boundary of a device area formed by the sensor device.

17. The input system according to claim 16, wherein the one or more processors, in the rendering process, if the sensor device does not display a portion of the seamless object within the device area, specify an area that includes the device area as the position of the seamless object.

18. The input system according to claim 16, wherein, in the rendering process, when the sensor device displays a part of the seamless object within the device area, the one or more processors specify an area excluding the device area as the position of the seamless object.

19. The input system of claim 16, wherein the rendering process includes an imparting process for imparting a visual effect to an outer object, which is a portion of the seamless object that is outside the device area, to enhance its distinctiveness from an inner object, which is a portion of the seamless object that is inside the device area.

20. The input system according to claim 19, wherein the applying process includes an unsharpening process that converts the outer object so that it is unsharp compared to the inner object.

21. The input system according to claim 20, wherein the unsharpening process is color processing that reduces brightness or saturation, masking processing that reduces spatial frequency characteristics, or image reduction processing that reduces an image size.

22. The input system according to claim 19, wherein the applying process includes an image transformation process for arranging the outer object in a transformed state.

23. The input system according to claim 22, wherein the image transformation process is an image reduction process that reduces an image size of the outer object, or a panoramic process that arranges the outer object in a curved state so as to surround the user.

24. An input method using an input system comprising: an electronic pen; a sensor device including a planar pen sensor capable of detecting a position and orientation of the electronic pen; a wearable display device that is a display device that can be worn to cover a user's eyes and is provided with a camera that captures an image of at least a space in front of the user; and one or more processors that perform display control on the wearable display device so as to display a virtual object within a display area of ​​the wearable display device, wherein the one or more processors perform the following steps: a detection step of performing image processing on an image signal acquired through imaging by the camera to detect the sensor device, and acquiring a device state quantity indicating the position and orientation of the sensor device on a display coordinate space defined for the display control; a calculation step of calculating a coordinate transformation rule for mapping from a sensor coordinate space defined by the sensor device to the display coordinate space, using the device state quantity acquired through the detection step; and a conversion step of converting a first pen state quantity indicating the position and orientation of the electronic pen on the sensor coordinate space into a second pen state quantity indicating the position and orientation of the electronic pen on the display coordinate space in accordance with the calculated coordinate transformation rule.

Images