Electronic pen and information processing device

Abstract

To make it possible to supply an electronic pen capable of generating tactile feedback with haptic data relating to the generation of tactile feedback.SOLUTION: An electronic pen capable of communicating with an information processing device includes: a vibration unit that generates tactile feedback to be provided to a user; a transmitting unit that transmits information indicating that the tactile feedback can be generated to the information processing device; a receiving unit that receives haptic data relating to the generation of the tactile feedback, the haptic data being transmitted from the information processing device in response to receiving the information transmitted from the transmitting unit; and a control unit that controls the vibration unit to generate the tactile feedback based on the haptic data received by the receiving unit.SELECTED DRAWING: Figure 15

Description

[Technical Field]

[0001] The present invention relates to a handwriting data generation device, a handwriting data reproduction device, and a digital ink data structure, and more particularly to a handwriting data generation device, a handwriting data reproduction device, and a digital ink data structure that use haptic feedback. [Background technology]

[0002] Tablet terminals are known that are configured to provide tactile feedback when operations such as selecting text displayed on a touch screen, inserting or positioning a cursor within the text, etc. Patent Document 1 discloses an example of such a tablet terminal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-118993 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide a handwritten data generation device, a handwritten data reproduction device, and a digital ink data structure that generate tactile feedback according to stroke data. [Means for solving the problem]

[0005] A handwriting data generation device according to the present invention is a handwriting data generation device including a processor, the processor associating tactile feedback with at least a portion of stroke data generated in response to handwriting input, and generating digital ink including the stroke data and haptic data indicative of the tactile feedback.

[0006] The handwriting data reproduction device according to the present invention is a handwriting data reproduction device comprising a display and a processor, wherein the processor acquires digital ink including stroke data and haptic data indicating tactile feedback, reproduces the stroke data to display the stroke data on the display, and controls a force-sense generating device to reproduce the tactile feedback indicated by the haptic data when the stroke data is reproduced.

[0007] The digital ink data structure of the present invention is used in a computer including a display, a memory unit, and a processor, and is a digital ink data structure stored in the memory unit, and includes stroke data and haptic data that associates haptic feedback with target stroke data among the stroke data.The digital ink data structure is used in a process in which the processor retrieves the contents of the haptic feedback from the memory unit when reproducing the stroke data to display it on the display and controlling a force-sense generating device to reproduce the haptic feedback indicated by the haptic data during the reproduction of the stroke data. [Effects of the Invention]

[0008] According to the present invention, it is possible to generate haptic feedback according to stroke data. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram showing the appearance of tablet terminals 1a and 1b according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing the internal configuration of tablet terminals 1a and 1b shown in FIG. [Figure 3] 2 is a diagram showing the data structure of digital ink DINK generated by the tablet terminal 1a shown in FIG. 1. FIG. [Figure 4]FIG. 5 is a diagram showing an example of the stroke data group 102 shown in FIG. [Figure 5] 5 is a diagram showing an example of haptic effect definition data 121 shown in FIG. 4. FIG. [Figure 6] FIG. 5 is a diagram showing an example of haptic data 105 shown in FIG. 4. [Figure 7] 1. FIG. 4 is a flowchart showing a haptic feedback providing process performed by the tablet terminal 1a shown in FIG. [Figure 8] 1. FIG. 4 is a flowchart showing a haptic feedback reproduction process performed by the tablet terminal 1b shown in FIG. [Figure 9] 9 is a flowchart showing details of the stroke data reproduction process shown in FIG. 8. FIG. [Figure 10] FIG. 10 is a diagram showing the structure of partial haptic data PHD according to the second embodiment of the present invention. [Figure 11] FIG. 10 is a diagram showing details of stroke data reproduction processing according to the second embodiment of the present invention. [Figure 12] FIG. 11 is a diagram showing an example of a stroke data group 102 according to a third embodiment of the present invention. [Figure 13] FIG. 11 is a flowchart showing stroke data generation processing performed by a tablet terminal 1a according to a third embodiment of the present invention. [Figure 14] FIG. 10 is a diagram showing a correspondence table that associates pen pressure values ​​with haptic data. [Figure 15] FIG. 11 is a flowchart showing haptic feedback reproduction processing performed by a tablet terminal 1b according to a third embodiment of the present invention. [Figure 16] FIG. 16 is a diagram illustrating a haptic feedback reproduction area set in step S53 of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0011] Fig. 1 is a diagram showing the appearance of tablet terminals 1a and 1b according to a first embodiment of the present invention, and Fig. 2 is a diagram showing the internal configuration of tablet terminals 1a and 1b.

[0012] As can be seen from FIG. 2, tablet terminals 1a and 1b, which are computers, have the same configuration and are connected via a communication network 2. In this embodiment, tablet terminal 1a is a "handwritten data generation device" for generating digital ink DINK and is configured to supply the generated digital ink DINK to tablet terminal 1b via communication network 2. Tablet terminal 1b is a "handwritten data reproduction device" for reproducing digital ink DINK and is configured to reproduce the digital ink DINK supplied from tablet terminal 1a. Of course, each of tablet terminals 1a and 1b may have the functions of both a "handwritten data generation device" and a "handwritten data reproduction device." Furthermore, the functions of a "handwritten data generation device" and a "handwritten data reproduction device" can also be implemented in computers other than tablet terminals, such as personal computers or smartphones.

[0013] 2, each of the tablet terminals 1a and 1b is configured to include a host processor 10, a storage unit 11, a display 12, a sensor 13, a sensor controller 14, a force-sense generating device 15, and a communication unit 16. While the following will focus on the tablet terminal 1a for detailed explanation, the same applies to the tablet terminal 1b. Note that the tablet terminal 1a may be configured to have a touch surface that receives touch from a pointer such as an electronic pen P or a finger, without having a display 12.

[0014] The host processor 10 is a central processing unit that controls the entire tablet terminal 1a, including the display 12 and the force-sense generating device 15, and serves to execute the operating system of the tablet terminal 1a and various applications such as drawing software by executing programs stored in the memory unit 11. The memory unit 11 is a storage device configured to be able to store any data, and includes a main storage device such as a DRAM (Dynamic Random Access Memory) and an auxiliary storage device such as a hard disk.

[0015] The display 12 is a display device that displays arbitrary data under the control of the host processor 10, and is, for example, a liquid crystal display or an organic EL display. The sensor 13 is a sensor pattern composed of a plurality of sensor electrodes each connected to a sensor controller 14, and is arranged inside the display surface of the display 12. As a result, in the tablet terminal 1a, the display surface of the display 12 becomes the touch surface. However, a touch surface may be provided at a position separate from the display surface of the display 12, in which case the sensor 13 is arranged inside the touch surface.

[0016] The sensor controller 14 is an integrated circuit that detects the position of a pointer, such as the electronic pen P or a finger shown in FIG. 1 , on the touch surface via the sensor 13 and has the function of communicating with the electronic pen P. The sensor controller 14 detects the position of the finger, for example, using a capacitance method, and detects the position of the electronic pen P and communicates with the electronic pen P, for example, using an active electrostatic method. Note that the sensor controller 14 may also be configured to detect the position of the electronic pen P and communicate with the electronic pen P using an electromagnetic induction method. When the sensor controller 14 detects the position of the finger or electronic pen P, or when it receives data from the electronic pen P through communication with the electronic pen P, the sensor controller 14 is configured to sequentially supply these positions or data to the host processor 10. Based on the positions and data thus supplied, the host processor 10 executes processes such as generating and drawing digital ink DINK and moving a cursor.

[0017] The active electrostatic method will be briefly described below. In the following, a signal transmitted from the sensor controller 14 to the electronic pen P will be referred to as an "uplink signal," and a signal transmitted from the electronic pen P to the sensor controller 14 will be referred to as a "downlink signal."

[0018] The sensor controller 14 transmits an uplink signal periodically, in principle at regular time intervals, using the sensor 13. The uplink signal thus transmitted contains a command that is an instruction to the electronic pen P.

[0019] The electronic pen P is composed of a pen tip electrode provided at the pen tip, a control unit that receives uplink signals and transmits downlink signals via this pen tip electrode, a pen pressure detection unit that detects the value of pressure applied to the pen tip (pen pressure value), a grip force detection unit that detects the value of pressure applied to the side of the housing of the electronic pen P (grip force value), a power supply that supplies the power necessary for the operation of the electronic pen P, and a memory unit that stores various data such as a pen ID for uniquely identifying the electronic pen P.

[0020] The control unit of the electronic pen P is configured to transmit a downlink signal in response to receiving the uplink signal. If the sensor controller 14 and the electronic pen P have not yet detected each other, the transmitted downlink signal will be a signal containing only a burst signal, which is an unmodulated carrier wave signal. On the other hand, if the sensor controller 14 and the electronic pen P have already detected each other, the transmitted downlink signal will be a signal containing a shorter burst signal and a data signal modulated with data corresponding to the command contained in the uplink signal. The data transmitted by the data signal may include the above-mentioned writing pressure value, grip force value, pen ID, etc.

[0021] Before detecting the electronic pen P, the sensor controller 14 attempts to detect the position of the electronic pen P over the entire touch surface (global scan) by receiving burst signals using all sensor electrodes included in the sensor 13. After detecting the position of the electronic pen P as a result, the sensor controller 14 updates the position of the electronic pen P by receiving burst signals using only one or more sensor electrodes in the vicinity of the detected position (local scan), and receives data transmitted by the electronic pen P by receiving and demodulating data signals.

[0022] Here, the electronic pen P may have multiple pen tip electrodes arranged in the pen axis direction and may be configured to transmit a burst signal from each pen tip electrode. In this case, the sensor controller 14 is configured to distinguish and receive the burst signals transmitted from each pen tip electrode to derive the respective positions and acquire a tilt angle indicating the inclination of the electronic pen P based on the results. The tilt angle is also supplied to the host processor 10, like other data, and used for processes such as generating and drawing digital ink DINK. Note that a gyroscope may be provided within the electronic pen P, and the tilt angle of the electronic pen P may be calculated from its output value. In this case, the electronic pen P may supply the gyroscope output value to the sensor controller 14, and the tilt angle may be calculated within the sensor controller 14. Alternatively, the tilt angle may be calculated within the electronic pen P, and the calculated tilt angle may be supplied from the electronic pen P to the sensor controller 14.

[0023] Returning to the explanation of each component shown in Figure 2, the force-sense generating device 15 is a device for providing tactile feedback to the user, and may be configured, for example, by a vibrator that vibrates in response to the waveform of an input current, or by a magnetic fluid whose hardness can be controlled by the waveform of an input current. In a typical example, the force-sense generating device 15 is disposed inside the housing of the tablet terminal 1a, and is configured to apply vibrations to the surface of the housing, including the touch surface.

[0024] The communication unit 16 is a communication device for connecting the host processor 10 to the communication network 2. The communication unit 16 in this embodiment is used to transmit and receive digital ink DINK between the tablet terminal 1a and the tablet terminal 1b.

[0025] 3 is a diagram showing the data structure of digital ink DINK generated by tablet terminal 1a. As shown in the figure, digital ink DINK is composed of ink data 100 and metadata 120. Of these, ink data 100 is composed of stroke data group 102, each of which includes multiple stroke data ST (see FIG. 4), each of which indicates the result of handwritten input (i.e., the trajectory of the pointer), and context data 101 used to draw each stroke data ST.

[0026] The context data 101 includes brush attribute data 103, a timestamp 104, and haptic data 105. The brush attribute data 103 stores data (brush attributes) required for drawing stroke data ST, such as a drawing color 107, transparency 108, and a tip shape 109 of the electronic pen P, in association with a brush ID 106. The timestamp 104 represents the time at which input of the first stroke data ST included in the digital ink DINK was initiated, expressed as the elapsed time from a predetermined time (e.g., January 1, 1970, 0:00:00). The haptic data 105 associates haptic feedback with one or more stroke data ST, and includes a timestamp 111, target stroke data identification data 112, and haptic feedback identification data 113, all associated with an area ID 110. Each piece of data constituting the haptic data 105 will be described in detail later.

[0027] 4 is a diagram showing an example of the stroke data group 102. As shown in the figure, each stroke data ST included in the stroke data group 102 is <trace> Tags and< / trace>The context data 101 has a structure including multiple pieces of position data PD between tags. As shown in FIG. 4, each piece of position data PD includes an X coordinate, a Y coordinate, a tilt angle Tilt, a pen pressure value F1, and a grip force value F2. Stroke data ST also includes a brush attribute reference value brushRef and a time offset timeOffset as attributes. The brush attribute reference value brushRef is set to the brush ID 106 described above. Each piece of stroke data ST is drawn in accordance with each brush attribute stored in the brush attribute data 103 in association with this brush ID 106. The time offset timeOffset is set to data representing the start time of input of the stroke data ST as the elapsed time from the time represented by the timestamp 104 in the context data 101.

[0028] Here, the sensor controller 14 usually detects the position and acquires the data at regular time intervals. Therefore, the host processor 10 can derive the moving speed of the indicator and the length of the stop time of the indicator from the amount of change in the coordinates included in the stroke data ST. The moving speed or the length of the stop time thus derived can be used to draw the stroke data ST.

[0029] Returning to Figure 3, metadata 120 is information added to ink data 100 and includes haptic effect definition data 121. Haptic effect definition data 121 is data that defines the specific content of haptic feedback (i.e., the content of the force generated by force-sense generation device 15), and includes, in association with definition ID 122, a wave file 123 (waveform data) that defines the input waveform to force-sense generation device 15 or its path information. When tablet terminal 1b reproduces haptic feedback, the reproduction of haptic feedback is achieved by supplying a current conforming to this wave file 123 to force-sense generation device 15.

[0030] FIG. 5 is a diagram showing an example of haptic effect definition data 121. "HE1," "HE2," and "HE3" shown in the figure each have a definition ID 122. The haptic effect definition data 121 in the figure includes wave files 123 (wave-file 20, wave-file 30) associated with "HE2" and "HE3." "HE1" includes the description "No haptics," which indicates that no haptic feedback is to be provided. In this way, it is also possible to define in haptic effect definition data 121 that no haptic feedback is to be provided.

[0031] 6 is a diagram showing an example of haptic data 105. The haptic data 105 shown in this figure corresponds to the digital ink shown in FIG. 1 (digital ink showing a letter from a father to a son).

[0032] 6, haptic data 105 is composed of one or more partial haptic data PHDs each including a different region ID 110. Each partial haptic data PHD includes a timestamp 111, target stroke data identification data 112, and haptic feedback identification data 113 in addition to region ID 110.

[0033] Area ID 110 is a serial number assigned to an area designated by the user as a target for haptic feedback (hereinafter referred to as a "haptics-imparting area"). FIGS. 1 and 6 show an example in which four haptics-imparting areas D1 to D4 have been designated by the user. Details will be described later, but the user designates each haptics-imparting area by inputting stroke data ST (shown by dashed lines in FIG. 1) that surrounds one or more stroke data ST using a pointer such as an electronic pen P.

[0034] The target stroke data identification data 112 identifies one or more stroke data ST included in the corresponding haptics-imparted region and is described by the sequential number of the stroke data ST within the digital ink DINK. Hereinafter, each of the one or more stroke data ST identified by the target stroke data identification data 112 will be referred to as the “target stroke data ST.” For example, in the haptics-imparted region D1 shown in FIG. 1, the target stroke data ST includes the series of stroke data ST representing the character string “I love you, Emma-chan,” the series of stroke data ST constituting the heart symbol, and the series of stroke data ST representing the colored circle (shown in gray in the drawing, but actually depicted in chromatic colors) that forms the background of the heart symbol. Similarly, in the haptics-imparted region D2 shown in FIG. 1, the target stroke data ST includes the series of stroke data ST constituting the cherry (including the leaves and colored portions).

[0035] The haptic feedback specification data 113 is data that defines the content of the haptic feedback to be played when the target stroke data ST is played, and is described using the definition ID 122 shown in Fig. 3. For example, in the case of the haptics-imparting area D1, since "HE3" is described in the haptic feedback specification data 113, the "wave-file 30" (see Fig. 5) corresponding to the definition ID 122 "HE3" becomes the content of the haptic feedback for the haptics-imparting area D1.

[0036] The timestamp 111 is timing information indicating the start timing of playback of the corresponding haptic feedback, and is described by the input start time of the earliest of the one or more corresponding target stroke data (i.e., the earliest time indicated by the time offset timeOffset shown in FIG. 4). In one example, the time offset timeOffset set for the earliest of the one or more corresponding target stroke data is directly set as the timestamp 111. The timestamp 111 also plays a role in determining the order of each haptic-imparting area.

[0037] Below, with reference to a flow diagram showing the processing performed by the tablet terminals 1a and 1b, the processing performed by the tablet terminal 1a as a handwritten data generation device and the processing performed by the tablet terminal 1b as a handwritten data reproduction device will be explained in more detail.

[0038] 7 is a flow diagram showing the haptic feedback providing process performed by the tablet terminal 1a. The process shown in the figure is executed by the host processor 10 of the tablet terminal 1a while the digital ink DINK to which the haptic feedback is to be provided is being displayed.

[0039] 7, the tablet terminal 1a first enters a haptic feedback providing mode in response to a user operation (step S1). After entering the haptic feedback providing mode, the tablet terminal 1a waits for the user to specify the haptic providing area described above. This specification is performed by inputting stroke data ST that surrounds one or more stroke data ST using a pointer such as an electronic pen P, as described above.

[0040] When the stroke data ST input by the pointer forms a closed region, the tablet device 1a accepts the designation of a haptics-imparting region (step S2). Then, based on the designated haptics-imparting region, one or more target stroke data ST to which haptic feedback is to be imparted is determined (step S3). Specifically, stroke data ST that is entirely contained within the designated haptics-imparting region may be determined as the target stroke data ST, or stroke data ST that is partially contained within the designated haptics-imparting region may be determined as the target stroke data ST.

[0041] Next, the tablet device 1a acquires the time offset timeOffset (see FIG. 4) of the earliest one of the one or more target stroke data that have been determined (step S4), and accepts the selection of the haptic feedback to be applied (step S5). The selection in step S5 is made by selecting one of the multiple definition IDs 122 stored in the haptic effect definition data 121 shown in FIG. 5. However, it is also possible to select multiple definition IDs 122 here. This point will be explained in detail later in the second embodiment.

[0042] Next, tablet device 1a generates the partial haptics data PHD shown in FIG. 6. Specifically, tablet device 1a first assigns area ID 110 to the haptics-imparting area received in step S2 and sets the area ID in the partial haptics data PHD. Then, in the partial haptics data PHD, timestamp 111, target stroke data identification data 112, and haptic feedback identification data 113 are set in association with this area ID 110. The time offset "timeOffset" acquired in step S4 is set in timestamp 111. The serial number of each of the one or more target stroke data ST determined in step S3 is set in target stroke data identification data 112. The definition ID 122 received in step S5 is set in haptic feedback identification data 113.

[0043] After completing the generation of the partial haptic data PHD, the tablet terminal 1a determines whether the designation of the haptics-imparting area is complete based on the presence or absence of a user operation indicating completion (step S7). If it determines that the designation is not complete, the tablet terminal 1a returns to step S2 and accepts the designation of the next haptics-imparting area. On the other hand, if it determines that the designation is complete, the tablet terminal 1a generates digital ink DINK including one or more partial haptic data PHDs generated up to that point (step S8). The above process completes the application of haptic feedback to the digital ink DINK. As can be understood from the description so far, the above haptic feedback application process can be performed by the user who wrote the digital ink DINK.

[0044] 8 is a flow diagram showing the haptic feedback reproduction process performed by the tablet terminal 1b. Note that the process shown in the figure is executed by the host processor 10 of the tablet terminal 1b.

[0045] First, tablet terminal 1b acquires digital ink DINK including haptic data 105 from tablet terminal 1a and displays it on its own display 12 (step S10). Next, tablet terminal 1b determines whether the user has pressed the PLAY_ALL button or the digital ink display area (step S11).

[0046] Here, the PLAY_ALL button is a button for inputting a user instruction to tablet terminal 1b to play all of the stroke data ST in the displayed digital ink DINK. In one example, tablet terminal 1b displays the PLAY_ALL button together with the digital ink DINK on display 12, and determines that the PLAY_ALL button has been pressed when the PLAY_ALL button is tapped with a pointer. The same applies to pressing the display area of ​​the digital ink DINK; tablet terminal 1b determines that the display area of ​​the digital ink DINK has been pressed when a position within the display area of ​​the digital ink DINK is tapped with a pointer.

[0047] If it is determined in step S11 that the PLAY_ALL button has been pressed, the tablet terminal 1b first stores the current time in the storage unit 11 shown in Fig. 2 (step S12), and then starts a stroke data playback process for all stroke data ST in the digital ink DINK as the playback target (step S13).

[0048] 9 is a flow diagram showing the details of the stroke data reproducing process. Having started the stroke data reproducing process, the tablet terminal 1b determines whether the reproducing start timing for any of the stroke data ST has arrived (step S30). Specifically, the tablet terminal 1b compares the time offset timeOffset of each stroke data ST to be reproduced with the elapsed time since the time stored in the storage unit 11 in step S12, and when stroke data ST is generated in which the latter exceeds the former, it determines that the reproducing start timing for that stroke data ST has arrived. When the tablet terminal 1b determines that the reproducing start timing for a certain stroke data ST has arrived, it starts reproducing that stroke data ST (step S31).

[0049] Here, reproduction of each stroke data ST is performed by plotting each coordinate included in the stroke data ST at the above-mentioned fixed time interval (the time interval for position detection by the sensor controller 14) and drawing a curve (e.g., a Catmull-Rom curve) that interpolates between the coordinates. To draw the curve, the tablet terminal 1b may use one or more of the tilt angle Tilt, the writing pressure value F1, the grip force value F2, the movement speed of the indicator, the length of time the indicator is stopped, and various brush attributes (the drawing color 107 of the stroke data ST, the drawing color 107 of the stroke data ST, the transparency 108 of the stroke data ST, the tip shape 109 of the electronic pen P, etc.). For example, the curve may be drawn using a line width corresponding to the writing pressure value F1 and the movement speed of the indicator, and a line color corresponding to the drawing color 107. The reproduction of the stroke data ST thus performed ends when the plotting of all coordinates included in the stroke data ST and the drawing of the corresponding curves are completed.

[0050] If it is determined in step S30 that the playback start timing for any stroke data ST has not arrived, or if step S31 has ended, tablet device 1b determines whether the playback start timing for any haptic feedback has arrived (step S32). Specifically, tablet device 1b compares the timestamp 111 of each partial haptic data PHD included in digital ink DINK with the elapsed time since the time stored in storage unit 11 in step S12. If partial haptic data PHD occurs for which the latter exceeds the former, tablet device 1b determines that the playback start timing for the haptic feedback included in that partial haptic data PHD has arrived. If tablet device 1b determines that the playback start timing for a certain haptic feedback has arrived, it first stops the supply of current to the force-sense generating device 15 to terminate the playback of the haptic feedback currently being played (step S33). Then, tablet device 1b starts the playback of the haptic feedback by starting the supply of current to force-sense generating device 15 based on the haptic feedback (specifically, the wave file) for which the playback start timing has arrived (step S34).

[0051] If it is determined in step S31 that the timing to start playback of any haptic feedback has not arrived, or if step S34 has ended, tablet device 1b determines whether the timing to end playback of the haptic feedback currently being played has arrived (step S35). Specifically, when playback of all stroke data ST included in partial haptic data PHD corresponding to the haptic feedback currently being played has been completed, tablet device 1b determines that the timing to end playback of the haptic feedback currently being played has arrived. When tablet device 1b determines that the timing to end playback of the haptic feedback currently being played has arrived, it stops supplying current to force-sense generating device 15 to end playback of the haptic feedback (step S36).

[0052] If it is determined in step S35 that the playback end timing for the haptic feedback being played has not yet arrived, and if step S36 has ended, the tablet terminal 1b determines whether playback of all stroke data ST to be played has been completed (step S37). If it is determined that playback has been completed, the stroke data playback process and the haptic feedback playback process are terminated, and if it is determined that playback has not been completed, the process returns to step S30 and is repeated.

[0053] Returning to Figure 8, if it is determined in step S11 that the display area of ​​the digital ink DINK has been pressed, the tablet terminal 1b first acquires stroke data ST corresponding to the pressed position (step S20). Specifically, the stroke data ST that passes through the pressed position may be acquired, or if there is no stroke data ST that passes through the pressed position, the stroke data ST that is closest to the pressed position may be acquired.

[0054] Next, tablet device 1b determines whether there is partial haptic data PHD containing target stroke data identification data 112 indicating the stroke data ST acquired in step S20 (step S21). If it determines that there is no target stroke data PHD, tablet device 1b terminates the haptic feedback reproduction process. On the other hand, if it determines that there is target stroke data PHD, tablet device 1b acquires the target stroke data PHD (step S22) and stores in storage unit 11 a time preceding the current time by the time indicated by timestamp 111 of the acquired partial haptic data PHD (step S23). This is to enable playback of the corresponding stroke data ST and the corresponding haptic feedback to begin immediately after the user taps the display area of ​​the digital ink DINK. Then, the tablet device starts the stroke data reproduction process, targeting all stroke data ST in the acquired partial haptic data PHD as the playback targets (step S24).

[0055] Details of the stroke data reproduction process started here are the same as those described with reference to Fig. 9. However, the determinations in steps S30 and S32 are made based on the time stored in storage unit 11 in step S23, instead of the time stored in storage unit 11 in step S12. Furthermore, the stroke data ST for which the reproduction start timing is determined in step S30 and the haptic feedback for which the reproduction start timing is determined in step S32 are only those included in the partial haptic data PHD acquired in step S22.

[0056] As described above, with the tablet terminals 1a, 1b and digital ink DINK according to this embodiment, the tactile feedback to be applied to one or more target stroke data ST can be set within the digital ink DINK, allowing the person who wrote the text to intentionally set the tactile feedback during playback.

[0057] Next, tablet terminals 1a and 1b according to a second embodiment of the present invention will be described. The tablet terminals 1a and 1b according to this embodiment differ from the tablet terminals 1a and 1b according to the first embodiment in the structure of the generated partial haptic data PHD and the specific contents of the stroke data reproduction process, but are otherwise similar to the tablet terminals 1a and 1b according to the first embodiment. The following description will focus on the differences from the tablet terminals 1a and 1b according to the first embodiment.

[0058] 10 is a diagram showing the structure of partial haptics data PHD according to this embodiment. As shown in the figure, partial haptics data PHD according to this embodiment differs from partial haptics data PHD according to the first embodiment shown in FIG. 6 in that haptic feedback specification data 113 is stored in association with pen pressure value F1 (specifically, range 114 of pen pressure value F1). To generate such partial haptics data PHD, tablet terminal 1a according to this embodiment is configured to allow the user to select definition ID 122 for each range 114 of pen pressure value F1 in step S5 of FIG. 7.

[0059] 11 is a diagram showing details of the stroke data reproduction process according to this embodiment. As can be seen by comparing with FIG. 9, in this embodiment, after the negative determination in step S32, a process of determining whether the timing for changing the haptic feedback has arrived (step S32a) is added.

[0060] To explain step S32a in more detail, after obtaining a negative determination result in step S32, tablet device 1b according to the present embodiment determines whether the timing to change the haptic feedback to be reproduced has arrived based on the pen pressure value F1 included in the currently reproduced position data PD and the range 114 of pen pressure values ​​F1 included in the partial haptic data PHD corresponding to the currently reproduced haptic feedback. If it is determined that the timing has arrived, steps S33 and S34 are executed, as in the case where a positive determination result is obtained in step S32. In step S34, haptic feedback corresponding to the pen pressure value F1 included in the currently reproduced position data PD is reproduced. If it is determined that the timing has not arrived, the process proceeds to step S35.

[0061] As described above, with the tablet terminals 1a, 1b and digital ink DINK according to this embodiment, the writing pressure value F1 is associated with the haptic feedback specification data 113 in the partial haptics data PHD, and the timing for changing the haptic feedback is determined based on the writing pressure value F1, making it possible to change the haptic feedback in response to changes in the writing pressure value F1. Therefore, a user reproducing digital ink can view the reproduced digital ink while realistically feeling the writing pressure of the user who wrote the digital ink.

[0062] Although the present embodiment describes an example in which the haptic feedback is changed according to the pen pressure value F1, it goes without saying that the haptic feedback may be changed based on other criteria. For example, the haptic feedback may be changed according to a combination of two or more data selected from the tilt angle Tilt, the grip force value F2, the movement speed of the indicator, the length of time the indicator is stopped, various brush attributes (the drawing color 107 of the stroke data ST, the drawing color 107 of the stroke data ST, the transparency 108 of the stroke data ST, the tip shape 109 of the electronic pen P, etc.), and a combination of these with the pen pressure value F1. Even in this case, the specific structure of the partial haptics data PHD and the stroke data reproduction process may be similar to those described in the present embodiment. However, in step S32a when the brush attribute is used as the criterion for changing the haptic feedback, it is determined whether the timing for changing the haptic feedback to be reproduced has arrived based on the brush attribute of the currently reproduced stroke data ST, instead of the pen pressure value F1 included in the currently reproduced position data PD.

[0063] In the first and second embodiments, an example has been described in which one force-sense generating device 15 is provided in the tablet terminal 1b, which is the handwritten data reproducing device, but it is also possible to divide the display 12 into multiple areas, provide a force-sense generating device 15 for each area, and reproduce haptic feedback by driving the force-sense generating device 15 for the area containing the stroke data ST being reproduced. In this way, it becomes possible to provide haptic feedback locally to only a part of the display surface, rather than to the entire surface.

[0064] The force-sense generating device 15 may also be provided in a device separate from the tablet terminal 1b. For example, if a microphone is connected to the tablet terminal 1b and the tablet terminal 1b is used as a karaoke machine by playing music in sync with the playback of digital ink indicating lyrics, placing the force-sense generating device 15 inside the microphone makes it possible to provide haptic feedback to the hand holding the microphone in accordance with the progress of the music. The force-sense generating device 15 can also be provided in various other devices, such as smart bracelets, smart fingers, smart rings, smart glasses, smart shirts with built-in heart rate sensors and breathing sensors, smart watches, Bluetooth (registered trademark) key trackers, smart shoes, smart socks, smart pants, smart belts, and baby controllers using SGPS (Simultaneous Global Positioning System) / GPRS (General Packet Radio Service).

[0065] Coordinate data defining the haptics-imparting region may be placed within the partial haptics data PHD, which allows the execution of the processing from step S22 onward shown in FIG. 8 when the haptics-imparting region is pressed.

[0066] In the first and second embodiments, one or more target stroke data ST to which haptic feedback is to be applied are determined based on a haptics-to-be-applied area designated by the user, but one or more target stroke data ST may be determined by other methods. For example, a semantic segmentation process may be performed on the digital ink DINK, and an area obtained based on the results (e.g., an area including one or more stroke data corresponding to a semantic paragraph) may be determined as a haptics-to-be-applied area, and one or more target stroke data ST may be determined based on the haptics-to-be-applied area.

[0067] Next, tablet terminals 1a and 1b according to a third embodiment of the present invention will be described. The tablet terminals 1a and 1b according to this embodiment differ from the tablet terminals 1a and 1b according to the first embodiment in that haptic data is arranged in the position data PD of each stroke data ST rather than in the context data 101, and in that the communication unit 16 also supports short-range wireless communication such as Bluetooth (registered trademark) and is configured to be able to communicate with the electronic pen P via short-range wireless communication, but are otherwise similar to the tablet terminals 1a and 1b according to the first embodiment. The following description will focus on the differences from the tablet terminals 1a and 1b according to the first embodiment.

[0068] FIG. 12 is a diagram showing an example of stroke data group 102 according to this embodiment. As can be seen by comparing with the example shown in FIG. 4, stroke data ST according to this embodiment differs from stroke data ST according to the first embodiment in that haptic data HEP is included in each piece of position data PD. Haptic data HEP according to this embodiment is data indicating the content of haptic feedback, and is described, for example, by the playback intensity of haptic feedback (hereinafter referred to as "haptic intensity"). In the following, the explanation will continue assuming that haptic data HEP is haptic intensity.

[0069] The communication unit 16 (see FIG. 2) of the tablet terminal 1b according to this embodiment is configured to be able to communicate via short-range wireless communication such as Bluetooth (registered trademark). Similarly, the electronic pen P is also configured to be able to communicate via short-range wireless communication. The electronic pen P according to this embodiment further has a built-in force-sense generating device similar to the force-sense generating device 15 described above. As will be described in more detail later, the tablet terminal 1b according to this embodiment is configured to control the electronic pen P using short-range wireless communication in order to cause the force-sense generating device within the electronic pen P to generate tactile feedback.

[0070] 13 is a flow diagram showing the stroke data generation process performed by the tablet terminal 1a according to this embodiment. Note that the process shown in the figure is executed by the host processor 10 of the tablet terminal 1a.

[0071] 13, the tablet terminal 1a first determines whether or not the position and data of the electronic pen P have been supplied from the sensor controller 14 (step S40). The tablet terminal 1a repeatedly executes step S40 until the position and data of the electronic pen P are supplied.

[0072] After determining in step S40 that the position and data of the electronic pen P have been supplied, the tablet terminal 1a next determines whether or not pen-down has occurred (step S41). In this determination, the tablet terminal 1a determines that pen-down has occurred if the writing pressure value F1 included in the data supplied from the sensor controller 14 has changed from 0 to a value greater than 0. When it has determined that pen-down has occurred, the tablet terminal 1a executes processing to start generating stroke data ST (step S42). Specifically, the header portion ( <trace>tag).

[0073] If it is determined in step S41 that pen down has not occurred, or if step S42 has been completed, the tablet terminal 1a determines whether pen up has occurred (step S43). In this determination, the tablet terminal 1a determines that pen up has occurred if the writing pressure value F1 included in the data supplied from the sensor controller 14 has changed from a value greater than 0 to 0. If the tablet terminal 1a determines that pen up has occurred, it executes processing to end generation of stroke data ST (step S42). Specifically, the footer portion (< / trace> tag) and return to step S40.

[0074] If it is determined in step S43 that pen-up has not occurred, the tablet terminal 1a acquires haptic data HEP corresponding to the pen pressure value F1 included in the supplied data (step S45).

[0075] 14 is a diagram showing a correspondence table that associates the pen pressure value F1 with the haptic data HEP, which is the haptic intensity. The tablet terminal 1a stores this correspondence table in advance, and in the process of step S45, the haptic data HEP is acquired according to this correspondence table.

[0076] Returning to Figure 13, the tablet device 1a that has acquired the haptic data HEP adds the position data PD that includes the acquired haptic data HEP to the stroke data ST that is being generated, and returns the process to step S40. Through the above process, stroke data ST that includes the haptic data HEP in the position data PD is generated between pen-down and pen-up.

[0077] 15 is a flow diagram showing the haptic feedback reproduction process performed by tablet terminal 1b according to this embodiment. The process shown in the figure is executed by host processor 10 of tablet terminal 1b. In this example, host processor 10 is configured to, in response to receiving a user's designation of position data PD, use a force-sense generating device in electronic pen P or tablet terminal 1b to reproduce haptic feedback indicated by haptic data HEP included in the designated position data PD.

[0078] More specifically, the tablet terminal 1b first draws one or more pieces of stroke data ST (step S50), which results in one or more pieces of stroke data ST being displayed on the display 12 of the tablet terminal 1b.

[0079] Next, tablet terminal 1b determines whether or not the user has performed a mode setting operation to enter the haptic feedback reproduction mode (step S51). This operation is, for example, clicking or tapping a button displayed by tablet terminal 1b on display 12. Tablet terminal 1b repeats the process of step S51 until it determines that the mode setting operation has been performed, and if it determines that the mode setting operation has been performed, it enters the haptic feedback reproduction mode and determines the reproduction method of the haptic feedback (step S52).

[0080] The method of reproducing haptic feedback is, for example, either a method of causing a force-sense generating device in electronic pen P to reproduce haptic feedback, or a method of causing force-sense generating device 15 in tablet terminal 1b to reproduce haptic feedback. For example, if the mode setting operation is performed by electronic pen P, tablet terminal 1b may determine whether or not electronic pen P has a built-in force-sense generating device from the content of the downlink signal transmitted by electronic pen P, and if it is determined that electronic pen P has a built-in force-sense generating device, may decide to cause the force-sense generating device in electronic pen P to reproduce haptic feedback, or if it is determined that electronic pen P does not have a built-in force-sense generating device, may decide to cause the force-sense generating device 15 in tablet terminal 1b to reproduce haptic feedback. Also, for example, if the mode setting operation is performed with a finger, tablet terminal 1b may decide to cause the force-sense generating device 15 in tablet terminal 1b to reproduce haptic feedback.

[0081] Next, the tablet terminal 1b sets a haptic feedback reproduction area around the drawing area of ​​the stroke data ST drawn in step S50 (step S53).

[0082] Fig. 16 is a diagram illustrating the haptic feedback reproduction area set in step S53. A curve 200 shown in Fig. 16(a) represents an approximation curve (specifically, a Bezier curve, a Catmull-Rom curve, or the like) obtained from a plurality of coordinates included in the stroke data ST. A circle 201 shown in Fig. 16(a) represents the position of each coordinate included in the stroke data ST and the magnitude of the pen pressure value F1 at that position.

[0083] When drawing the line stroke data ST in step S50, the tablet terminal 1b calculates envelopes 202 and 203 of the circle 201 at each coordinate. Then, as shown in FIG. 16(b), the area between the envelopes 202 and 203 is acquired as a drawing area 210 for the stroke data ST. The tablet terminal 1b draws the stroke data ST by filling the drawing area 210 with a given color, and as a result, the user can visually recognize the stroke data ST.

[0084] Having drawn the stroke data ST in this way, the tablet terminal 1b sets a haptic feedback reproduction area 211 that is invisible to the user in the drawing area 210 and its periphery (on both sides) in step S53. As a specific example, the tablet terminal 1b virtually sets circles obtained by increasing the radius of each of the above-mentioned circles 201 by a predetermined percentage (for example, 5%), calculates their envelopes 202a and 203a, and sets the area sandwiched between the envelopes 202a and 203a as the haptic feedback reproduction area 211. In this way, the tablet terminal 1b can reproduce haptic feedback in an area slightly larger than the drawing area 210. If such an effect is not required, the tablet terminal 1b can set the drawing area 210 as the haptic feedback reproduction area 211.

[0085] Returning to FIG. 15, the tablet terminal 1b that has set the haptic feedback reproduction area determines whether or not the user has performed a mode cancel operation to cancel the haptic feedback reproduction mode (step S54). This operation may also be, for example, clicking or tapping a button that the tablet terminal 1b has displayed on the display 12. If the tablet terminal 1b determines in step S54 that the mode cancel operation has been performed, it returns the process to step S51. On the other hand, if it determines that the mode cancel operation has not been performed, the tablet terminal 1b next determines whether or not the position of the indicator has been supplied from the sensor controller 14 (step S55).

[0086] If the tablet terminal 1b determines in step S55 that no data has been supplied, it returns to step S54 and continues processing. On the other hand, if the tablet terminal 1b determines that data has been supplied, it determines whether the supplied position is within the haptic feedback reproduction area set in step S53 (step S56). If the tablet terminal 1b determines that the position is not within the haptic feedback reproduction area, it returns to step S54 and continues processing. On the other hand, if the tablet terminal 1b determines that the position is within the haptic feedback reproduction area, it determines the haptic data HEP to be reproduced based on the supplied position (step S57). Specifically, it determines the position data PD that includes the coordinates closest to the supplied position from among one or more pieces of position data PD included in the stroke data ST being displayed, and determines the haptic data HEP included therein as the data to be reproduced.

[0087] Next, tablet terminal 1b determines whether the reproduction method determined in step S52 was a method of causing the force-sense generating device in electronic pen P to reproduce haptic feedback, or a method of causing the force-sense generating device 15 in tablet terminal 1b to reproduce haptic feedback (step S58). If it is determined to be the former, it transmits the haptic data HEP determined in step S57 to electronic pen P via short-range wireless communication (step S59), and if it is determined to be the latter, it supplies a current based on the haptic data HEP determined in step S57 to force-sense generating device 15 (step S60). As a result, haptic feedback corresponding to the position touched by the user with electronic pen P or a finger is reproduced by either electronic pen P or tablet terminal 1b.

[0088] Here, the waveform of the current supplied to the force-sense generating device as a result of tablet terminal 1b executing step S59 or step S60 may be set in advance in electronic pen P or tablet terminal 1b. However, it is also possible to accept waveform data designated by the user in the stroke data generation process shown in Fig. 13, and include information indicating that waveform data in the haptics data HEP. In this case, the corresponding wave file may be sent to electronic pen P in step S59.

[0089] As described above, with the tablet terminals 1a, 1b, electronic pen P, and digital ink DINK of this embodiment, haptic data HEP is placed within the position data PD of each stroke data ST, and tactile feedback indicated by the haptic data HEP included in the specified position data PD is reproduced in response to receiving a user's specification of position data PD. Therefore, when a user traces the displayed stroke data ST with the electronic pen P or a finger, it is possible to reproduce tactile feedback corresponding to the pen pressure value of the user when writing the part of the stroke data ST corresponding to the position where the electronic pen P or finger is touching.

[0090] Furthermore, according to the tablet terminals 1a, 1b, electronic pen P, and digital ink DINK of this embodiment, haptic data HEP is placed within the position data PD of the stroke data ST, so that tactile feedback can be reproduced even with electronic pens and tablet terminals that do not support the haptic data 105 or haptic effect definition data 121 shown in Figure 3.

[0091] In the present embodiment, an example has been described in which haptic feedback is reproduced when the user traces the displayed stroke data ST with the electronic pen P or a finger. However, haptic feedback may also be reproduced when the stroke data ST is reproduced. Specifically, when reproducing a portion of the stroke data ST corresponding to each piece of position data PD in the stroke data ST, haptic feedback indicated by the haptic data HEP included in that position data PD may be reproduced. In this way, the user reproducing the digital ink can view the reproduced digital ink while realistically feeling the pen pressure of the user who wrote the digital ink, as in the second embodiment.

[0092] In addition, in this embodiment, the haptic data HEP is calculated from the pen pressure value F1 in step S45 of FIG. 13, but the tablet terminal 1a may calculate the haptic data HEP from other values ​​such as the grip force value F2, the movement speed of the indicator, the tilt angle, and the gyro output value. Furthermore, the haptic data HEP may be calculated from the heart rate and blood oxygen level by attaching a device to the user to measure the heart rate and blood oxygen level and providing the measurement results to the tablet terminal 1a. In this case, the left column of FIG. 14 may contain a combination of values ​​to be referenced to calculate the haptic data HEP.

[0093] 15, haptic data HEP is transmitted to electronic pen P each time a position is supplied from sensor controller 14 to host processor 10, but tablet terminal 1b may transmit haptic data HEP to electronic pen P only when the content of haptic data HEP has changed, and electronic pen P may continue to reproduce tactile feedback based on the haptic data HEP received up to that point until new haptic data HEP is transmitted. This reduces the amount of communication between tablet terminal 1b and electronic pen P.

[0094] Furthermore, in the present embodiment, an example has been described in which values ​​such as the writing pressure value F1 are converted into haptic data HEP within tablet terminal 1b and the haptic data HEP is transmitted from tablet terminal 1b to electronic pen P, but it is also possible to convert values ​​such as the writing pressure value F1 into haptic data HEP within electronic pen P and transmit the values ​​such as the writing pressure value F1 from tablet terminal 1b to electronic pen P. In this case, it is preferable to store the correspondence table shown in FIG. 14 in electronic pen P in advance.

[0095] Furthermore, although the present embodiment has been described with reference to stroke data ST generated in response to input using an electronic pen P, the present invention is also applicable to stroke data ST generated in response to input using other types of indicators. For example, by determining haptic data HEP based on the movement speed of an air mouse operated in the air and arranging the determined haptic data HEP within position data PD, a user playing haptic feedback can realistically experience the movement speed of the air mouse. Similarly, by determining haptic data HEP based on the grip force value of a spray-type electronic pen and arranging the determined haptic data HEP within position data PD, a user playing haptic feedback can realistically experience the spray's emission intensity.

[0096] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and it goes without saying that the present invention can be embodied in various forms without departing from the spirit of the present invention. [Explanation of symbols]

[0097] 1a, 1b Tablet device 2. Communication Network 10 Host Processor 11 Storage section 12 Display 13 Sensors 14 Sensor Controller 15 Force generation device 16 Communications Department 100 ink data 101 Context Data 102 stroke data set 103 Brush attribute data 104 Timestamp 105,HEP Haptic Data 106 Brush ID 107 Drawing Color 108 Transparency 109 Tip shape of electronic pen P 110 Area ID 111 Timestamp 112 Target stroke data specific data 113 Haptic Feedback Specific Data 114 Pen pressure value F1 range 120 Metadata 121 Haptic effect definition data 122 Definition ID 123 wave files 200 Curve representing the trajectory 201 Circle representing pen pressure value 202,203,202a,203a envelope 210 Drawing area 211 Haptic Feedback Playback Area brushRef Brush attribute reference value D1~D4 Haptic area DINK Digital Ink F1 pen pressure value F2 Grip force value P Electronic pen PD Position Data PHD partial haptic data ST stroke data Tilt angle timeOffset time offset

Claims

1. An electronic pen capable of communicating with an information processing device, a vibration unit that generates haptic feedback to provide to a user; a transmitting unit that transmits information indicating that the haptic feedback can be generated to the information processing device; a receiving unit that receives haptic data related to generation of the haptic feedback, the haptic data being transmitted from the information processing device in response to reception of the information transmitted from the transmitting unit; a control unit that controls the vibration unit to generate the haptic feedback based on the haptic data received by the receiving unit; An electronic pen having

2. The electronic pen according to claim 1 , the receiving unit receives the haptic data from the information processing device via short-range wireless communication; An electronic pen characterized by:

3. The electronic pen according to claim 1 , the haptic data is data relating to the intensity of tactile feedback; An electronic pen characterized by:

4. The electronic pen according to claim 1 , the control unit drives the vibration unit based on waveform data related to the haptic data. An electronic pen characterized by:

5. The electronic pen according to claim 1 , The information is information indicating that the electronic pen has the vibration unit built in. An electronic pen characterized by:

6. The electronic pen according to claim 1 , the control unit continues to control the vibration unit based on the most recently received haptic data until the haptic data is changed. An electronic pen characterized by:

7. The electronic pen according to claim 1 , the control unit changes the vibration pattern of the vibration unit based on waveform data related to the haptic data. An electronic pen characterized by:

8. 3. The electronic pen according to claim 2, The short-range wireless communication is communication using Bluetooth (registered trademark). An electronic pen characterized by:

9. An information processing device, a receiving unit that receives information transmitted from the electronic pen; a processor that determines whether the electronic pen is capable of generating tactile feedback based on the information received by the receiving unit; a transmitter that transmits haptic data related to generation of the haptic feedback to the electronic pen when the processor determines that the haptic feedback can be generated; An information processing device having the above.

10. 10. The information processing device according to claim 9, the transmitting unit transmits the haptic data to the electronic pen by short-range wireless communication; 1. An information processing device comprising:

11. 10. The information processing device according to claim 9, the haptic data is data relating to the intensity of tactile feedback; 1. An information processing device comprising:

12. 10. The information processing device according to claim 9, the transmitting unit transmits waveform data relating to the haptic data to the electronic pen; 1. An information processing device comprising:

13. 10. The information processing device according to claim 9, the transmitting unit transmits the haptic data to the electronic pen when the content of the haptic data is changed.

1. An information processing device comprising:

14. The information processing device according to claim 10, The short-range wireless communication is communication using Bluetooth (registered trademark).

1. An information processing device comprising:

Images