Information processing system, information processing method, and program

Abstract

To allow a person to have a sensation close to the sensation that the user has when actually writing on a physical medium.SOLUTION: An information processing system includes: a contact parameter detection unit configured to detect at least one contact parameter corresponding to the contact state of a pen-type input device with respect to an operation target surface; and a feedback control unit configured to determine, on the basis of the contact parameters, the contact state of the pen tip of the pen-type input device with respect to the operation target surface and drive a haptic reproduction unit of the pen-type input device to reproduce haptics corresponding to a moving state when the contact state is determined to be the moving state of the pen tip, and drive the haptic reproduction unit to reproduce haptics corresponding to a movement start state at a timing when it is determined that the contact state of the pen tip of the pen-type input device has transitioned from a stationary state in which the pen tip is in contact with the operation target surface to the movement start state in which the pen tip starts moving.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] It is known that when inputting data to an interactive display system, a pen-shaped stylus (pen-type input device) that enables tactile feedback by generating vibrations that drive a tactile actuator is used as an input device (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-537395 Summary of the Invention [Problem to be solved by the invention]

[0004] The vibrations emitted by the pen-type input device simulate the tactile sensation of sliding the pen tip across a writing medium such as paper, allowing users using the pen-type input device to get a sensation similar to that of writing on a writing medium.

[0005] When inputting data using a pen-type input device, it is desirable to have a feeling as close as possible to that of writing on an actual writing medium. In consideration of the above-mentioned problems, the present invention aims to provide an input using a pen-type input device that provides a feeling similar to that of actually writing on a writing medium. [Means for solving the problem]

[0006] One aspect of the present invention that solves the above-mentioned problems is an information processing system that includes a contact parameter detection unit that detects one or more contact parameters corresponding to the contact state of a pen-type input device with respect to an operation surface, and a feedback control unit that determines the contact state of the pen tip of the pen-type input device with respect to the operation surface based on the contact parameters, and that, when it is determined that the contact state is a moving state in which the pen tip of the pen-type input device is moving while in contact with the operation surface, executes a first feedback control that drives a tactile reproduction unit in the pen-type input device to reproduce a tactile sensation corresponding to the moving state, and a second feedback control that drives the tactile reproduction unit to reproduce a tactile sensation corresponding to the moving start state when it is determined that the contact state is a moving start state in which the pen tip of the pen-type input device has started moving from a stationary state while in contact with the operation surface.

[0007] One aspect of the present invention is an information processing method in an information processing system, the information processing method including: a contact parameter detection step in which a contact parameter detection unit detects one or more contact parameters corresponding to a contact state of a pen-type input device with respect to an operation surface; and a feedback control step in which a feedback control unit determines a contact state of a pen tip of the pen-type input device with respect to the operation surface based on the contact parameters, and when it is determined that the contact state is a moving state in which the pen tip of the pen-type input device is moving while in contact with the operation surface, executes a first feedback control to drive a tactile reproduction unit in the pen-type input device to reproduce a tactile sensation corresponding to the moving state; and a second feedback control to drive the tactile reproduction unit to reproduce a tactile sensation corresponding to the moving start state when it is determined that the contact state is a moving start state in which the pen tip of the pen-type input device has started moving from a stationary state while in contact with the operation surface.

[0008] One aspect of the present invention is a program for causing a computer in an information processing system to function as a feedback control unit that executes: a contact parameter detection unit that detects one or more contact parameters corresponding to the contact state of a pen-type input device with respect to an operation surface; and a feedback control unit that determines the contact state of the pen tip of the pen-type input device with respect to the operation surface based on the contact parameters, and that, when it is determined that the contact state is a moving state in which the pen tip of the pen-type input device is moving while in contact with the operation surface, drives a tactile reproduction unit in the pen-type input device to reproduce a tactile sensation corresponding to the moving state; and a second feedback control that, when it is determined that the contact state is a movement start state in which the pen tip of the pen-type input device has started moving from a stationary state while in contact with the operation surface, drives the tactile reproduction unit to reproduce a tactile sensation corresponding to the movement start state. [Effects of the Invention]

[0009] According to the present invention, when inputting using a pen-type input device, it is possible to obtain an effect that the feeling of actually writing on a writing medium can be made to be closer to that of the user. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of the external configuration of an information processing system according to an embodiment of the present invention. [Figure 2] 10A and 10B are diagrams showing changes in the frictional force of the pen tip over time when a writing operation is started in this embodiment. [Figure 3] FIG. 2 is a diagram illustrating an outline of feedback control in this embodiment. [Figure 4] 10A and 10B are diagrams showing an example of a movement locus of a pen tip of a pen-type input device on a panel surface in this embodiment. [Figure 5] FIG. 1 is a diagram illustrating an example of the hardware configuration of an information processing apparatus and a pen-type input device according to an embodiment of the present invention. [Figure 6]1 is a diagram illustrating an example of the functional configuration of an information processing apparatus and a pen-type input device according to an embodiment of the present invention. [Figure 7] 10 is a diagram illustrating an example of a functional configuration corresponding to driving of a vibration unit and a sound output unit by a feedback signal in the present embodiment. FIG. [Figure 8] 10 is a flowchart illustrating an example of a processing procedure executed by the information processing device and the pen-type input device in relation to haptic feedback according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] 1 shows an example of the external configuration of an information processing system according to this embodiment. As shown in the figure, the information processing system according to this embodiment includes an information processing device 100 and a pen-type input device 200.

[0012] The information processing device 100 is capable of executing information processing in response to an input operation using a pen-type input device 200. The information processing device 100 in the figure is an example of a tablet terminal or a notebook-type personal computer.

[0013] The information processing device 100 includes a touch panel display unit 30. The touch panel display unit 30 is a component that combines a touch panel and a display unit. The touch panel display unit 30 displays an image on a panel surface (display surface: an example of a surface to be operated), and allows operation by contacting the panel surface with an operating object such as a pen-type input device or a finger.

[0014] The pen-type input device 200 is a pen-type input device that is used by a user to perform operations on the touch panel of the touch panel display unit 30 of the information processing device 100. The user holds the pen-type input device 200 and moves the pen tip so that it comes into contact with the panel surface of the touch panel display unit 30, thereby performing handwritten input operations for characters, pictures, figures, etc.

[0015] Note that, as an operation using the pen-type input device 200, a pointing operation on a user interface image displayed on the touch panel display unit 30 may also be possible.

[0016] The detection method of the pen-type input device 200 by the touch panel in the touch panel display unit 30 of this embodiment is not particularly limited, but examples thereof include a capacitance method and an electromagnetic induction method. In the following description, a case where the capacitance method is used will be taken as an example.

[0017] An application compatible with input operations using the pen-type input device 200 (pen operation compatible application) is installed in the information processing device 100. The pen operation compatible application may be capable of, for example, inputting characters by touching the tip of the pen-type input device 200 to the touch panel display unit 30, displaying characters or pictures drawn in response to handwriting input operations such as drawing, on the touch panel display unit 30, or performing processes such as converting characters or pictures drawn by the operation into data.

[0018] Furthermore, in the information processing system of this embodiment, the pen-type input device 200 emits vibrations and sounds (writing sounds) in response to an operation (writing operation) corresponding to writing. The vibrations and writing sounds emitted by the pen-type input device 200 allow the user to feel a tactile sensation similar to that felt when writing on a writing medium such as paper, and thus the user can obtain a sensation similar to that felt when writing on a writing medium with an actual writing implement.

[0019] In the following description, the vibrations and sounds generated when a writing operation is performed with the pen-type input device 200 will also be referred to as "tactile feedback."

[0020] Next, an overview of the haptic feedback in this embodiment will be described. Figure 2 shows the change in frictional force at the tip of a writing instrument over time as a user actually begins a writing operation. At time t0, the user has the tip of a writing instrument (e.g., a pencil) in contact with a writing medium such as paper, but the tip of the writing instrument is still stationary. Then, at time t1, a certain amount of time after time t0, the user begins to move the tip of the writing instrument over the writing medium in response to the start of writing. When the user starts moving the tip of the writing instrument, the tip of the writing instrument is pressed against the surface of the writing medium with a certain amount of force, generating static friction. Therefore, during the period from time t1 to time t2, even if the user applies force to move the tip of the writing instrument, the tip of the writing instrument remains stationary on the writing medium and the static friction increases over time. Then, at time t2, when the tip of the writing implement starts to move, the static friction force that had been increasing up until then suddenly decreases, and a certain kinetic friction force occurs in response to the movement of the tip of the writing implement. This change in friction force is observed as strong vibrations and sounds (writing sounds) in real writing.

[0021] When the pen tip of the pen-type input device 200 moves, tactile feedback in the form of vibration or sound may be provided at a constant level, but the level of the tactile feedback may be changeable depending on the state of movement of the pen tip. For example, the level of the tactile feedback may be changed so as to increase as the movement speed of the pen tip increases. However, when the tactile feedback level is changed according to the pen tip movement speed, the movement speed at the timing when the pen tip starts to move is low, so the tactile feedback at the timing when the pen tip starts to move is also small. In other words, changing the tactile feedback level according to the pen tip movement speed does not reproduce as tactile feedback the change in kinetic friction force according to the movement speed from the large static friction force that occurs instantaneously when the tip of the writing implement starts to move, as shown in Figure 2.

[0022] Therefore, in this embodiment, tactile feedback is controlled in accordance with the movement of the pen tip of the pen-type input device 200 as shown in FIG. 3, lines L1-1 and L1-2 show the change over time in the levels of two feedback signals output in response to the haptic feedback of this embodiment, and line L2 shows the change over time in the detected movement speed of the pen tip. Time t10 is the timing when the pen tip of the pen-type input device 200, which had been stationary up until that point, starts to move in response to the user's writing operation. In response to the start of pen tip movement at time t10, output of a first feedback signal begins, the level of which changes in response to the pen tip movement speed, as shown by line L1-1. At time t11 when the pen tip movement stops, the first feedback signal becomes zero level and output is stopped. Furthermore, in response to the start of movement of the pen tip at time t10, a second feedback signal having an instantaneous peak as shown by line L1-2 is also output.

[0023] In this manner, in this embodiment, from the time the pen tip starts to move until it stops, composite tactile feedback is provided by two feedback signals, the first feedback signal and the second feedback signal. Such tactile feedback not only reproduces vibrations and writing sounds corresponding to the pen tip movement speed, but also reproduces vibrations and writing sounds corresponding to the static friction force that momentarily increases at the timing when the pen tip of the pen-type input device 200 starts to move. As a result, when inputting using the pen-type input device 200, it becomes possible to approximate the sensation of actually writing on a writing medium.

[0024] 4 shows an example of a movement trajectory MT of the pen tip of the pen-type input device 200 on the panel surface PN due to a user's writing operation. The movement trajectory MT in the figure starts from a starting point p1, changes direction at direction change points p2, p3, p4, and p5 in this order, and ends at an end point p6. According to this movement trajectory MT, the movement of the pen tip is stopped at each of direction change points p2, p3, p4, and p5, and then restarted. In this case, haptic feedback based on the first feedback signal at a level corresponding to the movement speed from the start to the stop of the pen tip movement is provided for each of sections p1-p2, p2-p3, p3-p4, and p5-p6. Furthermore, haptic feedback based on the static friction force based on the second feedback signal is provided for each of the start point p1 and direction change points p2, p3, p4, and p5.

[0025] In this embodiment, the peak level of the second feedback signal may be a predetermined fixed value, or may be set based on a predetermined contact parameter among the detection values ​​(contact parameters) obtained by detecting a predetermined contact state of the pen tip with the panel surface. Specifically, the information processing device 100 of this embodiment can include, as contact parameters, the speed (movement speed) at which the pen tip moves on the panel surface, the pressure (writing pressure) when the pen tip is in contact with the panel surface, the movement direction of the pen tip on the panel surface, and the tilt angle of the pen-type input device 200 relative to the panel surface when the pen tip is in contact with the panel surface. The static friction force when starting to move the pen tip by writing increases according to the writing pressure when starting the movement. Therefore, the information processing device 100 may set the peak level of the second feedback signal according to the contact parameter as the writing pressure. Furthermore, the static friction force when starting to move the pen tip by a writing operation varies depending on, for example, the tilt angle of the pen-type input device 200 relative to the panel surface. Therefore, the information processing device 100 may set the peak level of the second feedback signal depending on the contact parameter as the tilt angle. Furthermore, the static friction force when starting to move the pen tip by a writing operation may also vary depending on, for example, the direction of movement of the pen tip on the panel surface in response to the writing operation. Therefore, the information processing device 100 may set the peak level of the second feedback signal depending on the contact parameter as the movement direction. Furthermore, the information processing device 100 may set the peak level of the second feedback signal by using two or more contact parameters in combination from among the contact parameters of the writing pressure, the tilt angle, and the movement direction.

[0026] An example of the hardware configuration of the information processing device 100 and the pen-type input device 200 will be described with reference to FIG. First, we will explain an example of the hardware configuration of an information processing device 100. The information processing device 100 in the figure includes a processor 11, a main memory 12, a flash memory 13, a peripheral device 14, a short-range communication unit 15, a baseband chip 21, a communication unit 22, an audio system 23, a microphone 24, a speaker 25, and a touch panel display unit 30.

[0027] The processor 11 is, for example, an application processor including a CPU (Central Processing Unit). The processor 11 controls the information processing device 100 as a whole.

[0028] Main memory 12 is a writable memory used as a read area for the executable program of processor 11 or as a work area for writing processing data for the executable program. Main memory 12 is composed of, for example, multiple DRAM (Dynamic Random Access Memory) chips. This executable program includes an OS (Operating System), various device drivers for operating peripheral hardware, various services / utilities, application programs (application software), etc.

[0029] The flash memory 13 is, for example, a flash EEPROM (Electrically Erasable Programmable Read Only Memory), and stores an OS, various drivers, various services / utilities, application programs (hereinafter sometimes referred to as applications), and various data.

[0030] The touch panel display unit 30 is a part that displays an image and allows an operation to be performed on the panel surface on which the image is displayed using the pen-type input device 200. The touch panel display unit 30 may also be operable using an operating object other than the pen-type input device 200, such as a finger, but here, an example will be described in which the operating object is the pen-type input device 200. The touch panel display unit 30 includes a display unit 31 and a panel sensor 32 . The display unit 31 is, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display, and displays an image based on drawing data (display data) output from the processor 11. The panel sensor 32 detects the contact state of the pen tip of the pen-type input device 200 with the panel surface of the display unit 31. Specifically, the panel sensor 32 may be capable of detecting the position on the panel surface where the pen tip of the pen-type input device 200 is in contact, the pressure of the pen tip in contact with the panel surface, the distance from the panel surface of the pen-type input device 200, etc. The detection method of such a panel sensor 32 is not particularly limited, but examples thereof include a capacitance method and an electromagnetic induction method. Furthermore, the panel sensor 32 may be configured to use a combination of such multiple detection methods.

[0031] The peripheral device 14 is, for example, a wireless local area network (WLAN) module, a global positioning system (GPS) module, and sensors such as an acceleration sensor.

[0032] The audio system 23 is, for example, an audio IC (Integrated Circuit), and inputs, records, plays back, and outputs sound data. For example, a microphone 24 and a speaker 25 are connected to the audio system 23. For example, the audio system 23 outputs sound data picked up by the microphone 24 to the processor 11 or the baseband chip 21. Furthermore, the audio system 23 converts sound data acquired from the processor 11 or the baseband chip 21 into a sound signal and outputs it to the speaker 25, for example.

[0033] The microphone 24 picks up sounds around the information processing device 100. For example, the microphone 24 picks up sounds such as the user's voice when audio integration with another terminal is performed. The speaker 25 outputs various sounds to the outside of the information processing device 100. For example, when audio integration with another terminal is performed, the speaker 25 outputs (emits) sounds received from the other terminal.

[0034] The baseband chip 21 is a dedicated IC that controls wireless communications such as 4G (fourth generation mobile communication system) and 5G (fourth generation mobile communication system). The baseband chip 21, for example, causes the audio system 23 to output sound data received by the communication unit 22 from the speaker 25 as sound. Furthermore, the baseband chip 21 acquires, for example, sound data collected by a microphone 24 via an audio system 23, and outputs the sound data via a mobile communication system using a communication unit 22. The baseband chip 21 may also be configured to exchange input and output data for data communication via the mobile communication system with the processor 11.

[0035] The communication unit 22 is a wireless communication device including an antenna for performing wireless communication via a mobile communication system.

[0036] The short-distance communication unit 15 is, for example, a Bluetooth (registered trademark) module, and performs short-distance wireless communication with the pen-type input device 200.

[0037] Next, with reference to the same FIG. 5, an example of the hardware configuration of the pen-type input device 200 will be described. The pen-type input device 200 includes an MCU 41, a short-range communication unit 42, a vibration unit 43 (an example of a tactile reproduction unit), a sound output unit 44 (an example of a tactile reproduction unit), and a flash memory 45.

[0038] The short-range communication unit 42 is, for example, a Bluetooth (registered trademark) module, and performs short-range wireless communication with the information processing device 100.

[0039] The MCU (Micro Controller Unit) 41 includes a CPU, memories such as ROM and RAM, and I / O related components, and executes control of the pen-type input device 200. The MCU 41 exchanges information via a short-range communication unit 42. The MCU 41 also controls the generation of vibrations by a vibration unit 43 and the output of sounds by a sound output unit 44 so that haptic feedback is provided in accordance with feedback signals (including a first feedback signal and a second feedback signal) transmitted from the information processing device 100.

[0040] The vibration unit 43 is a part that includes, for example, an actuator and vibrates in response to an input of a feedback signal from the MCU 41. The sound output unit 44 is a part that includes, for example, a speaker, and outputs sound in response to an input of a feedback signal from the MCU 41.

[0041] The flash memory 45 stores various data corresponding to the pen-type input device 200, such as programs and feedback signal data.

[0042] An example of the functional configuration of the information processing device 100 and the pen-type input device 200 will be described with reference to Fig. 6. In the figure, functional units that can be considered equivalent to the parts in the hardware configuration of Fig. 5 are assigned the same reference numerals as in Fig. 5, and descriptions thereof will be omitted as appropriate. The functions of the information processing device 100 shown in the figure may be realized by the processor 11 (FIG. 5) executing a program.

[0043] First, a description will be given of an example of the functional configuration of the information processing device 100. The information processing device 100 includes a communication unit 22, a touch panel display unit 30, a touch parameter detection unit 101, a control unit 102, and a storage unit 103 as functional units.

[0044] The contact parameter detection unit 101 detects the contact state of the pen tip of the pen-type input device 200 with the panel surface of the touch panel display unit 30 . The contact parameter detection unit 101 outputs parameters (contact parameters) detected in accordance with the contact state based on the detection signal output by the panel sensor 32. In the following description, the contact parameters will be exemplified by the aforementioned movement speed, writing pressure, movement direction, and tilt angle.

[0045] The control unit 102 executes various controls in the information processing device 100. The control unit 102 includes an application support processing unit 121 and a feedback control unit 122.

[0046] The application processing unit 121 executes processing corresponding to a pen operation compatible application. When a writing operation is performed with the pen-type input device 200 as an operation for a pen operation compatible application, pen operation information is input from the panel sensor 32 to the application compatible processing unit 121. In response to the input of the pen operation information, the application compatible processing unit 121 performs processing such as drawing.

[0047] The feedback control unit 122 executes feedback control, which in this embodiment is control that causes the pen-type input device 200 to generate vibration as tactile feedback. The feedback control unit 122 of this embodiment executes feedback control (second feedback control) for reproducing vibrations and sounds generated in response to changes in frictional force when the pen tip of the pen-type input device 200, which has been stationary in contact with the panel surface, starts to move, and feedback control (first feedback control) for reproducing vibrations and sounds generated when the pen tip of the pen-type input device 200 is moving. The feedback control unit 122 generates feedback control information based on the contact parameters output from the contact parameter detection unit 101. The feedback control information is information for controlling the drive control unit 221 of the pen-type input device 200 so as to generate tactile feedback in the form of vibration and sound. The feedback control unit 122 outputs (transmits) the generated feedback control information to the pen-type input device 200 via the communication unit 22.

[0048] The storage unit 103 stores various types of information corresponding to the information processing device 100 .

[0049] Next, a description will be given of an example of the functional configuration of the pen-type input device 200. The functions of the pen-type input device 200 shown in the figure may be realized by the MCU 41 (FIG. 5) executing a program. The pen-type input device 200 includes a short-range communication unit 42, a vibration unit 43, a sound output unit 44, a control unit 201, and a storage unit 203 as functional units.

[0050] The control unit 201 executes various controls in the pen-type input device 200. The control unit 201 includes a drive control unit 221. The drive control unit 221 controls the vibration unit 43 to generate vibration as haptic feedback, and also controls the sound output unit 44 to generate sound as haptic feedback. When controlling the vibration unit 43 and the sound output unit 44, the drive control unit 221 inputs the feedback signals (including the first feedback signal and the second feedback signal) stored in the feedback signal storage unit 231 to the vibration unit 43 and the sound output unit 44.

[0051] The storage unit 203 stores various data corresponding to the pen-type input device 200. The storage unit 203 stores a feedback signal storage unit 231. The feedback signal storage unit 231 stores waveforms that become feedback signals. The feedback signal storage unit 231 stores the waveform of a first feedback signal and the waveform of a second feedback signal. The first feedback signal and the second feedback signal stored in the feedback signal storage unit 231 may include a waveform of a vibration-corresponding feedback signal used to vibrate the vibration unit 43 and a waveform of a sound-corresponding feedback signal to be output as sound from the sound output unit 44, respectively.

[0052] The waveform of the second feedback signal may be, for example, an impulse waveform. That is, the second feedback signal may be an impulse signal. Alternatively, the waveform of the second feedback signal may be a waveform that has a steep peak at the time when vibration or sound is generated due to static friction when the pen tip starts to move.

[0053] An example of a functional configuration corresponding to driving of vibration section 43 and sound output section 44 by a feedback signal will be described with reference to FIG. In the information processing device 100, the contact parameter detection unit 101 includes a movement speed detection unit 111, a writing pressure detection unit 112, a movement direction detection unit 113, and an inclination angle detection unit 114.

[0054] The movement speed detection unit 111 detects the movement speed of the pen tip that moves while touching the panel surface. The movement speed detection unit 111 may detect the movement speed based on the amount of movement per unit time of the contact position of the pen tip detected by the panel sensor 32. The movement speed detection unit 111 outputs the detected movement speed parameter Dt1 to the feedback control unit 122.

[0055] The writing pressure detection unit 112 detects the pressure (writing pressure) of the pen tip in contact with the panel surface. The writing pressure detection unit 112 may detect the writing pressure based on the value of the capacitance at the contact position of the pen tip detected by the panel sensor 32. The writing pressure detection unit 112 outputs a writing pressure parameter Dt2 indicating the detected writing pressure to the feedback control unit 122.

[0056] The movement direction detection unit 113 detects the movement direction of the pen tip that moves while in contact with the panel surface. The movement speed detection unit 111 may detect the movement direction based on the movement direction of the contact position of the pen tip detected by the panel sensor 32. The movement direction detection unit 113 outputs a movement direction parameter Dt3 indicating the detected movement direction to the feedback control unit 122.

[0057] The tilt angle detection unit 114 detects the tilt angle of the main body of the pen-type input device 200 relative to the panel surface when the pen tip is in contact with the panel surface. The tilt angle detection unit 114 may detect the tilt angle based on the distribution of capacitance corresponding to the main body of the pen-type input device 200 detected around the contact position of the pen tip by the panel sensor 32. The tilt angle detection unit 114 outputs the detected tilt angle parameter Dt4 to the feedback control unit 122.

[0058] In the information processing device 100, the feedback control unit 122 generates feedback control information FB based on the input contact parameters (movement speed parameter Dt1, writing pressure parameter Dt2, movement direction parameter Dt3, and tilt angle parameter Dt4). In this embodiment, the feedback control unit 122 generates feedback control information FB corresponding to the vibration unit 43 and feedback control information FB corresponding to the sound output unit 44 separately. The feedback control unit 122 may control feedback control information that is common to the vibration unit 43 and the sound output unit 44 .

[0059] The feedback control information FB includes a level of the vibration-responsive first feedback signal (vibration-responsive first level) and a level of the sound-responsive first feedback signal (sound-responsive first level). The feedback control unit 122 may set the vibration-responsive first level and the sound-responsive first level based on the moving speed parameter Dt1. In addition, the feedback control information FB, which is output in accordance with the timing at which the pen tip starts to move on the panel surface, includes the level of the vibration-responsive second feedback signal (vibration-responsive second level) and the level of the sound-responsive second feedback signal (sound-responsive second level). In the following description, when there is no particular distinction between the vibration response level 1 and the sound response level 1, they will be referred to as the first level. Also, when there is no particular distinction between the vibration response level 2 and the sound response level 2, they will be referred to as the second level.

[0060] The feedback control unit 122 outputs (transmits) the generated feedback control information FB to the pen-type input device 200.

[0061] The drive control unit 221 drives the vibration unit 43 and the sound output unit 44 based on the input feedback control information FB.

[0062] In the figure, an example of the configuration of a vibration unit 43 and a sound output unit 44 is shown. The vibration unit 43 includes, for example, an amplifier 432 , an amplifier 433 , a mixer 434 , and an actuator 435 . The amplifier 432 receives a first feedback signal corresponding to vibration as a signal source S1-1. The signal source S1-1 is generated based on the waveform of the first feedback signal corresponding to vibration stored in the feedback signal storage unit 231. The amplifier 432 changes the input first feedback signal corresponding to vibration to a level instructed by the drive control unit 221 and outputs the signal. The amplifier 433 receives a second feedback signal corresponding to vibration as a signal source S2-1. The signal source S2-1 is generated based on the waveform of the second feedback signal corresponding to vibration stored in the feedback signal storage unit 231. The amplifier 433 changes the input second feedback signal corresponding to vibration to a level instructed by the drive control unit 221 and outputs the signal. Mixer 434 combines the first feedback signal output from amplifier 432 and the second feedback signal output from amplifier 433 . The actuator 435 vibrates in response to the feedback signal output from the mixer 434 .

[0063] In accordance with this configuration of the vibration unit 43, the drive control unit 221 controls the signal source S1-1 to input the vibration-responsive first feedback signal to the amplifier 432 based on the vibration-responsive first level included in the feedback control information FB, and controls the amplifier 432 to specify the level of the vibration-responsive first feedback signal. In addition, if the feedback control information FB includes a vibration-responsive second level, based on the vibration-responsive second level, control is performed to input a vibration-responsive second feedback signal from signal source S2-1 to amplifier 433, and control is performed to instruct amplifier 433 on the level of the vibration-responsive second feedback signal.

[0064] The sound output unit 44 includes, for example, an amplifier 442, an amplifier 443, a mixer 444, and a speaker 445. The amplifier 442 receives a first feedback signal corresponding to sound as a signal source S1-2. The signal source S1-2 is generated based on the waveform of the first feedback signal corresponding to sound stored in the feedback signal storage unit 231. The amplifier 442 changes the input first feedback signal corresponding to sound to a level instructed by the drive control unit 221 and outputs the signal. The amplifier 443 receives a second feedback signal corresponding to sound as a signal source S2-2. The signal source S2-1 is generated based on the waveform of the second feedback signal corresponding to sound stored in the feedback signal storage unit 231. The amplifier 443 changes the input second feedback signal corresponding to sound to a level instructed by the drive control unit 221 and outputs the signal. Mixer 444 combines the first feedback signal output from amplifier 442 and the second feedback signal output from amplifier 443 . The speaker 445 emits a sound corresponding to the feedback signal output from the mixer 444 .

[0065] In accordance with the configuration of the sound output unit 44, the drive control unit 221 controls the amplifier 442 to input the signal source S1-2 with the sound-corresponding first feedback signal based on the sound-corresponding first level included in the feedback control information FB, and controls the amplifier 442 to specify the level of the sound-corresponding first feedback signal. Furthermore, when the feedback control information FB includes a sound-corresponding second level, based on the sound-corresponding second level, control is performed to input the sound-corresponding second feedback signal from the signal source S2-2 to the amplifier 443, and control is performed to instruct the amplifier 443 on the level of the sound-corresponding second feedback signal.

[0066] An example of a processing procedure executed by the information processing device 100 and the pen-type input device 200 in relation to haptic feedback will be described with reference to the flowchart of FIG. First, an example of a processing procedure of the information processing device 100 will be described. When the processing in the figure is being executed, the feedback control unit 122 is configured to capture, for example, at each predetermined capture cycle (capture period), the contact parameters (movement speed parameter Dt1, writing pressure parameter Dt2, movement direction parameter Dt3, and tilt angle parameter Dt4) detected by the contact parameter detection unit 101. At this time, the feedback control unit 122 may capture one sample of contact parameters at each capture cycle (period), or may capture contact parameters corresponding to each of a plurality of sample cycles.

[0067] Step S100: In the information processing device 100, the feedback control unit 122 waits for the pen tip of the pen-type input device 200 to change from a state in which it is in contact with the panel surface and is stopped to a state in which it starts moving (movement start state) based on the contact parameters acquired at the current sample timing. In this case, the feedback control unit 122 may determine that the pen tip has started moving when, for example, the movement speed parameter Dt1 in the contact parameters has changed from zero to a value greater than zero. Alternatively, the feedback control unit 122 may determine that the pen tip has started moving when the movement direction parameter Dt3 has changed from a value that does not indicate a movement direction to a value that indicates a certain movement direction, due to the pen tip not moving (stopping). Furthermore, the feedback control unit 122 may determine whether the pen tip has started moving by using the movement speed parameter Dt1 and the movement direction parameter Dt3 in combination.

[0068] Step S102: When the feedback control unit 122 determines in step S100 that the pen tip has started moving, it sets the levels of the second feedback signals for vibration and sound (second levels: vibration-responsive second level, sound-responsive second level). The feedback control unit 122 may set the second level based on the writing pressure parameter Dt2. In actual writing, the vibrations and sounds generated when the tip of the writing implement starts to move increase as the writing pressure on the writing medium, such as paper, increases when the tip of the writing implement starts to move. Furthermore, when setting the second level, the feedback control unit 122 may also use, for example, acceleration calculated based on a plurality of movement speed parameters Dt1 obtained during the current capture period. Furthermore, when setting the second level, the feedback control unit 122 may use, for example, the type of writing implement or the type of medium to be written on set in a pen operation compatible application.

[0069] Step S104: Next, the feedback control unit 122 sets the levels of the first feedback signals corresponding to vibration and sound (first levels: vibration-related first level and sound-related first level). At this time, the feedback control unit 122 may set the first vibration response level and the first sound response level in accordance with the moving speed parameter Dt1 acquired at the timing of the current sample period. Furthermore, the feedback control unit 122 may use at least some of the writing pressure parameter Dt2, the movement direction parameter Dt3, and the tilt angle parameter Dt4 when setting the first vibration response level and the first sound response level. For example, the magnitude of vibration and sound generated in response to the movement of the pen tip in actual writing may also vary depending on the writing pressure. Furthermore, when setting the first level, the feedback control unit 122 may use, for example, the type of writing implement or the type of medium to be written on set in a pen operation compatible application.

[0070] Step S106: The feedback control unit 122 generates feedback control information FB including the second level set in step S102 and the first level set in step S104.

[0071] Step S108: The feedback control unit 122 transmits the feedback control information generated in step S106 to the pen-type input device 200.

[0072] Step S110: The feedback control unit 122 determines whether the movement of the pen tip that started in response to step S100 has stopped. The feedback control unit 122 may determine that the movement of the pen tip has stopped in response to the movement speed parameter Dt1, which had been greater than 0 until then, becoming zero. If it is determined that the pen tip is in a moving state (moving state), that is, the movement has not stopped, the process returns to step S104. That is, the feedback control unit 122 thereafter executes a loop process of generating feedback control information FB according to the state in which the pen tip is moving and transmitting the information to the pen-type input device 200 until the movement of the pen tip is stopped. The feedback control information FB generated in such a loop process does not include the second level. On the other hand, if it is determined that the movement of the pen tip has stopped, the process returns to step S100. By returning the process to step S100 in this way, the transmission of the feedback control information FB to the pen-type input device 200 is stopped until the movement of the pen tip starts again.

[0073] Next, an example of a processing procedure executed by the pen-type input device 200 will be described. Step S200: In the pen-type input device 200, the drive control unit 221 waits to receive the feedback control information FB transmitted from the information processing device 100 in step S108.

[0074] Step S202: When the feedback control information FB is received, the drive control unit 221 determines whether or not the currently received feedback control information FB includes the second level.

[0075] Step S204: If it is determined in step S202 that the feedback control information FB includes the second level, the drive control section 221 inputs the signal sources S2-1 and S2-2 as second feedback signals to the amplifiers 433 and 443, respectively.

[0076] Step S206: Furthermore, the drive control unit 221 instructs the amplifier 433 to use the second vibration response level, and instructs the amplifier 443 to use the second sound response level.

[0077] By executing the processes of steps S204 and S206 as described above, vibration and writing sound corresponding to the static friction force generated when the pen tip starts to move are reproduced. In addition, at this time, the strength of vibration and the volume of writing sound corresponding to the static friction force are obtained according to the writing pressure when the pen tip starts to move, for example.

[0078] Step S208: After the processing of step S206, or if it is determined in step S202 that the feedback control information FB does not include the second level, the drive control unit 221 inputs the signal sources S1-1 and S1-2 as first feedback signals to the amplifiers 432 and 442, respectively.

[0079] Step S210: Furthermore, the drive control section 221 instructs the amplifier 432 to use the vibration response first level, and instructs the amplifier 442 to use the sound response first level.

[0080] By executing the processes of steps S208 and S210 as described above, the tactile sensation is reproduced by changing the strength of vibration and the volume of the writing sound according to the moving speed of the pen tip during movement.

[0081] A modification of this embodiment will now be described. The haptic feedback may be provided by either vibration or sound. In this case, the information processing device 100 and the pen-type input device 200 may have a configuration corresponding to one of the vibration and sound haptic feedback, but not the other. Alternatively, whether vibration or sound is used for the haptic feedback may be set according to, for example, a selection operation by the user or the type of writing implement or medium set in a pen operation-compatible application.

[0082] The predetermined functional units of the information processing device 100 in the above embodiment may be provided in the pen type input device 200. Alternatively, the predetermined functional units of the pen type input device 200 in the above embodiment may be provided in the information processing device 100. Specifically, for example, the feedback control unit 122 may be provided in the pen-type input device 200. In this case, the information processing device 100 may be configured to transmit the contact parameters detected by the contact parameter detection unit 101 to the pen-type input device 200. Furthermore, for example, the drive control unit 221 may be provided in the information processing device 100. In this case, the drive control unit 221 may be configured to transmit to the pen-type input device 200 a signal instructing input of a feedback signal to the amplifiers (432, 433, 442, 443) and a signal instructing the level to be set in each of the amplifiers (432, 433, 442, 443).

[0083] Note that a program for implementing the functions of the information processing device 100, pen-type input device 200, etc. described above may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be loaded into a computer system and executed to perform processing as the information processing device 100, pen-type input device 200, etc. Here, "loading a program recorded on a recording medium into a computer system and executing it" includes installing the program on a computer system. The term "computer system" here includes hardware such as an OS and peripheral devices. The term "computer system" may also include multiple computer devices connected via a network, including communication lines such as the Internet, WAN, LAN, and dedicated lines. The term "computer-readable recording medium" refers to portable media such as floppy disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Thus, a recording medium storing a program may be a non-transitory recording medium such as a CD-ROM. The recording medium may also include internal or external recording media accessible from a distribution server for distributing the program. The program code stored on the distribution server's recording medium may be different from the program code executable on a terminal device. In other words, the format in which the program is stored on the distribution server does not matter as long as it can be downloaded from the distribution server and installed in a form that is executable on the terminal device. The program may be divided into multiple parts, each of which may be downloaded at different times and then combined on the terminal device, or each of the divided programs may be distributed by a different distribution server. Furthermore, the term "computer-readable recording medium" also includes a storage medium that stores a program for a certain period of time, such as volatile memory (RAM) within a computer system that serves as a server or client when a program is transmitted over a network. The program may also be one that realizes part of the functions described above.Furthermore, the above-mentioned functions may be realized in combination with a program already recorded in the computer system, that is, a so-called differential file (differential program). [Explanation of symbols]

[0084] 30 touch panel display unit, 31 display unit, 32 panel sensor, 43 vibration unit, 44 sound output unit, 100 information processing device, 101 contact parameter detection unit, 102 control unit, 103 memory unit, 111 movement speed detection unit, 112 pen pressure detection unit, 113 movement direction detection unit, 114 angle detection unit, 121 application corresponding processing unit, 122 feedback control unit, 200 pen type input device, 201 control unit, 203 memory unit, 221 drive control unit, 231 feedback signal memory unit

Claims

1. a contact parameter detection unit that detects one or more contact parameters corresponding to a contact state of the pen-type input device with respect to the operation surface; a feedback control unit that determines a contact state of the pen tip of the pen type input device with respect to the operation target surface based on the contact parameters, and that, when it is determined that the contact state is a moving state in which the pen tip of the pen type input device is moving while in contact with the operation target surface, executes a first feedback control to drive a tactile reproduction unit in the pen type input device so as to reproduce a tactile sensation corresponding to the moving state, and a second feedback control to drive the tactile reproduction unit at a timing when it is determined that the contact state is a movement start state in which the pen tip of the pen type input device has started moving from a stationary state while in contact with the operation target surface, so as to reproduce a tactile sensation corresponding to the movement start state; An information processing system comprising:

2. The haptic reproduction unit includes a vibration unit that generates vibrations in response to an input of a feedback signal. The information processing system according to claim 1 .

3. The haptic reproduction unit includes a sound output unit that outputs a sound in response to an input of a feedback signal. The information processing system according to claim 1 .

4. The second feedback control includes control for inputting a feedback signal corresponding to the second feedback control as an impulse signal to the haptic reproduction unit. The information processing system according to claim 1 .

5. The second feedback control includes control for inputting a feedback signal corresponding to the second feedback control, which has a predetermined peak level at a predetermined time according to the timing when the pen tip of the pen-type input device starts to move, to the tactile reproduction unit. The information processing system according to claim 1 .

6. The contact parameters are all or some of the following: a speed at which the pen tip moves on the operation surface; a pressure of the pen tip on the operation surface; a direction in which the pen tip moves on the operation surface; and a tilt angle of the pen-type input device with respect to the operation surface. The information processing system according to claim 1 .

7. The feedback control unit sets a level of the feedback signal corresponding to the second feedback control based on at least the pressure of the contact parameters. The information processing system according to claim 6.

8. The feedback control unit changes a level of the feedback signal corresponding to the first feedback control based on at least the speed of the contact parameters. The information processing system according to claim 6.

9. An information processing method in an information processing system, comprising: a contact parameter detection step in which a contact parameter detection unit detects one or more contact parameters corresponding to a contact state of the pen-type input device with respect to an operation target surface; a feedback control step in which a feedback control unit determines a contact state of the pen tip of the pen type input device with respect to the operation target surface based on the contact parameters, and when it is determined that the contact state is a moving state in which the pen tip of the pen type input device is moving while in contact with the operation target surface, executes a first feedback control to drive a tactile reproduction unit in the pen type input device so as to reproduce a tactile sensation corresponding to the moving state, and a second feedback control to drive the tactile reproduction unit at a timing when it is determined that the contact state is a movement start state in which the pen tip of the pen type input device has started moving from a stationary state while in contact with the operation target surface, so as to reproduce a tactile sensation corresponding to the movement start state; An information processing method including:

10. The computer in the information processing system a contact parameter detection unit that detects one or more contact parameters corresponding to a contact state of the pen-type input device with respect to the operation surface; a feedback control unit that determines a contact state of the pen tip of the pen type input device with respect to the operation target surface based on the contact parameters, and that, when it is determined that the contact state is a moving state in which the pen tip of the pen type input device is moving while in contact with the operation target surface, executes a first feedback control to drive a tactile reproduction unit in the pen type input device so as to reproduce a tactile sensation corresponding to the moving state, and a second feedback control to drive the tactile reproduction unit at a timing when it is determined that the contact state is a movement start state in which the pen tip of the pen type input device has started moving from a stationary state while in contact with the operation target surface, so as to reproduce a tactile sensation corresponding to the movement start state. A program to function as a

Images