Information processing system, information processing method, and program

Abstract

To provide a feeling similar to that of actually writing on a writing medium. [Solution] An information processing system is configured to detect one or more contact parameters corresponding to the contact state of a pen-type input device with respect to an operation target surface, and determine the contact state of the tip of the pen-type input device with respect to the operation target surface based on the contact parameters.When it is determined that the contact state is a moving state of the tip of the pen-type input device, a tactile reproduction unit in the pen-type input device is driven to reproduce a tactile sensation corresponding to the moving state, and a feedback control unit is provided to drive the tactile reproduction unit to reproduce a tactile sensation corresponding to the moving start state when it is determined that the contact state is a movement start state in which the tip of the pen-type input device has started moving from a stationary state while in contact with the operation target surface.

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] It is known that, when inputting data to an interactive display system, a pen-shaped stylus (pen-type input device) that enables tactile feedback by generating vibrations that drive a tactile actuator is used as an input device (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-537395 Summary of the Invention [Problem to be solved by the invention]

[0004] By emitting vibrations from the pen-type input device, the tactile sensation of sliding the tip of a pen across a writing medium such as paper can be simulated, allowing a user inputting data using the pen-type input device to get a sensation similar to that of writing on a writing medium.

[0005] When inputting data using a pen-type input device, it is desirable to have a sensation as close as possible to that of writing on an actual writing medium. In consideration of the above-mentioned problems, the present invention has an object to provide an input using a pen-type input device that provides a sensation similar to that felt when actually writing on a writing medium. [Means for solving the problem]

[0006] One aspect of the present invention that solves the above-mentioned problems is an information processing system that includes a contact parameter detection unit that detects one or more contact parameters corresponding to a contact state of a pen-type input device with respect to an operation target surface, and a feedback control unit that determines a contact state of a pen tip of the pen-type input device with respect to the operation target surface based on the contact parameters, and when it is determined that the contact state is a moving state in which the pen tip of the pen-type input device is moving while in contact with the operation target surface, executes a first feedback control that drives a tactile reproduction unit in the pen-type input device to reproduce a tactile sensation corresponding to the moving state, and a second feedback control that drives the tactile reproduction unit to reproduce a tactile sensation corresponding to the moving start state at the timing when it is determined that the contact state is a moving start state in which the pen tip of the pen-type input device has started moving from a stationary state while in contact with the operation target surface.

[0007] One aspect of the present invention is an information processing method in an information processing system, the information processing method including: a contact parameter detection step in which a contact parameter detection unit detects one or more contact parameters corresponding to a contact state of a pen-type input device with respect to an operation target surface; and a feedback control step in which a feedback control unit determines a contact state of a pen tip of the pen-type input device with respect to the operation target surface based on the contact parameters, and when it is determined that the contact state is a moving state in which the pen tip of the pen-type input device is moving while in contact with the operation target surface, a first feedback control is executed to drive a tactile reproduction unit in the pen-type input device to reproduce a tactile sensation corresponding to the moving state; and a second feedback control is executed to drive the tactile reproduction unit to reproduce a tactile sensation corresponding to the moving start state at the timing when it is determined that the contact state is a moving start state in which the pen tip of the pen-type input device has started moving from a stationary state while in contact with the operation target surface.

[0008] One aspect of the present invention is a program for causing a computer in an information processing system to function as a feedback control unit that executes: a contact parameter detection unit that detects one or more contact parameters corresponding to a contact state of a pen-type input device with respect to an operation target surface; a contact state of the tip of the pen-type input device with respect to the operation target surface based on the contact parameters; a first feedback control that drives a tactile reproduction unit in the pen-type input device to reproduce a tactile sensation corresponding to the movement state when it is determined that the contact state is a movement state in which the tip of the pen-type input device is moving while in contact with the operation target surface; and a second feedback control that drives the tactile reproduction unit to reproduce a tactile sensation corresponding to the movement start state when it is determined that the contact state is a movement start state in which the tip of the pen-type input device has started moving from a stationary state while in contact with the operation target surface. Effect of the Invention

[0009] According to the present invention, when performing input using a pen-type input device, it is possible to obtain an effect that the feeling of actually writing on a writing medium can be approximated. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing an example of the external configuration of an information processing system according to an embodiment of the present invention; [Diagram 2] 11A to 11C are diagrams showing changes in frictional force of the pen tip over time when starting a writing operation in this embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an overview of feedback control in the present embodiment. [Figure 4] 5A and 5B are diagrams showing an example of a movement trajectory of a pen tip of a pen-type input device on a panel surface in the present embodiment. [Diagram 5] FIG. 2 is a diagram illustrating an example of the hardware configuration of an information processing device and a pen-type input device according to the present embodiment. [Figure 6]1 is a diagram illustrating an example of a functional configuration of an information processing device and a pen-type input device according to an embodiment of the present invention. [Figure 7] 4 is a diagram illustrating an example of a functional configuration corresponding to driving of a vibration unit and a sound output unit by a feedback signal in the present embodiment. FIG. [Figure 8] 10 is a flowchart illustrating an example of a processing procedure executed by the information processing device and the pen-type input device in the present embodiment in relation to haptic feedback. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] 1 shows an example of the external configuration of an information processing system according to the present embodiment. As shown in the figure, the information processing system according to the present embodiment includes an information processing device 100 and a pen-type input device 200.

[0012] The information processing device 100 is capable of executing information processing in response to an input operation by a pen-type input device 200. The information processing device 100 in the figure is an example of a tablet terminal or a notebook personal computer.

[0013] The information processing device 100 includes a touch panel display unit 30. The touch panel display unit 30 is a portion in which a touch panel and a display unit are combined. The touch panel display unit 30 displays an image on a panel surface (a display surface: an example of a surface to be operated), and enables operation by contacting the panel surface with an operating object such as a pen-type input device or a finger.

[0014] The pen-type input device 200 is a pen-type input device that is used by a user to perform operations on the touch panel of the touch panel display unit 30 of the information processing device 100. A user holds the pen-type input device 200 and moves the pen tip so as to contact the panel surface of the touch panel display unit 30, thereby performing a handwritten input operation of characters, pictures, figures, and the like.

[0015] Note that as an operation using the pen-type input device 200, a pointing operation on a user interface image displayed on the touch panel display unit 30 may also be possible.

[0016] The detection method of the pen-type input device 200 by the touch panel in the touch panel display unit 30 of this embodiment is not particularly limited, but may be, for example, a capacitance method or an electromagnetic induction method. In the following description, a case where the capacitance method is used will be taken as an example.

[0017] An application compatible with input operations using the pen-type input device 200 (pen operation compatible application) is installed in the information processing device 100. The pen operation compatible application may be capable of, for example, inputting characters by touching the tip of the pen of the pen-type input device 200 to the touch panel display unit 30, displaying characters or pictures drawn in response to handwriting input operations such as drawing, and executing processes for converting characters or pictures drawn by the operation into data.

[0018] In the information processing system of this embodiment, the pen-type input device 200 generates vibrations and sounds (writing sounds) in response to an operation (writing operation) corresponding to writing. The vibrations and writing sounds generated by the pen-type input device 200 allow the user to feel a tactile sensation similar to that felt when writing on a writing medium such as paper, and the user can obtain a sensation similar to that felt when writing on a writing medium with an actual writing implement.

[0019] In the following description, the generation of vibrations and sounds when a writing operation is being performed with the pen-type input device 200 is also referred to as "tactile feedback."

[0020] Next, an overview of the haptic feedback in this embodiment will be described. Figure 2 shows the change in frictional force of the tip of a writing implement over time when a user actually starts a writing operation. At time t0, the user has the tip of a writing implement (e.g., a pencil) in contact with a writing medium such as paper, but the tip of the writing implement is still stationary and not moving. Then, at time t1, a certain amount of time after time t0, the user starts moving the tip of the writing implement over the writing medium in response to the start of writing. When the user starts moving the tip of the writing implement, the tip of the writing implement is in contact with the surface of the writing medium with a certain amount of force, generating a static friction force. Therefore, during the period from time t1 to time t2, even if the user applies force to move the tip of the writing implement, the tip of the writing implement remains stationary and does not move on the writing medium, and the static friction force increases over time. Then, when the tip of the writing implement starts to move at time t2, the static friction force that had been increasing up until that point suddenly decreases, and a certain kinetic friction force occurs in response to the movement of the tip of the writing implement. This change in friction force is observed as strong vibrations and sounds (writing sounds) in real writing.

[0021] When the pen tip of the pen-type input device 200 moves, haptic feedback may be given at a constant level by vibration or sound, but the level of the haptic feedback may be changeable depending on the movement state of the pen tip. For example, the level of the haptic feedback may be changed so as to increase as the movement speed of the pen tip increases. However, when changing the level of haptic feedback according to the pen tip movement speed, the movement speed at the timing when the pen tip starts to move is low, so the haptic feedback at the timing when the pen tip starts to move is also small. In other words, changing the level of haptic feedback according to the pen tip movement speed cannot reproduce as haptic feedback the change in kinetic friction force according to the movement speed from the large static friction force that occurs instantaneously when the tip of the writing implement starts to move, as shown in Figure 2.

[0022] Therefore, in this embodiment, tactile feedback is controlled in response to the movement of the pen tip of the pen type input device 200 as shown in FIG. 3, lines L1-1 and L1-2 show the change over time in the levels of two feedback signals output in response to the haptic feedback of this embodiment, and line L2 shows the change over time in the movement speed detected for the pen tip. Time t10 is the timing when the pen tip of the pen type input device 200, which had been stationary up until that point, starts to move in response to a writing operation by the user. In response to the start of the movement of the pen tip at time t10, output of a first feedback signal starts, the level of which changes in response to the moving speed of the pen tip as shown by line L1-1. At time t11 when the movement of the pen tip stops, the first feedback signal becomes zero level and output is stopped. In response to the start of the movement of the pen tip at time t10, a second feedback signal having an instantaneous peak as shown by line L1-2 is also output.

[0023] In this manner, in this embodiment, composite tactile feedback is provided by two feedback signals, the first feedback signal and the second feedback signal, from when the pen tip starts to move until it stops. This tactile feedback not only reproduces vibrations and writing sounds corresponding to the moving speed of the pen tip, but also reproduces vibrations and writing sounds corresponding to the static friction force that momentarily increases at the timing when the pen tip of the pen type input device 200 starts to move. As a result, when inputting using the pen type input device 200, it becomes possible to get closer to the sensation of actually writing on a writing medium.

[0024] 4 shows an example of a movement trajectory MT of the pen tip of the pen type input device 200 on the panel surface PN by a user's writing operation. The movement trajectory MT in the figure starts from a starting point p1, changes its movement direction at direction change points p2, p3, p4, and p5 in this order, and ends at an end point p6. Under such a movement trajectory MT, the movement of the pen tip is stopped once at each of the direction change points p2, p3, p4, and p5, and then the movement is restarted. In this case, haptic feedback based on the first feedback signal at a level according to the movement speed from the start of the pen tip movement to the stop is provided for each of the sections p1-p2, p2-p3, p3-p4, and p5-p6. In addition, haptic feedback based on the static friction force based on the second feedback signal is provided for each of the start point p1 and the direction change points p2, p3, p4, and p5.

[0025] In this embodiment, the peak level of the second feedback signal may be a predetermined fixed value, or may be set based on a predetermined contact parameter among detection values ​​(contact parameters) obtained by detecting a predetermined contact state of the pen tip with respect to the panel surface. Specifically, the information processing device 100 of this embodiment can include, as contact parameters, the speed (movement speed) at which the pen tip moves across the panel surface, the pressure (writing pressure) when the pen tip is in contact with the panel surface, the movement direction of the pen tip on the panel surface, and the inclination angle of the pen-type input device 200 with respect to the panel surface when the pen tip is in contact with the panel surface. The static friction force when the pen tip starts to move by a writing operation increases according to the writing pressure when the movement is about to start. Therefore, the information processing device 100 may set the peak level of the second feedback signal according to the contact parameter as the writing pressure. Furthermore, the static frictional force when the pen tip starts to move by a writing operation varies depending on, for example, the inclination angle of the pen type input device 200 with respect to the panel surface. Therefore, the information processing device 100 may set the peak level of the second feedback signal depending on the contact parameter as the inclination angle. In addition, the static friction force when the pen tip starts to move due to a writing operation may also change depending on, for example, the moving direction of the pen tip on the panel surface in response to the writing operation. Therefore, the information processing device 100 may set the peak level of the second feedback signal depending on the contact parameter as the moving direction. Furthermore, the information processing device 100 may set the peak level of the second feedback signal by using two or more contact parameters of the contact parameters of the pen pressure, the tilt angle, and the movement direction in a composite manner.

[0026] An example of the hardware configuration of the information processing device 100 and the pen-type input device 200 will be described with reference to FIG. First, a description will be given of an example of the hardware configuration of an information processing device 100. The information processing device 100 in the figure includes a processor 11, a main memory 12, a flash memory 13, a peripheral device 14, a short-range communication unit 15, a baseband chip 21, a communication unit 22, an audio system 23, a microphone 24, a speaker 25, and a touch panel display unit 30.

[0027] The processor 11 is, for example, an application processor including a CPU (Central Processing Unit). The processor 11 controls the entire information processing device 100.

[0028] The main memory 12 is a writable memory used as a read area for the execution program of the processor 11 or as a work area for writing processing data for the execution program. The main memory 12 is composed of, for example, a plurality of DRAM (Dynamic Random Access Memory) chips. The execution program includes an OS (Operating System), various device drivers for operating the hardware of peripheral devices, various services / utilities, application programs (application software), and the like.

[0029] The flash memory 13 is, for example, a flash EEPROM (Electrically Erasable Programmable Read Only Memory), and stores the OS, various drivers, various services / utilities, application programs (hereinafter sometimes referred to as applications), and various data.

[0030] The touch panel display unit 30 is a part that displays an image and allows an operation to be performed on the panel surface on which the image is displayed using the pen-type input device 200. The touch panel display unit 30 may also be operable using an operating object other than the pen-type input device 200, such as a finger, but here, an example will be described in which the operating object is the pen-type input device 200. The touch panel display unit 30 includes a display unit 31 and a panel sensor 32 . The display unit 31 is, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display, and displays an image based on drawing data (display data) output from the processor 11. The panel sensor 32 detects the contact state of the pen tip of the pen type input device 200 with the panel surface of the display unit 31. Specifically, the panel sensor 32 may be capable of detecting the position on the panel surface where the pen tip of the pen type input device 200 is in contact, the pressure of the pen tip in contact with the panel surface, the distance of the pen type input device 200 from the panel surface, etc. The detection method of such a panel sensor 32 is not particularly limited, but examples thereof include a capacitance type and an electromagnetic induction type. Furthermore, the panel sensor 32 may be configured to use a combination of such multiple detection methods.

[0031] The peripheral device 14 is, for example, a Wireless Local Area Network (WLAN) module, a Global Positioning System (GPS) module, and sensors such as an acceleration sensor.

[0032] The audio system 23 is, for example, an audio IC (Integrated Circuit), and inputs, records, plays back, and outputs sound data. For example, a microphone 24 and a speaker 25 are connected to the audio system 23. For example, the audio system 23 outputs sound data picked up by the microphone 24 to the processor 11 or the baseband chip 21. In addition, the audio system 23 converts audio data acquired from the processor 11 or the baseband chip 21 into an audio signal, and outputs it to the speaker 25, for example.

[0033] The microphone 24 picks up sounds around the information processing device 100. For example, the microphone 24 picks up sounds such as the user's voice when audio integration with another terminal is performed. The speaker 25 outputs various sounds to the outside of the information processing device 100. For example, the speaker 25 outputs (emits) sounds received from other terminals when performing audio integration with other terminals.

[0034] The baseband chip 21 is a dedicated IC that controls wireless communications such as 4G (fourth generation mobile communication system) and 5G (fourth generation mobile communication system). The baseband chip 21, for example, causes the audio data received by the communication unit 22 to be output as sound from the speaker 25 by the audio system 23. Furthermore, the baseband chip 21 acquires, for example, sound data collected by a microphone 24 via an audio system 23, and outputs the sound data via a mobile communication system by a communication unit 22. Furthermore, the baseband chip 21 may be configured to transmit and receive input and output data for data communication by the mobile communication system to and from the processor 11.

[0035] The communication unit 22 is a wireless communication device including an antenna for performing wireless communication via a mobile communication system.

[0036] The short-distance communication unit 15 is, for example, a Bluetooth (registered trademark) module, and performs short-distance wireless communication with the pen-type input device 200.

[0037] Next, with reference to the same FIG. 5, an example of the hardware configuration of the pen type input device 200 will be described. The pen type input device 200 includes an MCU 41, a near field communication unit 42, a vibration unit 43 (an example of a tactile reproduction unit), a sound output unit 44 (an example of a tactile reproduction unit), and a flash memory 45.

[0038] The short-range communication unit 42 is, for example, a Bluetooth (registered trademark) module, and performs short-range wireless communication with the information processing device 100.

[0039] The MCU (Micro Controller Unit) 41 includes a CPU, memories such as ROM and RAM, and I / O related components, and executes control of the pen type input device 200. The MCU 41 transmits and receives information via a short-range communication unit 42. The MCU 41 also controls the generation of vibrations in a vibration unit 43 and the output of sounds by a sound output unit 44 so that haptic feedback is performed in response to feedback signals (including a first feedback signal and a second feedback signal) transmitted from the information processing device 100.

[0040] The vibration unit 43 is a part that includes, for example, an actuator, and vibrates in response to an input of a feedback signal from the MCU 41. The sound output unit 44 includes, for example, a speaker, and outputs sound in response to an input of a feedback signal from the MCU 41.

[0041] The flash memory 45 stores various data corresponding to the pen type input device 200, such as programs and feedback signal data.

[0042] An example of the functional configuration of the information processing device 100 and the pen-type input device 200 will be described with reference to Fig. 6. In the figure, functional units that can be considered equivalent to the parts in the hardware configuration in Fig. 5 are given the same reference numerals as in Fig. 5, and the description thereof will be omitted as appropriate. The functions of the information processing device 100 shown in the figure may be realized by the processor 11 (FIG. 5) executing a program.

[0043] First, a description will be given of an example of the functional configuration of the information processing device 100. The information processing device 100 includes a communication unit 22, a touch panel display unit 30, a touch parameter detection unit 101, a control unit 102, and a storage unit 103 as functional units.

[0044] The contact parameter detection unit 101 detects the contact state of the pen tip of the pen type input device 200 with the panel surface of the touch panel display unit 30 . The contact parameter detection unit 101 outputs parameters (contact parameters) detected in accordance with the contact state based on the detection signal output by the panel sensor 32. In the following description, the contact parameters will be, for example, the above-mentioned movement speed, writing pressure, movement direction, and tilt angle.

[0045] The control unit 102 executes various controls in the information processing device 100. The control unit 102 includes an application support processing unit 121 and a feedback control unit 122.

[0046] The application processing unit 121 executes processing corresponding to a pen operation compatible application. When a writing operation is being performed with the pen type input device 200 as an operation for a pen operation compatible application, pen operation information is input from the panel sensor 32 to the application compatible processing unit 121. In response to the input of the pen operation information, the application compatible processing unit 121 performs processing such as drawing.

[0047] The feedback control unit 122 executes feedback control. The feedback control in this embodiment is control for causing the pen type input device 200 to generate vibration as haptic feedback. The feedback control unit 122 of this embodiment executes feedback control (second feedback control) for reproducing vibrations and sounds generated in response to changes in frictional force when the pen tip of the pen type input device 200, which has been stationary in contact with the panel surface, starts to move, and feedback control (first feedback control) for reproducing vibrations and sounds generated when the pen tip of the pen type input device 200 is moving. The feedback control unit 122 generates feedback control information based on the contact parameters output from the contact parameter detection unit 101. The feedback control information is information for controlling the drive control unit 221 of the pen type input device 200 so as to generate tactile feedback by vibration and sound. The feedback control unit 122 outputs (transmits) the generated feedback control information to the pen type input device 200 via the communication unit 22.

[0048] The storage unit 103 stores various types of information corresponding to the information processing device 100 .

[0049] Next, a description will be given of an example of the functional configuration of the pen type input device 200. The functions of the pen type input device 200 shown in the figure may be realized by the MCU 41 (FIG. 5) executing a program. The pen type input device 200 includes a short-distance communication unit 42, a vibration unit 43, a sound output unit 44, a control unit 201, and a storage unit 203 as functional units.

[0050] The control unit 201 executes various controls in the pen type input device 200. The control unit 201 includes a drive control unit 221. The drive control unit 221 controls the vibration unit 43 to generate vibration as haptic feedback, and also controls the sound output unit 44 to generate sound as haptic feedback. When controlling the vibration unit 43 and the sound output unit 44, the drive control unit 221 inputs the feedback signals (including the first feedback signal and the second feedback signal) stored in the feedback signal storage unit 231 to the vibration unit 43 and the sound output unit 44.

[0051] The storage unit 203 stores various data corresponding to the pen type input device 200. The storage unit 203 stores a feedback signal storage unit 231. The feedback signal storage unit 231 stores a waveform that becomes a feedback signal. The feedback signal storage unit 231 stores a waveform of a first feedback signal and a waveform of a second feedback signal. The first feedback signal and the second feedback signal stored in the feedback signal storage unit 231 may each include a waveform of a feedback signal corresponding to a vibration used to vibrate the vibration unit 43 and a waveform of a feedback signal corresponding to a sound to be output as a sound from the sound output unit 44.

[0052] The waveform of the second feedback signal may be, for example, an impulse waveform. That is, the second feedback signal may be an impulse signal. Alternatively, the waveform of the second feedback signal may be a waveform that has a steep peak at the time when vibration or sound is generated due to static friction force when the pen tip starts to move.

[0053] An example of a functional configuration corresponding to driving of vibration section 43 and sound output section 44 by a feedback signal will be described with reference to FIG. In the information processing device 100 , the contact parameter detection unit 101 includes a movement speed detection unit 111 , a writing pressure detection unit 112 , a movement direction detection unit 113 , and a tilt angle detection unit 114 .

[0054] The movement speed detection unit 111 detects the movement speed of the pen tip that moves while touching the panel surface. The movement speed detection unit 111 may detect the movement speed based on the amount of movement per unit time of the contact position of the pen tip detected by the panel sensor 32. The movement speed detection unit 111 outputs the detected movement speed parameter Dt1 to the feedback control unit 122.

[0055] The writing pressure detection unit 112 detects the pressure (writing pressure) of the pen tip in contact with the panel surface. The writing pressure detection unit 112 may detect the writing pressure based on the value of the capacitance at the contact position of the pen tip detected by the panel sensor 32. The writing pressure detection unit 112 outputs a writing pressure parameter Dt2 indicating the detected writing pressure to the feedback control unit 122.

[0056] The movement direction detection unit 113 detects the movement direction of the pen tip that moves while in contact with the panel surface. The movement speed detection unit 111 may detect the movement direction based on the movement direction of the contact position of the pen tip detected by the panel sensor 32. The movement direction detection unit 113 outputs a movement direction parameter Dt3 indicating the detected movement direction to the feedback control unit 122.

[0057] The tilt angle detection unit 114 detects the tilt angle of the pen type input device 200 body relative to the panel surface when the pen tip is in contact with the panel surface. The tilt angle detection unit 114 may detect the tilt angle based on the distribution of capacitance corresponding to the pen type input device 200 body detected around the contact position of the pen tip by the panel sensor 32. The tilt angle detection unit 114 outputs the detected tilt angle parameter Dt4 to the feedback control unit 122.

[0058] In the information processing device 100, the feedback control unit 122 generates feedback control information FB based on the input contact parameters (movement speed parameter Dt1, writing pressure parameter Dt2, movement direction parameter Dt3, and tilt angle parameter Dt4). In this embodiment, the feedback control unit 122 generates feedback control information FB corresponding to the vibration unit 43 and feedback control information FB corresponding to the sound output unit 44 separately. The feedback control section 122 may control feedback control information that is common to the vibration section 43 and the sound output section 44 .

[0059] The feedback control information FB includes a level of a first feedback signal corresponding to vibration (a vibration-responsive first level) and a level of a first feedback signal corresponding to sound (a sound-responsive first level). The feedback control unit 122 may set the vibration-responsive first level and the sound-responsive first level based on the moving speed parameter Dt1. In addition, the feedback control information FB, which is output in accordance with the timing at which the pen tip starts to move on the panel surface, includes the level of a second feedback signal corresponding to vibration (vibration-responsive second level) and the level of a second feedback signal corresponding to sound (sound-responsive second level). In the following description, when there is no particular distinction between the vibration response level 1 and the sound response level 1, they will be referred to as the first level. Also, when there is no particular distinction between the vibration response level 2 and the sound response level 2, they will be referred to as the second level.

[0060] The feedback control unit 122 outputs (transmits) the generated feedback control information FB to the pen type input device 200.

[0061] The drive control unit 221 drives the vibration unit 43 and the sound output unit 44 based on the input feedback control information FB.

[0062] In the figure, an example of the configuration of a vibration section 43 and a sound output section 44 is shown. The vibration section 43 includes, for example, an amplifier 432 , an amplifier 433 , a mixer 434 , and an actuator 435 . The amplifier 432 receives a first feedback signal corresponding to vibration as a signal source S1-1. The signal source S1-1 is generated based on the waveform of the first feedback signal corresponding to vibration stored in the feedback signal storage unit 231. The amplifier 432 changes the input first feedback signal corresponding to vibration to a level instructed by the drive control unit 221 and outputs the changed signal. The amplifier 433 receives a second feedback signal corresponding to vibration as a signal source S2-1. The signal source S2-1 is generated based on the waveform of the second feedback signal corresponding to vibration stored in the feedback signal storage unit 231. The amplifier 433 changes the input second feedback signal corresponding to vibration to a level instructed by the drive control unit 221 and outputs the changed signal. Mixer 434 mixes the first feedback signal output from amplifier 432 and the second feedback signal output from amplifier 433 . The actuator 435 vibrates in response to the feedback signal output from the mixer 434 .

[0063] In accordance with this configuration of the vibration unit 43, the drive control unit 221 performs control to input the signal source S1-1 based on the vibration-responsive first feedback signal to the amplifier 432 based on the vibration-responsive first level included in the feedback control information FB, and controls to instruct the amplifier 432 on the level of the vibration-responsive first feedback signal. In addition, when the feedback control information FB includes a vibration-responsive second level, based on the vibration-responsive second level, control is performed to input a vibration-responsive second feedback signal from signal source S2-1 to amplifier 433, and control is performed to instruct amplifier 433 on the level of the vibration-responsive second feedback signal.

[0064] The sound output unit 44 includes, for example, an amplifier 442, an amplifier 443, a mixer 444, and a speaker 445. The amplifier 442 receives a first feedback signal corresponding to sound as a signal source S1-2. The signal source S1-2 is generated based on the waveform of the first feedback signal corresponding to sound stored in the feedback signal storage unit 231. The amplifier 442 changes the input first feedback signal corresponding to sound to a level instructed by the drive control unit 221 and outputs the changed signal. The amplifier 443 receives a second feedback signal corresponding to sound as a signal source S2-2. The signal source S2-1 is generated based on the waveform of the second feedback signal corresponding to sound stored in the feedback signal storage unit 231. The amplifier 443 changes the input second feedback signal corresponding to sound to a level instructed by the drive control unit 221 and outputs the signal. Mixer 444 mixes the first feedback signal output from amplifier 442 and the second feedback signal output from amplifier 443 . The speaker 445 emits a sound according to the feedback signal output from the mixer 444 .

[0065] In accordance with such a configuration of the sound output unit 44, the drive control unit 221 performs control to input the signal source S1-2 based on the sound-related first level contained in the feedback control information FB to the amplifier 442 using the sound-related first feedback signal, and controls to instruct the amplifier 442 on the level of the sound-related first feedback signal. In addition, when the feedback control information FB includes a sound-responsive second level, based on the sound-responsive second level, control is performed to input a sound-responsive second feedback signal from signal source S2-2 to amplifier 443, and control is performed to instruct amplifier 443 on the level of the sound-responsive second feedback signal.

[0066] An example of a process performed by the information processing device 100 and the pen type input device 200 in relation to haptic feedback will be described with reference to the flowchart of FIG. First, an example of a processing procedure of the information processing device 100 will be described. When the processing in the figure is being executed, the feedback control unit 122 is configured to capture the contact parameters (movement speed parameter Dt1, writing pressure parameter Dt2, movement direction parameter Dt3, and tilt angle parameter Dt4) detected by the contact parameter detection unit 101, for example, for each predetermined capture cycle (capture period). At this time, the feedback control unit 122 may capture one sample of contact parameters for each capture cycle (period), or may capture contact parameters corresponding to each of a plurality of sample cycles.

[0067] Step S100: In the information processing device 100, the feedback control unit 122 waits for the pen tip of the pen type input device 200 to move from a stopped state in contact with the panel surface to a state where it starts moving (movement start state) based on the contact parameters acquired at the current sample timing. In this case, the feedback control unit 122 may determine that the pen tip has started moving when, for example, the movement speed parameter Dt1 in the contact parameters has changed from zero to a value greater than zero. Alternatively, the feedback control unit 122 may determine that the pen tip has started moving when the movement direction parameter Dt3 has changed from a value not indicating a movement direction to a value indicating a certain movement direction due to the pen tip not moving (stopping). Furthermore, the feedback control unit 122 may determine whether the pen tip has started moving by using the above-mentioned movement speed parameter Dt1 and movement direction parameter Dt3 in combination.

[0068] Step S102: When the feedback control unit 122 determines in step S100 that the pen tip has started moving, it sets the levels of the second feedback signals for vibration and sound (second level: vibration-responsive second level, sound-responsive second level). The feedback control unit 122 may set the second level based on the writing pressure parameter Dt2. In real writing, the vibration and sound generated when the tip of the writing implement starts to move increases as the writing pressure on the writing target medium, such as paper, increases when the tip of the writing implement starts to move. Furthermore, when setting the second level, the feedback control unit 122 may also use, for example, an acceleration calculated based on a plurality of movement speed parameters Dt1 obtained during the current capture period. Furthermore, when setting the second level, the feedback control unit 122 may use, for example, the type of writing implement or the type of medium to be written on, which is set in a pen operation compatible application.

[0069] Step S104: Next, the feedback control unit 122 sets the levels of the first feedback signals corresponding to vibration and sound (first levels: vibration-related first level and sound-related first level). At this time, the feedback control unit 122 may set each of the first vibration response level and the first sound response level in accordance with the moving speed parameter Dt1 acquired at the timing of the current sampling period. Furthermore, the feedback control unit 122 may use at least a part of the writing pressure parameter Dt2, the movement direction parameter Dt3, and the tilt angle parameter Dt4 when setting the first vibration response level and the first sound response level. For example, the magnitude of vibration and sound generated in response to the movement of the pen tip in real writing may change depending on the writing pressure. Furthermore, when setting the first level, the feedback control unit 122 may use, for example, the type of writing implement or the type of medium to be written on, which is set in a pen operation compatible application.

[0070] Step S106: The feedback control unit 122 generates feedback control information FB including the second level set in step S102 and the first level set in step S104.

[0071] Step S108: The feedback control unit 122 transmits the feedback control information generated in step S106 to the pen type input device 200.

[0072] Step S110: The feedback control unit 122 determines whether the movement of the pen tip started in response to step S100 has stopped. The feedback control unit 122 may determine that the movement of the pen tip has stopped in response to the movement speed parameter Dt1, which has been greater than 0 until now, becoming zero. When it is determined that the pen tip is in a moving state (moving state), that is, the movement has not stopped, the process returns to step S104. That is, the feedback control unit 122 thereafter executes a loop process of generating feedback control information FB according to the state in which the pen tip is moving and transmitting it to the pen type input device 200 until the movement of the pen tip is stopped. The feedback control information FB generated in such a loop process does not include the second level. On the other hand, if it is determined that the movement of the pen tip has stopped, the process returns to step S100. By returning the process to step S100 in this manner, the transmission of the feedback control information FB to the pen type input device 200 is stopped until the movement of the pen tip starts again.

[0073] Next, an example of a process procedure executed by the pen type input device 200 will be described. Step S200: In the pen type input device 200, the drive control unit 221 waits to receive the feedback control information FB transmitted from the information processing device 100 in step S108.

[0074] Step S202: When the feedback control information FB is received, the drive control unit 221 determines whether or not the currently received feedback control information FB includes the second level.

[0075] Step S204: If it is determined in step S202 that the feedback control information FB includes the second level, the drive control section 221 inputs the signal sources S2-1 and S2-2 as second feedback signals to the amplifiers 433 and 443, respectively.

[0076] Step S206: Furthermore, the drive control unit 221 instructs the amplifier 433 to use the second vibration-responsive level, and instructs the amplifier 443 to use the second sound-responsive level.

[0077] As described above, the processes of steps S204 and S206 are executed to reproduce the vibration and writing sound corresponding to the static friction force generated when the pen tip starts to move. In addition, at this time, the strength of the vibration and the volume of the writing sound corresponding to the static friction force are obtained according to the writing pressure when the pen tip starts to move, for example.

[0078] Step S208: After the process of step S206, or if it is determined in step S202 that the feedback control information FB does not include the second level, the drive control unit 221 inputs the signal sources S1-1 and S1-2 as first feedback signals to the amplifiers 432 and 442, respectively.

[0079] Step S210: Furthermore, the drive control unit 221 instructs the amplifier 432 to use the first vibration response level, and instructs the amplifier 442 to use the first sound response level.

[0080] By executing the processes of steps S208 and S210 as described above, the tactile sensation is reproduced by changing the strength of vibration and the volume of the writing sound according to the moving speed of the pen tip during movement.

[0081] A modification of this embodiment will now be described. The tactile feedback may be performed by either vibration or sound. In this case, the information processing device 100 and the pen-type input device 200 may have a configuration corresponding to one of the tactile feedbacks, vibration and sound, and may not have a configuration corresponding to the other tactile feedback. Alternatively, whether vibration or sound is used for the tactile feedback may be set according to, for example, a selection operation by a user, or a type of writing implement or a type of medium to be written on set in a pen operation compatible application.

[0082] A predetermined functional unit included in the information processing device 100 in the above embodiment may be provided in the pen type input device 200. Alternatively, a predetermined functional unit included in the pen type input device 200 in the above embodiment may be provided in the information processing device 100. Specifically, for example, the feedback control unit 122 may be provided in the pen type input device 200. In this case, the information processing device 100 may be configured to transmit the contact parameters detected by the contact parameter detection unit 101 to the pen type input device 200. Also, for example, the drive control unit 221 may be provided in the information processing device 100. In this case, the drive control unit 221 may transmit to the pen type input device 200 a signal instructing input of a feedback signal to the amplifiers (432, 433, 442, 443) and a signal instructing the level to be set in each of the amplifiers (432, 433, 442, 443).

[0083] In addition, a program for realizing the functions of the above-mentioned information processing device 100, the pen-type input device 200, etc. may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to perform processing as the above-mentioned information processing device 100, the pen-type input device 200, etc. Here, "reading a program recorded in a recording medium into a computer system and executing it" includes installing the program into a computer system. The "computer system" here includes hardware such as an OS and peripheral devices. In addition, the "computer system" may include a plurality of computer devices connected via a network including a communication line such as the Internet, a WAN, a LAN, or a dedicated line. In addition, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, an optical magnetic disk, a ROM, a CD-ROM, etc., and a storage device such as a hard disk built into a computer system. In this way, the recording medium storing the program may be a non-transient recording medium such as a CD-ROM. In addition, the recording medium also includes a recording medium provided inside or outside the computer system that can be accessed from a distribution server in order to distribute the program. The code of the program stored in the recording medium of the distribution server may be different from the code of the program in a format executable by a terminal device. That is, the format in which the program is stored in the distribution server does not matter as long as the program can be downloaded from the distribution server and installed in a form executable by the terminal device. The program may be divided into multiple parts, each of which may be downloaded at a different time and then combined on the terminal device, or each of the divided programs may be distributed by a different distribution server. Furthermore, the term "computer-readable recording medium" includes a storage medium that holds a program for a certain period of time, such as a volatile memory (RAM) inside a computer system that is a server or client when a program is transmitted over a network. The program may also be one that realizes part of the above-mentioned functions.Furthermore, the above-mentioned functions may be realized in combination with a program already recorded in the computer system, that is, a so-called differential file (differential program). [Explanation of symbols]

[0084] 30 touch panel display unit, 31 display unit, 32 panel sensor, 43 vibration unit, 44 sound output unit, 100 information processing device, 101 contact parameter detection unit, 102 control unit, 103 memory unit, 111 movement speed detection unit, 112 pen pressure detection unit, 113 movement direction detection unit, 114 angle detection unit, 121 application corresponding processing unit, 122 feedback control unit, 200 pen type input device, 201 control unit, 203 memory unit, 221 drive control unit, 231 feedback signal storage unit

Claims

1. a contact parameter detection unit that detects one or more contact parameters corresponding to a contact state of the pen-type input device with respect to the operation target surface; a feedback control unit that determines a contact state of a pen tip of the pen type input device with respect to the operation target surface based on the contact parameters, and when it is determined that the contact state is a moving state in which the pen tip of the pen type input device is moving while in contact with the operation target surface, executes a first feedback control to drive a tactile reproduction unit in the pen type input device so as to reproduce a tactile sensation corresponding to the moving state, and a second feedback control to drive the tactile reproduction unit at a timing when it is determined that the contact state is a movement start state in which the pen tip of the pen type input device has started moving from a stationary state while in contact with the operation target surface, so as to reproduce a tactile sensation corresponding to the movement start state; An information processing system comprising:

2. The haptic reproduction unit includes a vibration unit that generates vibrations in response to an input of a feedback signal. The information processing system according to claim 1 .

3. The haptic reproduction unit includes a sound output unit that outputs a sound in response to an input of a feedback signal. The information processing system according to claim 1 .

4. The second feedback control includes control for inputting a feedback signal corresponding to the second feedback control as an impulse signal to the haptic reproduction unit. The information processing system according to claim 1 .

5. The second feedback control includes control for inputting a feedback signal corresponding to the second feedback control, which has a predetermined peak level at a predetermined time according to a timing when the pen tip of the pen type input device starts to move, to the tactile reproduction unit. The information processing system according to claim 1 .

6. The contact parameters are all or some of the following: a speed at which the pen tip moves on the operation surface; a pressure of the pen tip on the operation surface; a direction in which the pen tip moves on the operation surface; and a tilt angle of the pen-type input device with respect to the operation surface. The information processing system according to claim 1 .

7. The feedback control unit sets a level of a feedback signal corresponding to a second feedback control based on at least the pressure of the contact parameters.

7. The information processing system according to claim 6.

8. The feedback control unit changes a level of a feedback signal corresponding to a first feedback control based on at least the speed of the contact parameters.

7. The information processing system according to claim 6.

9. An information processing method in an information processing system, comprising: a contact parameter detection step in which a contact parameter detection unit detects one or more contact parameters corresponding to a contact state of the pen type input device with respect to an operation target surface; a feedback control step of executing a first feedback control in which a feedback control unit determines a contact state of the pen tip of the pen type input device with respect to the operation target surface based on the contact parameters, and when it is determined that the contact state is a moving state in which the pen tip of the pen type input device is moving while in contact with the operation target surface, drives a tactile reproduction unit in the pen type input device so as to reproduce a tactile sensation corresponding to the moving state, and a second feedback control in which, at a timing when it is determined that the contact state is a movement start state in which the pen tip of the pen type input device starts moving from a stationary state while in contact with the operation target surface, drives the tactile reproduction unit so as to reproduce a tactile sensation corresponding to the movement start state; An information processing method comprising:

10. A computer in an information processing system, a contact parameter detection unit that detects one or more contact parameters corresponding to a contact state of the pen-type input device with respect to the operation target surface; a feedback control unit that executes a first feedback control for determining a contact state of a pen tip of the pen type input device with respect to the operation target surface based on the contact parameters, and when it is determined that the contact state is a moving state in which the pen tip of the pen type input device is moving while in contact with the operation target surface, driving a tactile reproduction unit in the pen type input device so as to reproduce a tactile sensation corresponding to the moving state, and a second feedback control for driving the tactile reproduction unit so as to reproduce a tactile sensation corresponding to the moving start state at a timing when it is determined that the contact state is a moving start state in which the pen tip of the pen type input device starts moving from a stationary state while in contact with the operation target surface. A program to function as a

Images