Information processing system, information processing method, and program

Abstract

The present invention configures an information processing system that can approach a feeling when writing on a writing medium in reality. The information processing system is provided with: a contact parameter detection unit that detects one or more contact parameters corresponding to a contact state of the pen-type input device with respect to an operation target surface; and a feedback control unit that, on the basis of the contact parameter, determines a contact state of a pen tip of the pen-shaped input device with respect to the operation target surface, and when it is determined that the contact state is a pen tip moving state of the pen-shaped input device, drives a tactile sense reproduction unit in the pen-shaped input device to reproduce a tactile sense corresponding to the moving state, when it is determined that the contact state is a movement start state in which the pen tip of the pen-shaped input device moves from a state in which the pen tip comes into contact with the operation target surface and is stationary, the tactile sense reproduction unit is driven to reproduce a tactile sense corresponding to the movement start state.

Description

TECHNICAL FIELD

[0001] The present application relates to an information processing system, an information processing method, and a program. BACKGROUND

[0002] It is known that, at the time of input to an interactive display system, a stylus-shaped input device (pen-shaped input device) capable of haptic feedback by generating vibration that drives a haptic actuator is used as an input device (for example, refer to Patent Literature 1).

[0003] Patent Literature 1: Japanese Patent Application Laid-Open No. 2017-537395

[0004] By causing the pen-shaped input device to vibrate, a feeling such as when a pen nib is slid on a writing medium such as paper is approximately obtained, and thus a user who inputs using the pen-shaped input device can obtain a feeling close to when writing on a writing medium.

[0005] At the time of input using the pen-shaped input device, there is an aspect in which it is preferable to approximate a feeling when writing on a real writing medium as much as possible. SUMMARY

[0006] Therefore, the present application considers the above-described problem, and aims to approximate a feeling when writing on a writing medium in reality at the time of input using a pen-shaped input device.

[0007] One way of the present application that solves the above-described problem is an information processing system including: a contact parameter detection unit that detects one or more contact parameters corresponding to a contact state of a pen-shaped input device with respect to an operation target surface; and a feedback control unit that determines a contact state of a pen nib of the pen-shaped input device with respect to the operation target surface based on the contact parameter, and executes first feedback control when it is determined that the contact state is a moving state in which the pen nib of the pen-shaped input device moves while being in contact with the operation target surface, and executes second feedback control when it is determined that the contact state is a moving start state in which movement is started from a state in which the pen nib of the pen-shaped input device is in contact with the operation target surface and is stationary, in which, in the first feedback control, a haptic reproduction unit in the pen-shaped input device is driven to reproduce a haptic sensation corresponding to the moving state, and in the second feedback control, the haptic reproduction unit is driven to reproduce a haptic sensation corresponding to the moving start state.

[0008] One embodiment of the present application is an information processing method in an information processing system, including: a contact parameter detection step in which a contact parameter detection unit detects one or more contact parameters corresponding to a contact state of a pen-shaped input device with respect to a surface of an operation object; and a feedback control step in which a feedback control unit determines a contact state of a pen tip of the pen-shaped input device with respect to the surface of the operation object based on the contact parameters, and when the contact state is determined to be a moving state in which the pen tip of the pen-shaped input device is in contact with the surface of the operation object while moving, performs first feedback control, and when the contact state is determined to be a moving start state in which movement is started from a state in which the pen tip of the pen-shaped input device is in contact with the surface of the operation object and is stationary, performs second feedback control, in which, in the first feedback control, a haptic reproduction unit in the pen-shaped input device is driven to reproduce a haptic sensation corresponding to the moving state, and in the second feedback control, the haptic reproduction unit is driven to reproduce a haptic sensation corresponding to the moving start state.

[0009] One embodiment of the present application is a program for causing a computer in an information processing system to function as a contact parameter detection unit and a feedback control unit, the contact parameter detection unit detecting one or more contact parameters corresponding to a contact state of a pen-shaped input device with respect to a surface of an operation object, and the feedback control unit determining a contact state of a pen tip of the pen-shaped input device with respect to the surface of the operation object based on the contact parameters, and when the contact state is determined to be a moving state in which the pen tip of the pen-shaped input device is in contact with the surface of the operation object while moving, performing first feedback control, and when the contact state is determined to be a moving start state in which movement is started from a state in which the pen tip of the pen-shaped input device is in contact with the surface of the operation object and is stationary, performing second feedback control, in which, in the first feedback control, a haptic reproduction unit in the pen-shaped input device is driven to reproduce a haptic sensation corresponding to the moving state, and in the second feedback control, the haptic reproduction unit is driven to reproduce a haptic sensation corresponding to the moving start state.

[0010] According to the present application, an effect of being able to approximate a feeling in reality when writing on a writing medium at the time of input using a pen-shaped input device is obtained. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a diagram showing an example of an external appearance structure of an information processing system in the present embodiment.

[0012] Figure 2 is a diagram showing a change in frictional force of a pen tip corresponding to a lapse of time at the time of starting a writing operation in the present embodiment.

[0013] Figure 3 is a diagram showing an outline of feedback control in the present embodiment.

[0014] Figure 4 is a diagram showing one example of a movement trajectory of a pen tip of a pen-shaped input device on a panel surface in the present embodiment.

[0015] Figure 5 is a diagram showing a hardware structure example of an information processing apparatus and a pen-shaped input device in the present embodiment.

[0016] Figure 6 is a diagram showing a functional structure example of an information processing apparatus and a pen-shaped input device in the present embodiment.

[0017] Figure 7 is a diagram showing a functional structure example corresponding to driving of a vibration section and a sound output section based on a feedback signal in the present embodiment.

[0018] Figure 8 is a flowchart showing a processing step example performed by an information processing apparatus and a pen-shaped input device in relation to haptic feedback in the present embodiment.

[0019] Explanation of Reference Numerals

[0020] 30 touch panel display section; 31 display section; 32 panel sensor; 43 vibration section; 44 sound output section; 100 information processing apparatus; 101 contact parameter detection section; 102 control section; 103 storage section; 111 movement speed detection section; 112 pen pressure detection section; 113 movement direction detection section; 114 angle detection section; 121 application corresponding processing section; 122 feedback control section; 200 pen-shaped input device; 201 control section; 203 storage section; 221 drive control section; 231 feedback signal storage section. DETAILED DESCRIPTION

[0021] Figure 1 is a diagram showing an example of an external appearance structure of an information processing system of the present embodiment. As shown in this diagram, the information processing system of the present embodiment has an information processing apparatus 100 and a pen-shaped input device 200.

[0022] The information processing apparatus 100 is capable of performing information processing in accordance with an input operation of the pen-shaped input device 200. The information processing apparatus 100 of this diagram shows an example of a tablet terminal or a notebook computer or the like.

[0023] The information processing apparatus 100 is provided with a touch panel display section 30. The touch panel display section 30 is a section in which a touch panel and a display section are combined. The touch panel display section 30 displays an image on a panel surface (display surface: one example of an operation target surface), and is capable of being operated by bringing a pen-shaped input device, a finger, or the like into contact with the panel surface.

[0024] The pen-shaped input device 200 is a pen-shaped input device used by a user to operate a touch panel in the touch panel display portion 30 of the information processing apparatus 100.

[0025] The user holds the pen-shaped input device 200 so that the pen tip contacts the panel surface of the touch panel display portion 30 and moves, thereby performing a character, drawing, figure, or the like handwriting input operation.

[0026] Further, as the operation using the pen-shaped input device 200, there can be a pointing operation or the like on a user interface image displayed in the touch panel display portion 30.

[0027] Further, the detection method of the pen-shaped input device 200 based on the touch panel in the touch panel display portion 30 according to the present embodiment is not particularly limited, and for example, an electrostatic capacity method, an electromagnetic induction method, or the like can be cited. In the following description, a case using the electrostatic capacity method is cited.

[0028] An application (pen operation corresponding application) corresponding to the input operation of the pen-shaped input device 200 is installed in the information processing apparatus 100.

[0029] The pen operation corresponding application can display a character, drawing, or the like drawn according to a character input, drawing, or the like handwriting input operation performed by contacting the pen tip of the pen-shaped input device 200 to the touch panel display portion 30, for example, in the touch panel display portion 30, and can perform a process of dataizing the character, drawing, or the like drawn by the operation.

[0030] Further, in the information processing system according to the present embodiment, the pen-shaped input device 200 emits vibration and sound (writing sound) corresponding to when a writing operation (writing operation) is performed. By the vibration and writing sound emitted by the pen-shaped input device 200, the user feels a tactile sensation close to when writing on a writing medium such as paper, and can obtain a feeling close to when writing on a writing medium by a writing tool of a real object.

[0031] In the following description, the generation of vibration and sound when a writing operation is performed by the pen-shaped input device 200 is also referred to as "tactile feedback".

[0032] Next, a summary of the tactile feedback in the present embodiment is described.

[0033] Figure 2represents a change in the friction of the writing tool tip corresponding to the passage of time at the time when the user starts the writing operation in reality. At time tO, the user contacts the writing tool tip (e.g., a pencil or the like) with a writing medium (e.g., paper or the like) but is in a state of not moving the writing tool tip and being stationary. Then, the user starts the operation of moving the writing tool tip on the writing medium according to the start of writing from time tO by a certain time to time tl.

[0034] Here, at the time of starting the operation of moving the writing tool tip, the writing tool tip is contacted in a manner that the writing tool tip is pressed on the writing medium surface with a certain degree of force, and thus static friction is generated. Therefore, during a period from time tl to time t2 by a certain time, even if the user applies a force to move the writing tool tip, the writing tool tip does not move on the writing medium and remains in a stationary state, while the static friction increases with the passage of time.

[0035] Moreover, if the writing tool tip starts to move up to time t2, the static friction that has increased up to this point is sharply reduced, and becomes a state of generating certain kinetic friction corresponding to the movement of the writing tool tip. Such a change in the friction is obtained as a stronger vibration, sound (writing sound) in reality.

[0036] At the time of movement of the pen tip of the pen-shaped input device 200, the haptic feedback based on the vibration, sound can also be provided at a certain level, but the level of the haptic feedback can also be changed according to the movement state of the pen tip. As one example, according to the increase in the movement speed of the pen tip, the level of the haptic feedback can be changed to be also large.

[0037] However, in the change in the level of the haptic feedback corresponding to the movement speed of the pen tip, the movement speed at the time of starting the movement of the pen tip is low, and thus the haptic feedback at the time of starting the movement of the pen tip is also small. That is, the change in the haptic feedback corresponding to the movement speed of the pen tip cannot be reproduced as the large static friction instantaneously generated at the time when the writing tool tip starts to move. Figure 2

[0038] Therefore, in the present embodiment, as shown below in Figure 3 , the control of the haptic feedback according to the movement of the pen tip of the pen-shaped input device 200 is performed.

[0039] In Figure 3 , lines LI-1, LI-2 show the changes in the levels of two feedback signals output corresponding to the haptic feedback of the present embodiment with the passage of time. In addition, line L2 represents the change corresponding to the passage of time with respect to the movement speed detected by the pen tip.

[0040] ​The time t10 is when the pen tip of the pen-shaped input device 200, which has been in a stationary state up to this point, starts to move in accordance with the user's writing operation. In accordance with the start of the movement of the pen tip at the time t10, the output of the first feedback signal, which varies in accordance with the speed of movement of the pen tip as shown by the line L1-1, starts. The first feedback signal stops outputting at zero level until the time tll when the movement of the pen tip stops.

[0041] In addition, in accordance with the start of the movement of the pen tip at the time t10, the second feedback signal, which momentarily has a peak value as shown by the line L1-2, is also output.

[0042] In this way, in the present embodiment, from the start of the movement of the pen tip to the stop thereof, the haptic feedback based on the combination of the two feedback signals, the first feedback signal and the second feedback signal, is performed.

[0043] Through such haptic feedback, in addition to the vibration and the reproduction of the writing sound corresponding to the speed of movement of the pen tip, the vibration and the reproduction of the writing sound corresponding to the momentarily increased stationary friction at the time when the movement of the pen tip of the pen-shaped input device 200 starts are also reproduced. As a result, at the time of input using the pen-shaped input device 200, it is possible to approach the feeling at the time of writing on a writing medium in reality.

[0044] Figure 4 One example of the movement trajectory MT of the pen tip of the pen-shaped input device 200 based on the user's writing operation on the panel face PN is shown. The movement trajectory MT of the figure starts from the start point pl, changes the direction of movement in the order of the direction change points p2, p3, p4, p5, and ends at the end point p6.

[0045] In accordance with such a movement trajectory MT, the movement of the pen tip temporarily stops after each direction change point p2, p3, p4, p5, and then starts to move again. In this case, the haptic feedback based on the first feedback signal, which varies in accordance with the speed of movement of the pen tip from the start to the stop of the movement, is performed in each section pl-p2, p2-p3, p3-p4, p5-p6. In addition, the haptic feedback based on the second feedback signal, which corresponds to the stationary friction, is performed at the start point pl and each direction change point p2, p3, p4, p5.

[0046] In the present embodiment, the peak level of the second feedback signal can also be a fixed value decided in advance. Alternatively, the peak level of the second feedback signal can be set based on a prescribed contact parameter among the detection values (contact parameters) obtained in accordance with the detection of a prescribed contact state of the pen tip with respect to the panel face.

[0047] Specifically, the information processing device 100 of this embodiment can list the speed at which the pen tip moves on the panel surface (moving speed), the pressure when the pen tip contacts the panel surface (pen pressure), the direction in which the pen tip moves on the panel surface, and the tilt angle of the pen-shaped input device 200 relative to the panel surface when the pen tip contacts the panel surface as contact parameters.

[0048] The static friction force when the pen tip starts moving by writing increases according to the pen pressure at the start of movement. Therefore, the information processing device 100 can set the peak level of the second feedback signal according to the contact parameter as the pen pressure.

[0049] Furthermore, the static friction force when the pen tip starts moving by writing varies depending on the tilt angle of the pen-shaped input device 200 relative to the panel surface. Therefore, the information processing apparatus 100 can set the peak level of the second feedback signal based on the contact parameter, which is the tilt angle.

[0050] Furthermore, the static friction force when the pen tip starts moving by a writing operation may also vary depending on the direction of movement of the pen tip on the panel surface corresponding to the writing operation. Therefore, the information processing device 100 may set the peak level of the second feedback signal based on the contact parameter serving as the movement direction.

[0051] In addition, the information processing device 100 may use two or more contact parameters of the pen pressure, tilt angle, and movement direction in combination to set the peak level of the second feedback signal.

[0052] Reference Figure 5 , an example of the hardware structure of the information processing apparatus 100 and the pen-shaped input device 200 is described.

[0053] First, an example of the hardware configuration of information processing device 100 will be described. Information processing device 100 shown in the figure includes a processor 11, main memory 12, flash memory 13, peripherals 14, a short-range communication unit 15, a baseband chip 21, a communication unit 22, an audio system 23, a microphone 24, a speaker 25, and a touch-panel display unit 30.

[0054] The processor 11 is, for example, an application processor including a CPU (Central Processing Unit) and controls the entire information processing device 100 .

[0055] The main memory 12 is a writable memory used as a read-in area of an execution program of the processor 11 or as a job area for writing processing data of the execution program. The main memory 12 is constituted by, for example, a plurality of DRAM (Dynamic Random Access Memory) chips. The execution program contains an OS (Operating System), various device drivers for hardware operation of peripheral device classes, various services / utilities, application programs (application software), and the like.

[0056] The flash memory 13 is, for example, a flash EEPROM (Electrically Erasable Programmable Read Only Memory) that stores an OS, various drivers, various services / utilities, application programs (hereinafter, sometimes referred to as applications), and various data.

[0057] The touch panel display section 30 is a section that displays an image and enables operation of a panel surface of the display image by the pen-shaped input device 200. The touch panel display section 30 can also be operated by an operating body other than the pen-shaped input device 200, such as a finger, for example, and a case where the operating body is the pen-shaped input device 200 is exemplified here.

[0058] The touch panel display section 30 is provided with a display section 31 and a panel sensor 32.

[0059] The display section 31 is, for example, a liquid crystal display, an organic EL (Electro-Luminescence) display, and displays an image based on drawing data (display data) output from the processor 11.

[0060] The panel sensor 32 detects a contact state of a pen tip of the pen-shaped input device 200 with respect to a panel surface of the display section 31. Specifically, the panel sensor 32 can be able to detect a position at which the pen tip of the pen-shaped input device 200 contacts on the panel surface, a pressure based on the pen tip contacting the panel surface, a distance of the pen-shaped input device 200 from the panel surface, and the like. As a detection method of such a panel sensor 32, there is no particular limitation, and an electrostatic capacity type or an electromagnetic induction type or the like can be exemplified. In addition, the panel sensor 32 can be configured to use a plurality of detection methods in combination.

[0061] The peripheral device 14 is, for example, a WLAN (Wireless Local Area Network) module, a GPS (Global Positioning System) module, and a sensor class such as an acceleration sensor, and the like.

[0062] The audio system 23 is, for example, an audio IC (Integrated Circuit) that performs input, recording, reproduction, and output of sound data. The audio system 23 is connected to, for example, a microphone 24 and a speaker 25. The audio system 23 outputs, for example, sound data of a sound picked up by the microphone 24 to the processor 11 or the baseband chip 21.

[0063] In addition, the audio system 23 converts, for example, sound data acquired from the processor 11 or the baseband chip 21 into a sound signal and outputs the sound signal to the speaker 25.

[0064] The microphone 24 picks up a sound around the information processing apparatus 100. The microphone 24 picks up, for example, a user's voice and the like when performing voice communication with another terminal.

[0065] The speaker 25 outputs various sounds to the outside of the information processing apparatus 100. The speaker 25 outputs, for example, a sound received from another terminal (plays back the sound) when performing voice communication with the other terminal.

[0066] The baseband chip 21 is, for example, a dedicated IC that controls wireless communication of 4G (4th Generation mobile communication system), 5G (5th Generation mobile communication system), and the like.

[0067] The baseband chip 21 outputs, for example, sound data received by the communication section 22 as a sound from the speaker 25 via the audio system 23.

[0068] In addition, the baseband chip 21 acquires, for example, sound data picked up by the microphone 24 via the audio system 23 and outputs the sound data via the communication section 22 using a mobile communication system. In addition, the baseband chip 21 can transmit and receive input and output data of data communication based on the mobile communication system between the processor 11.

[0069] The communication section 22 is a wireless communication device including an antenna that performs wireless communication based on a mobile communication system.

[0070] The close proximity communication section 15 is, for example, a Bluetooth (registered trademark) module that performs close proximity wireless communication with the pen-shaped input device 200.

[0071] Next, the same Figure 5 The hardware structure example of the pen-shaped input device 200 will be described.

[0072] The pen-shaped input device 200 includes an MCU 41, a close proximity communication section 42, a vibration section 43 (one example of a tactile sensation reproduction section), a sound output section 44 (one example of a tactile sensation reproduction section), and a flash memory 45.

[0073] The short-range communication unit 42 is, for example, a Bluetooth (registered trademark) module, and performs short-range wireless communication with the information processing device 100 .

[0074] The MCU (Micro Controller Unit) 41 includes a CPU, memory such as ROM and RAM, and I / O components, and controls the pen-shaped input device 200. The MCU 41 transmits and receives information to and from the short-range communication unit 42. Furthermore, the MCU 41 controls the generation of vibrations by the vibration unit 43 and the output of sound by the sound output unit 44, thereby providing tactile feedback in accordance with feedback signals (including the first feedback signal and the second feedback signal) transmitted from the information processing device 100.

[0075] The vibration unit 43 is provided with, for example, an actuator, and vibrates in response to input of a feedback signal from the MCU 41 .

[0076] The sound output unit 44 includes, for example, a speaker, and outputs sound in response to input of a feedback signal from the MCU 41 .

[0077] The flash memory 45 stores various data corresponding to the pen-type input device 200 , such as programs and feedback signal data.

[0078] Reference Figure 6 , the functional configuration example of the information processing apparatus 100 and the pen-shaped input device 200 is explained. Figure 5 The same functional parts in the hardware structure are marked with Figure 5 The same reference numerals are used and descriptions thereof are appropriately omitted.

[0079] The functions of the information processing device 100 shown in the figure can be performed by the processor 11 ( Figure 5 ) to execute the program.

[0080] First, a functional configuration example of the information processing device 100 will be described. The information processing device 100 includes a communication unit 22, a touch panel display unit 30, a contact parameter detection unit 101, a control unit 102, and a storage unit 103 as functional units.

[0081] The contact parameter detection unit 101 detects the contact state of the pen tip of the pen-shaped input device 200 with respect to the panel surface of the touch panel display unit 30 .

[0082] The contact parameter detection unit 101 outputs a parameter (contact parameter) detected according to the contact state based on the detection signal output by the panel sensor 32. In the following description, the contact parameter is exemplified by the aforementioned moving speed, pen pressure, moving direction, and tilt angle.

[0083] The control section 102 performs various controls in the information processing apparatus 100. The control section 102 includes an application corresponding processing section 121 and a feedback control section 122.

[0084] The application corresponding processing section 121 performs processing corresponding to an application in response to a pen operation.

[0085] As an operation with respect to the application corresponding to the pen operation, when a writing operation based on the pen-shaped input device 200 is performed, pen operation information is input from the panel sensor 32 to the application corresponding processing section 121. The application corresponding processing section 121 performs, for example, drawing and the like in response to the input pen operation information being input.

[0086] The feedback control section 122 performs feedback control. The feedback control in the present embodiment is control to generate vibration as a tactile feedback in the pen-shaped input device 200.

[0087] The feedback control section 122 of the present embodiment performs feedback control to reproduce vibration and sound generated in response to a change in friction when the nib of the pen-shaped input device 200, which was stationary in contact with the panel surface, starts to move (second feedback control), and feedback control to reproduce vibration and sound generated when the nib of the pen-shaped input device 200 moves (first feedback control).

[0088] The feedback control section 122 generates feedback control information based on the contact parameters output from the contact parameter detection section 101. The feedback control information is information to control the drive control section 221 of the pen-shaped input device 200 to generate tactile feedback based on vibration and sound. The feedback control section 122 outputs (transmits) the generated feedback control information to the pen-shaped input device 200 via the communication section 22.

[0089] The storage section 103 stores various information corresponding to the information processing apparatus 100.

[0090] Next, an example of the functional structure of the pen-shaped input device 200 will be described. The functions of the pen-shaped input device 200 shown in this drawing can be realized by the MCU 41 Figure 5 ) executing a program.

[0091] The pen-shaped input device 200 includes a close proximity communication section 42, a vibration section 43, a sound output section 44, a control section 201, and a storage section 203 as functional sections.

[0092] The control section 201 performs various controls in the pen-shaped input device 200. The control section 201 includes a drive control section 221.

[0093] The drive control section 221 controls the vibration section 43 to cause vibration as haptic feedback. In addition, the drive control section 221 controls the sound output section 44 to cause sound as haptic feedback.

[0094] The drive control section 221, when controlling the vibration section 43 and the sound output section 44, inputs the feedback signal (including the first feedback signal and the second feedback signal) stored in the feedback signal storage section 231 to the vibration section 43 and the sound output section 44.

[0095] The storage section 203 stores various data corresponding to the pen-shaped input device 200. The storage section 203 stores the feedback signal storage section 231.

[0096] The feedback signal storage section 231 stores waveforms that become feedback signals. The feedback signal storage section 231 stores waveforms of the first feedback signal and waveforms of the second feedback signal. In addition, the first feedback signal and the second feedback signal stored by the feedback signal storage section 231 can each include a waveform of a feedback signal corresponding to vibration used to vibrate the vibration section 43 and a waveform of a feedback signal corresponding to sound output from the sound output section 44 as sound.

[0097] In addition, the waveform of the second feedback signal can be, for example, a pulse waveform. That is, the second feedback signal can be a pulse signal. Alternatively, the waveform of the second feedback signal can be a waveform in which a time at which vibration, sound, is generated due to static friction when the pen tip starts to move has a sharp peak.

[0098] Reference Figure 7 An example of a functional configuration corresponding to driving of the vibration section 43 and the sound output section 44 based on a feedback signal will be described.

[0099] In the information processing apparatus 100, the contact parameter detection section 101 includes a movement speed detection section 111, a pen pressure detection section 112, a movement direction detection section 113, and a tilt angle detection section 114.

[0100] The movement speed detection section 111 detects the movement speed of the pen tip that moves while contacting the panel surface. The movement speed detection section 111 can detect the movement speed based on the amount of movement per unit time of the contact position of the pen tip detected by the panel sensor 32. The movement speed detection section 111 outputs the detected movement speed parameter Dt1 to the feedback control section 122.

[0101] The pen pressure detection section 112 detects the pressure (pen pressure) of the pen tip that contacts the panel surface. The pen pressure detection section 112 can detect the pen pressure based on the value of the electrostatic capacity at the contact position of the pen tip detected by the panel sensor 32. The pen pressure detection section 112 outputs the pen pressure parameter Dt2 indicating the detected pen pressure to the feedback control section 122.

[0102] The movement direction detection section 113 detects the movement direction of the pen tip that moves while being in contact with the panel surface. The movement speed detection section 111 can detect the movement direction based on the direction of movement of the contact position of the pen tip detected by the panel sensor 32. The movement direction detection section 113 outputs a movement direction parameter Dt3 indicating the detected movement direction to the feedback control section 122.

[0103] The tilt angle detection section 114 detects the tilt angle of the pen-shaped input device 200 main body with respect to the panel surface in the state where the pen tip is in contact with the panel surface. The tilt angle detection section 114 can detect the tilt angle based on the distribution of electrostatic capacitance corresponding to the pen-shaped input device 200 main body detected by the panel sensor 32 around the contact position of the pen tip. The tilt angle detection section 114 outputs a detected tilt angle parameter Dt4 to the feedback control section 122.

[0104] In the information processing apparatus 100, the feedback control section 122 generates feedback control information FB based on the input contact parameters (movement speed parameter Dt1, pen pressure parameter Dt2, movement direction parameter Dt3, tilt angle parameter Dt4).

[0105] In the present embodiment, the feedback control section 122 independently generates each of the feedback control information FB corresponding to the vibration section 43 and the feedback control information FB corresponding to the sound output section 44, respectively.

[0106] Further, the feedback control section 122 can also control feedback control information that is common in the vibration section 43 and the sound output section 44.

[0107] The feedback control information FB includes the level of the first feedback signal corresponding to vibration (vibration corresponding first level) and the level of the first feedback signal corresponding to sound (sound corresponding first level). The feedback control section 122 can set the vibration corresponding first level and the sound corresponding first level based on the movement speed parameter Dt1.

[0108] In addition, the feedback control information FB includes the level of the second feedback signal corresponding to vibration (vibration corresponding second level) and the level of the second feedback signal corresponding to sound (sound corresponding second level) with respect to information output at the time when the movement of the pen tip on the panel surface is started.

[0109] In the following description, in the case where the vibration corresponding first level and the sound corresponding first level are not particularly distinguished, it is described as the first level. In addition, in the case where the vibration corresponding second level and the sound corresponding second level are not particularly distinguished, it is described as the second level.

[0110] The feedback control section 122 outputs (transmits) the generated feedback control information FB to the pen-shaped input device 200.

[0111] The drive control unit 221 drives the vibration unit 43 and the sound output unit 44 based on the input feedback control information FB.

[0112] This figure shows a configuration example of the vibration unit 43 and the sound output unit 44 .

[0113] The vibration unit 43 includes, for example, an amplifier 432 , an amplifier 433 , a mixer 434 , and an actuator 435 .

[0114] Amplifier 432 receives the first feedback signal corresponding to vibration as signal source S1-1. Signal source S1-1 is generated based on the waveform of the first feedback signal corresponding to vibration stored in feedback signal storage unit 231. Amplifier 432 changes the input first feedback signal corresponding to vibration to the level specified by drive control unit 221 and outputs it.

[0115] Amplifier 433 receives the second feedback signal corresponding to vibration as signal source S2-1. Signal source S2-1 is generated based on the waveform of the second feedback signal corresponding to vibration stored in feedback signal storage unit 231. Amplifier 433 changes the input second feedback signal corresponding to vibration to the level specified by drive control unit 221 and outputs it.

[0116] The mixer 434 combines the first feedback signal output from the amplifier 432 and the second feedback signal output from the amplifier 433 .

[0117] The actuator 435 vibrates by the feedback signal output from the mixer 434 .

[0118] Corresponding to the structure of such a vibration unit 43, the drive control unit 221 controls the signal source S1-1 based on the first feedback signal corresponding to the vibration to be input to the amplifier 432 and controls the level of the first feedback signal corresponding to the vibration to be indicated to the amplifier 432 based on the first level of the vibration corresponding contained in the feedback control information FB.

[0119] In addition, when the feedback control information FB includes the vibration corresponding second level, based on the vibration corresponding second level, control is performed to input the signal source S2-1 based on the vibration corresponding second feedback signal to the amplifier 433 and control is performed to indicate the level of the vibration corresponding second feedback signal to the amplifier 433.

[0120] The sound output unit 44 includes, for example, an amplifier 442 , an amplifier 443 , a mixer 444 , and a speaker 445 .

[0121] The amplifier 442 is inputted with the sound corresponding first feedback signal as a signal source S1-2. The signal source S1-2 is generated based on the waveform of the sound corresponding first feedback signal stored in the feedback signal storage section 231. The amplifier 442 changes the inputted sound corresponding first feedback signal to the level instructed by the drive control section 221 and outputs it.

[0122] The amplifier 443 is inputted with the sound corresponding second feedback signal as a signal source S2-2. The signal source S2-1 is generated based on the waveform of the sound corresponding second feedback signal stored in the feedback signal storage section 231. The amplifier 443 changes the inputted sound corresponding second feedback signal to the level instructed by the drive control section 221 and outputs it.

[0123] The mixer 444 synthesizes the first feedback signal outputted from the amplifier 442 and the second feedback signal outputted from the amplifier 443.

[0124] The speaker 445 emits sound corresponding to the feedback signal outputted from the mixer 444.

[0125] Corresponding to the structure of such a sound output section 44, the drive control section 221 performs control to input the signal source S1-2 based on the sound corresponding first feedback signal to the amplifier 442 and control to instruct the amplifier 442 of the level of the sound corresponding first feedback signal based on the sound corresponding first level contained in the feedback control information FB.

[0126] In addition, in a case where the sound corresponding second level is contained in the feedback control information FB, control to input the signal source S2-2 based on the sound corresponding second feedback signal to the amplifier 443 and control to instruct the amplifier 443 of the level of the sound corresponding second feedback signal are performed based on the sound corresponding second level.

[0127] Referring to the flowchart of Figure 8 , the processing steps example of the information processing apparatus 100 and the pen-shaped input device 200 performed in relation to the tactile feedback will be described.

[0128] First, the processing steps example of the information processing apparatus 100 will be described. In performing the processing of this figure, the feedback control section 122 acquires the contact parameters (the moving speed parameter Dt1, the pen pressure parameter Dt2, the moving direction parameter Dt3, the tilt angle parameter Dt4) detected by the contact parameter detection section 101, for example, every prescribed acquisition period (acquisition period). At this time, the feedback control section 122 can acquire one sample of the contact parameters every acquisition period (period) or can acquire the contact parameters corresponding to each of a plurality of sampling periods.

[0129] Step S100: In the information processing apparatus 100, the feedback control section 122 waits for the state in which the pen tip of the pen-shaped input device 200 in contact with the panel surface and stopped becomes a state in which movement starts (movement start state) based on the contact parameter acquired at this sampling timing.

[0130] In this case, the feedback control section 122 can determine that the pen tip starts to move, for example, by the movement speed parameter Dt1 in the contact parameter changing from zero to a value larger than zero. Alternatively, the feedback control section 122 can determine that the pen tip starts to move based on the movement direction parameter Dt3 changing from a value not indicating a movement direction to a value indicating a certain movement direction by the pen tip not moving (stopping). In addition, the feedback control section 122 can determine whether or not the pen tip starts to move using the movement speed parameter Dt1 and the movement direction parameter Dt3 described above in combination.

[0131] Step S102: If it is determined in step S100 that the pen tip starts to move, the feedback control section 122 sets the levels of the respective second feedback signals of the vibration correspondence and the sound correspondence (second levels: vibration correspondence second level, sound correspondence second level).

[0132] The feedback control section 122 can set the second levels based on the pressure parameter Dt2. In actual writing, the vibration and the sound at the time when the leading end of the writing tool starts to move become larger as the pressure against the writing target medium such as paper at the time when the leading end of the writing tool starts to move becomes higher.

[0133] Also, the feedback control section 122 can use, for example, acceleration calculated based on a plurality of movement speed parameters Dt1 obtained during this acquisition period at the time of setting the second levels.

[0134] Also, the feedback control section 122 can use, for example, the kind of the writing tool, the kind of the medium of the writing target, and the like set in the pen operation corresponding application at the time of setting the second levels.

[0135] Step S104: Next, the feedback control section 122 sets the levels of the respective first feedback signals of the vibration correspondence and the sound correspondence (first levels: vibration correspondence first level and sound correspondence first level).

[0136] At this time, the feedback control section 122 can set each of the vibration correspondence first level and the sound correspondence first level based on the movement speed parameter Dt1 acquired at the timing of the sampling period this time.

[0137] Also, the feedback control section 122 can use at least one of the pen pressure parameter Dt2, the moving direction parameter Dt3, and the tilt angle parameter Dt4 at the time of setting the vibration corresponding first level and the sound corresponding first level. For example, the magnitude of the vibration and the sound generated in actual writing according to the movement of the nib can also change depending on the pen pressure.

[0138] Also, the feedback control section 122 can use, for example, the type of the writing tool set in the pen operation corresponding application, the type of the medium of the writing object, and the like at the time of setting the first level.

[0139] Step S106: The feedback control section 122 generates feedback control information FB containing the second level set in step S102 and the first level set in step S104.

[0140] Step S108: The feedback control section 122 transmits the feedback control information generated in step S106 to the pen-shaped input device 200.

[0141] Step S110: The feedback control section 122 determines whether or not the movement of the nib started in correspondence with step S100 is stopped. The feedback control section 122 can determine that the movement of the nib is stopped according to the moving speed parameter Dt1 becoming zero as a value larger than 0 up to this point.

[0142] In the case where it is determined that the nib is in a state of being moved (moving state), that is, the movement is not stopped, the processing returns to step S104. That is, the feedback control section 122 executes the loop processing of generating and transmitting the feedback control information FB corresponding to the state of the nib being moved until the movement of the nib is stopped. The feedback control information FB generated in such loop processing does not contain the second level.

[0143] On the other hand, in the case where it is determined that the movement of the nib is stopped, the processing returns to step S100. In this way, by returning the processing to step S100, the transmission of the feedback control information FB to the pen-shaped input device 200 is stopped until the movement of the nib is started again.

[0144] Next, the processing step example executed by the pen-shaped input device 200 will be described.

[0145] Step S200: In the pen-shaped input device 200, the drive control section 221 waits to receive the feedback control information FB transmitted from the information processing apparatus 100 in step S108.

[0146] Step S202: If the feedback control information FB is received, the drive control section 221 determines whether or not the second level is contained in the feedback control information FB received this time.

[0147] Step S204: In a case where it is determined in step S202 that the second level is included in the feedback control information FB, the drive control section 221 inputs the signal sources S2-1, S2-2, which are second feedback signals, to the amplifiers 433, 443, respectively.

[0148] Step S206: In addition, the drive control section 221 instructs the vibration corresponding to the second level to the amplifier 433 and instructs the sound corresponding to the second level to the amplifier 443.

[0149] As described above, by executing the processes of step S204 and step S206, the vibration corresponding to the static friction force generated at the start of the movement of the pen tip and the writing sound are reproduced. In addition, at this time, for example, the strength of the vibration and the size of the writing sound corresponding to the static friction force are obtained from the pen pressure at the start of the movement of the pen tip.

[0150] Step S208: After the process of step S206, or in a case where it is determined in step S202 that the second level is not included in the feedback control information FB, the drive control section 221 inputs the signal sources S1-1, S1-2, which are first feedback signals, to the amplifiers 432, 442, respectively.

[0151] Step S210: In addition, the drive control section 221 instructs the vibration corresponding to the first level to the amplifier 432 and instructs the sound corresponding to the first level to the amplifier 442.

[0152] As described above, by executing the processes of step S208 and step S210, the strength of the vibration and the size of the writing sound are changed in accordance with the moving speed of the pen tip in the movement and the haptics are reproduced.

[0153] Hereinafter, a modification of the present embodiment will be described.

[0154] The haptics feedback can be performed by either one of the vibration and the sound. In this case, the information processing apparatus 100 and the pen-shaped input device 200 can have a structure corresponding to the haptics feedback of one of the vibration and the sound and do not have a structure corresponding to the haptics feedback of the other. Alternatively, for example, which one of the vibration and the sound is used for the haptics feedback can be set in accordance with a selection operation of the user, a kind of the writing tool set in the pen operation corresponding application, and a kind of the medium of the writing object.

[0155] The prescribed functional section that the information processing apparatus 100 has in the above-described embodiment can be provided to the pen-shaped input device 200. Alternatively, the prescribed functional section that the pen-shaped input device 200 has in the above-described embodiment can be provided to the information processing apparatus 100.

[0156] Specifically, for example, the feedback control section 122 can be provided in the pen-shaped input device 200. In this case, the information processing apparatus 100 can transmit the contact parameter detected by the contact parameter detection section 101 to the pen-shaped input device 200.

[0157] In addition, for example, the drive control section 221 can be provided in the information processing apparatus 100. In this case, the drive control section 221 can transmit, to the pen-shaped input device 200, a signal instructing the input of the feedback signal to the amplifiers (432, 433, 442, 443), and a signal instructing the level set in each of the amplifiers (432, 433, 442, 443).

[0158] Further, a program for realizing the functions of the information processing apparatus 100, the pen-shaped input device 200, and the like described above can be recorded in a recording medium readable by a computer, and the processing as the information processing apparatus 100, the pen-shaped input device 200, and the like described above can be performed by causing a computer system to read in and execute the program recorded in the recording medium. Here, the "causing a computer system to read in and execute the program recorded in the recording medium" includes installing the program in the computer system. The "computer system" here includes an OS, hardware such as peripheral devices, and the like. In addition, the "computer system" can include a plurality of computer apparatuses connected via a network including a communication line such as the Internet, a WAN, a LAN, a dedicated line, and the like. In addition, the "recording medium readable by a computer" refers to a portable medium such as a floppy disk, a magneto-optical disk, a ROM, a CD-ROM, and the like, a storage device such as a hard disk built in a computer system, and the like. In this way, the recording medium storing the program can be a non-transitory recording medium such as a CD-ROM. In addition, the recording medium includes a recording medium provided inside or outside that can be accessed from a distribution server in order to distribute the program. The code of the program stored in the recording medium of the distribution server can be different from the code of the program that can be executed in the terminal apparatus. That is, as long as it is code that can be installed in a form that can be downloaded from the distribution server and executed in the terminal apparatus, it does not matter what form is stored in the distribution server. Further, the program can be divided into a plurality of pieces, and the terminal apparatus can be configured to be assembled after downloading each piece at different times, and the distribution server that distributes each piece of the divided program can be different. Furthermore, the "recording medium readable by a computer" also includes a structure that holds the program for a certain period of time, such as a server that transmits the program via a network, and a volatile memory (RAM) inside a computer system that becomes a client. In addition, the above-described program can be a structure for realizing a part of the above-described functions. Further, it can be a so-called differential file (differential program) that can realize the above-described functions in combination with a program already recorded in the computer system.

Claims

1. An information processing system comprising: a contact parameter detection section that detects one or more contact parameters corresponding to a contact state of a pen-shaped input device with respect to a surface of an operation object; and a feedback control section that determines a contact state of a pen nib of the pen-shaped input device with respect to the surface of the operation object based on the contact parameters, and executes first feedback control when it is determined that the contact state is a moving state in which the pen nib of the pen-shaped input device moves while being in contact with the surface of the operation object, and executes second feedback control when it is determined that the contact state is a moving start state in which the pen nib of the pen-shaped input device moves from a state in which the pen nib is in contact with the surface of the operation object and is stationary, wherein, in the first feedback control, a haptic reproduction section in the pen-shaped input device is driven to reproduce a haptic sensation corresponding to the moving state, and in the second feedback control, the haptic reproduction section is driven to reproduce a haptic sensation corresponding to the moving start state.

2. The information processing system according to claim 1, wherein the haptic reproduction section includes a vibration section that generates a vibration in accordance with an input of a feedback signal.

3. The information processing system according to claim 1, wherein the haptic reproduction section includes a sound output section that outputs a sound in accordance with an input of a feedback signal.

4. The information processing system according to any one of claims 1 to 3, wherein the second feedback control includes control that inputs a feedback signal corresponding to the second feedback control, which is a pulse signal, to the haptic reproduction section.

5. The information processing system according to any one of claims 1 to 3, wherein the second feedback control includes control that inputs a feedback signal corresponding to the second feedback control, which has a prescribed peak level at a prescribed time corresponding to a time when the pen nib of the pen-shaped input device starts moving, to the haptic reproduction section.

6. The information processing system according to any one of claims 1 to 3, wherein the contact parameters are all or a part of a speed at which the pen nib moves on the surface of the operation object, a pressure of the pen nib with respect to the surface of the operation object, a direction in which the pen nib moves on the surface of the operation object, and an inclination angle of the pen-shaped input device with respect to the surface of the operation object.

7. The information processing system according to claim 6, wherein the feedback control section sets a level of a feedback signal corresponding to the second feedback control based on at least the pressure among the contact parameters.

8. The information processing system according to claim 6, wherein the feedback control section changes a level of a feedback signal corresponding to the first feedback control based on at least the speed among the contact parameters.

9. An information processing method that is an information processing method in an information processing system, comprising: a contact parameter detection step in which a contact parameter detection section detects one or more contact parameters corresponding to a contact state of a pen-shaped input device with respect to a surface of an operation object; and ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ The feedback control step, based on the contact parameter, judges a contact state of a pen tip of the pen-shaped input device with respect to the operation object surface, and when the contact state is judged to be a moving state in which the pen tip of the pen-shaped input device is in contact with the operation object surface while moving, executes first feedback control, and when the contact state is judged to be a moving start state in which movement is started from a state in which the pen tip of the pen-shaped input device is in contact with the operation object surface and is stationary, executes second feedback control, in the first feedback control, drives a haptic reproduction section in the pen-shaped input device to reproduce a haptic sensation corresponding to the moving state, and in the second feedback control, drives the haptic reproduction section to reproduce a haptic sensation corresponding to the moving start state.

10. A program for causing a computer in an information processing system to function as a contact parameter detection section and a feedback control section, wherein the contact parameter detection section detects one or more contact parameters corresponding to a contact state of a pen-shaped input device with respect to an operation object surface, the feedback control section, based on the contact parameter, judges a contact state of a pen tip of the pen-shaped input device with respect to the operation object surface, and when the contact state is judged to be a moving state in which the pen tip of the pen-shaped input device is in contact with the operation object surface while moving, executes first feedback control, and when the contact state is judged to be a moving start state in which movement is started from a state in which the pen tip of the pen-shaped input device is in contact with the operation object surface and is stationary, executes second feedback control, in the first feedback control, drives a haptic reproduction section in the pen-shaped input device to reproduce a haptic sensation corresponding to the moving state, and in the second feedback control, drives the haptic reproduction section to reproduce a haptic sensation corresponding to the moving start state.

Images