Information processing system

Abstract

The present invention relates to an information processing system that does not impair the same feeling as when writing on a writing medium in reality when inputting using a pen-shaped input device. This information processing system is provided with: a sensor unit that detects the contact of a pen-shaped input device with an operation target surface and the presence of the pen-shaped input device within a detection limit distance on the operation target surface; a sound output unit that outputs sound corresponding to the inputted sound data; and a sound output control unit that inputs silent sound data to the sound output unit on the basis of a first state in which the sensor unit detects that the pen-shaped input device is detected within the detection limit distance, and that outputs sound data to the sound output unit on the basis of a second state in which the sensor unit detects that the pen-shaped input device is in contact with the operation target surface after the first state is reached. Audio data is inputted to the audio output unit.

Description

Technical Field

[0001] The present invention relates to information processing systems. Background Art

[0002] It is known that when inputting to an interactive display system, a pen-shaped stylus (pen-type input device) capable of providing tactile feedback by generating vibrations that drive a tactile actuator is used as an input device.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2017-537395.

[0004] By vibrating the pen-shaped input device, a user can obtain a tactile sensation similar to that of sliding the pen tip on a writing medium such as paper. Therefore, the user who uses the pen-shaped input device to input can obtain a feeling close to that of writing on a writing medium. Summary of the Invention

[0005] It is desirable that the input using a pen-shaped input device be as close as possible to the feeling of writing on a real writing medium.

[0006] Therefore, the present invention has been made in consideration of the above-mentioned problems, and an object thereof is to maintain the same feeling as when writing on a real writing medium when inputting using a pen-shaped input device.

[0007] One way of solving the above-mentioned problems in the present invention is an information processing system, comprising: a sensor unit, which detects the contact of a pen-shaped input device with an operation object surface, and detects the presence of the pen-shaped input device within a detection limit distance on the above-mentioned operation object surface; a sound output unit, which outputs sound corresponding to the input sound data; and a sound output control unit, which inputs silent sound data to the above-mentioned sound output unit based on a first state in which the pen-shaped input device is detected within the above-mentioned detection limit distance by the above-mentioned sensor unit, and inputs audible sound data to the above-mentioned sound output unit after becoming the above-mentioned first state based on a second state in which the pen-shaped input device is in contact with the operation object surface detected by the above-mentioned sensor unit.

[0008] According to the present invention, input using a pen-shaped input device can be performed without losing the same feeling as when writing on a real writing medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 This is a diagram showing an example of the external configuration of the information processing system in the first embodiment.

[0010] Figure 2 This is a diagram showing an example of a functional configuration corresponding to tactile feedback of the information processing apparatus and the pen-type input device in the first embodiment.

[0011] Figure 3 It is a diagram for explaining an example of a method for detecting a contact velocity in the first embodiment.

[0012] Figure 4 This is a flowchart showing an example of processing steps executed by the information processing apparatus and the pen-type input device in response to tactile feedback in the first embodiment.

[0013] Figure 5 It is a diagram for explaining an example of a method for detecting a contact velocity in the second embodiment.

[0014] Figure 6 This is a flowchart showing an example of processing steps executed by the information processing apparatus and the pen-type input device in response to tactile feedback in the second embodiment.

[0015] Figure 7 This is a diagram showing an example of a functional configuration corresponding to tactile and writing sound feedback of the information processing apparatus and the pen-shaped input device in the third embodiment.

[0016] Figure 8 This is a flowchart showing an example of processing steps executed by the pen-type input device in accordance with the third embodiment in response to tactile and writing sound feedback.

[0017] Figure 9 This is a diagram showing an example of a functional configuration corresponding to tactile and writing sound feedback of an information processing apparatus and a pen-shaped input device in a fourth embodiment.

[0018] Figure 10 This is a diagram illustrating a data input / output unit in the fourth embodiment.

[0019] Figure 11 This is a diagram illustrating the time delay of the sound output unit in the fourth embodiment.

[0020] Figure 12 This is a flowchart showing an example of processing steps executed by the information processing apparatus and the pen-type input device in accordance with tactile and writing sound feedback in the fourth embodiment.

[0021] Description of Reference Numerals

[0022] 100…information processing device; 101…communication unit; 102…touch panel display unit; 103…control unit; 104…storage unit; 105…speed detection unit; 106…sound output unit; 121…display unit; 122…touch panel; 131…application corresponding processing unit; 132…feedback control unit; 141…writing sound data storage unit; 161…queue; 200…pen-shaped input device; 201…communication unit; 202…vibration unit; 203…sound output unit; 204…control unit; 205…storage unit; 210…pen tip; 241…vibration control unit; 242…sound output control unit; 251…vibration waveform data storage unit; 252…writing sound data storage unit; 1221…panel surface. DETAILED DESCRIPTION

[0023] <First embodiment>

[0024] Figure 1 FIG. 1 shows an example of the external configuration of an information processing system according to the present embodiment. As shown in the figure, the information processing system according to the present embodiment includes an information processing apparatus 100 and a pen-shaped input device 200 .

[0025] The information processing apparatus 100 can execute information processing based on input operations using the pen-shaped input device 200. The information processing apparatus 100 in the figure shows an example in which the information processing apparatus 100 is a tablet terminal, a notebook personal computer, or the like.

[0026] The information processing device 100 includes a touch panel display unit 102. The touch panel display unit 102 is a combination of a touch panel and a display unit. The touch panel display unit 102 displays images on a display surface and can be operated by touching the display surface with an operating body such as a pen-shaped input device or a finger.

[0027] The pen-shaped input device 200 is a pen-shaped input device used by a user to operate the touch panel in the touch-panel display unit 102 of the information processing apparatus 100 .

[0028] The user holds the pen-shaped input device 200 , brings the pen tip into contact with the display surface of the touch-panel display portion 102 , and moves the pen to perform handwriting input operations such as text, drawing, and graphics.

[0029] Furthermore, an operation using the pen-shaped input device 200 may be a pointing operation on a user interface image displayed on the touch-panel display portion 102 .

[0030] The detection method of the pen-type input device 200 based on the touch panel in the touch panel display unit 102 of this embodiment is not particularly limited, and examples thereof include electrostatic capacitance, electromagnetic induction, etc. In the following description, the case of using the electrostatic capacitance method is taken as an example.

[0031] An application corresponding to input operation using the pen-shaped input device 200 (pen operation corresponding application) is installed in the information processing apparatus 100 .

[0032] The pen operation corresponding application can, for example, perform processing such as displaying text and drawings drawn on the touch panel display unit 102 by handwriting input operations such as text input and drawing performed by bringing the pen tip of the pen-shaped input device 200 into contact with the touch panel display unit 102, or digitizing the text and drawings drawn by the operation.

[0033] Furthermore, in the information processing system of this embodiment, the pen-shaped input device 200 vibrates in response to an operation corresponding to writing (writing operation). This vibration of the pen-shaped input device 200 allows the user to experience a tactile sensation similar to writing on a writing medium such as paper, allowing the user to obtain a sensation similar to writing on a writing medium using a physical writing utensil.

[0034] In the following description, the vibration generated when a writing operation is performed using the pen-shaped input device 200 is also referred to as "tactile feedback."

[0035] Figure 2 An example of the functional configuration of the information processing apparatus 100 and the pen-type input device 200 corresponding to the tactile feedback of this embodiment is shown.

[0036] First, a functional configuration example of information processing device 100 will be described. The functions of information processing device 100 shown in the figure are realized by executing programs by a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) included as hardware of information processing device 100.

[0037] An information processing device 100 in this figure includes a communication unit 101 , a touch panel display unit 102 , a control unit 103 , a storage unit 104 , and a speed detection unit 105 .

[0038] The communication unit 101 performs wireless communication with the pen-shaped input device 200. There is no particular limitation on the wireless communication method supported by the communication unit 101. Alternatively, the communication unit 101 may support wired communication with the pen-shaped input device 200.

[0039] The touch panel display portion 102 is a device in which a display portion 121 and a touch panel 122 (an example of a sensor portion) are combined.

[0040] The display unit 121 displays images under the control of the control unit 103 .

[0041] The touch panel 122 is provided in correspondence with the display surface of the display unit 121, and detects the position of the pen tip of the pen-shaped input device 200 and outputs the coordinates of the detected position as pen operation information. Specifically, the electrostatic capacitance touch panel 122 can detect the electrostatic capacitance generated by the contact (or proximity) of the pen tip of the pen-shaped input device 200 with the panel surface (an example of the operation target surface) through electrodes, thereby detecting the position (coordinates) of the pen tip on the panel surface.

[0042] The control unit 103 performs various controls in the information processing device 100. The control unit 103 includes an application-compatible processing unit 131 that performs processing corresponding to a pen operation-compatible application.

[0043] When a writing operation is performed using the pen-shaped input device 200 as an operation on a pen-operation-compatible application, pen operation information is input from the touch panel 122 to the application-compatible processing unit 131. The application-compatible processing unit 131 performs processing such as drawing based on the input pen operation information.

[0044] The storage unit 104 stores various information corresponding to the information processing device 100 .

[0045] The speed detection unit 105 detects the speed (contact speed) of the pen tip of the pen-shaped input device 200 when it contacts the panel surface based on a detection signal indicating capacitance generated in the touch panel 122 due to the proximity or contact of the pen-shaped input device 200 with the panel surface.

[0046] In addition, the speed detection unit 105 may be provided in the touch panel display unit 102 or in the control unit 103 .

[0047] Next, a functional configuration example of the pen-type input device 200 will be described. The functions of the pen-type input device 200 shown in the figure can be realized by, for example, causing a microprocessor such as an MCU (Micro Controller Unit) included in the pen-type input device 200 as hardware to execute a program.

[0048] The pen-type input device 200 in this figure includes a communication unit 201 , a vibration unit 202 , a control unit 204 , and a storage unit 205 .

[0049] The communication unit 201 is communicably connected to the communication unit 101 of the information processing device 100 .

[0050] The vibration unit 202 (actuator) is a portion that vibrates to provide a tactile sensation during a writing operation to a user holding the pen-shaped input device 200. The vibration of the vibration unit 202 is controlled by the vibration control unit 241.

[0051] The control unit 204 performs various controls in the pen-shaped input device 200. The control unit 204 includes a vibration control unit 241.

[0052] The vibration control unit 241 vibrates the vibration unit 202 based on the vibration waveform data stored in the vibration waveform data storage unit 251 .

[0053] The storage unit 205 stores various information related to the pen-shaped input device 200. The storage unit 205 includes a vibration waveform data storage unit 251. The vibration waveform data storage unit 251 stores vibration waveform data. The vibration waveform data is data representing a vibration waveform for causing the pen-shaped input device 200 to generate a desired vibration state.

[0054] The pen-shaped input device 200 of the information processing system of this embodiment, based on the above-described structure, can provide tactile feedback. Specifically, the pen-shaped input device 200 of this embodiment can vibrate in response to a writing operation performed while the pen tip is in contact with the surface of the touch-panel display unit 102 (touch panel 122). This vibration of the pen-shaped input device 200 during writing simulates the sensation of writing on a writing medium using a physical writing utensil.

[0055] As such tactile feedback, for example, if the vibration can be controlled by changing the intensity of the vibration according to the strength (contact strength) when the pen-shaped input device 200 contacts the panel surface, it can be closer to the feeling of writing on the writing medium with a real writing tool, and is therefore preferred.

[0056] Therefore, in the information processing system of this embodiment, the speed (panel contact speed) at which the front end (pen tip) of the pen-shaped input device 200 contacts the panel surface of the touch panel 122 due to a user's operation is detected. As the contact speed increases, the contact strength also increases, so there is a correlation between the contact speed and the contact strength. Therefore, the panel contact speed can be processed as the contact strength of the pen-shaped input device 200 relative to the panel surface of the touch panel 122. Therefore, the pen-shaped input device 200 of this embodiment vibrates with an intensity corresponding to the detected contact speed regarding the vibration generated at the time of contact with the panel surface of the touch panel 122.

[0057] Reference Figure 3An example of a contact velocity detection method in this embodiment will be described. In the touch panel 122 of this embodiment, which utilizes an electrostatic capacitance method, even if the pen tip 210 of the pen-shaped input device 200 is not in contact with the panel surface 1221, a change in electrostatic capacitance occurs when the pen tip 210 is within a certain detectable range d1 from the panel surface 1221. This allows the position of the pen tip 210 to be detected and a detection signal to be output. Furthermore, the electrostatic capacitance varies depending on the distance between the pen tip 210 and the panel surface 1221 within the detectable range d1.

[0058] When the user starts writing on the panel surface 1221 using the pen-shaped input device 200 , the user brings the pen tip 210 of the pen-shaped input device 200 into contact with the panel surface 1221 . Figure 3 The figure shows the change in the positional relationship between the pen tip 210 and the panel surface 1221 according to the passage of time when the user attempts to bring the pen tip 210 into contact with the panel surface 1221 .

[0059] When the user wants to bring the pen tip 210 of the pen-shaped input device 200 into contact with the panel surface 1221 , the user moves the pen tip 210 closer to the panel surface 1221 from a state where the pen tip 210 of the pen-shaped input device 200 is separated from the panel surface 1221 by a certain distance.

[0060] exist Figure 3 In the embodiment, first, the pen-shaped input device 200 approaches the panel surface 1221 from a position farther from the panel surface 1221 than the detectable range d1 , and the pen tip 210 reaches the detectable range d1 at time t1 .

[0061] In addition, in this embodiment, a speed detection target range d2 is determined. The speed detection target range d2 is a range from the panel surface 1221 to a predetermined distance shorter than the detectable range d1.

[0062] After time t1, the pen-shaped input device 200 approaches the panel surface 1221, and at time t2, the pen tip 210 reaches the speed detection target range d2. The touch panel 122 detects (samples) operations on the panel surface 1221 at a predetermined sampling rate. The sampling number based on the sampling at time t2 is "m." The sampling number is, for example, a number that is incremented by 1 each time a sample is taken. Thus, the sampling number m corresponding to the timing when the pen tip 210 reaches the speed detection target range d2 becomes the sampling number corresponding to the timing when the pen tip 210 is at the speed detection start position.

[0063] After time t2, as shown by the passage of time between times t3 and t4, the pen tip 210 of the pen-shaped input device 200 continues to approach the panel surface 1221. During this period, the electrostatic capacitance generated in the touch panel 122 changes according to the distance between the pen tip 210 and the panel surface 1221.

[0064] After time t4, the pen tip 210 of the pen-shaped input device 200 continues to approach the panel surface 1221, and at time t5, the pen tip 210 contacts the panel surface 1221. The sampling number at this point is "n." The velocity detection unit 105 can determine whether the pen tip 210 is in contact with the panel surface 1221 by, for example, detecting the electrostatic capacitance generated corresponding to the contact between the pen tip 210 and the panel surface 1221. Thus, the sampling number n corresponding to the timing of the pen tip 210 contacting the panel surface 1221 becomes the sampling number corresponding to the timing when the pen tip 210 reaches the end of velocity detection.

[0065] The speed detection unit 105 detects the contact speed of the pen-shaped input device 200 based on the contact between the pen tip 210 and the panel surface 1221. The speed detection unit 105 can determine the contact speed by dividing the speed detection target range d2 by the time the pen tip 210 moves within the speed detection target range d2 (corresponding to the period from time t2 to time t5). Specifically, the speed detection unit 105 can calculate the contact speed V using the following equation 1. In equation 1, sr is the sampling rate, m is the sampling number corresponding to the speed detection start position, and n is the sampling number corresponding to the speed detection end position.

[0066] V = d2 × sr / (n-m) (Formula 1)

[0067] Furthermore, when the touch panel 122 detects the presence of the pen tip 210 within the detectable range d1 or the speed detection target range d2, for example, it can change the sampling rate to a predetermined rate higher than the sampling rate set when the pen tip 210 is not detected. This change in the sampling rate can reduce the error between the sampling timing of sampling number m and the actual timing when the pen tip 210 reaches the speed detection start position, as well as the error between the sampling timing of sampling number n and the actual timing when the pen tip 210 contacts the panel surface 1221. This reduction in timing error improves the accuracy of the detected contact velocity.

[0068] Reference Figure 4 An example of processing steps executed by the information processing apparatus 100 and the pen-type input device 200 according to the present embodiment in response to tactile feedback will be described with reference to the flowchart of FIG.

[0069] First, an example of a processing procedure executed by the information processing device 100 will be described.

[0070] Step S100: In the information processing device 100, the speed detection unit 105 waits for the start of detection of the pen tip 210 by the touch panel 122. The speed detection unit 105 may be as follows: Figure 3As at time t1 , when the pen tip 210 reaches the detectable range d1 , capacitance is generated on the touch panel 122 , and when the detection signal corresponding to the generated capacitance is output, it is determined that detection of the pen tip 210 has started.

[0071] Step S102: The speed detection unit 105 changes the sampling rate of the touch panel 122 to increase based on the start of detection of the pen tip 210 by the touch panel 122. In other words, the process in this figure shows an example of changing the sampling rate based on the presence of the pen tip 210 in the detectable range d1.

[0072] Step S104: After the processing of step S102, the speed detection unit 105 waits for the distance from the pen tip 210 to the panel surface 1221 (the distance between the pen tip and the panel) to be less than the speed detection target range d2. In other words, the speed detection unit 105 waits for the distance from the pen tip 210 to the panel surface 1221 to be less than the speed detection target range d2. Figure 3 The state corresponding to time t1 in becomes the state corresponding to time t2.

[0073] Furthermore, during the waiting period in step S104, the distance between the pen tip and the panel may not be less than the velocity detection range d2, and the pen tip 210 may return to being outside the detectable range d1. In such a case, the sampling rate changed in step S102 may be returned to the default setting, and the processing of this figure may be terminated.

[0074] Step S106: The speed detection unit 105 obtains a sampling number m indicating the sampling executed at the timing when the distance between the pen tip and the panel becomes smaller than the speed detection target range d2. The sampling number m corresponds to the speed detection start position.

[0075] Step S108 : Next, the speed detection unit 105 waits for the pen tip 210 to come into contact with the panel surface 1221 .

[0076] During the waiting period in step S108, the pen tip 210 may not be in contact with the panel surface 1221 and may return to a state outside the detectable range d1. In such a case, the sampling rate changed in step S102 may be returned to the default state before the processing of this figure is terminated.

[0077] Step S110: The velocity detection unit 105 acquires a sampling number n indicating the sampling executed at the timing when the pen tip 210 is determined to be in contact with the panel surface 1221. The sampling number n corresponds to the velocity detection end position.

[0078] Step S112 : The speed detection unit 105 calculates the contact speed using, for example, Equation 1, using the sampling number m corresponding to the speed detection start position acquired in step S106 and the sampling number n corresponding to the speed detection end position acquired in step S110 .

[0079] Step S114 : The speed detection unit 105 transmits the contact speed calculated in step S112 to the pen-type input device via the communication unit 101 .

[0080] Next, an example of a processing procedure executed by the pen-shaped input device 200 will be described.

[0081] Step S200 : In the pen-shaped input device 200 , the communication unit 201 receives the contact speed transmitted from the information processing apparatus 100 in step S114 .

[0082] Step S202 : In the pen-type input device 200 , the vibration control unit 241 sets the vibration intensity when vibrating the pen-type input device 200 according to the timing of contact between the pen tip 210 and the panel surface 1221 based on the contact speed received in step S200 .

[0083] Step S204: The vibration control unit 241 drives the vibration unit 202 at the vibration intensity set in step S202 using the vibration waveform data stored in the vibration waveform data storage unit 251. By vibrating the vibration unit 202 in this manner, the pen-shaped input device 200 vibrates at an intensity corresponding to the contact intensity when the pen tip 210 contacts the panel surface 1221.

[0084] For example, after the user brings the pen tip 210 of the pen-shaped input device 200 into contact with the panel surface 1221, they can then perform a writing operation by moving the pen tip 210 while maintaining contact with the panel surface 1221. In this case, although the corresponding processing is not shown in the figure, tactile feedback corresponding to the movement of the pen tip 210 is also provided to the pen-shaped input device 200.

[0085] <Second embodiment>

[0086] Next, the second embodiment will be described. When the user temporarily brings the pen tip 210 of the pen-shaped input device 200 into contact with the panel surface 1221 to perform a writing operation, the pen tip 210 may be slightly separated from the panel surface 1221 and then brought into contact with the panel surface 1221 again. In this case, the pen tip 210 may be temporarily separated from the panel surface 1221 and then brought into contact with the panel surface 1221 within a range smaller than the speed detection target range d2. In such a case, the pen tip 210 does not move outside the detectable range d1 and contacts the panel surface 1221, so Figure 4 In the processing step example, the processing after step S106 is not performed and the contact speed cannot be detected.

[0087] Therefore, in this embodiment, the contact speed can be appropriately detected based on the movement of the pen tip 210 within a range smaller than the speed detection target range d2.

[0088] Reference Figure 5 , an example of a method for detecting the contact speed corresponding to the movement of the pen tip 210 within a range smaller than the speed detection target range d2 will be described.

[0089] For example, Figure 5 The moment t11 is based on Figure 3 At time t5, a certain time has passed since the pen tip 210 was in contact with the panel surface 1221. During the period from time t5 to time t11, the pen tip 210 is maintained in contact with the panel surface 1221. In addition, during the period from time t5 to time t11, the pen tip 210 may be moved on the panel surface 1221 by, for example, a writing operation.

[0090] At time t11, the user begins to separate pen tip 210 from panel surface 1221. From time t11 through time t12 to time t13, pen tip 210 moves in a direction away from panel surface 1221 (separation direction), and the distance from panel surface 1221 (separation distance) gradually increases over time. In velocity detection unit 105, touch panel 122 detects the distance from panel surface 1221 to pen tip 210 based on a detection signal indicating changes in electrostatic capacitance corresponding to the distance from panel surface 1221 to pen tip 210.

[0091] At time t13, the user starts to reverse the direction of movement of pen tip 210 so far, moving pen tip 210 toward the direction (approaching direction) approaching panel surface 1221. After time t13, pen tip 210 moves toward panel surface 1221, and contacts panel surface 1221 at time t15 after time t14.

[0092] In response to this movement of the pen tip 210, the speed detection unit 105 determines that the movement direction of the pen tip 210 reversed from the separating direction to the approaching direction at time t13 within a range smaller than the speed detection target range d2. The speed detection unit 105 acquires sampling number m, corresponding to the speed detection start position, at the time when the movement direction of the pen tip 210 reversed from the separating direction to the approaching direction. Furthermore, at time t15 when the pen tip 210 made contact with the panel surface 1221, the speed detection unit 105 acquires sampling number n, corresponding to the speed detection end position. Using the thus acquired sampling numbers m and n and the distance between the pen tip 210 and the panel surface 1221 at time t13, when the movement direction of the pen tip 210 reversed from the separating direction to the approaching direction, the speed detection unit 105 calculates the contact speed using Equation 1. Thus, the speed detection unit 105 of this embodiment can detect the contact speed of the pen tip 210 when it makes contact with the panel surface 1221 within a range smaller than the speed detection target range d2.

[0093] Reference Figure 6 An example of a processing procedure executed by the information processing device 100 according to the present embodiment in response to the detection of the contact velocity will be described with reference to the flowchart of FIG.

[0094] In this figure, the processing of steps S300 to S314 is the same as Figure 4 Steps S100 to S114 are the same.

[0095] Step S316: Through the processing of steps S300 to S314, Figure 3 As shown, the touch speed caused by the pen tip moving from the outside of the detectable range d1 and contacting the panel surface 1221 is detected and sent. Figure 4 Through the processing of steps S200 to S204 , the pen-shaped input device 200 is given tactile feedback, so that it vibrates at a strength corresponding to the contact speed at the timing of contact with the panel surface 1221 .

[0096] On this basis, in this embodiment, when the process of step S314 is executed, the speed detection unit 105 determines whether the distance between the pen tip and the panel is greater than or equal to the speed detection target range d2 based on the subsequent writing operation of the user.

[0097] Step S318: According to the state that the distance between the pen tip and the panel is less than the speed detection object range d2, as shown in FIG. Figure 5 As illustrated at time t13 , the speed detection unit 105 determines whether the moving direction of the pen tip 210 is reversed from the separating direction to the approaching direction.

[0098] If it is determined that the moving direction of the pen tip 210 has not been reversed from the separating direction to the approaching direction, the process returns to step S316 .

[0099] Step S320: When it is determined in step S318 that the moving direction of the pen tip 210 is reversed from the separation direction to the approach direction, the speed detection unit 105 obtains the sampling number m representing the sampling performed by the touch panel 122 according to the timing of the reversal to the approach direction as the sampling number corresponding to the start position of the speed detection.

[0100] Step S322: After the processing of step S320, the speed detection unit 105 waits for the pen tip 210 to contact the panel surface 1221. In this case, Figure 4 Similarly, when waiting in step S108, in this step S322, if the pen tip 210 does not contact the panel surface 1221 and returns to a state outside the detectable range d1, the processing of this figure can be ended after returning the sampling rate changed in step S302 to the default state.

[0101] If the pen tip 210 contacts the panel surface 1221, the speed detection unit 105 returns the process to step S310. In this case, the speed detection unit 105 executes the process of steps S310 to S314, thereby detecting the contact speed of the pen tip 210 when it contacts the panel surface 1221 based on the state that the distance between the pen tip and the panel is less than the speed detection target range d2, and the detected contact speed is sent to the pen-shaped input device 200. The pen-shaped input device 200 executes steps S200 to S204 ( Figure 4 Thus, the pen-shaped input device 200 vibrates according to the intensity of the contact between the pen tip 210 and the panel surface 1221 at the timing when the pen tip 210 contacts the panel surface 1221.

[0102] Step S324: If it is determined in step S316 that the distance between the pen tip and the panel is greater than or equal to the speed detection target range d2, the speed detection unit 105 determines whether detection of the pen tip 210 has been completed on the touch panel 122. Detection of the pen tip 210 is determined to have been completed if the pen tip 210 is outside the detectable range d1, no electrostatic capacitance is generated on the panel surface 1221, and no detection signal corresponding to the electrostatic capacitance is output.

[0103] If it is determined that the detection of the pen tip 210 has not been completed, the process returns to step S316 .

[0104] If it is determined that the detection of the pen tip 210 has been completed, the process of this figure is terminated. When the process of this figure is terminated, the sampling rate of the touch panel 122 that was changed in step S302 is returned to the default state.

[0105] <Third embodiment>

[0106] Next, the third embodiment will be described. In the information processing system of this embodiment, the pen-shaped input device 200 can output, in addition to vibrations based on tactile feedback, the sound produced when writing on a writing medium using a physical writing utensil (writing sound) in response to a writing operation. Outputting the writing sound corresponding to a writing operation using the pen-shaped input device 200 is also referred to as "writing sound feedback."

[0107] By providing writing sound feedback in this manner, when writing on the touch panel 122 using the pen-shaped input device 200 , the user can more closely resemble the feeling of writing on a writing medium using a real writing implement.

[0108] Figure 7 FIG. 1 shows an example of the functional configuration of an information processing device 100 and a pen-shaped input device 200 corresponding to the tactile and writing sound feedback of this embodiment. Figure 2 The same parts are marked with the same reference numerals and the description thereof is omitted. Figure 2 The differences are explained.

[0109] In this figure, the structure of the information processing device 100 is the same as that of the Figure 2 same.

[0110] The pen-shaped input device 200 in the figure includes a sound output unit 203. The pen-shaped input device 200 in the figure also includes a sound output control unit 242 in the control unit 204 and a writing sound data storage unit 252 in the storage unit 205.

[0111] The sound output unit 203 includes, for example, a speaker, and outputs writing sounds under the control of the sound output control unit 242. Alternatively, the sound output unit 203 may output sounds such as electronic sounds and voices.

[0112] The sound output control unit 242 inputs the audio data as the writing sound data stored in the writing sound data storage unit 252 to the sound output unit 203 , and the writing sound is output from the sound output unit 203 .

[0113] The writing sound data storage unit 252 stores writing sound data. The writing sound data is audio data produced to reproduce the writing sound.

[0114] In this embodiment, the information processing device 100 can execute Figure 4 or Figure 6 The same processing is performed as the processing corresponding to the tactile and writing sound feedback.

[0115] Figure 8 The flowchart shows an example of processing steps executed by the pen-type input device 200 according to this embodiment in response to tactile and writing sound feedback.

[0116] Step S400: In the pen-shaped input device 200, the communication unit 201 receives the Figure 4 Step S114 or Figure 6 In step S314 , the contact velocity is transmitted from the information processing device 100 .

[0117] Step S402 : In the pen-type input device 200 , the vibration control unit 241 sets the vibration intensity when vibrating the pen-type input device 200 according to the timing of contact between the pen tip 210 and the panel surface 1221 based on the contact speed received in step S400 .

[0118] Step S404 : Furthermore, the sound output control unit 242 sets the volume of the writing sound output from the pen-shaped input device 200 (writing volume) according to the timing of contact between the pen tip 210 and the panel surface 1221 based on the contact speed received in step S400 .

[0119] Step S406 : The vibration control unit 241 drives the vibration unit 202 at the vibration intensity set in step S402 using the vibration waveform data stored in the vibration waveform data storage unit 251 .

[0120] Step S408: Furthermore, the sound output control unit 242 uses the writing sound data stored in the writing sound data storage unit 252 to output the writing sound from the sound output unit 203 at the writing volume set in step S404.

[0121] Although corresponding processing is not shown in the figure, tactile and writing sound feedback can be provided to the pen-shaped input device 200 according to the movement of the pen tip 210 in contact with the panel surface 1221.

[0122] Furthermore, the functions of the sound output unit 203 and the sound output control unit 242 are provided in the information processing apparatus 100 , so that writing sound feedback can be performed in the information processing apparatus.

[0123] As a variation of this embodiment, the application-compatible processing unit 131 of the information processing device 100 can perform drawing in accordance with the writing operation based on the contact speed detected by the speed detection unit 105. Specifically, for example, when using a writing implement such as a pen as a writing implement to perform drawing in accordance with the writing operation, if the contact speed is greater than a predetermined value, the pen contacts the writing medium with strong momentum, thereby enabling drawing such that ink (and thus paint, ink, etc.) splashes on the writing medium.

[0124] Furthermore, in each of the above-described embodiments, the tactile feedback of the timing of the contact between the pen tip 210 and the panel surface 1221 may be implemented by changing the vibration intensity according to the contact speed, and also by changing the vibration time, vibration waveform, and the like.

[0125] <Fourth embodiment>

[0126] Next, a fourth embodiment will be described. In this embodiment, the information processing apparatus 100 is configured to output writing sound as writing sound feedback in addition to the output of writing sound by the pen-shaped input device 200 .

[0127] In this way, the information processing device 100 can output writing sounds, thereby simultaneously outputting writing sounds from the information processing device 100 and from the pen-shaped input device 200. This increases the writing volume and easily provides writing sound feedback at a sufficient writing volume. Furthermore, the writing volume can be easily changed during the output of writing sounds from the information processing device 100. Therefore, for example, the writing volume can be adjusted to approximate the sensation of writing on a writing medium using a physical writing utensil, or to suit the user's preferences.

[0128] In addition, the information processing device 100 is capable of outputting writing sounds, so that even when the user uses a pen-shaped input device that does not have the writing sound feedback function, tactile feedback based on the vibration of the pen-shaped input device and tactile and writing sound feedback based on the writing sound feedback of the information processing device 100 can be achieved.

[0129] The following description of this embodiment illustrates a case where a pen-shaped input device also has a writing sound feedback function. Furthermore, the following description of this embodiment illustrates a configuration that omits the functions of changing the vibration intensity and writing volume based on the contact velocity in the previous embodiments.

[0130] Figure 9 FIG. 1 shows an example of a functional structure of an information processing device 100 and a pen-shaped input device 200 corresponding to tactile and writing sound feedback according to an embodiment. Figure 7 The same parts are marked with the same reference numerals and the description thereof is omitted. Figure 7 The differences are explained.

[0131] The information processing device 100 shown in the figure includes a sound output unit 106. The sound output unit 106 includes, for example, a voice circuit and a speaker, and outputs writing sounds under the control of a feedback control unit 132 (an example of a sound output control unit). Alternatively, the sound output unit 106 may output sounds such as electronic sounds and speech.

[0132] Furthermore, the control unit 103 of the information processing device 100 of this embodiment includes a feedback control unit 132 (an example of a sound output control unit). The feedback control unit 132 performs control corresponding to tactile and writing sound feedback. Specifically, the feedback control unit 132 controls the start and end of tactile and writing sound feedback in the pen-shaped input device 200. Furthermore, the feedback control unit 132 inputs the audio data of writing sound data stored in the writing sound data storage unit 141 into the sound output unit 106, thereby outputting the writing sound from the sound output unit 106 and controlling the writing sound feedback in the information processing device 100.

[0133] Figure 10 This diagram shows queue 161 included in the internal circuitry of audio output unit 106. Queue 161 inputs audio data AD and outputs it, for example, using a FIFO (First-In-First-Out) system. In audio output unit 106, the audio data output from queue 161 is converted into an analog audio signal, for example, through D / A conversion, amplified by an amplifier circuit, and output as audio from a speaker.

[0134] Return instructions to Figure 9 Furthermore, the storage unit 104 of the information processing device 100 of this embodiment includes a writing sound data storage unit 141. The writing sound data storage unit 141 stores writing sound data.

[0135] The structure of the pen-shaped input device 200 in this embodiment is similar to Figure 7 same.

[0136] In addition, in the information processing device 100 of this embodiment, as shown in the figure, the Figure 7 The speed detection unit 105 is provided.

[0137] To save power, the audio output unit 106 included in the information processing device 100 of this embodiment enters a sleep mode when not performing audio output processing. Specifically, the audio output unit 106 is set to a sleep state when no audio data is input, and is activated from the sleep state to an active state upon input of audio data, thereby commencing audio output processing.

[0138] Figure 11 An example of state transition of the audio output unit 106 according to the input of audio data is shown.

[0139] When audio data is input while the audio output unit 106 is in a sleep state, a warm-up wait time (warm-up period Tw) occurs. Next, after warm-up period Tw has passed, a wait time (system delay period Ts) corresponding to signal processing in the audio output unit 106 occurs. After system delay period Ts has passed, audio output begins. That is, when audio data is input while the audio output unit 106 is in a sleep state, a time delay (delay) based on warm-up period Tw and system delay period Ts occurs before the audio corresponding to the input audio data is output.

[0140] On the other hand, if sound data is input while the sound output unit 106 is in an operating state after being activated from a sleep state, the warm-up period Tw does not occur. In this case, sound output begins after the system delay period Ts has passed. In other words, if sound data is input while the sound output unit 106 is in an operating state, a time delay (delay) based on the system delay period Ts occurs before the sound corresponding to the input sound data is output.

[0141] Thus, the time delay from the timing when audio data is input to the timing when audio corresponding to the input audio data is output is longer in the sleep state than in the operation state.

[0142] For example, in the information processing device 100, when outputting writing sounds, sound data may be input to the sound output unit 106 at the time the pen tip of the pen-shaped input device 200 contacts the panel surface 1221 of the touch-panel display unit 102. However, in such a configuration, the sound output unit 106 is activated from a sleep state when the sound data is input. In this case, the sound corresponding to the sound data input to the sound output unit 106 is output after a relatively long delay based on the warm-up period Tw and the system delay period Ts. In this case, the user may hear the writing sound at a somewhat delayed time after contacting the pen tip of the pen-shaped input device 200 with the panel surface 1221 of the touch-panel display unit 102. While the output of the writing sound allows the user to experience the same sensation as writing on a real writing medium, this sensation is impaired by the aforementioned delay in the writing sound.

[0143] Therefore, the feedback control unit 132 of the information processing device 100 according to this embodiment performs control corresponding to the start of output of writing sound (sound output start control) as follows, thereby shortening the delay of the writing sound output from the sound output unit 106 .

[0144] Refer again Figure 3The voice start control in this embodiment will be described below. When a user wishes to start writing with the pen-shaped input device 200, they move the pen tip of the pen-shaped input device 200 closer to the panel surface 1221 from a position outside the detectable range d1 (an example of the detection limit distance). As the pen tip of the pen-shaped input device 200 approaches the panel surface 1221, if the pen tip reaches the detectable range d1, the touch panel 122 begins detecting the pen tip.

[0145] Upon starting detection of the pen tip, feedback control unit 132 of information processing device 100 starts inputting silent data into queue 161 of audio output unit 106, thereby starting audio output control. Silence data is, for example, audio data corresponding to silence caused by a zero volume level.

[0146] By inputting the silent data, the sound output unit 106 is activated from the previous sleep state and starts Figure 11 The operation starts during the warm-up period Tw shown.

[0147] When a certain time has passed since the pen tip detection was started as described above, the pen tip of the pen-shaped input device comes into contact with the panel surface 1221. Thus, when the touch panel 122 detects that the pen tip is in contact with the panel surface 1221, the feedback control unit 132 switches the sound data input to the queue 161 from the previously silent data to the writing sound data stored in the writing sound data storage unit 141 (an example of sound data).

[0148] By switching from silent data to writing sound data in this way, the sound output unit 106 can be activated at a timing prior to the contact between the pen tip of the pen-shaped input device 200 and the panel surface 1221. Therefore, at the time the pen tip contacts the panel surface 1221, the sound output unit 106 may sometimes become capable of sound output after a delay based on the warm-up period Tw and the system delay period Ts. In this case, the sound output unit 106 outputs the writing sound data input at the time the pen tip contacts the panel surface 1221 as writing sound without any delay. As a result, the user can hear the writing sound output from the information processing device 100 at the time the pen tip contacts the panel surface 1221, with virtually no perceptible delay.

[0149] In addition, at the time when the pen tip contacts the panel surface 1221, it is possible that the sound output unit 106 has not yet passed the delay time based on the preheating period Tw and the system delay period Ts. However, at the time when the pen tip contacts the panel surface 1221, a certain degree of delay time (preheating period Tw and system delay period Ts) has already passed, so it is possible to output the writing sound after a short delay time. In this case, the writing sound is output after a certain degree of delay time from the time when the pen tip contacts the panel surface 1221, but the delay time is short, so the user's auditory sense hardly feels the delay, and even if a delay is felt, it is not easy to feel a sense of disharmony.

[0150] Thus, in this embodiment, the delay of the writing sound output from the information processing apparatus 100 is eliminated or alleviated. As a result, when the user writes with the pen-shaped input device 200, the user can maintain the same feeling as when writing on a real writing medium.

[0151] Reference Figure 12 The flowchart of FIG. 1 will be used to describe an example of processing steps executed by the information processing apparatus 100 and the pen-type input device 200 according to this embodiment in response to tactile and writing sound feedback.

[0152] First, an example of a processing procedure executed by the information processing device 100 will be described.

[0153] Step S500: In the information processing device 100, the feedback control unit 132 waits for the state (an example of the first state) in which the detection of the pen tip 210 by the touch panel 122 is started. Figure 4 Similarly, the process of step S100 can be performed according to the pen tip 210 reaching the detectable range d1 ( Figure 3 At time t1 ), electrostatic capacitance is generated in touch panel 122 , and the start of detection of pen tip 210 is determined by the start of output of a detection signal corresponding to the generated electrostatic capacitance.

[0154] Step S502: If it is determined in step S500 that the detection of the pen tip 210 by the touch panel 122 has started, the feedback control unit 132 starts inputting silent data to the queue 161 of the sleep-state sound output unit 106. The sound output unit 106 is activated by the start of the silent data input.

[0155] Step S504 : After starting input of silent data to the queue 161 in step S502 , the feedback control unit 132 waits for the touch panel 122 to detect that the pen tip is in contact with the panel surface 1221 (an example of the second state).

[0156] Furthermore, if the pen tip is not in contact with the panel surface 1221 in step S504 and is not detected due to movement outside the detectable range d1, the feedback control unit 132 may stop the input of silent data up to that point and then return the process to step S500. In this case, the sound output unit 106 may enter a sleep state in response to the cessation of the input of silent data.

[0157] Step S506: If it is detected in step S504 that the pen tip is in contact with the panel surface 1221, the feedback control unit 132 starts inputting writing sound data into the queue 161 stored in the writing sound data storage unit 141, replacing the silent data thus far.

[0158] Step S508: In addition, based on the detection of the pen tip contacting the panel surface 1221 in step S504, the feedback control unit 132 sends a feedback start command to the pen-shaped input device 200. The feedback start command instructs the pen-shaped input device 200 to start tactile and writing sound feedback.

[0159] Step S510 : After the processing of steps S506 and S508 , the feedback control unit 132 waits for the touch panel 122 to detect that the pen tip is separated from the panel surface 1221 (an example of the third state).

[0160] Step S512: Upon detecting the pen tip being separated from the panel surface 1221 in step S510, the feedback control unit 132 stops inputting the writing sound data started in step S506 to the queue 161. Upon stopping the input of the writing sound data, the sound output unit 106 enters the sleep state.

[0161] Step S514: Based on the detection of the pen tip being separated from the panel surface 1221 in step S510, the feedback control unit 132 sends a feedback stop command to the pen-shaped input device 200. The feedback stop command instructs the pen-shaped input device 200 to stop the tactile and writing sound feedback.

[0162] Next, an example of a processing procedure executed by the pen-shaped input device 200 will be described.

[0163] Step S600 : In the pen-shaped input device 200 , the control unit 204 receives the feedback start command transmitted from the information processing apparatus 100 in step S508 .

[0164] Step S602 : In response to receiving the feedback start command in step S600 , the vibration control unit 241 starts vibrating the vibration unit 202 at, for example, a predetermined vibration intensity.

[0165] Step S604: In addition, in response to receiving the feedback start command in step S600, the sound output control unit 242 inputs the writing sound data stored in the writing sound data storage unit 252 to the sound output unit 203, thereby starting the output of the writing sound.

[0166] As described above, by executing the processing of steps S602 and S604 , the pen-shaped input device 200 starts the tactile and writing sound feedback operations at the same timing as when the pen tip of the pen-shaped input device 200 is considered to be in contact with the panel surface 1221 .

[0167] Step S606 : At a certain timing after the tactile and writing sound feedback operations are terminated in steps S602 and S604 , the control unit 204 receives the feedback stop command transmitted from the information processing device 100 in step S514 .

[0168] Step S608 : upon receiving the feedback stop command, the vibration control unit 241 stops driving the vibration unit 202 .

[0169] Step S610 : In addition, upon receiving the feedback stop command, the sound output control unit 242 stops outputting the writing sound from the sound output control unit 242 .

[0170] In addition, according to the above Figure 12 If it is detected through step S510 that the pen tip of the pen-shaped input device 200 is separated from the panel surface 1221, the writing sound feedback of the information processing device 100 and the tactile and writing sound feedback of the pen-shaped input device 200 are immediately stopped through the processing of steps S512 and S514.

[0171] However, depending on the content of the writing, the pen tip may be briefly separated from the panel surface 1221, and then immediately returned to a state where the pen tip is in contact with the panel surface 1221. Under such circumstances, if the information processing device 100's writing sound feedback and the pen-shaped input device 200's tactile and writing sound feedback were stopped each time the pen tip separated from the panel surface 1221, the user might experience a sense of disharmony, which would impair the feeling of writing on a real writing medium. In other words, when the pen tip of the pen-shaped input device 200 is separated from the panel surface 1221 for a short period of time, it is sometimes preferable to continue the information processing device 100's writing sound feedback and the pen-shaped input device 200's tactile and writing sound feedback.

[0172] Therefore, as a modification of the present embodiment, it may be determined in step S510 whether the pen tip of the pen-shaped input device 200 has been separated from the panel surface 1221 for a predetermined time or longer.

[0173] By adopting such a process, even when the user writes with the pen tip of the pen-shaped input device 200 immediately contacting the panel surface 1221 even after separating from the panel surface 1221, the information processing device 100 provides continuous feedback on the writing sound and the pen-shaped input device 200 provides continuous feedback on the writing sound. This reduces the user's discomfort and the feeling of writing on a real writing medium.

[0174] Alternatively, a program for implementing the functions of the aforementioned information processing device 100, pen-shaped input device 200, etc. may be recorded on a computer-readable recording medium, and the processing of the aforementioned information processing device 100, pen-shaped input device 200, etc. may be performed by having a computer system read and execute the program recorded on the recording medium. Here, "having a computer system read and execute the program recorded on the recording medium" includes installing the program on the computer system. The "computer system" herein includes hardware such as the operating system and peripheral devices. Furthermore, the "computer system" may include multiple computer devices connected via a network, including communication lines such as the Internet, a wide area network (WAN), a local area network (LAN), or dedicated lines. Furthermore, the term "computer-readable recording medium" refers to portable media such as floppy disks, magnetic optical disks, ROMs, and CD-ROMs, as well as storage devices such as HDDs and solid-state drives (SSDs) built into the computer system. Thus, the recording medium storing the program may be a non-transitory recording medium such as a CD-ROM. Furthermore, the recording medium also includes internal or external recording media accessible from a distribution server for distributing the program. The code of the program stored in the recording medium of the distribution server may also be a code different from the code of the program in a form that can be executed in the terminal device. That is, as long as it is a code that can be downloaded from the distribution server and installed in a form that can be executed in the terminal device, the form stored in the distribution server does not matter. In addition, the program may be divided into multiple parts, and the structure formed by combining the terminal devices after being downloaded at different times, and the distribution server for each of the divided programs may be different. Moreover, the so-called "computer-readable recording medium" also includes a structure that keeps the program for a certain period of time, such as a server in the case of sending a program via a network, or a volatile memory (RAM) inside a computer system that becomes a client. In addition, the above-mentioned program may also be a structure for realizing a part of the above-mentioned functions. Furthermore, it may also be a so-called differential file (differential program) that can realize the above-mentioned functions by combining them with a program already recorded in the computer system.

Claims

1. An information processing system comprising: a sensor portion for detecting contact of the pen-shaped input device with the operation target surface and detecting the presence of the pen-shaped input device within a detection limit distance on the operation target surface; a sound output unit that outputs sound corresponding to the input sound data; as well as The sound output control unit inputs silent sound data to the sound output unit based on the first state in which the pen-shaped input device is detected within the detection limit distance by the sensor unit, and inputs audible sound data to the sound output unit after the first state is reached based on the second state in which the pen-shaped input device is detected to be in contact with the operation object surface by the sensor unit.

2. The information processing system according to claim 1, wherein The sound output control unit stops input of sound data to the sound output unit when the sensor unit detects that the second state has shifted to a third state in which the pen-type input device is separated from the operation target surface.

3. The information processing system according to claim 2, wherein: The audio output control unit stops input of audio data to the audio output unit in response to the third state continuing for a predetermined time from the timing at which the second state changes to the third state.

Images