Information processing device, information processing system, and computer program

Abstract

An information processing device 1 comprises a calculation unit 8 for evaluating an input operation from a user. The calculation unit acquires trajectory data M1 corresponding to a virtual target, calculates a first coordinate G1 corresponding to the trajectory data, uses the input operation from the user as a basis to calculate a second coordinate G2 corresponding to an input trajectory K1 input to an input unit 2 so as to follow the trajectory data for each instance of a predetermined timing, calculates a degree of coincidence between the first and second coordinates, uses audio as a basis to generate first feedback information F1 in which an output state is adjusted in accordance with the degree of coincidence, causes an output unit 3 to output the first feedback information, and, when the degree of coincidence satisfies a predetermined condition, generates second feedback information indicating an evaluation of the input operation from the user and causes the output unit to output the second feedback information.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing system, and a computer program for training a user's physical function based on their hearing, for example, for brain function training through hearing. [Background technology]

[0002] The technology shown in Figures 1 to 18 did not exist. Furthermore, Patent Document 1 proposes an information processing device that acquires information about the user's operation content and provides feedback information to the user. According to the technology described in Patent Document 1, a predetermined action of the user is detected based on information about the user's operation content, and the timing for providing feedback information from the time of detection is calculated according to the content of the predetermined action, and the feedback information can be output at the calculated timing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2024-109398 [Non-patent literature]

[0004] [Non-Patent Document 1] “Audiomotor Perceptual Training Enhances Speech Intelligibility in Background Noise”, Jonathon P. Whitton, Kenneth E. Hancock, Jeffrey M. Shannon, Daniel B. Polley, Current Biology, 11.6th.2017 Summary of the Invention [Problem to be solved by the invention]

[0005] In order to train a user's physical function based on hearing, it is necessary to have the user perform an operation based on hearing. The technology described in Patent Document 1 provides feedback information when it is determined that a predetermined movement has been detected, but is not intended to be used for training a user's physical function based on hearing.

[0006] The present invention aims to provide an information processing device, an information processing system, and a computer program that are capable of, for example, performing brain function training through hearing in order to train a user's physical functions based on their hearing. [Means for solving the problem]

[0007] 1 to 18. In order to achieve the above object, an information processing system according to one embodiment of the present invention has the functions described in Figures 1 to 18. Another embodiment of the present invention is an information processing device that includes a calculation unit that evaluates an input operation of a user, wherein the calculation unit acquires trajectory data corresponding to a virtual target, calculates first coordinates corresponding to the trajectory data, calculates second coordinates corresponding to an input trajectory input to an input unit so as to follow the trajectory data at predetermined timings based on the input operation of the user, calculates a degree of coincidence between the first coordinates and the second coordinates, generates first feedback information based on voice, the output state of which has been adjusted in accordance with the degree of coincidence, and causes an output unit to output the first feedback information, and, if the degree of coincidence satisfies a predetermined condition, generates second feedback information indicating an evaluation of the input operation to the user, and causes the output unit to output the second feedback information. [Effects of the Invention]

[0008] According to the present invention, it is possible to perform physical function training based on the user's hearing, for example, brain function training through hearing. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an overview (1) of an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an overview (2) of an information processing system according to one embodiment of the present invention. [Figure 3] FIG. 3 is a diagram showing an overview (3) of an information processing system according to one embodiment of the present invention. [Figure 4] FIG. 4 is a diagram showing an overview (4) of an information processing system according to one embodiment of the present invention. [Figure 5] FIG. 5 is a diagram showing an overview (5) of an information processing system according to one embodiment of the present invention. [Figure 6] FIG. 6 is a diagram showing an overview (6) of an information processing system according to one embodiment of the present invention. [Figure 7] FIG. 7 is a diagram showing an overview (7) of an information processing system according to one embodiment of the present invention. [Figure 8] FIG. 8 is a diagram showing an overview (8) of an information processing system according to one embodiment of the present invention. [Figure 9] FIG. 9 is a diagram showing an overview (9) of an information processing system according to one embodiment of the present invention. [Figure 10] 1 is a diagram showing an overview (10) of an information processing system according to an embodiment of the present invention. [Figure 11] FIG. 1 is a diagram showing an overview (11) of an information processing system according to an embodiment of the present invention. [Figure 12] FIG. 1 is a diagram showing an overview (12) of an information processing system according to one embodiment of the present invention. [Figure 13] FIG. 1 is a diagram showing an overview (13) of an information processing system according to one embodiment of the present invention. [Figure 14] FIG. 1 is a diagram showing an overview (14) of an information processing system according to an embodiment of the present invention. [Figure 15] FIG. 1 is a diagram showing an overview (15) of an information processing system according to an embodiment of the present invention. [Figure 16] FIG. 1 is a diagram showing an overview (16) of an information processing system according to an embodiment of the present invention. [Figure 17] FIG. 1 is a diagram showing an overview (17) of an information processing system according to one embodiment of the present invention. [Figure 18]FIG. 1 is a diagram showing an overview (18) of an information processing system according to one embodiment of the present invention. [Figure 19] 1 is a block diagram showing a configuration of an information processing system according to an embodiment of the present invention. [Figure 20] FIG. 10 is a diagram showing an example of trajectory data displayed on a display unit. [Figure 21] 10A and 10B are diagrams illustrating a method for calculating the degree of coincidence between first coordinates and second coordinates. [Figure 22] FIG. 10 is a diagram showing a state in which feedback information is output. [Figure 23] 10 is a flowchart showing the flow of processing executed in the information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0010] An overview of an information processing system according to an embodiment of the present invention will be described below. FIG. 1 is a diagram showing an overview (1) of an information processing system according to one embodiment of the present invention. An overview (1) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0011] FIG. 2 is a diagram showing an overview (2) of an information processing system according to one embodiment of the present invention. An overview (2) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0012] FIG. 3 is a diagram showing an overview (3) of an information processing system according to one embodiment of the present invention. An overview (3) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0013] FIG. 4 is a diagram showing an overview (4) of an information processing system according to one embodiment of the present invention. An overview (4) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0014] FIG. 5 is a diagram showing an overview (5) of an information processing system according to one embodiment of the present invention. An overview (5) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0015] FIG. 6 is a diagram showing an overview (6) of an information processing system according to one embodiment of the present invention. An overview (6) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0016] FIG. 7 is a diagram showing an overview (7) of an information processing system according to one embodiment of the present invention. An overview (7) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0017] FIG. 8 is a diagram showing an overview (8) of an information processing system according to one embodiment of the present invention. An overview (8) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0018] FIG. 9 is a diagram showing an overview (9) of an information processing system according to one embodiment of the present invention. An overview (9) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0019] FIG. 10 is a diagram showing an overview (10) of an information processing system according to one embodiment of the present invention. An overview of an information processing system (10) according to one embodiment of the present invention is as shown in FIG.

[0020] FIG. 11 is a diagram showing an overview (11) of an information processing system according to one embodiment of the present invention. An overview (11) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0021] FIG. 12 is a diagram showing an overview (12) of an information processing system according to one embodiment of the present invention. An overview (12) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0022] FIG. 13 is a diagram showing an overview (13) of an information processing system according to one embodiment of the present invention. An overview (13) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0023] FIG. 14 is a diagram showing an overview (14) of an information processing system according to one embodiment of the present invention. An overview (14) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0024] FIG. 15 is a diagram showing an overview (15) of an information processing system according to one embodiment of the present invention. An overview (15) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0025] FIG. 16 is a diagram showing an overview (16) of an information processing system according to one embodiment of the present invention. An overview (16) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0026] FIG. 17 is a diagram showing an overview (17) of an information processing system according to one embodiment of the present invention. An overview (17) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0027] FIG. 18 is a diagram showing an overview (18) of an information processing system according to one embodiment of the present invention. An overview (18) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0028] Although one embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and modifications, improvements, etc. within the scope of achieving the object of the present invention are included in the present invention.

[0029] Furthermore, for example, the above-described series of processes can be executed by hardware or software. In other words, the functional configurations of FIGS. 1 to 18 are merely exemplary and are not particularly limited. That is, it is sufficient for the information processing system to have the functionality to execute the above-described series of processes as a whole, and the functional blocks used to realize these functions are not particularly limited to the examples of FIGS. 1 to 18. Furthermore, the locations of the functional blocks and databases are not particularly limited to those of FIGS. 1 to 18 and may be arbitrary. For example, at least some of the functional blocks and databases required to execute various processes may be transferred to a user terminal or the like. Conversely, the functional blocks and databases of the user terminal may be transferred to a server or the like. Furthermore, one functional block may be configured as hardware alone, software alone, or a combination thereof.

[0030] When a series of processes is executed by software, the programs constituting the software are installed onto a computer or the like from a network or a recording medium. The computer may be a computer incorporated into dedicated hardware. The computer may also be a computer capable of executing various functions by installing various programs, such as a server, a general-purpose smartphone, or a personal computer.

[0031] The recording medium containing such a program may be configured as a removable medium (not shown) that is distributed separately from the device main body in order to provide the program to users, etc., or may be configured as a recording medium that is pre-installed in the device main body and provided to users, etc. The computer program for executing the processing of each unit of the processor that constitutes the calculation unit 8 of the information processing system S may be provided in a form recorded on a computer-readable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. The computer program may also be provided as a program product. The specific configuration of the information processing system will be described below.

[0032] 19, the information processing system S includes an information processing device 1 operated by a user and a server device 20 communicably connected via a network W. The information processing system S is configured to enable training of the user's physical functions, including hearing, based on the user's input operations. In the information processing system S, at least one of the information processing device 1 and the server device 20 is configured to evaluate the user's input operations.

[0033] The server device 20 is configured by an information communication device, such as a personal computer, on which a computer program can be installed. The server device 20 includes a calculation unit 21 that executes processes necessary for calculations and various controls. The calculation unit 21 is configured by at least one hardware processor, such as a CPU (Central Processing Unit). The calculation unit 21 may be realized by hardware (including circuitry), such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware.

[0034] The server device 20 includes a storage unit 22 that stores data and computer programs necessary for calculations and controls, and a database including training questions and answers. The storage unit 22 is configured with a non-transitory storage medium such as a hard disk drive (HDD) or a flash memory. The server device 20 includes a communication unit 23 that can be connected to a network W for communication. The communication unit 23 is configured with, for example, a communication device that can be connected to the network W via a wired or wireless connection. The calculation unit 21 communicates with the information processing device 1 via the network W and transmits training data.

[0035] The calculation unit 21 may perform calculation processing based on training data and cause the information processing device 1 to output the calculation results via the network W. The calculation unit 21 may perform calculation processing in cooperation with the information processing device 1 via the network W. The calculation unit 21 may transmit an updated computer program to the information processing device 1.

[0036] The information processing device 1 is configured by an information communication device, such as a smartphone or a personal computer, on which a computer program can be installed. The information processing device 1 may be a device configured specifically for training the physical functions of a user. The information processing device 1 includes a calculation unit 8 that executes processes necessary for calculations and various controls. The calculation unit 8 is configured by at least one hardware processor such as a CPU. The calculation unit 8 may be realized by hardware such as an LSI, ASIC, FPGA, or GPU, or may be realized by a combination of software and hardware.

[0037] The information processing device 1 includes a storage unit 9 that stores data and computer programs necessary for calculations and controls, as well as a database including training questions and answers (described below). The storage unit 9 is configured with a non-transitory storage medium such as a hard disk drive or flash memory. The information processing device 1 includes a communication unit 6 that can be connected to a network W. The communication unit 6 is configured with, for example, a communication device that can be connected to the network W via a wired or wireless connection. The calculation unit 8 communicates with a server device 20 via the network W to receive training data.

[0038] The information processing device 1 includes an input unit 2 that accepts user operations. The input unit 2 is configured by a device capable of inputting user input operations, such as a touch panel, touchpad, mouse, or keyboard. The input unit 2 is provided in the information processing device 1. The input unit 2 may be configured by a device that is communicatively connected to the information processing device 1 via a wireless or wired connection. The input unit 2 may be configured by a controller having a cross key or joystick, or a controller based on an acceleration sensor that detects the user's body movements. Any acceleration sensor can be used as long as it can detect the user's body movements. The input unit 2 may be configured by a camera, or the user's movements may be input by image recognition of image data by the calculation unit 8. The input unit 2 may be configured to input voice based on the user's speech, recognize the content of the speech, and perform input operations. The input unit 2 may be configured to input specific voice information, recognize the content of the voice information generated based on a combination of one or more parameters such as frequency, acoustic power, acoustic spectrum, and emission pattern, and perform input operations.

[0039] The input unit 2 may be configured with a motion capture device that recognizes markers attached to the user's body and detects the user's body movements. The input unit 2 may be configured to detect the user's body based on a pressure sensor or a vibration sensor. The input unit 2 may be configured to accept input operations based on a device that acquires signals non-invasively and / or invasively from the user's body, such as an electromyography (EMG), an electroencephalogram (EEG), or an electrooculography (EOG). The input unit 2 may be configured with any device that can accept two-dimensional or three-dimensional input operations by the user.

[0040] The information processing device 1 includes an output unit 3 that outputs information. The output unit 3 is provided in the information processing device 1. The output unit 3 may be configured by a device communicatively connected to the information processing device 1. The output unit 3 includes, for example, a display unit 4 that outputs information based on an image. The display unit 4 is configured by a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The display unit 4 may be configured by a touch panel that can accept input operations. In this case, the display unit 4 also functions as the input unit 2. The display unit 4 is communicatively connected to the information processing device 1, and may be configured by a projector that can project images, or a head-mounted display device that is worn on the user's head.

[0041] The output unit 3 includes a speaker 5 that outputs sound. The speaker 5 outputs information based on the sound. The output unit 3 may be configured as a device that applies stimuli to the user's body non-invasively and / or invasively. The output unit 3 may be configured as a device that applies physical and / or neural stimuli based on one or more combinations of visual stimuli, auditory stimuli, vibration stimuli, olfactory stimuli, taste stimuli, temperature stimuli, electrical stimuli, magnetic stimuli, gravitational stimuli, acceleration stimuli, pressure stimuli, brain wave stimuli, balance sense stimuli, muscle stimuli, visceral stimuli, light stimuli, sound stimuli, visual cortex stimuli, auditory cortex stimuli, intestinal bacteria administration, drug administration, etc.

[0042] The following describes a process for training a user, which is executed in the information processing device 1. In this embodiment, a process in which the calculation unit 8 evaluates an input operation performed by a user on a touch panel is illustrated. The calculation unit 8 starts training, for example, in a state in which the trajectory of a virtual target is not displayed on the display unit 4. In the training, the user performs an input operation in which the user traces the trajectory of the virtual target with a finger on the display unit 4 to estimate the trajectory of the virtual target. The calculation unit 8 compares the input trajectory input based on the user's input operation with the trajectory of the virtual target, scores the accuracy of the input operation, and provides feedback to the user. The processing of the calculation unit 8 may be executed by the server device 20.

[0043] As shown in FIG. 20 , the display unit 4 is configured with a touch panel that accepts input operations for training by the user. The calculation unit 8 launches a training application based on the user's input operations. The calculation unit 8 acquires trajectory data M1 corresponding to the virtual target from the storage unit 9. The calculation unit 8 may acquire the trajectory data M1 from the server device 20. The virtual target is, for example, a character, a figure, or the like, represented by a plurality of linear trajectories. The calculation unit 8 suggests the virtual target to the user by, for example, outputting a quiz-style question to the output unit 3. The trajectory data M1 is data indicating the spatial position of the virtual target. The trajectory data M1 may be fixed over time or may be dynamic over time. The trajectory data M1 may change over time. The calculation unit 8 executes a calculation process to match the trajectory data M1 of the virtual target with the coordinates of the display unit 4. The calculation unit 8 calculates first coordinates corresponding to the trajectory data. The calculation unit 8 converts the first coordinates into data corresponding to a position on the screen of the display unit 4 and makes them correspond to the display unit 4. The calculation unit 8 calculates second coordinates in the virtual space corresponding to the input trajectory displayed on the display unit 4, based on the input operation input to the input unit 2. In the illustrated example, the first coordinates and the second coordinates are, for example, two-dimensional coordinates. The first coordinates and the second coordinates may be coordinates of two or more dimensions.

[0044] The calculation unit 8 calculates a plurality of first coordinates that match the trajectory data M1. The calculation unit 8 does not display the trajectory data M1 of the virtual target based on the first coordinates on the display unit 4, and outputs instruction information to the output unit 3 that instructs the user to perform an input operation to trace the trajectory data M1 of the virtual target. The calculation unit 8 may not only not display the trajectory data M1 on the display unit 4, but also display at least a portion of the trajectory data M1 on the display unit 4. The calculation unit 8 may display at least a portion of the trajectory data M1 on the display unit 4 when a predetermined condition is met, such as at a predetermined timing from the start of the input operation, within a predetermined time from the start of the input operation, or when a predetermined input operation is performed. The calculation unit 8 may switch between a state in which at least a portion of the trajectory data M1 is displayed on the display unit 4 and a state in which it is not displayed. The instruction information may be output in any form, such as text information, graphic information, audio information, or electrical stimulation, as long as it instructs the user to perform an input operation. The calculation unit 8 may generate information MA indicating the start position of the trajectory data M1 and output it from the output unit 3. The start position may be the beginning of the trajectory data M1 or any position other than the beginning of the trajectory data M1. The calculation unit 8 may not output the information MA indicating the start position of the trajectory data M1 from the output unit 3. The calculation unit 8 may display the information MA indicating the start position of the trajectory data M1 and then cause it to disappear from the output unit 3 after a predetermined time has elapsed. A user starts an input operation by tracing the trajectory data M1 on the display unit 4 using an input means such as a finger or a stylus. The user may start the input operation from any position on the trajectory data M1. The user continuously and / or discretely traces the display unit 4 to generate an input trajectory. The calculation unit 8 displays an input trajectory K1 input based on the user's input operation on the display unit 4. The calculation unit 8 may not only display the input trajectory K1 on the display unit 4, but may also not display the input trajectory K1 on the display unit 4. The calculation unit 8 may display at least a part of the input trajectory K1 on the display unit 4. The calculation unit 8 may switch between a state in which at least a part of the input trajectory K1 is displayed on the display unit 4 and a state in which it is not displayed. The input operation may be performed not only by tracing the display unit 4, but also by other methods based on the configuration of the input unit 2.

[0045] As shown in FIG. 21 , the calculation unit 8 generates a second coordinate G2 of the input trajectory K1 input to the input unit 2 at a predetermined timing so as to follow the trajectory data M1 based on the user's input operation. The predetermined timing may be, for example, a sampling period for accepting the input operation, or may be continuous. The second coordinate G2 indicates a spatial position in the virtual space relative to the first coordinate G1. The calculation unit 8 may generate the second coordinate G2 continuously or discretely. The calculation unit 8 may calculate an input velocity V of the input operation. For example, the calculation unit 8 calculates the input velocity V based on the path length of the input trajectory K1 input per unit time. The calculation unit 8 calculates the input velocity V using the average, median, maximum, minimum, etc. of the input velocity over a predetermined period. The calculation unit 8 generates image data of the input trajectory K1 based on the second coordinate G2 and displays the image data on the display unit 4. The calculation unit 8 generates image data of the input trajectory K1 based on, for example, a line diagram.

[0046] The calculation unit 8 calculates a degree of matching indicating the spatial relationship between the first coordinate G1 and the second coordinate G2 at each predetermined timing while the input operation is being performed. The degree of matching is a score indicating the degree to which the first coordinate G1 and the second coordinate G2 match. The calculation unit 8 calculates the degree of matching based on, for example, the relative distance D between the first coordinate G1 and the second coordinate G2. The calculation unit 8 calculates the degree of matching so that the smaller the distance D, the higher the degree of matching. The calculation unit 8 may calculate the degree of matching based on a combination of one or more of the relative distance, relative angle, relative velocity, relative acceleration, etc., in the relationship between the first coordinate G1 and the second coordinate G2. The calculation unit 8 may also calculate the degree of matching using other methods as long as they can calculate the relationship between the first coordinate G1 and the second coordinate G2. The calculation unit 8 may calculate the degree of matching at a certain moment, or may calculate the degree of matching at each predetermined time interval based on the average, median, maximum, or minimum value.

[0047] The calculation unit 8 generates feedback information indicating an evaluation of the input operation to the user according to the degree of coincidence, and causes the output unit 3 to output the feedback information. The calculation unit 8 causes the output unit 3 to output the feedback information based on a combination of at least one of, for example, audio information, text information, and image information. The calculation unit 8 may output the feedback information from the output unit 3 based on physical and / or neurological stimulation based on a combination of one or more of visual stimulation, auditory stimulation, vibration stimulation, olfactory stimulation, taste stimulation, temperature stimulation, electrical stimulation, magnetic stimulation, gravitational stimulation, acceleration stimulation, pressure stimulation, brain wave stimulation, balance stimulation, muscle stimulation, visceral stimulation, light stimulation, sound stimulation, visual cortex stimulation, auditory cortex stimulation, administration of intestinal bacteria, drug administration, etc.

[0048] As shown in FIG. 22 , the calculation unit 8 outputs first feedback information F1 based on audio information to the output unit 3 according to the degree of matching. The calculation unit 8 outputs one or more pieces of first feedback information F1 midway between the start and end of input of the input trajectory K1. For example, the calculation unit 8 generates first feedback information F1 based on audio, with the output state adjusted according to the degree of matching, and outputs the first feedback information F1 to the output unit 3. For example, the calculation unit 8 outputs the first feedback information F1 based on audio information so that the sound level changes according to the degree of matching. The first feedback information F1 based on audio information is generated using an alarm sound, language, or the like. For example, the calculation unit 8 may generate the first feedback information F1 by adjusting sound-related parameters such as volume, frequency, and audio spectrum so that the sound is comfortable for the user. For example, the calculation unit 8 outputs the first feedback information F1 so that the sound level increases as the degree of matching increases. The calculation unit 8 generates the first feedback information F1 so that the sound level decreases as the degree of matching decreases. The calculation unit 8 may not only adjust acoustic parameters according to the degree of match, but also adjust the output state of the first feedback information F1 based on any method. The calculation unit 8 outputs the first feedback information F1 at any time interval. The calculation unit 8 may output the first feedback information F1 continuously or intermittently. The calculation unit 8 may generate the first feedback information F1 based on linguistic information. The calculation unit 8 may generate the first feedback information F1 based on linguistic information that guides the position, direction, and speed of the user's input operation so as to increase the degree of match.

[0049] The calculation unit 8 generates not only first feedback information F1 based on audio information, but also second feedback information F2 to improve the effectiveness of the user's training based on the input operation, and outputs the information to the display unit 4. For example, when the degree of match satisfies a predetermined condition, the calculation unit 8 generates second feedback information F2 based on image information and outputs the information to the display unit 4. The calculation unit 8 outputs one or more pieces of second feedback information F2 midway from the start to the end of the input of the input trajectory K1. For example, based on a comparison result between the degree of match and a preset threshold, the calculation unit 8 outputs the second feedback information F2 to the display unit 4 when a predetermined condition is met that the degree of match is greater than the threshold. The calculation unit 8 generates the second feedback information F2 based on one or more combinations of outputtable information such as text information, graphic information, and audio information. For example, the calculation unit 8 outputs the second feedback information F2 based on image information to the display unit 4. For example, the calculation unit 8 increases the number of figures, such as stars, displayed on the display unit 4 as the degree of match increases. The calculation unit 8 generates second feedback information F2 for any number of graphics and outputs it to the display unit 4. If the calculation unit 8 generates second feedback information F2 including audio information, it may output the audio information from the speaker 5. The calculation unit 8 may generate second feedback information F2 based on not only graphics but also the color of the entire screen, the color of a portion of the screen, or the brightness of the screen. The calculation unit 8 may change visual information such as the thickness or color of the line of the input trajectory K1 in accordance with the output of the second feedback information F2. The calculation unit 8 outputs the second feedback information F2 at any time interval. The calculation unit 8 may output the second feedback information F2 based on any number of output times or output frequency. The calculation unit 8 may output the second feedback information F2 based on the number of output times or output frequency that the user can predict, or based on the number of output times or output frequency that the user cannot predict.

[0050] The calculation unit 8 may provide a predetermined delay state from the timing at which the input operation is performed, and output the second feedback information F2 to the output unit 3. Here, the timing at which the input operation is performed includes one or more timings in one input trajectory K1, such as the timing at which the input operation is performed at every predetermined sampling period, the timing at which the input operation is performed at every predetermined distance on the path of the input trajectory K1, or the timing at which the input operation is performed at every predetermined position and / or every predetermined area from the starting position. The delay state includes the concept of a temporal and / or spatial delay after the timing at which the input operation is performed, such as a time delay from the input timing or a distance delay after moving an arbitrary distance from the input position.

[0051] The calculation unit 8 may cause the output unit 3 to output the second feedback information F2 at the timing when the input operation is performed (delay time=0). The calculation unit 8 may cause the output unit 3 to output the second feedback information at any amount or frequency. The calculation unit 8 may cause the output unit 3 to output the second feedback information F2 when the degree of match reaches a maximum value. The calculation unit 8 may cause the output unit 3 to output the second feedback information F2 at a predetermined time or within a predetermined time range while the input operation is being performed. The calculation unit 8 may cause the output unit 3 to output the second feedback information F2 based on a delay time (seconds) that is equal to or greater than 0. The calculation unit 8 may cause the display unit 4 to display the second feedback information F2 including image information based on the delay time, and may cause the speaker 5 to output the second feedback information F2 including audio information based on the delay time (seconds). The calculation unit 8 may output the second feedback information F2 including image information and the second feedback information F2 including audio information at the same timing based on the same delay time, or at different timings based on different delay times. The calculation unit 8 may output the second feedback information F2 when a predetermined number of input operations, which marks a milestone in one task, are completed. The calculation unit 8 may output the second feedback information F2 when the input trajectory K1 reaches a predetermined area. The calculation unit 8 may output the second feedback information F2 when the trajectory data M1 reaches a predetermined area.

[0052] The calculation unit 8 may output the second feedback information F2 when the degree of matching meets a predetermined condition, such as when the degree of matching reaches a predetermined value, when the degree of matching is within a predetermined range, when the degree of matching is outside a predetermined range, etc. The calculation unit 8 may output the second feedback information F2 when the input trajectory K1 meets a predetermined condition, such as predetermined coordinate values, shape, position, velocity, acceleration value, within a predetermined range, or outside a predetermined range. The calculation unit 8 may output the second feedback information F2 when a movable display image other than the input trajectory K1 changes to a predetermined color or shape at a predetermined timing.

[0053] The calculation unit 8 may output the second feedback information F2 when a combination of one or more conditions is met, such as when the user touches the input unit 2, when the degree of contact with the input unit 2 meets a predetermined condition, when the user no longer touches the input unit 2, or when a predetermined or greater acceleration is applied to the information processing device 1 or the input unit 2. The calculation unit 8 may output the second feedback information F2 when the state of the input operation meets a predetermined condition, such as a predetermined value, within a predetermined range, or outside a predetermined range, that increases the degree of match.

[0054] The calculation unit 8 may output the second feedback information F2 when the user performs a specific operation, such as tapping on the display unit 4, pressing with a pressure greater than or equal to a predetermined pressure, pressing with a pressure less than a predetermined pressure, removing the operating means from the display unit 4, or shaking the information processing device 1. The calculation unit 8 may output the second feedback information F2 when the user makes a specific sound. The calculation unit 8 may output the second feedback information F2 when the user's hearing function has a specific performance level or when the calculation unit 8 recognizes a response that exceeds or falls below the specific performance level. The calculation unit 8 may output the second feedback information F2 when the input trajectory K1 moves to a predetermined position. The calculation unit 8 may output the second feedback information F2 when a calculated value, such as a score related to the input operation of the input trajectory K1, meets a predetermined condition, such as a specific value, a value within a predetermined range, or outside a predetermined range. The calculation unit 8 may output the second feedback information F2 when a specific sound is played.

[0055] The calculation unit 8 calculates the input speed V at which the input trajectory K1 is input at predetermined intervals and compares the input speed with a threshold. The calculation unit 8 may output second feedback information F2 to the output unit 3, including information for adjusting the input speed V, based on the comparison result between the input speed V and a preset threshold. It is generally known, based on test results and surveys such as subjective evaluations of hearing in daily life, that when a user is given feedback that stimulates dopamine production in the brain, the effectiveness of training for auditory physical functions such as hearing ability (e.g., hearing ability in noise (unit: dB)) is enhanced. For example, when the user is experiencing first feedback information F1 based on audio information, the calculation unit 8 may output second feedback information F2 including information for guiding the user to slow down the input speed V of the input operation compared to the threshold.

[0056] The calculation unit 8 may output second feedback information F2 to the output unit 3, the second feedback information F2 including information for adjusting not only the input velocity V but also the position of the second coordinate G2 and the input acceleration. Information for adjusting the input velocity V includes, for example, information for delaying the input operation compared to the current state, information for speeding up the input operation compared to the current state, information for maintaining the input operation, etc. For example, when the calculation unit 8 determines that the input trajectory K1 generated by the user's input operation is within a predetermined range from the end of the trajectory data M1, the calculation unit 8 generates information notifying the user of the end and outputs the information from the output unit 3. The calculation unit 8 may generate information notifying the user of the end at any timing and output the information from the output unit 3.

[0057] 23 is a flowchart showing the process flow of an information processing method executed in information processing device 1. The information processing method is executed based on a computer program that can be installed in a computer mounted on information processing device 1. The computer program causes a calculation unit 8 included in information processing device 1 to execute the following processes.

[0058] The calculation unit 8 acquires trajectory data M1 corresponding to the virtual target from the storage unit 9 or the server device 20 (S100). The calculation unit 8 calculates a first coordinate G1 corresponding to the trajectory data M1 (S102). The calculation unit 8 accepts an input operation from the user (S104). Based on the input operation, the calculation unit 8 calculates a second coordinate G2 corresponding to an input trajectory K1 input to the input unit 2 so as to follow the trajectory data at predetermined intervals (S106). The calculation unit 8 calculates the degree of coincidence between the first coordinate G1 and the second coordinate G2 (S108). The calculation unit 8 generates first feedback information whose output state has been adjusted according to the degree of coincidence based on the voice, and causes the output unit 3 to output the first feedback information (S109).

[0059] The calculation unit 8 determines whether the calculated degree of match is greater than a preset threshold (S110). If the degree of match is equal to or less than the threshold, the calculation unit 8 returns the process to S104. If the degree of match is greater than the threshold, the calculation unit 8 generates feedback information and outputs it from the output unit 3 (S112). The calculation unit 8 determines whether an end condition is met (S114), and if the end condition is met, the process ends. Here, according to the technology described in Non-Patent Document 1, feedback information is output after the user has finished tracing the trajectory data. In contrast, according to the information processing device 1, first feedback information F1 and second feedback information F2 are output while the user is tracing the trajectory data, making it possible to obtain an evaluation of the input operation in real time.

[0060] As described above, the information processing device 1 can perform physical function training based on the user's hearing, for example, brain function training through hearing. The information processing device 1 can perform brain function training through the user's hearing by determining the content of the user's input operation, generating feedback information based on the determination result, and guiding the input operation. The information processing device 1 can train the user's physical function based on hearing by generating first feedback information F1 based on voice. The information processing device 1 can improve the effectiveness of the user's training by generating second feedback information F2 to prompt the input operation to a predetermined state. The information processing device 1 can perform brain function training through the user's hearing by guiding the input operation based on the second feedback information F2.

[0061] Figure 1: Training to improve hearing Figure 2: Children can enjoy training their hearing by finding letters and shapes based on sound and then completing quizzes using the letters and shapes they have found. - Training auditory perception using sound can easily become monotonous and repetitive, and it is easy to get bored of continuing it. Therefore, by combining auditory training with the discovery of letters and shapes and quizzes, auditory training can be continued in a fun way. In auditory training, the user is asked to trace their finger across the screen to find visually hidden letters or shapes. By changing the sound depending on the distance between the hidden letters or shapes and the finger's position, the user can find the hidden letters or shapes by relying on the sound. By concentrating on the sound and moving their finger in this way, auditory training is achieved. The sounds and tracing actions used in auditory training will utilize neumo's existing technology. The application documents are reproduced later in this document. Furthermore, there is a step where you rotate the letters and shapes you find by tracing them with your finger, and a step where you solve a quiz using the letters and shapes you find. Examples of specific steps are shown on the following pages. Figure 3. Revealing letters and completing words: Step (1) Find letters by tracing lines based on sounds. What the user sees task The characters you trace are broken down into a set of lines, rotated at any angle. Rotation is not required. Trace the N lines that make up the character in order. You can trace all N lines in order, or just a few M lines. The lines can be straight or curved, and any shape is fine. You can also trace multiple lines in one stroke. The letters to be traced are hidden visually, so you need to trace the lines by listening to the sound. When tracing, the starting point of the line will be displayed, so start tracing from that starting point and reach the end point. When you reach the end point, the next line tracing problem will begin. The starting point can be a point, or it can be something that indicates a certain range, such as a circle. The range can also be indicated by a shape other than a circle. The end point can be visually represented by a point or a line, or it can be invisible. The end point can also be found without being displayed. Once you have finished tracing, you can check the path you have traced. You can choose not to display it while tracing, or you can display the path in order as you trace the question. The correct answer line may be displayed, or you may proceed without it being displayed. What the user should follow Figure 4. Raising letters to complete words: Step (2) Rotating letters to make them upright What the user sees task If you do not trace all of the N lines that make up the character to be traced, the lines that you did not trace will be displayed as the correct answer. Rotate the character based on the sound to make it turn upside down. By changing the sound depending on the angle at which the character is off from the correct position, you can find the correct angle by relying on the sound. This step is not required and can be skipped. What the user should follow Make a sound to let you know that a certain direction is the correct answer Figure 5. Making letters appear and completing words: Step (3) Select the traced letters What the user sees task Answer what characters you traced. You can choose from multiple options, or answer freely using keyboard input, voice input, handwriting input, etc. If you get the answer wrong, there are ways to answer the question either once or multiple times. If the answer is incorrect, the correct answer may or may not be revealed. If the correct answer is revealed, the next question will be easier, so this can be used to adjust the difficulty setting. This step is not required and can be skipped. Repeat steps (1) to (3) depending on the number of characters in the question. Figure 6. Making the letters appear and completing the word: Step (4) Guess the kanji from the characters you have obtained. What the user sees task · Ask a quiz using the letters you found in steps (1) to (3). In the quiz to guess the reading of a kanji, the characters found in steps (1) to (3) are used to display part of the reading for the kanji, and you must guess the missing characters. The kanji to be read can be a single character, or it can be a multi-character word, a four-character idiom, or a sentence. The characters found in steps (1) to (3) can be displayed as part of the pronunciation, or can be used as one of the candidates. -The quiz where you have to guess the blank letters is answered from the options. There is also a way to input it. If the answer is correct, a sound or visual will be displayed to let the user know that they got the answer right. For example, the sound or visual of fireworks can be presented. Other means besides fireworks are also acceptable. If the answer is incorrect, the user will be informed that they got the answer wrong. If the answer is incorrect, there are two options: displaying the correct answer or prompting the user to give the next answer. Figure 7 The rotation step can be done in a different way The step of tracing and rotating the letters or shapes you find can be replaced with the other methods shown below, or can be omitted. By using sound to help you find the correct location, you can train your auditory sense. You can also use a method where you rely on visuals or vibrations to find the location without using sound. Move the piece on the screen to the correct position in 1D, 2D, or 3D space. For example, it can be used in cases where you need to place the piece in the correct position, like in a jigsaw puzzle. - Scale text and shapes to fit the correct size. Rotate in 3D space around the X-axis or Y-axis to align with the correct position -Flick letters and shapes with your fingers, or operate a spring like a pinball to send them flying, hitting the correct target or putting them in a hole. You can also use sound to help you find the correct direction and strength. -Find the correct stroke order for kanji and characters. If the stroke order is correct, a correct sound will be played, and if the stroke order is incorrect, an incorrect sound will be played. Figure 8 Calculating the score of the tracing result The step of tracing letters or shapes to find them can also provide a function that gives feedback to the user by quantifying how accurately the tracing was done. The quantification method can be one of the following methods, or it can be another method. Use spatial distance ■There is a method to quantify the distance between the correct letter or shape and the traced line. 1. Convert the traced line into a set of points 2. For each point, calculate the distance to the nearest correct character or shape 3. The calculated distances can be added together to obtain the cumulative difference. 4. You can calculate the average distance and the difference. 5. You can also calculate using only the calculated distances within a certain range. For example, you can use only the values ​​close to the correct answer (values ​​less than or equal to 1) as the cumulative value. Or, you can use the values ​​far from the correct answer (values ​​greater than or equal to 10) as a cumulative value and deduct points. ■There is a method to quantify the time it takes to successfully trace the correct letters and shapes. 1. Define the range of correct characters and shapes. The range of correct answers can be either a binary value of 1 or 0, or a decimal value between 1 and 0, or a gradient range such as 100 to 0. 2. By taking the total tracing time as the denominator and the time spent tracing the correct position as the denominator, the time spent tracing the correct position can be quantified. In this case, you can also calculate the value using the slope defined in 1. ■There is a method to quantify the difference between the position where the tracing is completed and the end point of the correct letter or shape. Instead of using data from the tracing path, you can calculate the distance between the position at the end of the tracing and the end point of the correct character or figure, and use that as a numerical value to evaluate the result. For the steps of rotating letters and shapes, it is also possible to provide a function that quantifies how accurately the answer was given and gives feedback to the user. The smaller the difference between the correct angle and the angle you answered, the higher the score. Figure 9: A variety of patterns can be used for quizzes using letters and shapes found by tracing. We have provided examples of how to trace characters, but you can also use any of the following shapes, not just letters. - Japanese hiragana, katakana, kanji, numbers Alphabet (may include language-specific characters other than the basic alphabet, such as French spelling symbols and ligatures or German umlauts) Other language characters (Simplified Chinese, Traditional Chinese, Arabic, Devanagari, Greek, Hangul, etc.) Abstract shapes (circles, triangles, etc.). 1D, 2D, or 3D shapes are acceptable. -Line segments in photos and illustrations (any shape that can be expressed with lines, such as anime characters, animal illustrations, icons, products, constellations, family crests and coats of arms, tangrams, and topography of countries and prefectures, can be included) - Find the right path through maps and mazes There are various ways to create quizzes using the letters and shapes you have discovered, such as: - By tracing the steps, you can find hiragana, katakana, numbers, alphabets, and other characters of various languages, and then you will be given a quiz like the one below. The following is just an example, and you can make other quizzes as well. It can be a quiz where the answer can be found just by tracing the characters, or a quiz where you need to answer with only partial knowledge. ■ Guess the reading of kanji and compound words ■ Make a crossword puzzle and guess where the sentence or word goes ■ Use the characters you find by tracing to find or create kanji, words, or sentences. You can rearrange the characters, or find radicals to create kanji. You can also choose from the Hyakunin Isshu. ■ Fill-in-the-blank quiz using the letters you traced ■ Calculation problems and number puzzles using numbers found by tracing ■ Trace the characters and guess what they are. ■ Trace and find the letters and guess what they stand for - Find shapes and illustrations in the tracing steps and take quizzes like the following ■A quiz to guess what shape is hidden. You can trace all the lines, or just some of the lines, with some of the lines already displayed. Or you can have only a portion of the shape be displayed and have to guess the whole shape. For example, you could have the question guess what animal it is or what constellation it is. ■It can also be a quiz where you have to guess the correct answer by combining multiple shapes that you find by tracing. ■You can also use jigsaw puzzles, block puzzles, or Tetris, where you trace and combine the shapes you find. ■You can also have them compare the size of the shapes they trace and find similar shapes. - Find the correct path by tracing steps in a map or maze. ■A quiz where you have to guess the destination on the map or the route to the exit of the maze by tracing the path through the map or maze as a hint. - A quiz on listening to words in a noisy environment, where you can trace and find letters to use as hints to answer questions, is also good. Figure 10. Factors for enjoying and continuing training (1) - Increase motivation by visualizing tracing accuracy and keeping records -Give feedback in the form of a numerical value on how well the child traces letters and shapes. - Record the letters and shapes you have traced in the past as a history and review them later to see the improvement in your listening comprehension. Badges are awarded based on the difficulty and accuracy of the tracing. Various badges are available (high accuracy, tracing difficult sounds, tracing in a short amount of time, high quiz correct answer rate, etc.), and can be collected. Motivation is increased by showing the next badge that can be obtained. · Motivate yourself by sharing and communicating with others -You can share your tracing score and the quiz answer screen with others. You can share your results with other users who are registered as friends within the app, or via email, SNS, LINE (registered trademark), and other means outside the app. - Motivate yourself by notifying you when you receive likes or comments on your shared results · Allow students to feel a sense of accomplishment by indicating the difficulty and level of the questions The difficulty level is defined by the target sound and distracting sounds used in the tracing steps, allowing you to understand the level. If you can trace it neatly, you can move up to a higher level. As a result of hearing training, you can reach a higher level and feel a sense of accomplishment. For example, once you have cleared all the questions equivalent to Kanji Kentei Grade 5, you can move on to questions equivalent to Kanji Kentei Grade 4. By making it possible to quantify the level of difficulty, you can create a sense of accomplishment. Similar levels will be defined for other quizzes, such as crossword puzzles. For quizzes involving shapes or illustrations, define the difficulty level based on the detail of the shapes to be traced and whether the shapes are easy to understand. -Increase motivation by showing achievements compared to others -Your ranking is displayed by comparing how well you traced the same question with others. There are several types of rankings available, such as rankings by character (e.g. "a"), overall rankings for multiple questions, monthly rankings, rankings by difficulty level, rankings by quiz question, and rankings by quiz level. - Shows your ear age by comparing tracing accuracy performance with people of the same age and generation. - Brain age is displayed by comparing the degree of accuracy of the Wiz compared to people of the same age or generation. Figure 11: Factors for enjoying and continuing training (2) · Be aware of time You can set a time limit for each question to help students concentrate on their work. You may provide a suggested time for solving the problem. If the speed is too fast, the training effect will be reduced, so it is a good idea to make the child aware of the time in order to encourage them to trace slowly. In this case, you can combine messages such as "Take it slower" with the time. - You can visualize your efforts by displaying the training time for each day. -You can increase the score if you get consecutive correct answers If you train every day, you can increase your score. When tracing the line, if the child keeps tracing almost the correct answer, you can give them motivational feedback such as "Keep Going!!!" or "Perfect**!" Figure 12: Improved training effectiveness by setting the difficulty level according to ability By measuring the user's hearing level in advance using a system other than this system and incorporating the results as input, it is possible to present questions of an appropriate difficulty level to the person's hearing ability and hearing condition. By providing questions of an appropriate difficulty level to suit the person's ability, the effectiveness of the training can be improved. The following information can be used as input. Pure tone audiometry results Speech hearing test results Test results for hearing in noise (speech comprehension tests in steady noise, such as digits-in-noise, HINT, J-HINT, OLSA, and J-Matrix; or speech comprehension tests in noisy environments, such as digits-in-multi-talker-babble, Speech-in-multi-talker-babble, and QuicksIN) Left and right hearing difference measurement results - Auditory temporal processing measurements (measurement of thresholds using an amplitude modulation (AM) detection task, a task measuring the ability to process time differences between the ears using NOSJT and NOST signals, frequency modulation threshold detection task, gap detection task, TFS1, TFS2, TFS-LF, interaural time difference detection task, frequency modulation direction detection threshold for seep sounds) -Measurements of cognitive function, such as working memory and executive function -Electroencephalogram (ABR) results - As the user's hearing ability improves through continued training, by changing the difficulty level according to the training results, it is possible to present questions of a level appropriate to the person's hearing ability and hearing condition. If the tracing accuracy exceeds the level-up threshold, the target sound and distracting sound will be changed to allow for more difficult training. If the tracing accuracy falls below the threshold for level reduction, the target sound or distracting sound will be changed to allow for easier training. The above changes can be determined by any method, such as by measuring the changes based on the results of a single training session, by judging from the results of multiple consecutive sessions, or by averaging the results of multiple sessions (for example, the average value of one day's training). - Set the difficulty level taking into account the strengths / weaknesses of the target sound and the strengths / weaknesses of the interfering sounds. ・When it comes to target sounds, each person has their strengths and weaknesses, for example, they may be good at hearing changes in the pitch of pure tones but not so good at hearing human voices. If you set the level for a target sound that you are good at, you will not be able to hear the target sound that you are not good at. The overall level will not be raised unless the weak pair sound is also cleared to a certain level. Similarly, if there are strong and weak interfering sounds, the overall level of the interfering sounds will not be raised any further until the weak interfering sound reaches a certain level. The difference in level between the target sound and the interference sound should not be too great. For example, if the target sound is level 20 and the interference sound is level 8, you must increase the interference level to at least 10 in order to level up the target sound. Figure 13 Other functions (1) - In addition to existing technologies, the following methods may also be used for tracing. Use a stylus instead of your finger Use devices or software that detect eye gaze to replace eye tracking. The tracing action is detected using a device equipped with a sensor, such as a Nintendo Switch (registered trademark) controller. Other controllers available include those from existing products such as Meta Quest (registered trademark), Apple Vision Pro (registered trademark), Vive (registered trademark), PlayStation (registered trademark), XBOX (registered trademark), and pico, as well as specially developed controller devices. The following methods may be used to present the correct answer: - Plays a sound to indicate the correct answer when the correct position is traced - Vibrate your device when you are in the right spot - When the user has traced the correct position, some visual element is displayed (the screen lights up, a message is displayed, a character is displayed, the screen appears to vibrate, etc.) - The thickness and color of the line displayed changes depending on whether you trace the correct position or the incorrect position, making it easier to see the correct position. -The following methods may be used to indicate incorrect answers: - Plays an incorrect sound if you trace the wrong spot - Vibrate your device if you swipe in the wrong place If you swipe in the wrong place, some kind of visual element will be displayed (the screen will light up, a message will appear, a character will appear, the screen will appear to vibrate, etc.) The following may be used to present the correct and incorrect answers: - Present immediately after tracing -Present when you are continuously tracing the correct or incorrect position (for example, tracing the correct position for 3 seconds) - Show the correct or incorrect answer multiple times when tracing the location. For example, nothing is shown the first time, and then from the second time onwards. The number of times can be changed to three or four times. It is also possible to show only the second time, and not show anything again from the third time onwards. -It would also be good to have the device vibrate only when you pass through the correct spot while tracing and then return to the same spot again, so that you know that it is the correct spot. -The correct or incorrect answer will be displayed a few seconds after tracing. - Present only during certain conditions, such as during bonus stages, fever periods, or when using certain items Figure 14 Other functions (2) You can also add effects to the tracing to make it more fun. Draw a line where you are tracing Displaying visual elements in a location other than the area you are tracing (such as characters appearing or moving, pictures being colored, or sparkling visual elements being displayed) -Music plays only for the question you are tracing, the number of notes in the music being played increases or decreases, and the speed and BPM of the music change The tracing action can be done by drawing lines freely, or by filling in a grid. The grid can be any shape, such as a triangle, square, hexagon, or octagon. Figure 15 neumo's unique technology (1) Training using head-related transfer functions to adjust the left and right sound positions with your fingers -Distinguish the location of sound sources in one-, two-, and three-dimensional space using sound Listen to the direction from which the sound is coming and make some kind of adjustment to make it seem like the sound is coming from the front. By carefully listening to the location of the sound source and making adjustments, you can train your auditory temporal processing. For example, in one-dimensional space, a slider is displayed on the smartphone screen, and by operating the slider, the left and right position of the sound is controlled so that it comes from the center. In two-dimensional space, by tapping with your finger on the smartphone screen, the sound source is controlled to come directly in front of you by listening to the position above, below, left and right of the tapped position. In three-dimensional space, a VR headset is used, or the smartphone's AR function, gyro (angular velocity) sensor, and magnetic sensor are used to control the sound source to come directly in front of you in three-dimensional space. ·device Any device that can change the position of sound in one, two or three dimensions is fine, such as a smartphone, tablet, PC, game console, VR, or real-world speaker. - Any device capable of producing stereo or 3D sound, such as stereo speakers, earphones, or headphones, is acceptable. Target sound type The target sound can be a generated pure tone, sound sweep sound, or time-frequency modulated sound (spectro-temporally modulated sound), such as FM modulated sound, AM modulated sound, musical instrument sound, music, human voice, animal sounds, natural sounds, etc. When using music, use music that balances predictability and uncertainty. Automatic generation can also stimulate dopamine production and improve learning efficiency. Interference sounds By using distracting sounds to mask the target sound, the training becomes more difficult and more effective. Distracting sounds can be white noise, human voices like speech bubbles, rain, crowds, music, or other sounds that can interfere with the target sound. To effectively improve hearing in everyday life, it is also a good idea to use fluctuating noise based on a statistical model of fluctuations in volume and acoustic features in actual cafes, bars, etc. Figure 16 neumo's unique technology (2) Operation while listening to sound The following methods are available for operating while listening to the target sound: -Tracing continuously on the tablet / smartphone screen to move to a single point -Continuously trace the screen of your tablet or smartphone to track the moving target sound and keep it meeting the target condition. - Use the analog joystick on the gamepad to move continuously through the virtual space -Continuous movement in the virtual space using the analog joystick on the touch screen -Users themselves walk and move within the physical space ·How to operate Hold your smartphone horizontally and tilt it Hold your smartphone as you normally would and tilt it - Hold the smartphone and move mainly horizontally (search for objects that appear on the screen using AR) Move your arm while wearing the Apple watch (such as stroking a parakeet or cat) - Wear a VR headset and turn your neck or tilt your head -Operation using VR controller - Movements such as Tai Chi and yoga (detect movements by processing camera images) Record your voice with a microphone to detect the movements of your jaw muscles, tongue, lips, vocal cords, and respiratory muscles -Exercise operation method · Hand and arm (finger) exercise → Smartphone touch screen · Hand / arm movements while holding the smartphone → Angular velocity sensor (gyro) of the smartphone · Hand and arm movements while wearing Apple watch (registered trademark) → Angular velocity sensor (gyro) of Apple watch (registered trademark) Voice (jaw, tongue, lips, vocal cords, respiratory muscle exercise) → smartphone microphone ·Various body movements →Detect movements by processing camera images · Walking and moving from place to place → Camera images are processed to detect movement and position -Tilt and turn your head → VR headset motion sensor ·Hand / arm movements →VR controller Figure 17 neumo's unique technology (3) Explicit application of multimodal stimulation By combining sound with vibration, multimodal stimulation can be achieved, enhancing the effectiveness of training. In situations where the signal-to-noise ratio between the target sound and the interfering sound is poor and it is difficult to hear using hearing alone, by converting the target sound into vibration while preserving as much of its acoustic features as possible and adding it as a somatosensory sensation, it is possible to improve listening ability through the law of reverse effect and further worsen the audio-to-noise ratio. By boosting listening ability with multimodal stimulation in situations where the signal-to-noise ratio is so low that it is impossible to hear using hearing alone, it is possible to train listening in difficult auditory environments and improve learning effects. Vibration stimulation unrelated to the acoustic features of the target sound can also improve listening ability in the same way as described above through integration with somatosensory stimulation in the auditory pathway. Therefore, it is possible to train listening in a similarly harsh auditory environment and improve learning effects. By adding weak noise vibrations, stochastic resonance can be generated, improving listening ability even in difficult signal-to-noise ratios. This means that listening training can be performed in similarly difficult auditory environments, enhancing learning effects. RWM (pulse width modulation), PAM (pulse amplitude modulation), or a combination of these can be used to convert target sounds into vibrations while preserving acoustic features as much as possible. As vibration stimuli unrelated to the acoustic features of the target sound, vibrations synchronized with the rhythm of the target sound, vibrations linked to user operations, or vibrations at a fixed cycle or randomly regardless of the target sound or user operations are also acceptable. Figure 18 neumo's unique technology (4) Meditation and hearing training can be performed simultaneously by drawing circles and shapes while listening to sound. A common method of meditation involves relaxing and focusing on deep breathing, but by tracing a circle with your finger on the smartphone screen while concentrating on listening to a target sound, you can meditate and train your hearing at the same time. - The target sound changes in conjunction with the circular movement of your finger. For example: If you trace the correct circle with your finger, the correct sound will be played. If the object gets out of alignment, the sound will change to let you know that it has deviated from the circle. Define the speed of the circle tracing and if it is moving at the right speed The right sound is played and the sound can be changed when the speed is too fast or too slow. Circular motion The circle may or may not be visible on the screen. You can also draw a circle in space using a VR controller or motion capture equipment. -Change the size of the circle or use a pattern other than a circle You can change the size of the circle midway. You can also have them trace shapes other than circles, such as squares and stars, or letters and pictures. The target sound change can be any sound that can be distinguished between correct and incorrect sounds. For example, the following sounds can be used: The correct sound is the sound that comes from the front, and the sound that comes from the wrong position comes from a position in 1D, 2D, or 3D space that is shifted from the front, so you can tell in which direction on the circle the sound is shifted. - By changing pitch, volume, speed, sweep sound, rhythm, phoneme, type of instrument, etc., you can distinguish between correct and incorrect sounds. A guide is good If the child misses the mark when tracing the circle, guidance can be provided by text displayed on the screen or audio. In addition to voice and text, guides may also use sound effects, flashing screens, character and icon displays, vibrations, etc. You can give feedback on the speed of tracing the circle by saying things like, "Go slower" or "That's a good pace." It is more effective to train slowly while paying attention to the target sound than to trace the circle roughly. - Vibration enhances the effect of meditation By providing a constant vibration during a task, the entrainment effect can lower the heart rate and enhance the meditative effect. The user's heart rate can be measured using a smartwatch or smartphone camera, and the vibration can be set to a slightly slower rate than the heart rate. It is also possible to provide vibration based on the general heart rate without measuring the heart rate. Sound effects To prevent boredom from monotonous tasks, you can add variety by playing bell sounds, animal sounds, natural sounds, music, etc. Interference sounds In addition to the sound linked to the circular movement of the fingers, a constant distracting sound can be played, making it difficult to hear, enhancing the training effect. -The same interference sound as in Technique (1) can be used. [Explanation of symbols]

[0062] 1 Information processing device, 2 Input unit, 3 Output unit, 4 Display unit, 5 Speaker, 6 Communication unit, 8 Calculation unit, 9 Memory unit, 20 Server device, 21 Calculation unit, 22 Memory unit, 23 Communication unit, D Distance, F1 First feedback information, F2 Second feedback information, G1 First coordinate, G2 Second coordinate, K1 Input trajectory, M1 Trajectory data, MA Information, S Information processing system, V Input speed, W Network

Claims

1. a calculation unit capable of executing brain function training through hearing to train a physical function based on the hearing of the user based on an input operation of the user; The calculation unit Obtaining trajectory data corresponding to the virtual target; calculating a first coordinate corresponding to the trajectory data; calculating, at predetermined timings, second coordinates corresponding to the input trajectory input to the input unit so as to follow the trajectory data, based on the input operation of the user; calculating a degree of coincidence between the first coordinates and the second coordinates; generating, based on the speech, first feedback information whose output state has been adjusted in accordance with the degree of coincidence, and causing an output unit to output the first feedback information; If the degree of coincidence satisfies a predetermined condition, second feedback information indicating an evaluation of the input operation to the user is generated, and the second feedback information is output by the output unit with a predetermined delay from a predetermined timing at which the input operation is performed. Information processing device.

2. The predetermined delay state is: Including the concept of delaying the input operation in time and / or space after the timing of the input operation, The information processing device according to claim 1 .

3. The calculation unit calculating an input state in which the input trajectory is input at each predetermined timing; outputting the second feedback information including information for adjusting the input state based on a comparison result between the input state and a setting condition to the output unit; The information processing device according to claim 1 .

4. The calculation unit outputting an interference sound from the output unit; The information processing device according to claim 1 .

5. An information processing system including an information processing device that accepts input operations from a user, and a server device communicably connected to the information processing device, At least one of the information processing device and the server device includes a calculation unit capable of executing brain function training through hearing to train a physical function based on the hearing of the user, based on the input operation of the user; The calculation unit Obtaining trajectory data corresponding to the virtual target; calculating a first coordinate corresponding to the trajectory data; calculating, at predetermined timings, second coordinates corresponding to the input trajectory input to the input unit so as to follow the trajectory data, based on the input operation of the user; calculating a degree of coincidence between the first coordinates and the second coordinates; generating, based on the speech, first feedback information whose output state has been adjusted in accordance with the degree of coincidence, and causing an output unit to output the first feedback information; If the degree of coincidence satisfies a predetermined condition, second feedback information indicating an evaluation of the input operation to the user is generated, and the second feedback information is output by the output unit with a predetermined delay from a predetermined timing at which the input operation is performed. Information processing system.

6. A computer program installed in a computer mounted on an information processing device capable of performing brain function training through hearing, for training a user's physical function based on the user's hearing, based on an input operation of the user, Obtaining trajectory data corresponding to the virtual target; calculating a first coordinate corresponding to the trajectory data; calculating, at predetermined timings, second coordinates corresponding to the input trajectory input to the input unit so as to follow the trajectory data, based on the input operation of the user; calculating a degree of coincidence between the first coordinates and the second coordinates; generating, based on the speech, first feedback information whose output state has been adjusted in accordance with the degree of coincidence, and causing an output unit to output the first feedback information; If the degree of coincidence satisfies a predetermined condition, second feedback information indicating an evaluation of the input operation to the user is generated, and the second feedback information is output by the output unit with a predetermined delay from a predetermined timing at which the input operation is performed. causing the computer to execute a process; Computer program.

Images