Information processing device, information processing system, and computer program

Abstract

An information processing device 1 comprises a calculation unit 8 for evaluating an input operation from a user. The calculation unit acquires trajectory data M1 corresponding to a virtual target, calculates a first coordinate G1 corresponding to the trajectory data, uses the input operation from the user as a basis to calculate a second coordinate G2 corresponding to an input trajectory K1 input to an input unit 2 so as to follow the trajectory data for each instance of a predetermined timing, calculates a degree of coincidence between the first and second coordinates, uses audio as a basis to generate first feedback information F1 in which an output state is adjusted in accordance with the degree of coincidence, causes an output unit 3 to output the first feedback information, and, when the degree of coincidence satisfies a predetermined condition, generates second feedback information indicating an evaluation of the input operation from the user and causes the output unit to output the second feedback information.

Description

Information processing device, information processing system, and computer program

[0001] The present disclosure relates to an information processing device, an information processing system, and a computer program for training a user's hearing-based physical function, for example, for brain function training through hearing.

[0002] The technology shown in Figures 1 to 18 did not exist. Furthermore, Patent Document 1 proposes an information processing device that acquires information on the content of a user's operation and provides feedback information to the user. According to the technology described in Patent Document 1, a predetermined action of the user is detected based on information on the content of the user's operation, and the timing for providing feedback information from the time of detection is calculated according to the content of the predetermined action, and the feedback information can be output at the calculated timing.

[0003] JP 2024-109398 A

[0004] “Audiomotor Perceptual Training Enhances Speech Intelligibility in Background Noise”, Jonathon P. Whitton, Kenneth E. Hancock, Jeffrey M. Shannon, Daniel B. Polley, Current Biology, 11.6th.2017

[0005] In order to train a user's physical function based on hearing, it is necessary for the user to perform an operation based on hearing. The technology described in Patent Document 1 provides feedback information when it is determined that a predetermined movement has been detected, but is not intended to be used for training a user's physical function based on hearing.

[0006] The present invention aims to provide an information processing device, an information processing system, and a computer program that are capable of, for example, performing brain function training through hearing in order to train a user's physical functions based on their hearing.

[0007] 1 to 18. Another aspect of the present invention is an information processing system including a calculation unit that evaluates an input operation of a user, the calculation unit acquiring trajectory data corresponding to a virtual target, calculating first coordinates corresponding to the trajectory data, calculating second coordinates corresponding to an input trajectory input to an input unit so as to follow the trajectory data at predetermined timings based on the input operation of the user, calculating a degree of coincidence between the first coordinates and the second coordinates, generating first feedback information based on a voice, the output state of which has been adjusted in accordance with the degree of coincidence, and causing an output unit to output the first feedback information, and, if the degree of coincidence satisfies a predetermined condition, generating second feedback information indicating an evaluation of the input operation to the user, and causing the output unit to output the second feedback information.

[0008] According to the present invention, it is possible to perform physical function training based on the user's hearing, for example, brain function training through hearing.

[0009] FIG. 1 is a diagram showing an outline (1) of an information processing system according to an embodiment of the present invention. FIG. 2 is a diagram showing an outline (2) of an information processing system according to an embodiment of the present invention. FIG. 3 is a diagram showing an outline (4) of an information processing system according to an embodiment of the present invention. FIG. 4 is a diagram showing an outline (5) of an information processing system according to an embodiment of the present invention. FIG. 5 is a diagram showing an outline (6) of an information processing system according to an embodiment of the present invention. FIG. 6 is a diagram showing an outline (7) of an information processing system according to an embodiment of the present invention. FIG. 7 is a diagram showing an outline (8) of an information processing system according to an embodiment of the present invention. FIG. 9 is a diagram showing an outline (10) of an information processing system according to an embodiment of the present invention. FIG. 11 is a diagram showing an outline (12) of an information processing system according to an embodiment of the present invention. FIG. 13 is a diagram showing an outline (14) of an information processing system according to an embodiment of the present invention. FIG. 15 is a diagram showing an outline (16) of an information processing system according to an embodiment of the present invention. FIG. 17 is a diagram showing an outline (18) of an information processing system according to an embodiment of the present invention. FIG. 1 is a block diagram showing the configuration of an information processing system according to an embodiment of the present invention. FIG. 1 is a diagram showing an example of trajectory data displayed on a display unit. It is a figure explaining a method of calculating the degree of coincidence between a first coordinate and a second coordinate It is a figure showing a state in which feedback information is output It is a flowchart showing the flow of processing executed in an information processing device.

[0010] An overview of an information processing system according to one embodiment of the present invention will be described below. Fig. 1 is a diagram showing an overview (1) of an information processing system according to one embodiment of the present invention. The overview (1) of an information processing system according to one embodiment of the present invention is as shown in Fig. 1.

[0011] 2 is a diagram showing an outline (2) of an information processing system according to one embodiment of the present invention. The outline (2) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0012] 3 is a diagram showing an outline (3) of an information processing system according to one embodiment of the present invention. The outline (3) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0013] 4 is a diagram showing an outline (4) of an information processing system according to one embodiment of the present invention. The outline (4) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0014] 5 is a diagram showing an outline (5) of an information processing system according to one embodiment of the present invention. The outline (5) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0015] 6 is a diagram showing an outline (6) of an information processing system according to one embodiment of the present invention. The outline (6) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0016] 7 is a diagram showing an outline (7) of an information processing system according to one embodiment of the present invention. The outline (7) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0017] 8 is a diagram showing an outline (8) of an information processing system according to one embodiment of the present invention. The outline (8) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0018] 9 is a diagram showing an outline (9) of an information processing system according to one embodiment of the present invention. The outline (9) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0019] 10 is a diagram showing an overview (10) of an information processing system according to one embodiment of the present invention. The overview (10) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0020] 11 is a diagram showing an overview (11) of an information processing system according to one embodiment of the present invention. The overview (11) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0021] 12 is a diagram showing an overview (12) of an information processing system according to one embodiment of the present invention. The overview (12) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0022] 13 is a diagram showing an outline (13) of an information processing system according to one embodiment of the present invention. The outline (13) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0023] 14 is a diagram showing an overview (14) of an information processing system according to one embodiment of the present invention. The overview (14) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0024] 15 is a diagram showing an overview (15) of an information processing system according to one embodiment of the present invention. The overview (15) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0025] 16 is a diagram showing an overview (16) of an information processing system according to one embodiment of the present invention. The overview (16) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0026] 17 is a diagram showing an overview (17) of an information processing system according to one embodiment of the present invention. The overview (17) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0027] 18 is a diagram showing an overview (18) of an information processing system according to one embodiment of the present invention. The overview (18) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0028] Although one embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and modifications, improvements, etc. within the scope of achieving the object of the present invention are included in the present invention.

[0029] For example, the above-described series of processes can be executed by hardware or software. In other words, the functional configurations shown in Figures 1 to 18 are merely illustrative and are not particularly limited. That is, it is sufficient for the information processing system to have the functionality to execute the above-described series of processes as a whole, and the functional blocks used to realize these functions are not particularly limited to the examples shown in Figures 1 to 18. Furthermore, the locations of the functional blocks and databases are not particularly limited to those shown in Figures 1 to 18 and may be arbitrary. For example, at least some of the functional blocks and databases required to execute various processes may be transferred to a user terminal or the like. Conversely, the functional blocks and databases of the user terminal may be transferred to a server or the like. Furthermore, a single functional block may be configured as hardware alone, software alone, or a combination thereof.

[0030] When a series of processes is executed by software, the programs constituting the software are installed onto a computer or the like from a network or a recording medium. The computer may be a computer incorporated into dedicated hardware. The computer may also be a computer capable of executing various functions by installing various programs, such as a server, a general-purpose smartphone, or a personal computer.

[0031] The recording medium containing such a program may be configured as a removable medium (not shown) that is distributed separately from the device main body in order to provide the program to users, etc., or may be configured as a recording medium that is pre-installed in the device main body and provided to users, etc. The computer program for executing the processing of each unit of the processor that constitutes the calculation unit 8 of the information processing system S may be provided in a form recorded on a computer-readable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. The computer program may also be provided as a program product. The specific configuration of the information processing system will be described below.

[0032] 19, the information processing system S includes an information processing device 1 operated by a user and a server device 20 communicably connected via a network W. The information processing system S is configured to enable training of the user's physical functions, including hearing, based on input operations by the user. In the information processing system S, at least one of the information processing device 1 and the server device 20 is configured to evaluate the input operations of the user.

[0033] The server device 20 is configured by an information communication device, such as a personal computer, on which a computer program can be installed. The server device 20 includes a calculation unit 21 that executes processes necessary for calculations and various controls. The calculation unit 21 is configured by at least one hardware processor, such as a central processing unit (CPU). The calculation unit 21 may be realized by hardware (including circuitry), such as a large-scale integration (LSI), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a graphics processing unit (GPU), or may be realized by a combination of software and hardware.

[0034] The server device 20 includes a storage unit 22 that stores data and computer programs necessary for calculations and control, and a database including training questions and answers. The storage unit 22 is configured with a non-transitory storage medium such as a hard disk drive (HDD) or a flash memory. The server device 20 includes a communication unit 23 that can be connected to a network W. The communication unit 23 is configured with, for example, a communication device that can be connected to the network W via a wired or wireless connection. The calculation unit 21 communicates with the information processing device 1 via the network W and transmits training data.

[0035] The calculation unit 21 may perform calculation processing based on training data and cause the information processing device 1 to output the calculation results via the network W. The calculation unit 21 may perform calculation processing in cooperation with the information processing device 1 via the network W. The calculation unit 21 may transmit an updated computer program to the information processing device 1.

[0036] The information processing device 1 is configured as an information communication device, such as a smartphone or a personal computer, on which a computer program can be installed. The information processing device 1 may be a device specifically configured for training the user's physical functions. The information processing device 1 includes a calculation unit 8 that executes processing necessary for calculations and various controls. The calculation unit 8 is configured by at least one hardware processor, such as a CPU. The calculation unit 8 may be realized by hardware, such as an LSI, an ASIC, an FPGA, or a GPU, or may be realized by a combination of software and hardware.

[0037] The information processing device 1 includes a storage unit 9 that stores data and computer programs necessary for calculations and control, as well as a database including training questions and answers (described below). The storage unit 9 is configured with a non-transitory storage medium such as a hard disk drive or flash memory. The information processing device 1 includes a communication unit 6 that can be connected to a network W. The communication unit 6 is configured with, for example, a communication device that can be connected to the network W via a wired or wireless connection. The calculation unit 8 communicates with a server device 20 via the network W to receive training data.

[0038] The information processing device 1 includes an input unit 2 that accepts user operations. The input unit 2 is configured with a device capable of inputting user input operations, such as a touch panel, touchpad, mouse, or keyboard. The input unit 2 is provided in the information processing device 1. The input unit 2 may be configured with a device communicatively connected to the information processing device 1 via a wireless or wired connection. The input unit 2 may be configured with a controller having a cross key or joystick, or a controller based on an acceleration sensor that detects the user's body movements. Any acceleration sensor can be used as long as it can detect the user's body movements. The input unit 2 may be configured with a camera, or the user's movements may be input by image recognition of image data by the calculation unit 8. The input unit 2 may be configured to input voice based on the user's speech, recognize the content of the speech, and perform input operations. The input unit 2 may be configured to input specific voice information, recognize the content of the voice information generated based on a combination of one or more parameters, such as frequency, acoustic power, acoustic spectrum, and emission pattern, and perform input operations.

[0039] The input unit 2 may be configured with a motion capture device that recognizes markers attached to the user's body and detects the user's body movements. The input unit 2 may be configured to detect the user's body based on a pressure sensor or a vibration sensor. The input unit 2 may be configured to accept input operations based on a device that acquires signals from the user's body non-invasively and / or invasively, such as an electromyography (EMG), an electroencephalogram (EEG), or an electrooculography (EOG). The input unit 2 may be configured with any device that can accept two-dimensional or three-dimensional input operations by the user.

[0040] The information processing device 1 includes an output unit 3 that outputs information. The output unit 3 is provided in the information processing device 1. The output unit 3 may be configured by a device communicatively connected to the information processing device 1. The output unit 3 includes, for example, a display unit 4 that outputs information based on an image. The display unit 4 is configured by a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The display unit 4 may be configured by a touch panel that can accept input operations. In this case, the display unit 4 also functions as the input unit 2. The display unit 4 is communicatively connected to the information processing device 1 and may be configured by a projector that can project images or a head-mounted display device that is worn on the user's head.

[0041] The output unit 3 includes a speaker 5 that outputs sound. The speaker 5 outputs information based on the sound. The output unit 3 may be configured as a device that applies stimuli to the user's body non-invasively and / or invasively. The output unit 3 may be configured as a device that applies physical and / or neural stimuli based on one or more combinations of visual stimuli, auditory stimuli, vibration stimuli, olfactory stimuli, taste stimuli, temperature stimuli, electrical stimuli, magnetic stimuli, gravitational stimuli, acceleration stimuli, pressure stimuli, brain wave stimuli, balance sense stimuli, muscle stimuli, visceral stimuli, light stimuli, sound stimuli, visual cortex stimuli, auditory cortex stimuli, intestinal bacteria administration, drug administration, etc.

[0042] The following describes a process for training a user, which is executed in the information processing device 1. In this embodiment, a process in which the calculation unit 8 evaluates an input operation performed by a user on a touch panel is illustrated. The calculation unit 8 starts training, for example, in a state in which the trajectory of a virtual target is not displayed on the display unit 4. In the training, the user performs an input operation in which the user traces the trajectory of the virtual target with their finger on the display unit 4 to estimate the trajectory of the virtual target. The calculation unit 8 compares an input trajectory input based on the user's input operation with the trajectory of the virtual target, scores the accuracy of the input operation, and provides feedback to the user. The process of the calculation unit 8 may be executed by the server device 20.

[0043] As shown in FIG. 20 , the display unit 4 is configured with a touch panel that accepts input operations for training by the user. The calculation unit 8 launches a training application based on the user's input operations. The calculation unit 8 acquires trajectory data M1 corresponding to a virtual target from the storage unit 9. The calculation unit 8 may also acquire the trajectory data M1 from the server device 20. The virtual target represents, for example, characters, figures, etc., based on multiple linear trajectories. The calculation unit 8 suggests the virtual target to the user by, for example, outputting a quiz-style question to the output unit 3. The trajectory data M1 is data indicating the spatial position of the virtual target. The trajectory data M1 may be fixed over time or may be dynamic over time. The trajectory data M1 may change over time. The calculation unit 8 performs a calculation process to match the trajectory data M1 of the virtual target with the coordinates of the display unit 4. The calculation unit 8 calculates first coordinates corresponding to the trajectory data. The calculation unit 8 converts the first coordinates into data corresponding to a position on the screen of the display unit 4 and makes them correspond to the display unit 4. The calculation unit 8 calculates second coordinates in the virtual space corresponding to the input trajectory displayed on the display unit 4, based on the input operation input to the input unit 2. In the illustrated example, the first coordinates and the second coordinates are, for example, two-dimensional coordinates. The first coordinates and the second coordinates may be coordinates of two or more dimensions.

[0044] The calculation unit 8 calculates a plurality of first coordinates that match the trajectory data M1. The calculation unit 8 does not display the trajectory data M1 of the virtual target based on the first coordinates on the display unit 4, and instead outputs instruction information to the output unit 3 that instructs the user to perform an input operation to trace the trajectory data M1 of the virtual target. The calculation unit 8 may not only not display the trajectory data M1 on the display unit 4, but also display at least a portion of the trajectory data M1 on the display unit 4. The calculation unit 8 may display at least a portion of the trajectory data M1 on the display unit 4 when predetermined conditions are met, such as at a predetermined timing from the start of the input operation, within a predetermined time from the start of the input operation, or when a predetermined input operation is performed. The calculation unit 8 may switch between a state in which at least a portion of the trajectory data M1 is displayed on the display unit 4 and a state in which it is not displayed. The instruction information may be output in any form, such as text information, graphic information, audio information, or electrical stimulation, as long as it instructs the user to perform an input operation. The calculation unit 8 may generate information MA indicating the start position of the trajectory data M1 and output it from the output unit 3. The start position may be the beginning of the trajectory data M1 or any position other than the beginning of the trajectory data M1. The calculation unit 8 may not output the information MA indicating the start position of the trajectory data M1 from the output unit 3. The calculation unit 8 may display the information MA indicating the start of the trajectory data M1 and then cause it to disappear from the output unit 3 after a predetermined time has elapsed. The user starts an input operation by tracing the trajectory data M1 on the display unit 4 using an input means such as a finger or a stylus. The user may start the input operation from any position on the trajectory data M1. The user continuously and / or discretely traces the display unit 4 to generate an input trajectory. The calculation unit 8 displays the input trajectory K1 input based on the user's input operation on the display unit 4. The calculation unit 8 may not only display the input trajectory K1 on the display unit 4, but may also not display the input trajectory K1 on the display unit 4. The calculation unit 8 may display at least a part of the input trajectory K1 on the display unit 4. The calculation unit 8 may switch between a state in which at least a part of the input trajectory K1 is displayed on the display unit 4 and a state in which it is not displayed. The input operation may be performed not only by tracing the display unit 4, but also by other methods based on the configuration of the input unit 2.

[0045] As shown in FIG. 21 , the calculation unit 8 generates a second coordinate G2 of the input trajectory K1 input to the input unit 2 so as to follow the trajectory data M1 at a predetermined timing based on the user's input operation. The predetermined timing may be, for example, a sampling period for accepting the input operation, or may be continuous. The second coordinate G2 indicates a spatial position in the virtual space relative to the first coordinate G1. The calculation unit 8 may generate the second coordinate G2 continuously or discretely. The calculation unit 8 may calculate an input velocity V of the input operation. For example, the calculation unit 8 calculates the input velocity V based on the path length of the input trajectory K1 input per unit time. The calculation unit 8 calculates the input velocity V using the average, median, maximum, minimum, or the like of the input velocity over a predetermined period. The calculation unit 8 generates image data of the input trajectory K1 based on the second coordinate G2 and displays the image data on the display unit 4. The calculation unit 8 generates image data of the input trajectory K1 based on, for example, a line diagram.

[0046] The calculation unit 8 calculates a degree of match indicating the spatial relationship between the first coordinate G1 and the second coordinate G2 at each predetermined timing while the input operation is being performed. The degree of match is a score indicating the degree to which the first coordinate G1 and the second coordinate G2 match. The calculation unit 8 calculates the degree of match based on, for example, the relative distance D between the first coordinate G1 and the second coordinate G2. The calculation unit 8 calculates the degree of match such that the smaller the distance D, the higher the degree of match. The calculation unit 8 may calculate the degree of match based on a combination of one or more of the relative distance, relative angle, relative velocity, relative acceleration, etc., in the relationship between the first coordinate G1 and the second coordinate G2. The calculation unit 8 may also calculate the degree of match using other methods as long as they can calculate the relationship between the first coordinate G1 and the second coordinate G2. The calculation unit 8 may calculate the degree of match at a certain moment, or may calculate the degree of match at each predetermined time interval based on the average, median, maximum, or minimum value.

[0047] The calculation unit 8 generates feedback information indicating an evaluation of the input operation to the user according to the degree of coincidence, and causes the output unit 3 to output the feedback information. The calculation unit 8 causes the output unit 3 to output the feedback information based on information combining at least one of, for example, audio information, text information, and image information. The calculation unit 8 may output the feedback information from the output unit 3 based on physical and / or neurological stimulation based on a combination of one or more of visual stimulation, auditory stimulation, vibration stimulation, olfactory stimulation, taste stimulation, temperature stimulation, electrical stimulation, magnetic stimulation, gravitational stimulation, acceleration stimulation, pressure stimulation, brain wave stimulation, balance stimulation, muscle stimulation, visceral stimulation, light stimulation, sound stimulation, visual cortex stimulation, auditory cortex stimulation, administration of intestinal bacteria, drug administration, etc.

[0048] As shown in FIG. 22 , the calculation unit 8 outputs first feedback information F1 based on audio information to the output unit 3 according to the degree of matching. The calculation unit 8 outputs one or more pieces of first feedback information F1 midway between the start and end of input of the input trajectory K1. For example, the calculation unit 8 generates first feedback information F1 based on audio, with the output state adjusted according to the degree of matching, and outputs the first feedback information F1 to the output unit 3. For example, the calculation unit 8 outputs the first feedback information F1 based on audio information so that the sound level changes according to the degree of matching. The first feedback information F1 based on audio information is generated using an alarm sound, language, or the like. For example, the calculation unit 8 may generate the first feedback information F1 by adjusting sound-related parameters such as volume, frequency, and audio spectrum so that the sound is comfortable for the user. For example, the calculation unit 8 outputs the first feedback information F1 so that the sound level increases as the degree of matching increases. For example, the calculation unit 8 generates the first feedback information F1 so that the sound level decreases as the degree of matching decreases. The calculation unit 8 may not only adjust acoustic parameters according to the degree of match, but also adjust the output state of the first feedback information F1 based on any method. The calculation unit 8 outputs the first feedback information F1 at any time interval. The calculation unit 8 may output the first feedback information F1 continuously or intermittently. The calculation unit 8 may generate the first feedback information F1 based on linguistic information. The calculation unit 8 may generate the first feedback information F1 based on linguistic information that guides the position, direction, and speed of the user's input operation so as to increase the degree of match.

[0049] The calculation unit 8 generates not only first feedback information F1 based on audio information, but also second feedback information F2 to improve the effectiveness of the user's training based on the input operation, and outputs the information to the display unit 4. For example, when the degree of match satisfies a predetermined condition, the calculation unit 8 generates second feedback information F2 based on image information and outputs the information to the display unit 4. The calculation unit 8 outputs one or more pieces of second feedback information F2 midway between the start and end of input of the input trajectory K1. For example, based on a comparison result between the degree of match and a preset threshold, the calculation unit 8 outputs the second feedback information F2 to the display unit 4 when a predetermined condition is met, that is, the degree of match is greater than the threshold. The calculation unit 8 generates the second feedback information F2 based on one or more combinations of outputtable information such as text information, graphic information, and audio information. For example, the calculation unit 8 outputs the second feedback information F2 based on image information to the display unit 4. For example, the calculation unit 8 increases the number of shapes, such as stars, displayed on the display unit 4 as the degree of match increases. The calculation unit 8 generates second feedback information F2 for any number of graphics and outputs it to the display unit 4. If the calculation unit 8 generates second feedback information F2 including audio information, the calculation unit 8 may output the audio information from the speaker 5. The calculation unit 8 may generate second feedback information F2 based on not only graphics but also a change in the color of the entire screen, a change in the color of a partial area of ​​the screen, or a change in the brightness of the screen. The calculation unit 8 may change visual information such as the thickness or color of the line of the input trajectory K1 in accordance with the output of the second feedback information F2. The calculation unit 8 outputs the second feedback information F2 at any time interval. The calculation unit 8 may output the second feedback information F2 based on any number of output times or output frequency. The calculation unit 8 may output the second feedback information F2 based on a number of output times or output frequency that the user can predict, or based on a number of output times or output frequency that the user cannot predict.

[0050] The calculation unit 8 may provide a predetermined delay state from the timing at which the input operation is performed, and output the second feedback information F2 to the output unit 3. Here, the timing at which the input operation is performed includes one or more timings in one input trajectory K1, such as the timing at which the input operation is performed at every predetermined sampling period, the timing at which the input operation is performed at every predetermined distance on the path of the input trajectory K1, or the timing at which the input operation is performed at every predetermined position and / or every predetermined area from the starting position. The delay state includes a concept of a temporal and / or spatial delay after the timing at which the input operation is performed, such as a time delay from the input timing or a distance delay after moving an arbitrary distance from the input position.

[0051] The calculation unit 8 may cause the output unit 3 to output the second feedback information F2 at the timing when the input operation is performed (delay time = 0). The calculation unit 8 may cause the output unit 3 to output the second feedback information at any amount or frequency. The calculation unit 8 may cause the output unit 3 to output the second feedback information F2 when the degree of match reaches a maximum value. The calculation unit 8 may cause the output unit 3 to output the second feedback information F2 when a predetermined time or a predetermined time range occurs while the input operation is being performed. The calculation unit 8 may cause the output unit 3 to output the second feedback information F2 based on a delay time (seconds) that is equal to or greater than 0. The calculation unit 8 may cause the display unit 4 to display the second feedback information F2 including image information based on the delay time, and may cause the speaker 5 to output the second feedback information F2 including audio information based on the delay time (seconds). The calculation unit 8 may output the second feedback information F2 including image information and the second feedback information F2 including audio information at the same timing based on the same delay time, or at different timings based on different delay times. The calculation unit 8 may output the second feedback information F2 when a predetermined number of input operations, which marks a milestone in one task, are completed. The calculation unit 8 may output the second feedback information F2 when the input trajectory K1 reaches a predetermined area. The calculation unit 8 may output the second feedback information F2 when the trajectory data M1 reaches a predetermined area.

[0052] The calculation unit 8 may output the second feedback information F2 when the degree of matching satisfies a predetermined condition, such as when the degree of matching reaches a predetermined value, when the degree of matching is within a predetermined range, when the degree of matching is outside a predetermined range, etc. The calculation unit 8 may output the second feedback information F2 when the input trajectory K1 satisfies a predetermined condition, such as predetermined coordinate values, shape, position, velocity, acceleration value, within a predetermined range, or outside a predetermined range. The calculation unit 8 may output the second feedback information F2 when a movable display image other than the input trajectory K1 changes to a predetermined color or shape at a predetermined timing.

[0053] The calculation unit 8 may output the second feedback information F2 when a combination of one or more conditions is met, such as when the user touches the input unit 2, when the degree of contact with the input unit 2 meets a predetermined condition, when the user no longer touches the input unit 2, or when a predetermined or greater acceleration is applied to the information processing device 1 or the input unit 2. The calculation unit 8 may output the second feedback information F2 when the state of the input operation meets a predetermined condition, such as a predetermined value, within a predetermined range, or outside a predetermined range, at which the degree of match is high.

[0054] The calculation unit 8 may output the second feedback information F2 when the user performs a specific operation, such as tapping on the display unit 4, pressing with a pressure greater than or equal to a predetermined pressure, pressing with a pressure less than a predetermined pressure, removing the operating means from the display unit 4, or shaking the information processing device 1. The calculation unit 8 may output the second feedback information F2 when the user utters a specific sound. The calculation unit 8 may output the second feedback information F2 when the user's hearing function has a specific performance or when the calculation unit 8 recognizes a response that exceeds or falls below the specific performance. The calculation unit 8 may output the second feedback information F2 when the input trajectory K1 moves to a predetermined position. The calculation unit 8 may output the second feedback information F2 when a calculated value, such as a score related to the input operation of the input trajectory K1, meets a predetermined condition, such as a specific value, a value within a predetermined range, or outside a predetermined range. The calculation unit 8 may output the second feedback information F2 when a specific sound is reproduced.

[0055] The calculation unit 8 calculates the input speed V at which the input trajectory K1 is input at predetermined intervals and compares the input speed with a threshold. The calculation unit 8 may output second feedback information F2 to the output unit 3, including information for adjusting the input speed V, based on the comparison result between the input speed V and a preset threshold. It is generally known, based on test results and surveys such as subjective evaluations of hearing in daily life, that when a user is given feedback that stimulates dopamine production in the brain, the effectiveness of training for auditory physical functions such as hearing ability (e.g., hearing ability in noise (unit: dB)) is enhanced. For example, when the user is experiencing first feedback information F1 based on audio information, the calculation unit 8 may output second feedback information F2 including information for guiding the user to slow the input speed V of the input operation compared to the threshold.

[0056] The calculation unit 8 may output second feedback information F2 to the output unit 3, the second feedback information F2 including information for adjusting not only the input velocity V but also the position of the second coordinate G2 and the input acceleration. Information for adjusting the input velocity V includes, for example, information for delaying the input operation compared to the current state, information for speeding up the input operation compared to the current state, information for maintaining the input operation, etc. For example, when the calculation unit 8 determines that the input trajectory K1 generated by the user's input operation is within a predetermined range from the end of the trajectory data M1, the calculation unit 8 generates information notifying the user of the end and outputs the information from the output unit 3. The calculation unit 8 may generate information notifying the user of the end at any timing and output the information from the output unit 3.

[0057] 23 is a flowchart showing the flow of processing of an information processing method executed in information processing device 1. The information processing method is executed based on a computer program that can be installed in a computer mounted on information processing device 1. The computer program causes a calculation unit 8 of information processing device 1 to execute the following processes.

[0058] The calculation unit 8 acquires trajectory data M1 corresponding to the virtual target from the storage unit 9 or the server device 20 (S100). The calculation unit 8 calculates a first coordinate G1 corresponding to the trajectory data M1 (S102). The calculation unit 8 accepts an input operation from the user (S104). Based on the input operation, the calculation unit 8 calculates a second coordinate G2 corresponding to the input trajectory K1 input to the input unit 2 so as to follow the trajectory data at predetermined intervals (S106). The calculation unit 8 calculates the degree of coincidence between the first coordinate G1 and the second coordinate G2 (S108). The calculation unit 8 generates first feedback information, the output state of which has been adjusted according to the degree of coincidence, based on the voice, and causes the output unit 3 to output the first feedback information (S109).

[0059] The calculation unit 8 determines whether the calculated degree of match is greater than a preset threshold (S110). If the degree of match is equal to or less than the threshold, the calculation unit 8 returns the process to S104. If the degree of match is greater than the threshold, the calculation unit 8 generates feedback information and outputs it from the output unit 3 (S112). The calculation unit 8 determines whether an end condition is met (S114), and if the end condition is met, the process ends. Here, according to the technology described in Non-Patent Document 1, feedback information is output after the user has finished tracing the trajectory data. In contrast, according to the information processing device 1, first feedback information F1 and second feedback information F2 are output while the user is tracing the trajectory data, thereby enabling an evaluation of the input operation to be obtained in real time.

[0060] As described above, the information processing device 1 can perform physical function training based on the user's hearing, such as brain function training through hearing. The information processing device 1 can perform brain function training through the user's hearing by determining the content of the user's input operation, generating feedback information based on the determination result, and guiding the input operation. The information processing device 1 can train the user's physical function based on hearing by generating first feedback information F1 based on voice. The information processing device 1 can improve the effectiveness of the user's training by generating second feedback information F2 to prompt the input operation to a predetermined state. The information processing device 1 can perform brain function training through the user's hearing by guiding the input operation based on the second feedback information F2.

[0061] Figure 1: Training to improve hearing. Figure 2: Fun hearing training involves finding letters and shapes using sound and completing quizzes using the discovered letters and shapes. Auditory training using sound can easily become monotonous and repetitive, making it difficult to continue. Therefore, combining auditory training with letter and shape discovery and quizzes makes auditory training fun and continuous. Auditory training involves having users trace their finger across the screen to find visually hidden letters and shapes. By changing the sound depending on the distance between the finger and the hidden letters or shapes, users can find them using the sound. In this way, auditory training is achieved by concentrating on the sound and moving their finger. The sounds and tracing actions used in auditory training utilize neumo's existing technology. The application documents are reproduced later in this document. Furthermore, there is a step in which the user rotates the letters and shapes they find by tracing with their finger, and a step in which they use the discovered letters and shapes to solve quizzes. Specific examples of these steps are provided on the following pages. Figure 3: Making letters appear and completing words: Step (1) Find letters by tracing lines using the sound What the user sees Task - The letters to be traced are broken down into a set of lines, rotated at any angle. Rotation is not required. - Trace the N lines that make up the letter in order. You can trace all N lines in order, or just a few M lines. The lines can be straight or curved, and any shape is fine. You can also trace multiple lines in one stroke. - The letters to be traced are visually hidden, and you need to trace the lines using the sound. - When tracing, the starting point of the line is displayed, so you start tracing from that starting point and reach the end point. Once you reach the end point, the task of tracing the next line begins. - The starting point can be a point, or it can be something that indicates a certain area, such as a circle. The area can be indicated by a shape other than a circle. - The end point can be visually represented by a point or line, or it can be invisible. You can also search for the end point without displaying it. Once you have finished tracing, you can check the path you have traced. You can either not display it while tracing, or display the path in order as you trace.- The correct line may be displayed, or the user may proceed without it. What the user should trace Figure 4 Making letters appear to complete the word: Step (2) Rotating letters to make them upright What the user sees Task - If not all N lines that make up the letter to be traced are traced, the lines that were not traced are displayed as the correct answer. - Using sound as a guide, rotate the letter to make it upright. By changing the sound depending on the angle at which the letter's upright position is deviated from the correct position, the user can use the sound to find the correct angle. - This step is not required and can be skipped. What the user should trace A sound will indicate that a certain direction is the correct answer. Figure 5 Making letters appear to complete the word: Step (3) Selecting the letter that was traced What the user sees Task - Answer what letter you traced. There are several ways to answer, such as selecting from multiple options, or freely answering by keyboard input, voice input, or handwriting input.・If the answer is incorrect, there are both methods that allow only one answer and methods that allow multiple answers. ・If the answer is incorrect, the correct answer may or may not be revealed. Since revealing the correct answer makes the next question easier, this can be used to adjust the difficulty setting. ・This step is not required and can be skipped. ・Repeat steps (1) to (3) depending on the number of characters in the question. Figure 6: Make the characters appear and complete the word: Step (4) Guess the kanji from the characters obtained. What the user sees Task ・A quiz is given using the characters found in steps (1) to (3). ・In a quiz to guess the reading of a kanji, the characters found in steps (1) to (3) are used to display part of the reading for the kanji, and the user must guess the missing characters. ・The kanji to be read can be a single character, or it can be a multi-character word, four-character idiom, or sentence.・The characters found in steps (1) to (3) can be displayed as part of the reading, or can be used as one of the candidates. ・For the quiz where you have to guess the character in the blank, you can choose from the options. You can also choose not to display options and let the user enter the answer freely.- If the answer is correct, a sound or visual will be displayed to indicate the answer. For example, the sound or visual of fireworks can be presented. Other methods besides fireworks are also acceptable. If the answer is incorrect, the user is notified that the answer is incorrect. If the answer is incorrect, both the correct answer and the next answer can be displayed. Figure 7: The rotation step can be performed in a different way. - The step of rotating the letters or shapes found by tracing can be replaced with another method shown below, or it can be omitted. The location of the correct answer can be found using sound, which can be used to train the auditory sense. It is also possible to use a method where the location is found without using sound, relying on visuals or vibrations. - The screen can be moved to match the correct position in one, two, or three-dimensional space. For example, it can be used in cases such as a jigsaw puzzle, where pieces are placed in the correct position. - The letters or shapes can be enlarged or reduced to match the correct size. - Rotate in a 3D space centered on the X and Y axes and align with the correct position. - Flick letters or shapes with your finger or operate a spring like a pinball to send them flying and hit the correct target or put them in the hole. The correct direction and strength can also be determined by sound. - Find the correct stroke order for kanji or characters. A correct sound can be played if the stroke order is correct, and an incorrect sound can be played if the stroke order is incorrect. Figure 8 Calculating the score for the tracing result - For the step of tracing and finding letters or shapes, a function can be provided that quantifies how accurately the tracing was performed and provides feedback to the user. One of the following methods can be used to quantify the score, or other methods can be used.・Use spatial distance ■There is a method of quantifying the distance between the correct letter or shape and the traced line. ・1. Convert the traced line into a set of points. ・2. For each point, calculate the distance to the nearest correct letter or shape. ・3. Add up the calculated distances to obtain a cumulative difference value. ・4. The average of the calculated distances can be used as the difference value. ・5. It is also possible to calculate using only calculated distances within a certain range. For example, only those close to the correct answer (number 1 or less) are used as the cumulative value. Or, those far from the correct answer (number 10 or more) can be accumulated and used as a deduction system.■One method is to quantify the time it takes to successfully trace the correct letter or shape. ・1. Define the range of the correct answer for the letter or shape. The range can be either a binary value of 1 or 0, or a decimal value between 1 and 0, or a sloped range such as 100 to 0. ・2. By using the total tracing time as the denominator and the time spent tracing the correct position as the denominator, the time spent tracing the correct position can be quantified. In this case, the slope defined in 1 can also be used to calculate the value. ■One method is to quantify the difference between the position at which tracing ends and the end position of the correct letter or shape. ・Rather than using data from the tracing path, it is possible to calculate the distance between the position at which tracing ends and the end position of the correct letter or shape, and use this as a numerical value to evaluate the result. ・It is also possible to provide a function that quantifies how accurately the steps to rotate the letter or shape were answered and provide feedback to the user. - The smaller the difference between the correct angle and the angle you answered, the higher the score. Figure 9: Various patterns can be used for quizzes using letters and shapes found by tracing. - We have provided an example of how to trace letters, but any of the following shapes are acceptable, not just letters: - Japanese hiragana, katakana, kanji, and numbers - Alphabet (this may include language-specific characters other than the basic alphabet, such as French spelling symbols and ligatures or German umlauts) - Characters of other languages ​​(any language characters are acceptable, including simplified Chinese, traditional Chinese, Arabic, Devanagari, Greek Mandarin, and Hangul) - Abstract shapes (circles, triangles, etc.). One-, two-, or three-dimensional shapes are acceptable. - Line segments in photographs and illustrations (any shape that can be expressed with lines can be included, such as anime characters, animal illustrations, icons, products, constellations, family crests and coats of arms, tangrams, and the topography of countries and prefectures) - The correct route in a map or maze - There are various ways to create quizzes using the letters and shapes that have been discovered, such as the following.・By tracing the steps, you can find hiragana, katakana, numbers, the alphabet, and characters from various other languages, and then ask questions like the following. The following is just one example, and other quizzes can be made. You can choose quizzes where the answer can be determined just by tracing the characters you find, or quizzes where you need to answer with only partial knowledge. ■Guess the reading of kanji or compound words ■Make it a crosswood puzzle and guess where the character or word goes ■Use the characters you trace to find or create kanji, words, or sentences. You can rearrange the characters, or find radicals to create kanji. You can also choose a poem from the Hyakunin Isshu. ■Fill-in-the-blank quiz using the characters you trace ■Math problems and number puzzles using the numbers you trace ■Trace kanji or characters and guess what they are ■Guess what the characters you trace and find are an abbreviation for ・By tracing the steps, you can find shapes and illustrations, and then ask questions like the following ■A quiz to guess what shape is hidden. You can trace all the lines, or just some of the lines, with some of the lines displayed beforehand. Or you can have only some of the lines displayed and have to predict the whole picture and guess. For example, you can make questions such as guessing what animal it is or what constellation it is. ■It can also be a quiz where you guess the correct answer by combining multiple shapes you find by tracing. ■It can also be a jigsaw puzzle, block puzzle, or Tetris where you combine shapes you find by tracing. ■You can have students compare the sizes of shapes they find by tracing, or find shapes that are similar. ・It can also be a Wiz game where students find the correct path by tracing steps on a map or maze. ■A quiz where students guess the destination on the map or the route to the exit of a maze, using the ability to trace and find the path on a map or maze as a hint. ・It can also be a quiz where students listen to words in a noisy environment, using the letters they find by tracing as a hint to answer. Figure 10 Elements for making training enjoyable and continuous (1) ・Increase motivation by visualizing the accuracy of tracing and keeping a record ・Give feedback by quantifying how well they were able to trace letters and shapes ・Record letters and shapes that they have traced in the past as a history, allowing them to look back on it later, allowing students to feel an improvement in their listening ability.- Badges are awarded based on the difficulty and accuracy of the tracing. Various badges (high accuracy, tracing a difficult sound, tracing in a short time, high quiz correct answer rate, etc.) are available and can be collected. Motivation is increased by displaying the next badge that could be obtained. - Sharing and communication with others increases motivation. - Tracing scores and the quiz correct answer screen can be shared with others. Results can be shared with other users registered as friends within the app, or via email, social media, LINE (registered trademark), and other external platforms. - Motivation is increased by notifying users when they receive a "like" or comment on a shared result. - Presenting the difficulty and level of the problem allows a sense of accomplishment. - The difficulty level is defined by the target sound and distracting sounds used in the tracing step, and can be understood as a level. The more clearly you trace, the higher the level you can move up to. Reaching a higher level as a result of hearing training provides a sense of accomplishment.・For Kanji quizzes, for example, once all questions equivalent to Kanji Kentei Grade 5 have been cleared, the next step should be to challenge questions equivalent to Kanji Kentei Grade 4. This allows the level of difficulty to be understood numerically, allowing a sense of accomplishment. The same level definition is used for other quizzes, such as crossword puzzles. ・For shape or illustration quizzes, the difficulty level is defined according to the fineness of the shapes to be traced and whether they are easy to understand. ・Motivation is enhanced by displaying the level of achievement compared to others. ・Rankings are displayed by comparing how well a person traced the same problem with a numerical value. Multiple types of rankings can be used, such as rankings for each character such as "a," overall rankings for multiple questions, monthly rankings, rankings by difficulty level, rankings for each quiz question, and rankings by quiz level. ・Ear age is displayed by comparing tracing accuracy with that of people of the same age or generation. ・Brain age is displayed by comparing the degree of accuracy of Wis compared with that of people of the same age or generation. Figure 11 Elements for Enjoyable and Continuous Training (2) ・Make people aware of time・Setting a time limit for each problem can encourage concentration. ・Suggesting the time required to solve a problem can be displayed. ・Training too quickly reduces the effectiveness of training, so it is possible to encourage time-consciousness by encouraging users to trace slowly. In this case, it is possible to combine messages such as "Take it easy" with the time. ・Displaying the daily training time can visualize efforts. ・Increasing the score for consecutive correct answers can increase the score. ・Increasing the score for daily training can also provide motivating feedback such as "Keep Going!!!" or "Perfect**!" when tracing a line segment almost correctly. Figure 12. Enhancing Training Effectiveness by Setting Difficulty Levels According to Ability. ・Measuring the user's hearing level in advance using a system separate from this system and incorporating the results as input can present problems of an appropriate difficulty level for each individual's hearing ability and hearing condition. Training effectiveness can be enhanced by adjusting the difficulty level to match each individual's ability. The following information can be used as input.・Pure tone audiometry results ・Speech audiometry results ・Auditory ability in noise measurements (speech audibility tests in steady noise, such as digits-in-noise, HINT, J-HINT, OLSA, J-Matrix, or speech audibility tests in noisy environments, such as digits-in-multi-talker-babble, speech-in-multi-talker-babble, QuicksIN) ・Left-right hearing difference measurements ・Auditory time processing measurements (measurement of thresholds using an amplitude modulation (AM) detection task, a task measuring time difference processing ability between the ears using NOSJT or NOST signals, frequency modulation threshold detection task, gap detection task, TFS1, TFS2, TFS-LF, interaural time difference detection task, frequency modulation direction detection threshold for seep sound) ・Cognitive function measurements, such as working memory and executive function ・Electroencephalography (EEG) results (ABR)・Since the user's hearing ability improves with continued training, by adjusting the difficulty level according to the training results, it is possible to present questions of a level appropriate to the individual's hearing ability and hearing condition. ・If the tracing accuracy exceeds the level-up threshold, the target sound or interfering sound is changed to allow for more difficult training. ・If the tracing accuracy falls below the level-down threshold, the target sound or interfering sound is changed to allow for less difficult training. ・These changes can be made based on the results of a single training session, multiple consecutive sessions, or multiple averages (e.g., the average for one day's training). ・The difficulty level is set taking into account the user's strengths / weaknesses for the target sound and the interfering sounds. ・Each person has strengths and weaknesses when it comes to target sounds; for example, they may be good at pitch changes in pure tones but have difficulty hearing human voices. If the level is set based on the user's strong target sound, they will be unable to hear the weak target sound. The overall level cannot be raised unless the user can also clear the weak paired sounds to a certain level. Similarly, if there are strong and weak interference sounds, the overall level of the interference sounds will not increase any further until the weak interference sound reaches a certain level. The difference in level between the target sound and the interference sounds will also be prevented from becoming too great. For example, if the target sound is level 20 and the interference sound is level 8, a constraint is set that the interference level must be increased to at least 10 in order to increase the level of the target sound. Figure 13 Other Functions (1) ・In addition to existing technology, the following methods can also be used for tracing: ・Use a stylus instead of a finger. ・Use equipment or software that detects gaze to replace the gaze tracing operation. ・Detect tracing using a device equipped with sensors, such as the Nintendo Switch (registered trademark) controller. Other controllers that can be used include existing products such as Meta Quest (registered trademark), Apple Vision Pro (registered trademark), Vive (registered trademark), PlayStation (registered trademark), XBOX (registered trademark), and pico, as well as specially developed controller devices.・The following may be used as methods to indicate the correct answer: ・Play a sound indicating the correct answer if the user has traced the correct position. ・Vibrate the device if the user has traced the correct position. ・Present some kind of visual element if the user has traced the correct position (the screen lights up, a message is displayed, a character is displayed, the screen appears to vibrate, etc.) ・Make it easier to see the correct position by changing the thickness or colour of the line displayed when the correct position is traced and when the incorrect position is traced. ・The following may be used as methods to indicate an incorrect answer: ・Play a sound indicating the incorrect answer if the user has traced the wrong place. ・Vibrate the device if the user has traced the wrong place. ・Present some kind of visual element if the user has traced the wrong place (the screen lights up, a message is displayed, a character is displayed, the screen appears to vibrate, etc.) ・The following may be used as the timing to indicate the above correct and incorrect answers: ・Present the answer immediately after the tracing action・Present when the correct or incorrect position is traced continuously (for example, tracing the correct position for 3 seconds) ・Present when the correct or incorrect position is traced multiple times. For example, nothing is presented the first time, and then a presentation is made from the second time onwards. The number of times can be changed to 3 or 4 times, etc. It is also possible to present only the second time, and not present again from the third time onwards. ・It is also possible to vibrate only when the user passes over the correct position while tracing and then returns to the same place again, so that the user knows that this is the correct position.・The correct or incorrect answer is displayed a few seconds after it is traced. ・It is displayed only under certain conditions, such as during the bonus stage or fever period, or when an item is used. Figure 14 Other functions (2) ・The fun can be increased by adding effects to the tracing action. ・A line is drawn at the tracing position. ・Visual elements are displayed in a location other than the tracing position (a character appears or moves, a picture is colored, a sparkling visual element is displayed, etc.). ・Music is played only for the question being traced, the number of notes in the music being played increases or decreases, or the speed or BPM of the music changes.- Tracing can be done by freely drawing lines or by filling in a grid. The grid can be any shape, including triangles, squares, hexagons, and octagons. Figure 15: neumo's unique technology (1) Training to adjust the left and right sound position using a finger using head-related transfer functions. - Distinguish the location of a sound source in one-dimensional, two-dimensional, and three-dimensional space using sound. - Distinguish the direction from which the sound is coming and perform some kind of manipulation to make the sound appear to be coming from directly in front. By carefully listening to the location of the sound source and performing some manipulation, auditory temporal processing can be trained. - For example, in one-dimensional training, a slider is displayed on the smartphone screen and manipulated to change the left and right position of the sound so that it is heard from the center. In two-dimensional training, by tapping the smartphone screen with a finger, the user can distinguish the above, below, left, and right positions of the tapped location and manipulate the sound so that it is directly in front. For 3D, a VR headset or a smartphone's AR function, gyro (angular velocity) sensor, or magnetic sensor can be used to position the sound source directly in front of the subject in 3D space. Devices: Any device capable of changing the position of sound in 1D, 2D, or 3D is acceptable, including smartphones, tablets, PCs, game consoles, VR, and real-world speakers. Devices capable of achieving stereo or spatial sound, such as stereo speakers, earphones, and headphones, are also acceptable. Target Sound Types: Target sounds can be generated as pure tones, sound sweeps, or spectrally modulated sounds, including FM-modulated sounds, AM-modulated sounds, musical instruments, music, human voices, animal sounds, and natural sounds. When using music, using or automatically generating music that balances predictability and uncertainty can stimulate dopamine production and improve learning efficiency. Interfering Sounds: Using interfering sounds to mask the target sound can increase the difficulty and enhance training effectiveness. Interfering sounds can be noises such as white noise, human voices such as speech bubbles, the sound of rain, crowds, music, or other sounds that interfere with the target sound.・To effectively improve hearing in everyday life, it is also a good idea to use fluctuating noise based on a statistical model of fluctuations in volume and acoustic features in actual cafes, bars, etc. Figure 16 neumo's unique technology (2) Operations performed while listening to sound ・The following are possible methods of operation while listening to the target sound: ・Continuously tracing on the tablet / smartphone screen to move to a certain point ・Continuously tracing on the tablet / smartphone screen to follow the moving target sound and keep it meeting the target conditions ・Continuously moving in the virtual space using the analog joystick on the gamepad ・Continuously moving in the virtual space using the analog joystick on the touch screen ・The user moves by walking within the physical space ・Operation methods ・Hold the smartphone horizontally and tilt it ・Hold the smartphone in the normal way to look at it and tilt it ・Hold the smartphone and move mainly horizontally (search for an object that appears on the screen in AR) ・Move the arm wearing the Apple watch (registered trademark) (such as stroking a parakeet or cat) ・Turning the neck and tilting the head while wearing a VR headset ・Operations using a VR controller・Movements such as tai chi and yoga (movements are detected by image processing from camera images) ・Voice is recorded with a microphone and the movements of the jaw muscles, tongue, lips, vocal cords, and respiratory muscles are detected. ・Operation methods using movement ・Hand and arm (finger) movements → Touch screen of smartphone ・Hand and arm movements holding a smartphone → Angular velocity sensor (gyro) of smartphone ・Hand and arm movements wearing an Apple Watch (registered trademark) → Angular velocity sensor (gyro) of Apple Watch (registered trademark) ・Voice (movement of jaw, tongue, lips, vocal cords, and respiratory muscles) → Microphone of smartphone ・Various body movements → Movement is detected by image processing from camera images ・Walking to different locations → Movement and position are detected by image processing from camera images ・Tilting and turning the head → Motion sensor of VR headset ・Hand and arm movements → VR controller Figure 17 neumo's unique technology (3) Explicit application of multimodal stimulation ・By playing sound and combining it with vibration, multimodal stimulation can be provided, enhancing the effectiveness of training.In situations where the signal-to-noise ratio (SNR) between the target sound and the interfering sound is poor and difficult to hear using hearing alone, converting the target sound into vibrations while preserving as much of its acoustic features as possible and adding it as a somatosensory sensation can improve listening ability and further worsen the speech-to-noise ratio through the law of reverse effect. Boosting listening ability with multimodal stimulation at SNRs that are difficult to hear using hearing alone can train listening in difficult auditory environments and improve learning outcomes. Vibration stimulation unrelated to the target sound's acoustic features can also improve listening ability in a similar way through the integration pathway with somatosensory stimulation in the auditory pathway. Therefore, it is possible to train listening in difficult auditory environments and improve learning outcomes. Adding weak noise vibrations can induce stochastic resonance and improve listening ability even at SNRs that are difficult to hear. Therefore, it is possible to train listening in difficult auditory environments and improve learning outcomes.・RWM (pulse width modulation) or PAM (pulse amplitude modulation), or a combination of these, can be used to convert the target sound into vibration while preserving the acoustic features as much as possible. ・Vibration stimuli unrelated to the acoustic features of the target sound can include vibrations synchronized with the rhythm of the target sound, vibrations linked to user operations, and vibrations that are periodic or random regardless of the target sound or user operations. Figure 18: neumo's unique technology (4) Meditation and hearing training can be performed simultaneously by drawing circles or shapes while listening to sound. ・A common method of meditation is to relax and focus on deep breathing, but by concentrating on listening to the target sound and tracing a circle on a smartphone screen with your finger, you can meditate and train your hearing at the same time. ・The target sound can be changed in conjunction with the movement of drawing a circle with your finger. For example: ・When the finger is tracing the correct position on the circle, the correct sound is played, and if the position is off, the sound is changed to let you know that you have deviated from the circle. ・The speed of the movement of tracing the circle is defined, and when it is moving at the correct speed,The correct sound can be played, and the sound can be changed if the speed is too fast or too slow. - Circular drawing action - The circle can be displayed on the screen and visible, or it can be invisible. - It is also possible to draw a circle in space using a VR controller or motion capture equipment. - The size of the circle can be changed, or patterns other than circles can be used. - The size of the circle can be changed along the way. - It is also possible to have the user trace shapes other than circles, such as squares or stars, or letters or pictures. - The target sound can be changed in any way as long as it allows the user to distinguish between correct and incorrect sounds. For example, the following sounds can be used: - The correct sound is one that comes from the front, while an incorrect sound comes from a position in 1D, 2D, or 3D space that is shifted from the front, allowing the user to determine which direction on the circle it is shifted in. - By changing the pitch, volume, speed, sweep sound, rhythm, phonemes, type of instrument, etc., it is possible to distinguish between correct and incorrect sounds. - A guide can be provided. - When the user makes an incorrect circle tracing action, guidance can be provided by text or audio displayed on the screen.・In addition to voice and text, guides can also use sound effects, flashing screens, characters, icon displays, vibrations, etc. ・Feedback on the speed of tracing the circle can be given by saying things like "slower" or "good pace." It is more effective to train slowly while paying attention to the target sound than to simply trace the circle roughly. ・Enhancing the meditative effect by providing vibrations ・By providing a constant vibration during the task, the heart rate can be lowered through an entrainment effect, enhancing the meditative effect. The user's heart rate can be measured using a smartwatch or smartphone camera, and the vibrations can be set slightly slower than the heart rate. It is also possible to provide vibrations based on the general heart rate without measuring the heart rate. ・Sound effects ・To prevent boredom from monotonous tasks, variation can be added by playing bell sounds, animal calls, natural sounds, music, etc. ・Disturbing sounds ・By playing a constant disturbing sound in addition to the sound linked to the circular movement of the finger, it becomes more difficult to hear, enhancing the training effect ・The same disturbing sounds as in technique (1) can be used

[0062] REFERENCE SIGNS LIST 1 Information processing device, 2 Input unit, 3 Output unit, 4 Display unit, 5 Speaker, 6 Communication unit, 8 Calculation unit, 9 Memory unit, 20 Server device, 21 Calculation unit, 22 Memory unit, 23 Communication unit, D Distance, F1 First feedback information, F2 Second feedback information, G1 First coordinate, G2 Second coordinate, K1 Input trajectory, M1 Trajectory data, MA Information, S Information processing system, V Input speed, W Network

Claims

1. An information processing device comprising: a calculation unit that evaluates an input operation of a user, wherein the calculation unit: acquires trajectory data corresponding to a virtual target; calculates a first coordinate corresponding to the trajectory data; calculates a second coordinate corresponding to an input trajectory input to an input unit so as to follow the trajectory data at each predetermined timing based on the input operation of the user; calculates a degree of agreement between the first coordinate and the second coordinate; generates first feedback information based on voice, the output state of which is adjusted according to the degree of agreement, and causes an output unit to output the first feedback information; and if the degree of agreement satisfies a predetermined condition, generates second feedback information indicating an evaluation of the input operation to the user, and causes the output unit to output the second feedback information.

2. The information processing device according to claim 1, wherein the calculation unit causes the output unit to output the second feedback information with a predetermined delay from a predetermined timing at which the input operation is performed.

3. The information processing device according to claim 1, wherein the calculation unit calculates an input state in which the input trajectory is input at each predetermined timing, and outputs to the output unit the second feedback information including information for adjusting the input state based on a comparison result between the input state and a set condition.

4. An information processing system comprising an information processing device that accepts input operations by a user, and a server device communicatively connected to the information processing device, wherein at least one of the information processing device and the server device has a calculation unit that evaluates the input operation of the user, wherein the calculation unit: acquires trajectory data corresponding to a virtual target; calculates a first coordinate corresponding to the trajectory data; calculates a second coordinate corresponding to an input trajectory input to an input unit so as to follow the trajectory data at each predetermined timing based on the input operation of the user; calculates a degree of agreement between the first coordinate and the second coordinate; generates first feedback information based on voice, the output state of which is adjusted according to the degree of agreement, and causes an output unit to output the first feedback information; and, if the degree of agreement satisfies a predetermined condition, generates second feedback information indicating an evaluation of the input operation to the user, and causes the output unit to output the second feedback information.

5. A computer program installed in a computer mounted on an information processing device that evaluates a user's input operation, the computer program causing the computer to execute the following processes: acquire trajectory data corresponding to a virtual target; calculate a first coordinate corresponding to the trajectory data; calculate a second coordinate corresponding to an input trajectory input to an input unit so as to follow the trajectory data at each specified timing based on the input operation of the user; calculate a degree of agreement between the first coordinate and the second coordinate; generate first feedback information based on voice, the output state of which is adjusted according to the degree of agreement, and cause an output unit to output the first feedback information; and, if the degree of agreement satisfies a specified condition, generate second feedback information indicating an evaluation of the input operation to the user, and cause the output unit to output the second feedback information.

Images