Information processing device, information processing system, and computer program

Abstract

An information processing device 1 comprises a calculation unit 8 that evaluates an input operation of a user, the calculation unit generating prediction information that causes the user to predict a virtual target, causing an output unit to output the prediction information, acquiring trajectory data corresponding to the virtual target, calculating first coordinates corresponding to the trajectory data, calculating second coordinates corresponding to an input trajectory input to the input unit so as to follow the trajectory data for each predetermined timing on the basis of the input operation of the user, calculating a degree of matching indicating a spatial relationship between the first coordinates and the second coordinates, generating audio information associated with the input operation so as to change according to the degree of matching, and causing the output unit to output the audio information.

Description

Information processing device, information processing system, and computer program

[0001] The present disclosure relates to an information processing device, an information processing system, and a computer program for training a user's hearing-based physical function, for example, for brain function training through hearing.

[0002] The technology shown in Figures 1 to 18 did not exist. Furthermore, Patent Document 1 proposes an information processing device that acquires information on the content of a user's operation and provides feedback information to the user. According to the technology described in Patent Document 1, a predetermined action of the user is detected based on information on the content of the user's operation, and the timing for providing feedback information from the time of detection is calculated according to the content of the predetermined action, and the feedback information can be output at the calculated timing.

[0003] JP 2024-109398 A

[0004] “Audiomotor Perceptual Training Enhances Speech Intelligibility in Background Noise”, Jonathon P. Whitton, Kenneth E. Hancock, Jeffrey M. Shannon, Daniel B. Polley, Current Biology, 11.6th.2017

[0005] It has been known that in brain function training, predicting upcoming events increases the production of dopamine in the brain, enhancing the effectiveness of subsequent training. However, no effective technique for predicting future events in brain function training via auditory perception has been established.

[0006] The present invention aims to provide an information processing device, an information processing system, and a computer program that are capable of, for example, performing brain function training through hearing in order to train a user's physical functions based on their hearing.

[0007] In order to achieve the above object, an information processing system according to one aspect of the present invention has the functions described in Figures 1 to 18. Another aspect of the present invention is an information processing system including a calculation unit that evaluates an input operation by a user, wherein the calculation unit generates prediction information that allows the user to predict a virtual target, causes an output unit to output the prediction information, acquires trajectory data corresponding to the virtual target, calculates first coordinates corresponding to the trajectory data, calculates second coordinates corresponding to an input trajectory input to the input unit so as to follow the trajectory data at predetermined timings based on the input operation by the user, calculates a degree of coincidence that indicates a spatial relationship between the first coordinates and the second coordinates, generates sound information linked to the input operation so as to change according to the degree of coincidence, and outputs the sound information to the output unit.

[0008] According to the present invention, it is possible to perform physical function training based on the user's hearing, for example, brain function training through hearing.

[0009] FIG. 1 is a diagram showing an outline (1) of an information processing system according to an embodiment of the present invention. FIG. 2 is a diagram showing an outline (2) of an information processing system according to an embodiment of the present invention. FIG. 3 is a diagram showing an outline (4) of an information processing system according to an embodiment of the present invention. FIG. 5 is a diagram showing an outline (5) of an information processing system according to an embodiment of the present invention. FIG. 6 is a diagram showing an outline (7) of an information processing system according to an embodiment of the present invention. FIG. 8 is a diagram showing an outline (8) of an information processing system according to an embodiment of the present invention. FIG. 9 is a diagram showing an outline (10) of an information processing system according to an embodiment of the present invention. FIG. 11 is a diagram showing an outline (12) of an information processing system according to an embodiment of the present invention. FIG. 13 is a diagram showing an outline (14) of an information processing system according to an embodiment of the present invention. FIG. 15 is a diagram showing an outline (16) of an information processing system according to an embodiment of the present invention. FIG. 17 is a diagram showing an outline (18) of an information processing system according to an embodiment of the present invention. FIG. 1 is a block diagram showing the configuration of an information processing system according to an embodiment of the present invention. FIG. 1 is a diagram showing an example of predicted information displayed on a display unit. FIG. 1 is a diagram showing an example of trajectory data displayed on a display unit. FIG. 2 is a diagram explaining a method for calculating the degree of coincidence between a first coordinate and a second coordinate. FIG. 3 is a diagram showing a state in which feedback information is output. FIG. 4 is a diagram showing an example of a display image for prompting a user to input a predetermined task related to a virtual target after an input operation is completed. FIG. 5 is a flowchart showing the flow of processing executed in an information processing device.

[0010] An overview of an information processing system according to one embodiment of the present invention will be described below. Fig. 1 is a diagram showing an overview (1) of an information processing system according to one embodiment of the present invention. The overview (1) of an information processing system according to one embodiment of the present invention is as shown in Fig. 1.

[0011] 2 is a diagram showing an outline (2) of an information processing system according to one embodiment of the present invention. The outline (2) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0012] 3 is a diagram showing an outline (3) of an information processing system according to one embodiment of the present invention. The outline (3) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0013] 4 is a diagram showing an outline (4) of an information processing system according to one embodiment of the present invention. The outline (4) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0014] 5 is a diagram showing an outline (5) of an information processing system according to one embodiment of the present invention. The outline (5) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0015] 6 is a diagram showing an outline (6) of an information processing system according to one embodiment of the present invention. The outline (6) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0016] 7 is a diagram showing an outline (7) of an information processing system according to one embodiment of the present invention. The outline (7) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0017] 8 is a diagram showing an outline (8) of an information processing system according to one embodiment of the present invention. The outline (8) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0018] 9 is a diagram showing an outline (9) of an information processing system according to one embodiment of the present invention. The outline (9) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0019] 10 is a diagram showing an overview (10) of an information processing system according to one embodiment of the present invention. The overview (10) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0020] 11 is a diagram showing an overview (11) of an information processing system according to one embodiment of the present invention. The overview (11) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0021] 12 is a diagram showing an overview (12) of an information processing system according to one embodiment of the present invention. The overview (12) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0022] 13 is a diagram showing an outline (13) of an information processing system according to one embodiment of the present invention. The outline (13) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0023] 14 is a diagram showing an overview (14) of an information processing system according to one embodiment of the present invention. The overview (14) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0024] 15 is a diagram showing an overview (15) of an information processing system according to one embodiment of the present invention. The overview (15) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0025] 16 is a diagram showing an overview (16) of an information processing system according to one embodiment of the present invention. The overview (16) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0026] 17 is a diagram showing an overview (17) of an information processing system according to one embodiment of the present invention. The overview (17) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0027] 18 is a diagram showing an overview (18) of an information processing system according to one embodiment of the present invention. The overview (18) of an information processing system according to one embodiment of the present invention is as shown in FIG.

[0028] Although one embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and modifications, improvements, etc. within the scope of achieving the object of the present invention are included in the present invention.

[0029] For example, the above-described series of processes can be executed by hardware or software. In other words, the functional configurations shown in Figures 1 to 18 are merely illustrative and are not particularly limited. That is, it is sufficient for the information processing system to have the functionality to execute the above-described series of processes as a whole, and the functional blocks used to realize these functions are not particularly limited to the examples shown in Figures 1 to 18. Furthermore, the locations of the functional blocks and databases are not particularly limited to those shown in Figures 1 to 18 and may be arbitrary. For example, at least some of the functional blocks and databases required to execute various processes may be transferred to a user terminal or the like. Conversely, the functional blocks and databases of the user terminal may be transferred to a server or the like. Furthermore, a single functional block may be configured as hardware alone, software alone, or a combination thereof.

[0030] When a series of processes is executed by software, the programs constituting the software are installed onto a computer or the like from a network or a recording medium. The computer may be a computer incorporated into dedicated hardware. The computer may also be a computer capable of executing various functions by installing various programs, such as a server, a general-purpose smartphone, or a personal computer.

[0031] The recording medium containing such a program may be configured as a removable medium (not shown) that is distributed separately from the device main body in order to provide the program to users, etc., or may be configured as a recording medium that is pre-installed in the device main body and provided to users, etc. The computer program for executing the processing of each unit of the processor that constitutes the calculation unit 8 of the information processing system S may be provided in a form recorded on a computer-readable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. The computer program may also be provided as a program product. The specific configuration of the information processing system will be described below.

[0032] 19, the information processing system S includes an information processing device 1 operated by a user and a server device 20 communicably connected via a network W. The information processing system S is configured to enable training of the user's physical functions, including hearing, based on input operations by the user. In the information processing system S, at least one of the information processing device 1 and the server device 20 is configured to evaluate the input operations of the user.

[0033] The server device 20 is configured by an information communication device, such as a personal computer, on which a computer program can be installed. The server device 20 includes a calculation unit 21 that executes processes necessary for calculations and various controls. The calculation unit 21 is configured by at least one hardware processor, such as a central processing unit (CPU). The calculation unit 21 may be realized by hardware (including circuitry), such as a large-scale integration (LSI), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a graphics processing unit (GPU), or may be realized by a combination of software and hardware.

[0034] The server device 20 includes a storage unit 22 that stores data and computer programs necessary for calculations and control, and a database including training questions and answers. The storage unit 22 is configured with a non-transitory storage medium such as a hard disk drive (HDD) or a flash memory. The server device 20 includes a communication unit 23 that can be connected to a network W. The communication unit 23 is configured with, for example, a communication device that can be connected to the network W via a wired or wireless connection. The calculation unit 21 communicates with the information processing device 1 via the network W and transmits training data.

[0035] The calculation unit 21 may perform calculation processing based on training data and cause the information processing device 1 to output the calculation results via the network W. The calculation unit 21 may perform calculation processing in cooperation with the information processing device 1 via the network W. The calculation unit 21 may transmit an updated computer program to the information processing device 1.

[0036] The information processing device 1 is configured as an information communication device, such as a smartphone or a personal computer, on which a computer program can be installed. The information processing device 1 may be a device specifically configured for training the user's physical functions. The information processing device 1 includes a calculation unit 8 that executes processing necessary for calculations and various controls. The calculation unit 8 is configured by at least one hardware processor, such as a CPU. The calculation unit 8 may be realized by hardware, such as an LSI, an ASIC, an FPGA, or a GPU, or may be realized by a combination of software and hardware.

[0037] The information processing device 1 includes a storage unit 9 that stores data and computer programs necessary for calculations and control, as well as a database including training questions and answers (described below). The storage unit 9 is configured with a non-transitory storage medium such as a hard disk drive or flash memory. The information processing device 1 includes a communication unit 6 that can be connected to a network W. The communication unit 6 is configured with, for example, a communication device that can be connected to the network W via a wired or wireless connection. The calculation unit 8 communicates with a server device 20 via the network W to receive training data.

[0038] The information processing device 1 includes an input unit 2 that accepts user operations. The input unit 2 is configured with a device capable of inputting user input operations, such as a touch panel, touchpad, mouse, or keyboard. The input unit 2 is provided in the information processing device 1. The input unit 2 may be configured with a device communicatively connected to the information processing device 1 via a wireless or wired connection. The input unit 2 may be configured with a controller having a cross key or joystick, or a controller based on an acceleration sensor that detects the user's body movements. Any acceleration sensor can be used as long as it can detect the user's body movements. The input unit 2 may be configured with a camera, or the user's movements may be input by image recognition of image data by the calculation unit 8. The input unit 2 may be configured to input voice based on the user's speech, recognize the content of the speech, and perform input operations. The input unit 2 may be configured to input specific voice information, recognize the content of the voice information generated based on a combination of one or more parameters, such as frequency, acoustic power, acoustic spectrum, and emission pattern, and perform input operations.

[0039] The input unit 2 may be configured with a motion capture device that recognizes markers attached to the user's body and detects the user's body movements. The input unit 2 may be configured to detect the user's body based on a pressure sensor or a vibration sensor. The input unit 2 may be configured to accept input operations based on a device that acquires signals from the user's body non-invasively and / or invasively, such as an electromyography (EMG), an electroencephalogram (EEG), or an electrooculography (EOG). The input unit 2 may be configured with any device that can accept two-dimensional or three-dimensional input operations by the user.

[0040] The information processing device 1 includes an output unit 3 that outputs information. The output unit 3 is provided in the information processing device 1. The output unit 3 may be configured by a device communicatively connected to the information processing device 1. The output unit 3 includes, for example, a display unit 4 that outputs information based on an image. The display unit 4 is configured by a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The display unit 4 may be configured by a touch panel that can accept input operations. In this case, the display unit 4 also functions as the input unit 2. The display unit 4 is communicatively connected to the information processing device 1 and may be configured by a projector that can project images or a head-mounted display device that is worn on the user's head.

[0041] The output unit 3 includes a speaker 5 that outputs sound. The speaker 5 outputs information based on the sound. The output unit 3 may be configured as a device that applies stimuli to the user's body non-invasively and / or invasively. The output unit 3 may be configured as a device that applies physical and / or neural stimuli based on one or more combinations of visual stimuli, auditory stimuli, vibration stimuli, olfactory stimuli, taste stimuli, temperature stimuli, electrical stimuli, magnetic stimuli, gravitational stimuli, acceleration stimuli, pressure stimuli, brain wave stimuli, balance sense stimuli, muscle stimuli, visceral stimuli, light stimuli, sound stimuli, visual cortex stimuli, auditory cortex stimuli, intestinal bacteria administration, drug administration, etc.

[0042] The following describes a process for training a user, which is executed in the information processing device 1. In this embodiment, a process in which the calculation unit 8 evaluates an input operation performed by a user on a touch panel is illustrated. The calculation unit 8 starts training, for example, in a state in which the trajectory of a virtual target is not displayed on the display unit 4. In the training, the user performs an input operation in which the user traces the trajectory of the virtual target with their finger on the display unit 4 to estimate the trajectory of the virtual target. The calculation unit 8 compares an input trajectory input based on the user's input operation with the trajectory of the virtual target, scores the accuracy of the input operation, and provides feedback to the user. The process of the calculation unit 8 may be executed by the server device 20.

[0043] As shown in FIG. 20 , the display unit 4 is configured with a touch panel that accepts input operations for training by the user. The calculation unit 8 launches a training application based on the user's input operations. The calculation unit 8 acquires, from the storage unit 9, a dataset including prediction information for predicting a virtual target and trajectory data M1 corresponding to the virtual target. The calculation unit 8 may acquire the dataset from the server device 20. The virtual target is, for example, a representation of characters, figures, etc. based on multiple linear trajectories. The calculation unit 8 generates prediction information for allowing the user to predict the virtual target in a format compatible with the output unit 3. The calculation unit 8 generates the prediction information in the form of, for example, an image, text, audio, etc., so as to remind the user of the trajectory data M1.

[0044] It is generally known that when a user is prompted to predict an upcoming event, dopamine is produced in the brain, enhancing the effectiveness of subsequent auditory training. Therefore, the calculation unit 8 outputs prediction information to the output unit 3 so as to prompt the user to predict the shape of the virtual target before accepting an input operation. The calculation unit 8 generates, for example, a display image of the prediction information MQ in the form of a quiz, and outputs the image to the output unit 3. The prediction information MQ may be generated in various formats, such as the quiz format, as well as in the following examples. The prediction information MQ may be generated in any format as long as it can suggest a virtual target.

[0045] For example, if the virtual target is a character, the prediction information MQ may be generated based on a display image that allows the user to predict the shape of the character. The display image that allows the user to predict the shape of the character presents a word and allows the user to guess how to read the word. The calculation unit 8 may generate prediction information in which at least some of the readings of a language are not explicitly stated, such as four-character idioms, difficult-to-read kanji, modern terms, proverbs, homonyms, antonyms, similar words, compound kanji readings, difficult-to-read place names, seasonal words, idiomatic phrases, fish names, animal names, country names, color names, kanji representing digits of numbers, health-related words, and phonetic symbols for words.

[0046] The calculation unit 8 may generate predictive information MQ that presents the reading, meaning, and explanation of a word and allows the user to guess at least a part of the original word from the reading. The calculation unit 8 may generate predictive information that presents a state in which at least a part of the corresponding word is not specified after rearranging hiragana and katakana. The calculation unit 8 may generate predictive information based on a question that prompts the user to imagine the shape of a word. The calculation unit 8 may generate predictive information that prompts the user to guess the shape of a character to be filled in a blank or margin based on a question to fill in a blank or margin.

[0047] The calculation unit 8 may generate prediction information MQ that presents four-character idioms, difficult-to-read kanji, modern terms, proverbs, homonyms, antonyms / synonyms, difficult-to-read kanji, compound kanji readings, difficult-to-read place names, seasonal words, idiomatic phrases, bird names, fish names, color names, and states in which some of the characters representing digits of numbers are hidden. When generating prediction information using words, the calculation unit 8 may generate prediction information MQ using a combination of not only Japanese but also one or more other foreign languages. The calculation unit 8 may generate prediction information MQ that predicts word expressions between one or more languages.

[0048] The calculation unit 8 may present a task of rearranging letters to form a word and may generate prediction information MQ that prompts the user to fill in missing letters to complete the word. When the virtual target is a figure, the calculation unit 8 may generate prediction information MQ based on a question format using words related to the figure. The calculation unit 8 may generate prediction information MQ that prompts the user to complete a task using a puzzle. The calculation unit 8 may generate prediction information MQ based not only on words but also on figures. The calculation unit 8 may generate prediction information MQ by associating figures with words. The calculation unit 8 may generate prediction information MQ that prompts the user to predict a figure by presenting a direct explanation of the figure or an indirect explanation of the figure.

[0049] The calculation unit 8 may generate prediction information MQ based on a puzzle using graphics. The calculation unit 8 may set the virtual target in a different posture relative to its normal state, or may generate prediction information MQ for the virtual target being in a rotated state. The calculation unit 8 may generate prediction information MQ for characters with different stroke counts depending on the type of font (e.g., "ri") and for the characters to be displayed in different fonts. The calculation unit 8 may generate prediction information MQ indicating the shape of the virtual target based on linguistic descriptions and / or geometric descriptions. The calculation unit 8 may perform one or more of an action of deforming, rotating, or moving the virtual target over time, and generate prediction information MQ including information suggesting the action to be performed. The calculation unit 8 may generate prediction information MQ in which the trajectory of the virtual target is displayed in a lighter color than the normal display color, and display the prediction information MQ on the display unit 4. Alternatively, the calculation unit 8 may generate prediction information MQ in which the trajectory of the virtual target is displayed in a dashed line, and display the prediction information MQ on the display unit 4. The calculation unit 8 may display a virtual target on the display unit 4 and MQ-generate prediction information so that the displayed virtual target disappears after a predetermined time.

[0050] As shown in Fig. 21 , the calculation unit 8 causes the display unit 4 to display a display image for inputting trajectory data M1. The trajectory data M1 is data indicating a virtual target in a virtual space based on its spatial position. The trajectory data M1 is expressed by first coordinates in a coordinate system of the virtual space. The trajectory data M1 may be fixed over time or may be dynamic over time. The trajectory data M1 may also change over time.

[0051] The calculation unit 8 executes a calculation process to make the trajectory data M1 of the virtual target correspond to the display unit 4. The calculation unit 8 calculates first coordinates corresponding to the trajectory data. The calculation unit 8 converts the first coordinates into data corresponding to a position on the screen of the display unit 4 and makes the data correspond to the display unit 4. The calculation unit 8 calculates second coordinates in the virtual space corresponding to the input trajectory displayed on the display unit 4, based on an input operation input to the input unit 2. In the illustrated example, the first coordinates and the second coordinates are, for example, two-dimensional coordinates. The first coordinates and the second coordinates may be coordinates of two or more dimensions.

[0052] The calculation unit 8 calculates first coordinates that match the trajectory data M1. When the trajectory data includes multiple trajectories, the calculation unit 8 calculates multiple first coordinates. The calculation unit 8 does not display the trajectory data M1 of the virtual target based on the first coordinates on the display unit 4, and instead outputs instruction information to the output unit 3, instructing the user to perform an input operation to trace the trajectory data M1 of the virtual target. The calculation unit 8 may not only not display the trajectory data M1 on the display unit 4, but also display at least a portion of the trajectory data M1 on the display unit 4. The calculation unit 8 may display at least a portion of the trajectory data M1 on the display unit 4 when predetermined conditions are met, such as at a predetermined timing from the start of the input operation, within a predetermined time from the start of the input operation, or when a predetermined input operation is performed. The calculation unit 8 may switch between a state in which at least a portion of the trajectory data M1 is displayed on the display unit 4 and a state in which it is not displayed. The instruction information may be output in any form, such as text information, graphic information, audio information, or electrical stimulation, as long as it instructs the user to perform an input operation. The calculation unit 8 may generate information MA indicating the start position of the trajectory data M1 and output it from the output unit 3. The start position may be the start end of the trajectory data M1 or any position other than the start end of the trajectory data M1. The calculation unit 8 does not have to output the information MA indicating the start position of the trajectory data M1 from the output unit 3.

[0053] The calculation unit 8 may display information MA indicating the start position of the trajectory data M1, and then cause it to disappear from the output unit 3 after a predetermined time has elapsed. The calculation unit 8 may start an input operation without displaying information MA indicating the beginning of the trajectory data M1. A user starts an input operation by tracing the trajectory data M1 on the display unit 4 using an input means such as a finger or a stylus. The user may start the input operation from any position on the trajectory data M1. The user continuously and / or discretely traces the display unit 4 to generate an input trajectory. The calculation unit 8 displays an input trajectory K1 input based on the user's input operation on the display unit 4. The calculation unit 8 may not only display the input trajectory K1 on the display unit 4, but may also not display the input trajectory K1 on the display unit 4. The calculation unit 8 may display at least a portion of the input trajectory K1 on the display unit 4. The calculation unit 8 may switch between a state in which at least a portion of the input trajectory K1 is displayed on the display unit 4 and a state in which it is not displayed. The input operation may be performed not only by tracing the display unit 4 but also by other methods depending on the configuration of the input unit 2 .

[0054] As shown in FIG. 22 , the calculation unit 8 generates a second coordinate G2 of the input trajectory K1 input to the input unit 2 so as to follow the trajectory data M1 at predetermined timings based on the user's input operation. The predetermined timing may be, for example, a sampling period for accepting the input operation, or may be continuous. The second coordinate G2 indicates a spatial position in the virtual space relative to the first coordinate G1. The calculation unit 8 may generate the second coordinate G2 continuously or discretely. The calculation unit 8 may calculate an input velocity V of the input operation. For example, the calculation unit 8 calculates the input velocity V based on the path length of the input trajectory K1 input per unit time. The calculation unit 8 calculates the input velocity V using the average, median, maximum, minimum, or the like of the input velocity over a predetermined period. The calculation unit 8 generates image data of the input trajectory K1 based on the second coordinate G2 and displays the image data on the display unit 4. The calculation unit 8 generates image data of the input trajectory K1 based on, for example, a line diagram.

[0055] The calculation unit 8 calculates a degree of match indicating the spatial relationship between the first coordinate G1 and the second coordinate G2 at each predetermined timing while the input operation is being performed. The degree of match is a score indicating the degree to which the first coordinate G1 and the second coordinate G2 match. The calculation unit 8 calculates the degree of match based on, for example, the relative distance D between the first coordinate G1 and the second coordinate G2. The calculation unit 8 calculates the degree of match such that the smaller the distance D, the higher the degree of match. The calculation unit 8 may calculate the degree of match based on a combination of one or more of the relative distance, relative angle, relative velocity, relative acceleration, etc., in the relationship between the first coordinate G1 and the second coordinate G2. The calculation unit 8 may also calculate the degree of match using other methods as long as they can calculate the relationship between the first coordinate G1 and the second coordinate G2. The calculation unit 8 may calculate the degree of match at a certain moment, or may calculate the degree of match at each predetermined time interval based on the average, median, maximum, or minimum value.

[0056] The calculation unit 8 generates feedback information indicating an evaluation of the input operation to the user according to the degree of coincidence, and causes the output unit 3 to output the feedback information. The calculation unit 8 causes the output unit 3 to output the feedback information based on information combining at least one of, for example, audio information, text information, and image information. The calculation unit 8 may output the feedback information from the output unit 3 based on physical and / or neurological stimulation based on a combination of one or more of visual stimulation, auditory stimulation, vibration stimulation, olfactory stimulation, taste stimulation, temperature stimulation, electrical stimulation, magnetic stimulation, gravitational stimulation, acceleration stimulation, pressure stimulation, brain wave stimulation, balance stimulation, muscle stimulation, visceral stimulation, light stimulation, sound stimulation, visual cortex stimulation, auditory cortex stimulation, administration of intestinal bacteria, drug administration, etc.

[0057] As shown in FIG. 23 , the calculation unit 8 outputs feedback information F1 based on the audio information in accordance with the degree of match. For example, the calculation unit 8 outputs the feedback information F1 based on the audio information so that the sound level changes in accordance with the degree of match. The calculation unit 8 generates audio information linked to an input operation so that the sound level changes in accordance with the degree of match. The calculation unit 8 outputs the feedback information F1 based on the audio information to the output unit. The calculation unit 8 generates the feedback information F1 based on the audio information using an alarm sound, language, or the like. The calculation unit 8 may generate the feedback information F1 by adjusting acoustic parameters such as volume, frequency, and audio spectrum so that the feedback information F1 is a sound that is comfortable for the user. For example, the calculation unit 8 outputs the feedback information F1 so that the sound level increases as the second coordinates approach the first coordinates, i.e., as the degree of match increases.

[0058] The calculation unit 8 may generate feedback information F1 such that, for example, the farther the second coordinate is from the first coordinate, i.e., the lower the degree of similarity, the lower the sound level. The calculation unit 8 outputs the feedback information F1 at any time interval. The calculation unit 8 may generate feedback information F1 based on linguistic information. The calculation unit 8 may generate feedback information F1 based on linguistic information that guides the position, direction, and speed of the user's input operation so as to increase the degree of similarity. When the calculation unit 8 determines that the input trajectory K1 generated by the user's input operation is within a predetermined range from the end of the trajectory data M1, the calculation unit 8 generates information notifying the user of the end and outputs it from the output unit 3. The above-mentioned condition for the end is an example, and the calculation unit 8 may generate information notifying the user of the end at any timing and output it from the output unit 3.

[0059] 24 , when it is determined that the input operation is to end, the calculation unit 8 executes a process of having the user input a predetermined task related to the virtual target, and causes the process to be output to the output unit 3. The calculation unit 8 generates an input image so as to cause the user to input the predetermined task at any timing when it is determined that the input operation is to end, such as the timing when the input operation is to end, the timing a predetermined time after it is determined that the input operation is to end, or the timing when it is determined that the input operation has been completed to a predetermined percentage or more, and causes the display unit 4 to display the input image.

[0060] For example, the calculation unit 8 may generate correct answer information regarding the virtual target and output it to the output unit 3 at the timing when it is determined that the input operation has ended. If the virtual target is based on a shape, the calculation unit 8 may generate correct answer information MR indicating the correct answer for the shape. The calculation unit 8 may generate correct answer information MR based on a display image indicating the shape of the virtual target, or may generate correct answer information MR based on a display image presenting information indicating the shape of the virtual target or correct answer information based on audio information. If the virtual target is a character that constitutes part of the presented word, the calculation unit 8 may generate correct answer information MR based on a display image presenting the correct answer for the character (see FIG. 24(A)).

[0061] When the virtual target is a character that constitutes part of a word, the calculation unit 8 may generate correct answer information MS related to the word. The calculation unit 8 may generate correct answer information MS related to the word based on one or more combinations of the pronunciation, meaning, explanation, synonyms / antonyms, source, usage, etc. of the presented word. The calculation unit 8 may generate correct answer information MS based on a display image that presents information related to the word, or may generate correct answer information based on audio information that conveys information related to the word.

[0062] The calculation unit 8 may generate correct answer input information MT for prompting the user to input the correct answer of the virtual target predicted by the user based on the input trajectory K1 at a predetermined timing when the input operation ends, and output the generated information to the output unit 3 (see FIG. 24B ). When the virtual target is a character, the calculation unit 8 may generate correct answer input information MT based on a display image for a quiz or puzzle using the characters. For example, the calculation unit 8 may generate correct answer input information MT based on a fill-in-the-blank quiz or puzzle that presents one or more blank spaces and prompts the user to enter characters corresponding to the input trajectory K1 into the blank spaces. The calculation unit 8 may generate correct answer input information MT based on a multiple-choice method to prompt the user to enter characters corresponding to the input trajectory K1 into the blank spaces. The calculation unit 8 may also generate correct answer input information MT again to prompt the user to enter characters corresponding to the input trajectory K1 into the blank spaces.

[0063] The calculation unit 8 may generate correct answer input information MT in the form of a quiz, prompting the user to answer where the characters corresponding to the input trajectory K1 correspond to in the reading of a presented word. The calculation unit 8 may generate correct answer input information MT in the form of a quiz, prompting the user to select where the characters corresponding to the input trajectory K1 should go in a hidden location. The calculation unit 8 may generate correct answer input information MT in the form of a quiz, prompting the user to fill in a blank space with the characters corresponding to the input trajectory K1 as the reading of a word different from the presented word. The calculation unit 8 may generate correct answer input information MT in the form of a quiz, prompting the user to connect, rearrange, or combine the characters corresponding to the input trajectory K1 to form a character, word, or sentence. The calculation unit 8 may generate correct answer input information MT in the form of a quiz, prompting the user to guess what word a presented word is based on a predetermined number of input trajectories K1 that is smaller than the number of characters of the original word.

[0064] For example, the calculation unit 8 may present a plurality of shapes including a correct answer for the shape of the virtual target, generate correct answer input information MT for allowing the user to select a shape of the virtual target, and output the generated information to the output unit 3. When the calculation unit 8 receives an input operation from the user to select the correct answer, the calculation unit 8 may output the correct virtual target shape to the output unit 3. The calculation unit 8 may generate correct answer input information based on a display image that presents options representing the shapes of the virtual target. When an input operation is performed to select the correct answer from the options, the calculation unit 8 may generate a display image that displays the correct answer and display the image on the display unit 4.

[0065] The calculation unit 8 may generate a problem MU using a virtual target at a predetermined timing when the input operation ends, and output the problem MU to the output unit 3 (see FIG. 24(C)). When the virtual target is a figure, the calculation unit 8 may generate a problem MU that prompts the user to answer what kind of figure it is. The calculation unit 8 may generate an input image that accepts a figure answer, and display it on the display unit 4. The calculation unit 8 may generate an input image for the figure answer based on the method of the input unit 2, such as a keyboard or touch panel, and display it on the display unit 4. The calculation unit 8 may generate an input image that prompts the user to select a figure answer from options, and display it on the display unit 4.

[0066] The calculation unit 8 may generate a problem MU based on a puzzle that divides a figure corresponding to a virtual target, generates a plurality of divided images showing portions of the figure, and arranges the divided images at predetermined positions to estimate the overall image of the figure. The calculation unit 8 may generate a problem MU that prompts the subject to answer a figure that is topologically the same as the figure corresponding to the virtual target. If the virtual target is a letter that constitutes part of a word, the calculation unit 8 may generate a problem MU related to a word. If the virtual target is in the shape of a number, the calculation unit 8 may generate a problem MU based on an arithmetic problem or a number puzzle using the numbers corresponding to the virtual target.

[0067] 25 is a flowchart showing the flow of processing of an information processing method executed in information processing device 1. The information processing method is executed based on a computer program that can be installed in a computer mounted on information processing device 1. The computer program causes a calculation unit 8 included in information processing device 1 to execute the following processes.

[0068] The calculation unit 8 acquires, from the storage unit 9 or the server device 20, a dataset including prediction information for predicting a virtual target and trajectory data M1 corresponding to the virtual target, and generates prediction information (S100). The calculation unit 8 outputs the prediction information to the output unit 3 (S102). The calculation unit 8 acquires trajectory data M1 corresponding to the virtual target from the dataset (S104). The calculation unit 8 calculates a first coordinate G1 corresponding to the trajectory data M1 (S106). The calculation unit 8 accepts an input operation from the user (S108). Based on the input operation, the calculation unit 8 calculates a second coordinate G2 corresponding to the input trajectory K1 input to the input unit 2 at predetermined intervals so as to follow the trajectory data (S110).

[0069] The calculation unit 8 calculates the degree of coincidence between the first coordinate G1 and the second coordinate G2 (S112). The calculation unit 8 generates feedback information based on the voice information according to the degree of coincidence (S114). The calculation unit 8 causes the output unit 3 to output the feedback information based on the voice information (S116). The calculation unit 8 determines whether the input of the input trajectory has ended (S118), and if the end condition is not satisfied, returns the process to S108. If the calculation unit 8 determines in S118 that the input of the input trajectory has ended, it executes one or more processes shown in S120, S122, and S124.

[0070] The calculation unit 8 outputs correct answer input information MT to the output unit 3 at a predetermined timing when the input operation ends, causing the user to input the correct answer for the virtual target predicted by the user (S120). The calculation unit 8 outputs correct answer information MR regarding the virtual target to the output unit 3 at a predetermined timing when the input operation ends (S122). The calculation unit 8 outputs a problem MU using the virtual target to the output unit 3 at a predetermined timing when the input operation ends (S124). The calculation unit 8 may execute one or more of the processes shown in S120, S122, and S124 in any order.

[0071] As described above, the information processing device 1 can perform physical function training based on the user's hearing, for example, brain function training through hearing. The information processing device 1 can present predictive information to the user, allowing the user to predict upcoming events and stimulating the production of dopamine in the brain, thereby enhancing the effectiveness of subsequent auditory training. The information processing device 1 can output feedback information while the user is tracing trajectory data, thereby enabling evaluation of the input operation in real time. The information processing device 1 can perform brain function training for the user by having the user perform a predetermined task after completing the input operation.

[0072] Figure 1: Training to improve hearing. Figure 2: Fun hearing training involves finding letters and shapes using sound and completing quizzes using the discovered letters and shapes. Auditory training using sound can easily become monotonous and repetitive, making it difficult to continue. Therefore, combining auditory training with letter and shape discovery and quizzes makes auditory training fun and continuous. Auditory training involves having users trace their finger across the screen to find visually hidden letters and shapes. By changing the sound depending on the distance between the finger and the hidden letters or shapes, users can find them using the sound. In this way, auditory training is achieved by concentrating on the sound and moving their finger. The sounds and tracing actions used in auditory training utilize neumo's existing technology. The application documents are reproduced later in this document. Furthermore, there is a step in which the user rotates the letters and shapes they find by tracing with their finger, and a step in which they use the discovered letters and shapes to solve quizzes. Specific examples of these steps are provided on the following pages. Figure 3: Making letters appear and completing words: Step (1) Find letters by tracing lines using the sound What the user sees Task - The letters to be traced are broken down into a set of lines, rotated at any angle. Rotation is not required. - Trace the N lines that make up the letter in order. You can trace all N lines in order, or just a few M lines. The lines can be straight or curved, and any shape is fine. You can also trace multiple lines in one stroke. - The letters to be traced are visually hidden, and you need to trace the lines using the sound. - When tracing, the starting point of the line is displayed, so you start tracing from that starting point and reach the end point. Once you reach the end point, the task of tracing the next line begins. - The starting point can be a point, or it can be something that indicates a certain area, such as a circle. The area can be indicated by a shape other than a circle. - The end point can be visually represented by a point or line, or it can be invisible. You can also search for the end point without displaying it. Once you have finished tracing, you can check the path you have traced. You can either not display it while tracing, or display the path in order as you trace.- The correct line may be displayed, or the user may proceed without it. What the user should trace Figure 4 Making letters appear to complete the word: Step (2) Rotating letters to make them upright What the user sees Task - If not all N lines that make up the letter to be traced are traced, the lines that were not traced are displayed as the correct answer. - Using sound as a guide, rotate the letter to make it upright. By changing the sound depending on the angle at which the letter's upright position is deviated from the correct position, the user can use the sound to find the correct angle. - This step is not required and can be skipped. What the user should trace A sound will indicate that a certain direction is the correct answer. Figure 5 Making letters appear to complete the word: Step (3) Selecting the letter that was traced What the user sees Task - Answer what letter you traced. There are several ways to answer, such as selecting from multiple options, or freely answering by keyboard input, voice input, or handwriting input.・If the answer is incorrect, there are both methods that allow only one answer and methods that allow multiple answers. ・If the answer is incorrect, the correct answer may or may not be revealed. Since revealing the correct answer makes the next question easier, this can be used to adjust the difficulty setting. ・This step is not required and can be skipped. ・Repeat steps (1) to (3) depending on the number of characters in the question. Figure 6: Make the characters appear and complete the word: Step (4) Guess the kanji from the characters obtained. What the user sees Task ・A quiz is given using the characters found in steps (1) to (3). ・In a quiz to guess the reading of a kanji, the characters found in steps (1) to (3) are used to display part of the reading for the kanji, and the user must guess the missing characters. ・The kanji to be read can be a single character, or it can be a multi-character word, four-character idiom, or sentence.・The characters found in steps (1) to (3) can be displayed as part of the reading, or can be used as one of the candidates. ・For the quiz where you have to guess the character in the blank, you can choose from the options. You can also choose not to display options and let the user enter the answer freely.- If the answer is correct, a sound or visual will be displayed to indicate the answer. For example, the sound or visual of fireworks can be presented. Other methods besides fireworks are also acceptable. If the answer is incorrect, the user is notified that the answer is incorrect. If the answer is incorrect, both the correct answer and the next answer can be displayed. Figure 7: The rotation step can be performed in a different way. - The step of rotating the letters or shapes found by tracing can be replaced with another method shown below, or it can be omitted. The location of the correct answer can be found using sound, which can be used to train the auditory sense. It is also possible to use a method where the location is found without using sound, relying on visuals or vibrations. - The screen can be moved to match the correct position in one, two, or three-dimensional space. For example, it can be used in cases such as a jigsaw puzzle, where pieces are placed in the correct position. - The letters or shapes can be enlarged or reduced to match the correct size. - Rotate in a 3D space centered on the X and Y axes and align with the correct position. - Flick letters or shapes with your finger or operate a spring like a pinball to send them flying and hit the correct target or put them in the hole. The correct direction and strength can also be determined by sound. - Find the correct stroke order for kanji or characters. A correct sound can be played if the stroke order is correct, and an incorrect sound can be played if the stroke order is incorrect. Figure 8 Calculating the score for the tracing result - For the step of tracing and finding letters or shapes, a function can be provided that quantifies how accurately the tracing was performed and provides feedback to the user. One of the following methods can be used to quantify the score, or other methods can be used.・Use spatial distance ■There is a method of quantifying the distance between the correct letter or shape and the traced line. ・1. Convert the traced line into a set of points. ・2. For each point, calculate the distance to the nearest correct letter or shape. ・3. Add up the calculated distances to obtain a cumulative difference value. ・4. The average of the calculated distances can be used as the difference value. ・5. It is also possible to calculate using only calculated distances within a certain range. For example, only those close to the correct answer (number 1 or less) are used as the cumulative value. Or, those far from the correct answer (number 10 or more) can be accumulated and used as a deduction system.■One method is to quantify the time it takes to successfully trace the correct letter or shape. ・1. Define the range of the correct answer for the letter or shape. The range can be either a binary value of 1 or 0, or a decimal value between 1 and 0, or a sloped range such as 100 to 0. ・2. By using the total tracing time as the denominator and the time spent tracing the correct position as the denominator, the time spent tracing the correct position can be quantified. In this case, the slope defined in 1 can also be used to calculate the value. ■One method is to quantify the difference between the position at which tracing ends and the end position of the correct letter or shape. ・Rather than using data from the tracing path, it is possible to calculate the distance between the position at which tracing ends and the end position of the correct letter or shape, and use this as a numerical value to evaluate the result. ・It is also possible to provide a function that quantifies how accurately the steps to rotate the letter or shape were answered and provide feedback to the user. - The smaller the difference between the correct angle and the angle you answered, the higher the score. Figure 9: Various patterns can be used for quizzes using letters and shapes found by tracing. - We have provided an example of how to trace letters, but any of the following shapes are acceptable, not just letters: - Japanese hiragana, katakana, kanji, and numbers - Alphabet (this may include language-specific characters other than the basic alphabet, such as French spelling symbols and ligatures or German umlauts) - Characters of other languages ​​(any language characters are acceptable, including simplified Chinese, traditional Chinese, Arabic, Devanagari, Greek Mandarin, and Hangul) - Abstract shapes (circles, triangles, etc.). One-, two-, or three-dimensional shapes are acceptable. - Line segments in photographs and illustrations (any shape that can be expressed with lines can be included, such as anime characters, animal illustrations, icons, products, constellations, family crests and coats of arms, tangrams, and the topography of countries and prefectures) - The correct route in a map or maze - There are various ways to create quizzes using the letters and shapes that have been discovered, such as the following.・By tracing the steps, you can find hiragana, katakana, numbers, the alphabet, and characters from various other languages, and then ask questions like the following. The following is just one example, and other quizzes can be made. You can choose quizzes where the answer can be determined just by tracing the characters you find, or quizzes where you need to answer with only partial knowledge. ■Guess the reading of kanji or compound words ■Make it a crosswood puzzle and guess where the character or word goes ■Use the characters you trace to find or create kanji, words, or sentences. You can rearrange the characters, or find radicals to create kanji. You can also choose a poem from the Hyakunin Isshu. ■Fill-in-the-blank quiz using the characters you trace ■Math problems and number puzzles using the numbers you trace ■Trace kanji or characters and guess what they are ■Guess what the characters you trace and find are an abbreviation for ・By tracing the steps, you can find shapes and illustrations, and then ask questions like the following ■A quiz to guess what shape is hidden. You can trace all the lines, or just some of the lines, with some of the lines displayed beforehand. Or you can have only some of the lines displayed and have to predict the whole picture and guess. For example, you can make questions such as guessing what animal it is or what constellation it is. ■It can also be a quiz where you guess the correct answer by combining multiple shapes you find by tracing. ■It can also be a jigsaw puzzle, block puzzle, or Tetris where you combine shapes you find by tracing. ■You can have students compare the sizes of shapes they find by tracing, or find shapes that are similar. ・It can also be a Wiz game where students find the correct path by tracing steps on a map or maze. ■A quiz where students guess the destination on the map or the route to the exit of a maze, using the ability to trace and find the path on a map or maze as a hint. ・It can also be a quiz where students listen to words in a noisy environment, using the letters they find by tracing as a hint to answer. Figure 10 Elements for making training enjoyable and continuous (1) ・Increase motivation by visualizing the accuracy of tracing and keeping a record ・Give feedback by quantifying how well they were able to trace letters and shapes ・Record letters and shapes that they have traced in the past as a history, allowing them to look back on it later, allowing students to feel an improvement in their listening ability.- Badges are awarded based on the difficulty and accuracy of the tracing. Various badges (high accuracy, tracing a difficult sound, tracing in a short time, high quiz correct answer rate, etc.) are available and can be collected. Motivation is increased by displaying the next badge that could be obtained. - Sharing and communication with others increases motivation. - Tracing scores and the quiz correct answer screen can be shared with others. Results can be shared with other users registered as friends within the app, or via email, social media, LINE (registered trademark), and other external platforms. - Motivation is increased by notifying users when they receive a "like" or comment on a shared result. - Presenting the difficulty and level of the problem allows a sense of accomplishment. - The difficulty level is defined by the target sound and distracting sounds used in the tracing step, and can be understood as a level. The more clearly you trace, the higher the level you can move up to. Reaching a higher level as a result of hearing training provides a sense of accomplishment.・For Kanji quizzes, for example, once all questions equivalent to Kanji Kentei Grade 5 have been cleared, the next step should be to challenge questions equivalent to Kanji Kentei Grade 4. This allows the level of difficulty to be understood numerically, allowing a sense of accomplishment. The same level definition is used for other quizzes, such as crossword puzzles. ・For shape or illustration quizzes, the difficulty level is defined according to the fineness of the shapes to be traced and whether they are easy to understand. ・Motivation is enhanced by displaying the level of achievement compared to others. ・Rankings are displayed by comparing how well a person traced the same problem with a numerical value. Multiple types of rankings can be used, such as rankings for each character such as "a," overall rankings for multiple questions, monthly rankings, rankings by difficulty level, rankings for each quiz question, and rankings by quiz level. ・Ear age is displayed by comparing tracing accuracy with that of people of the same age or generation. ・Brain age is displayed by comparing the degree of accuracy of Wis compared with that of people of the same age or generation. Figure 11 Elements for Enjoyable and Continuous Training (2) ・Make people aware of time・Setting a time limit for each problem can encourage concentration. ・Suggesting the time required to solve a problem can be displayed. ・Training too quickly reduces the effectiveness of training, so it is possible to encourage time-consciousness by encouraging users to trace slowly. In this case, it is possible to combine messages such as "Take it easy" with the time. ・Displaying the daily training time can visualize efforts. ・Increasing the score for consecutive correct answers can increase the score. ・Increasing the score for daily training can also provide motivating feedback such as "Keep Going!!!" or "Perfect**!" when tracing a line segment almost correctly. Figure 12. Enhancing Training Effectiveness by Setting Difficulty Levels According to Ability. ・Measuring the user's hearing level in advance using a system separate from this system and incorporating the results as input can present problems of an appropriate difficulty level for each individual's hearing ability and hearing condition. Training effectiveness can be enhanced by adjusting the difficulty level to match each individual's ability. The following information can be used as input.・Pure tone audiometry results ・Speech audiometry results ・Auditory ability in noise measurements (speech audibility tests in steady noise, such as digits-in-noise, HINT, J-HINT, OLSA, J-Matrix, or speech audibility tests in noisy environments, such as digits-in-multi-talker-babble, speech-in-multi-talker-babble, QuicksIN) ・Left-right hearing difference measurements ・Auditory time processing measurements (measurement of thresholds using an amplitude modulation (AM) detection task, a task measuring time difference processing ability between the ears using NOSJT or NOST signals, frequency modulation threshold detection task, gap detection task, TFS1, TFS2, TFS-LF, interaural time difference detection task, frequency modulation direction detection threshold for seep sound) ・Cognitive function measurements, such as working memory and executive function ・Electroencephalography (EEG) results (ABR)・Since the user's hearing ability improves with continued training, by adjusting the difficulty level according to the training results, it is possible to present questions of a level appropriate to the individual's hearing ability and hearing condition. ・If the tracing accuracy exceeds the level-up threshold, the target sound or interfering sound is changed to allow for more difficult training. ・If the tracing accuracy falls below the level-down threshold, the target sound or interfering sound is changed to allow for less difficult training. ・These changes can be made based on the results of a single training session, multiple consecutive sessions, or multiple averages (e.g., the average for one day's training). ・The difficulty level is set taking into account the user's strengths / weaknesses for the target sound and the interfering sounds. ・Each person has strengths and weaknesses when it comes to target sounds; for example, they may be good at pitch changes in pure tones but have difficulty hearing human voices. If the level is set based on the user's strong target sound, they will be unable to hear the weak target sound. The overall level cannot be raised unless the user can also clear the weak paired sounds to a certain level. Similarly, if there are strong and weak interference sounds, the overall level of the interference sounds will not increase any further until the weak interference sound reaches a certain level. ・The difference in level between the target sound and the interference sounds will also be kept from becoming too great. For example, if the target sound is level 20 and the interference sound is level 8, a constraint is set that the interference level must be raised to at least 10 in order to level up the target sound. Figure 13 Other Functions (1) ・In addition to existing technology, the following methods can also be used for tracing: ・Use a stylus instead of a finger ・Use equipment or software that detects gaze to replace the gaze tracing operation ・Detect tracing actions using equipment equipped with sensors, such as the Nintendo Switch (registered trademark) controller. Other controllers that can be used include existing products such as Meta Quest (registered trademark), Apple Vision Pro (registered trademark), Vive (registered trademark), PlayStation (registered trademark), XBOX (registered trademark), and pico, as well as specially developed controller devices.・The following may be used as methods to indicate the correct answer: ・Play a sound indicating the correct answer if the user has traced the correct position. ・Vibrate the device if the user has traced the correct position. ・Present some kind of visual element if the user has traced the correct position (the screen lights up, a message is displayed, a character is displayed, the screen appears to vibrate, etc.) ・Make it easier to see the correct position by changing the thickness or colour of the line displayed when the correct position is traced and when the incorrect position is traced. ・The following may be used as methods to indicate an incorrect answer: ・Play a sound indicating the incorrect answer if the user has traced the wrong place. ・Vibrate the device if the user has traced the wrong place. ・Present some kind of visual element if the user has traced the wrong place (the screen lights up, a message is displayed, a character is displayed, the screen appears to vibrate, etc.) ・The following may be used as the timing to indicate the above correct and incorrect answers: ・Present the answer immediately after the tracing action・Present when the correct or incorrect position is traced continuously (for example, tracing the correct position for 3 seconds) ・Present when the correct or incorrect position is traced multiple times. For example, nothing is presented the first time, and then a presentation is made from the second time onwards. The number of times can be changed to 3 or 4 times, etc. It is also possible to present only the second time, and not present again from the third time onwards. ・It is also possible to vibrate only when the user passes over the correct position while tracing and then returns to the same place again, so that the user knows that this is the correct position.・The correct or incorrect answer is displayed a few seconds after it is traced. ・It is displayed only under certain conditions, such as during the bonus stage or fever period, or when an item is used. Figure 14 Other functions (2) ・The fun can be increased by adding effects to the tracing action. ・A line is drawn at the tracing position. ・Visual elements are displayed in a location other than the tracing position (a character appears or moves, a picture is colored, a sparkling visual element is displayed, etc.). ・Music is played only for the question being traced, the number of notes in the music being played increases or decreases, or the speed or BPM of the music changes.- Tracing can be done by freely drawing lines or by filling in a grid. The grid can be any shape, including triangles, squares, hexagons, and octagons. Figure 15: neumo's unique technology (1) Training to adjust the left and right sound position using a finger using head-related transfer functions. - Distinguish the location of a sound source in one-dimensional, two-dimensional, and three-dimensional space using sound. - Distinguish the direction from which the sound is coming and perform some kind of manipulation to make the sound appear to be coming from directly in front. By carefully listening to the location of the sound source and performing some manipulation, auditory temporal processing can be trained. - For example, in one-dimensional training, a slider is displayed on the smartphone screen and manipulated to change the left and right position of the sound so that it is heard from the center. In two-dimensional training, by tapping the smartphone screen with a finger, the user can distinguish the above, below, left, and right positions of the tapped location and manipulate the sound so that it is directly in front. For 3D, a VR headset or a smartphone's AR function, gyro (angular velocity) sensor, or magnetic sensor can be used to position the sound source directly in front of the subject in 3D space. Devices: Any device capable of changing the position of sound in 1D, 2D, or 3D is acceptable, including smartphones, tablets, PCs, game consoles, VR, and real-world speakers. Devices capable of achieving stereo or spatial sound, such as stereo speakers, earphones, and headphones, are also acceptable. Target Sound Types: Target sounds can be generated as pure tones, sound sweeps, or spectrally modulated sounds, including FM-modulated sounds, AM-modulated sounds, musical instruments, music, human voices, animal sounds, and natural sounds. When using music, using or automatically generating music that balances predictability and uncertainty can stimulate dopamine production and improve learning efficiency. Interfering Sounds: Using interfering sounds to mask the target sound can increase the difficulty and enhance training effectiveness. Interfering sounds can be noises such as white noise, human voices such as speech bubbles, the sound of rain, crowds, music, or other sounds that interfere with the target sound.・To effectively improve hearing in everyday life, it is also a good idea to use fluctuating noise based on a statistical model of fluctuations in volume and acoustic features in actual cafes, bars, etc. Figure 16 neumo's unique technology (2) Operations performed while listening to sound ・The following are possible methods of operation while listening to the target sound: ・Continuously tracing on the tablet / smartphone screen to move to a certain point ・Continuously tracing on the tablet / smartphone screen to follow the moving target sound and keep it meeting the target conditions ・Continuously moving in the virtual space using the analog joystick on the gamepad ・Continuously moving in the virtual space using the analog joystick on the touch screen ・The user moves by walking within the physical space ・Operation methods ・Hold the smartphone horizontally and tilt it ・Hold the smartphone in the normal way to look at it and tilt it ・Hold the smartphone and move mainly horizontally (search for an object that appears on the screen in AR) ・Move the arm wearing the Apple watch (registered trademark) (such as stroking a parakeet or cat) ・Turning the neck and tilting the head while wearing a VR headset ・Operations using a VR controller・Movements such as tai chi and yoga (movements are detected by image processing from camera images) ・Voice is recorded with a microphone and the movements of the jaw muscles, tongue, lips, vocal cords, and respiratory muscles are detected. ・Operation methods using movement ・Hand and arm (finger) movements → Touch screen of smartphone ・Hand and arm movements holding a smartphone → Angular velocity sensor (gyro) of smartphone ・Hand and arm movements wearing an Apple Watch (registered trademark) → Angular velocity sensor (gyro) of Apple Watch (registered trademark) ・Voice (movement of jaw, tongue, lips, vocal cords, and respiratory muscles) → Microphone of smartphone ・Various body movements → Movement is detected by image processing from camera images ・Walking to different locations → Movement and position are detected by image processing from camera images ・Tilting and turning the head → Motion sensor of VR headset ・Hand and arm movements → VR controller Figure 17 neumo's unique technology (3) Explicit application of multimodal stimulation ・By playing sound and combining it with vibration, multimodal stimulation can be provided, enhancing the effectiveness of training.In situations where the signal-to-noise ratio (SNR) between the target sound and the interfering sound is poor and difficult to hear using hearing alone, converting the target sound into vibrations while preserving as much of its acoustic features as possible and adding it as a somatosensory sensation can improve listening ability and further worsen the speech-to-noise ratio through the law of reverse effect. Boosting listening ability with multimodal stimulation at SNRs that are difficult to hear using hearing alone can train listening in difficult auditory environments and improve learning outcomes. Vibration stimulation unrelated to the target sound's acoustic features can also improve listening ability in a similar way through the integration pathway with somatosensory stimulation in the auditory pathway. Therefore, it is possible to train listening in difficult auditory environments and improve learning outcomes. Adding weak noise vibrations can induce stochastic resonance and improve listening ability even at SNRs that are difficult to hear. Therefore, it is possible to train listening in difficult auditory environments and improve learning outcomes.・Methods that convert target sounds into vibrations while preserving acoustic features as much as possible include RWM (pulse width modulation) and PAM (pulse amplitude modulation), or a combination of these. ・Vibration stimuli unrelated to the target sound's acoustic features can include vibrations synchronized with the rhythm of the target sound, vibrations linked to user operation, and regular or random vibrations unrelated to the target sound or user operation. Figure 18: neumo's unique technology (4) Simultaneous meditation and hearing training by drawing circles or shapes while listening to sound. ・A common meditation method involves relaxing and concentrating on deep breathing. However, by concentrating on listening to the target sound and tracing a circle with your finger on a smartphone screen, you can simultaneously meditate and train your hearing. ・The target sound can be changed in conjunction with the action of drawing a circle with your finger. For example: ・The correct sound is played when the finger is tracing the correct position on the circle, and if the finger is off-position, the sound is changed to indicate that the user is off-center.・The speed of the movement of tracing the circle can be defined, and the correct sound will be played if the movement is at the correct speed, and the sound can be changed if the speed is too fast or too slow. ・The movement of drawing a circle ・The circle can be displayed on the screen and visible, or it can be invisible. ・It is also possible to draw a circle in space using a VR controller or motion capture equipment. ・The size of the circle can be changed, or a pattern other than a circle can be used. ・The size of the circle can be changed midway through. ・It is also possible to have the child trace shapes other than circles, such as squares or stars, or letters or pictures. ・The target sound can be changed in any way, as long as it allows the user to distinguish between correct and incorrect sounds. For example, the following sounds are available. The correct sound is one that comes from the front, while the off-center sound comes from a position offset in one-, two-, or three-dimensional space, indicating the direction of the offset on the circle. Correct and off-center sounds can be distinguished by varying pitch, volume, speed, sweep sounds, rhythm, phonemes, and instrument type. A guide is also acceptable. When the subject makes an error while tracing the circle, on-screen guidance or audio can be used. In addition to audio and text, guidance can also be provided through sound effects, flashing screens, characters, icons, or vibrations. Feedback on the speed of tracing the circle can be provided, such as "Go slower" or "Good pace." Training slowly while paying attention to the target sound is more effective than simply tracing the circle roughly. Vibration enhances the meditative effect. By providing a constant vibration during the task, the entrainment effect can lower the heart rate and enhance the meditative effect. The user's heart rate can be measured using a smartwatch or smartphone camera, and the vibrations can be made a little slower than the heart rate. It is also possible to vibrate based on the general heart rate without measuring the heart rate. ・Sound effects ・To prevent boredom from monotonous work, you can add variety by playing bell sounds, animal cries, natural sounds, music, etc. ・Disturbing sounds ・By playing a certain disturbing sound in addition to the sound linked to the circular movement of the finger, it becomes difficult to hear, and the training effect can be improved ・The same disturbing sounds as in technique (1) can be used

[0073] 1 Information processing device, 2 Input unit, 3 Output unit, 4 Display unit, 5 Speaker, 6 Communication unit, 8 Calculation unit, 9 Memory unit, 20 Server device, 21 Calculation unit, 22 Memory unit, 23 Communication unit, D Distance, F1 Feedback information, G1 First coordinate, G2 Second coordinate, K1 Input trajectory, M1 Trajectory data, MA Information, MQ Prediction information, MR, MS Correct answer information, MT Correct answer input information, MU Question, S Information processing system, V Input speed, W Network

Claims

1. An information processing device comprising a calculation unit that evaluates a user's input operation, wherein the calculation unit: generates prediction information that allows the user to predict a virtual target; outputs the prediction information to an output unit; acquires trajectory data corresponding to the virtual target; calculates a first coordinate corresponding to the trajectory data; calculates, at a predetermined timing, a second coordinate corresponding to an input trajectory input to the input unit so as to follow the trajectory data based on the input operation of the user; calculates a degree of agreement that indicates a spatial relationship between the first coordinate and the second coordinate; generates audio information linked to the input operation so as to change according to the degree of agreement; and outputs the audio information to the output unit.

2. The information processing device according to claim 1, wherein the calculation unit outputs to the output unit correct answer input information for inputting the correct answer of the virtual target predicted by the user at a predetermined timing when the input operation ends.

3. The information processing device according to claim 1, wherein the calculation unit causes the output unit to output correct answer information regarding the virtual target at a predetermined timing when the input operation ends.

4. The information processing device according to claim 1, wherein the calculation unit causes the output unit to output a question using the virtual target at a predetermined timing when the input operation ends.

5. An information processing system comprising an information processing device that accepts input operations by a user, and a server device communicatively connected to the information processing device, wherein at least one of the information processing device and the server device has a calculation unit that evaluates the input operation of the user, the calculation unit: generates prediction information that allows the user to predict a virtual target, outputs the prediction information to an output unit, acquires trajectory data corresponding to the virtual target, calculates first coordinates corresponding to the trajectory data, calculates second coordinates corresponding to an input trajectory input to the input unit so as to follow the trajectory data based on the input operation of the user at each predetermined timing, calculates a degree of coincidence indicating a spatial relationship between the first coordinates and the second coordinates, generates audio information linked to the input operation so as to change according to the degree of coincidence, and outputs the audio information to the output unit.

6. A computer program installed in a computer mounted on an information processing device that evaluates a user's input operation, the computer program causing the computer to execute the following processes: generate prediction information that allows the user to predict a virtual target; output the prediction information to an output unit; obtain trajectory data corresponding to the virtual target; calculate a first coordinate corresponding to the trajectory data; calculate a second coordinate corresponding to an input trajectory input to an input unit so as to follow the trajectory data based on the user's input operation at each predetermined timing; calculate a degree of agreement indicating a spatial relationship between the first coordinate and the second coordinate; generate audio information linked to the input operation so as to change according to the degree of agreement; and output the audio information to an output unit.