Signal processing device, cognitive function improvement system, signal processing method, and program

Abstract

A signal processing device according to one embodiment of the present disclosure is provided with: an output unit that outputs an audio signal generated by amplitude modulation, the audio signal having a change in amplitude corresponding to the frequency of a gamma wave; a storage unit that stores information of the audio signal output by the output unit; and a transmission unit that transmits a notification based on the information stored by the storage unit to a device external to the signal processing device.

Description

Technical Field

[0001] The present disclosure relates to a signal processing device, a cognitive function improvement system, a signal processing method, and a program. Background Art

[0002] There is a research report that when a living being is made to perceive pulsed sound stimulation at a frequency of about 40 times per second to induce gamma waves in the brain of the living being, it is effective in improving the cognitive function of the living being (see Non-Patent Document 1).

[0003] Gamma waves are neural oscillations whose frequencies are included in the gamma frequency band (25 Hz to 140 Hz) among neural oscillations obtained by capturing periodic neural activities of the cerebral cortex using electrophysiological methods such as electroencephalography and magnetoencephalography.

[0004] Patent Document 1 discloses that the volume is adjusted by increasing or decreasing the amplitude of a sound wave or a sound channel to produce a rhythmic stimulation corresponding to a stimulation frequency for inducing tuning of brain waves.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application No. 2020-501853

[0008] Non-patent literature

[0009] Non-patent literature 1: Multi-sensory Gamma Stimulation Ameliorates Alzheimer's-Associated Pathology and Improves Cognition Cell 2019Apr 4; 177(2): 256-271.e22.doi:10.1016 / j.cell.2019.02.014. Summary of the invention

[0010] Problem that the invention aims to solve

[0011] When a user who expects to improve cognitive function listens to a sound based on an amplitude-modulated audio signal, stakeholders sometimes want to understand the user's listening status. For example, sometimes the user's family members feel uneasy when they cannot confirm that the user is listening to the sound. Medical stakeholders who perform treatment based on listening to the sound cannot provide appropriate treatment instructions when they cannot confirm whether the user (patient) is listening to the sound. If the user forgets to listen to the sound and has no opportunity to review the listening status, it may be difficult for the user to develop the habit of listening to the sound.

[0012] An object of the present disclosure is to provide a technology that enables a relevant person to understand a user's listening status to a sound corresponding to an amplitude-modulated audio signal.

[0013] Solutions for solving problems

[0014] A signal processing device in one embodiment of the present disclosure includes: an output unit that outputs an audio signal generated by amplitude modulation, wherein the audio signal has a change in amplitude corresponding to the frequency of a gamma wave; a storage unit that stores information of the audio signal output by the output unit; and a sending unit that sends a notification based on the information stored in the storage unit to a device outside the signal processing device. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a block diagram showing the configuration of the audio system according to the present embodiment.

[0016] Figure 2 It is a block diagram showing the structure of the signal processing device according to this embodiment.

[0017] Figure 3 It is a block diagram showing the structure of a stakeholder terminal according to the present embodiment.

[0018] Figure 4 This is an explanatory diagram of one mode of this embodiment.

[0019] Figure 5 2 is a diagram showing the data structure of the listening history database according to the present embodiment.

[0020] Figure 6 This is a flowchart of the audio signal processing according to this embodiment.

[0021] Figure 7 This is a flowchart of the information notification process according to the present embodiment.

[0022] Figure 8 : is a diagram showing an example of a score function used in the information notification process of the present embodiment.

[0023] Fig. 9 This is a diagram showing an example of presentation of target information in the information notification process according to the present embodiment.

[0024] Fig.10 This is a diagram showing an example of presentation of target information in the information notification process according to the present embodiment.

[0025] Fig.11 This is a diagram showing an example of presentation of target information in the information notification process according to the present embodiment.

[0026] Fig.12 This is a flowchart of the information notification process of Modification Example 1.

[0027] Fig.13 This is a block diagram showing the structure of an audio system according to Modification Example 2.

[0028] Fig.14 This is a flowchart of audio signal processing according to Modification Example 2. DETAILED DESCRIPTION

[0029] Hereinafter, one embodiment of the present invention will be described in detail based on the drawings. In the drawings for describing the embodiments, the same components are denoted by the same reference numerals in principle, and their repeated descriptions are omitted.

[0030] (1) Audio system structure

[0031] The structure of the audio system is described. Figure 1 This is a block diagram showing the configuration of the audio system according to the present embodiment.

[0032] like Figure 1 As shown, the audio system 1 includes a signal processing device 10 , an audio output device 30 , a sound source device 50 , and a stakeholder terminal 70 .

[0033] The signal processing device 10 and the sound source device 50 are connected to each other via a prescribed interface capable of transmitting an audio signal. The interface is, for example, SPDIF (Sony Philips Digital Interface), HDMI (registered trademark) (High-Definition Multimedia Interface), a pin connector (RCA pin) or an audio interface for headphones. The interface may also be a wireless interface using Bluetooth (registered trademark) or the like. The signal processing device 10 and the audio output device 30 are similarly connected to each other via a prescribed interface. The audio signal in this embodiment includes either or both of an analog signal and a digital signal.

[0034] The signal processing device 10 and the stakeholder terminal 70 may be connected to each other via a network (such as the Internet or an intranet) or via a predetermined interface (such as a wireless interface using Bluetooth, etc.) Alternatively, the signal processing device 10 and the stakeholder terminal 70 may be connected via an external device (such as a server) not shown.

[0035] The signal processing device 10 performs audio signal processing on the input audio signal obtained from the sound source device 50. The audio signal processing performed by the signal processing device 10 includes at least modulation processing of the audio signal (details will be described later). In addition, the audio signal processing performed by the signal processing device 10 can include conversion processing of the audio signal (for example, separation, extraction or synthesis). Moreover, the audio signal processing performed by the signal processing device 10 can also include amplification processing of the audio signal, for example, similar to an AV amplifier. The signal processing device 10 sends the output audio signal generated by the audio signal processing to the audio output device 30. The signal processing device 10 is an example of an information processing device.

[0036] The audio output device 30 generates sound corresponding to the output audio signal acquired from the signal processing device 10. The audio output device 30 is, for example, a loudspeaker (which may include a speaker with a built-in amplifier (active speaker)), headphones, earphones, or a smart speaker.

[0037] The audio output device 30 can also be configured as one device together with the signal processing device 10. Specifically, the signal processing device 10 and the audio output device 30 can be installed in a TV, a radio, a music player, an AV amplifier, a speaker, a headphone, an earphone, a smart phone, a PC (including a tablet PC. The same applies hereinafter) or a smart speaker. The signal processing device 10 and the audio output device 30 constitute a cognitive function improvement system.

[0038] The sound source device 50 sends an input audio signal to the signal processing device 10. The sound source device 50 is, for example, a TV, a radio, a music player, a smartphone, a PC, an electronic musical instrument, a telephone, a game machine, an amusement machine, or a device that transmits an audio signal through broadcasting or information communication.

[0039] The stakeholder terminal 70 can be installed in the form of a smartphone or a PC. The stakeholder terminal 70 has a function of receiving a notification of information from the signal processing device 10 and presenting the notified information to the user (stakeholder) of the stakeholder terminal 70 .

[0040] In the following description, a relevant person refers to, for example, at least one of the following persons.

[0041] The user of the audio output device 30 (that is, a person who listens to the sound in order to improve cognitive function, hereinafter referred to as a "patient")

[0042] ·Family members, friends or acquaintances of the patient

[0043] · Medical personnel (e.g. the patient's doctor, nurse, medical technician)

[0044] · Caregivers (e.g., the patient's responsible caregiver)

[0045] The creator or provider of the content corresponding to the input audio signal (described later)

[0046] The producer or provider of the content corresponding to the input audio signal

[0047] Provider of the signal processing device 10

[0048] ·Managers of facilities used by patients

[0049] Insurance companies

[0050] Other patients who belong to the same community as the patient

[0051] System provider

[0052] ・A third party designated by the above parties

[0053] The stakeholder terminal 70 is owned by the stakeholder. The "possession" here is not limited to ownership, and can mean temporary exclusive use for operation. The stakeholder terminal 70 can also be shared by multiple stakeholders. In addition, the signal processing device 10 can also notify information to multiple stakeholder terminals 70.

[0054] (1-1) Structure of Signal Processing Device

[0055] The structure of the signal processing device will be described. Figure 2 It is a block diagram showing the structure of the signal processing device according to this embodiment.

[0056] like Figure 2 As shown, the signal processing device 10 includes a storage device 11, a processor 12, an input / output interface 13, and a communication interface 14. The signal processing device 10 is connected to a display 21.

[0057] The storage device 11 is configured to store programs and data. The storage device 11 is, for example, a combination of a ROM (Read Only Memory), a RAM (Random Access Memory), and a storage device (such as a flash memory or a hard disk). The programs and data may be provided via a network or recorded in a computer-readable recording medium.

[0058] The program includes, for example, the following programs.

[0059] ·OS (operating system) programs

[0060] · Programs that execute information processing applications

[0061] The data includes, for example, the following data.

[0062] · Databases referenced in information processing

[0063] Data obtained by executing information processing (i.e., the results of executing information processing)

[0064] The processor 12 is a computer that implements the functions of the signal processing device 10 by reading and executing the program stored in the storage device 11. In addition, at least part of the functions of the signal processing device 10 may be implemented by one or more dedicated circuits. The processor 12 is, for example, at least one of the following.

[0065] CPU (Central Processing Unit)

[0066] GPU (Graphic Processing Unit)

[0067] ASIC (Application Specific Integrated Circuit)

[0068] FPGA (Field Programmable Array)

[0069] DSP (Digital Signal Processor)

[0070] The input / output interface 13 is configured to obtain information (eg, user instructions) from an input device connected to the signal processing apparatus 10 , and output information (eg, images or control signals) to an output device connected to the signal processing apparatus 10 .

[0071] The input device is, for example, the sound source device 50, a physical button, a keyboard, a pointing device, a touch panel, or a combination thereof.

[0072] The output device is, for example, the display 21, the audio output device 30, a light emitting element, a speaker different from the audio output device 30, or a combination thereof.

[0073] Furthermore, the input / output interface 13 can include hardware for signal processing, such as an A / D converter, a D / A converter, an amplifier, a mixer, and a filter.

[0074] The communication interface 14 is configured to control communication between the signal processing device 10 and an external device (for example, the audio output device 30 , the sound source device 50 , or the stakeholder terminal 70 ).

[0075] The display 21 is configured to display an image (still image or moving image). The display 21 is, for example, a liquid crystal display or an organic EL display. Alternatively, the display 21 may be a seven-segment display.

[0076] (1-2) Structure of the Stakeholder Terminal

[0077] The structure of the stakeholder terminal is described. Figure 3 It is a block diagram showing the structure of a stakeholder terminal according to the present embodiment.

[0078] like Figure 3 As shown, the stakeholder terminal 70 includes a storage device 71, a processor 72, an input / output interface 73, and a communication interface 74. The stakeholder terminal 70 is connected to a display 81.

[0079] The storage device 71 is configured to store programs and data. The storage device 71 is, for example, a combination of ROM, RAM, and a storage device (such as a flash memory or a hard disk). The programs and data may be provided via a network or recorded in a computer-readable recording medium.

[0080] The program includes, for example, the following programs.

[0081] ·OS program

[0082] · Programs that execute information processing applications

[0083] The data includes, for example, the following data.

[0084] · Databases referenced in information processing

[0085] ·The results of information processing

[0086] The processor 72 is a computer that implements the functions of the stakeholder terminal 70 by reading and executing the program stored in the storage device 71. In addition, at least part of the functions of the stakeholder terminal 70 may be implemented by one or more dedicated circuits. The processor 72 is, for example, at least one of the following.

[0087] CPU

[0088] GPU

[0089] ASIC

[0090] FPGA

[0091] DSP

[0092] The input / output interface 73 is configured to acquire information (eg, user instructions) from an input device connected to the stakeholder terminal 70 and output information (eg, images) to an output device connected to the stakeholder terminal 70 .

[0093] The input device is, for example, a physical button, a keyboard, a pointing device, a touch panel, or a combination thereof.

[0094] The output device is, for example, a display 81, a speaker, or a combination thereof.

[0095] The communication interface 74 is configured to control communication between the stakeholder terminal 70 and an external device (for example, the signal processing device 10 ).

[0096] The display 81 is configured to display an image (still image or moving image). The display 81 is, for example, a liquid crystal display or an organic EL display.

[0097] (2) One mode of implementation

[0098] One mode of this embodiment will be described. Figure 4 This is an explanatory diagram of one mode of this embodiment.

[0099] like Figure 4 As shown, the signal processing device 10 obtains an audio signal (input audio signal) from the sound source device 50. The signal processing device 10 performs amplitude modulation processing on the audio signal. The signal processing device 10 outputs the generated audio signal (output audio signal) to the audio output device 30. The output audio signal has a change in amplitude corresponding to the frequency of the gamma wave, so when the patient PA1 listens to the sound based on the output audio signal, a gamma wave is induced in the brain of the patient PA1 (the brain wave is synchronized with the gamma frequency). As a result, the effect of improving cognitive function (for example, treatment or prevention of cognitive dysfunction) can be expected. On the other hand, the signal processing device 10 saves the listening history (an example of "information of the output audio signal") of the patient PA1 listening to the sound used to improve cognitive function (the sound emitted by the audio output device 30 based on the output audio signal). As an example, the signal processing device 10 stores information in a listening history database to be described later, the information being information about what kind of amplitude modulation was performed on an audio signal having what kind of characteristic amount to generate an output audio signal, and for how long and at what playback volume the patient PA1 listened to the sound based on the output audio signal.

[0100] The signal processing device 10 generates information related to the patient PA1's listening condition of the sound for improving cognitive function (hereinafter referred to as "target information") based on the stored listening history, and notifies the stakeholder terminal 70 of the target information. The stakeholder terminal 70 presents the target information to the stakeholder PC 2. Thus, the stakeholder PC 2 can understand the patient PA1's listening condition of the sound.

[0101] Thereby, the stakeholder PC2 can obtain various effects according to the attributes of the stakeholder PC2.

[0102] For example, a family member or a medical professional can encourage the patient PA1 to improve the listening condition of the patient PA1 based on the notified target information.

[0103] Medical stakeholders can confirm whether patient PA1 is acting in accordance with the treatment policy based on the notified target information, create or modify the treatment policy related to the cognitive function of patient PA1 based on the listening status, or diagnose the cognitive function of patient PA1 based on the listening status.

[0104] The patient PA1 can reflect on his / her own listening status of the sound based on the notified target information.

[0105] The facility manager can effectively manage the patient PA1 who is a facility user by using the notified target information.

[0106] The insurance company can estimate a reasonable insurance premium for the patient PA1 by using the notified target information.

[0107] The provider of the signal processing device 10 can use the notified object information to conduct research and development for improving the improvement effect of the signal processing device 10 on cognitive function (for example, a more effective method for generating and providing output audio signals, etc.). In addition, the object information notified to the corresponding stakeholder PC 2 can also be anonymized. In addition, the patient can freely choose whether to provide the object information in exchange for a reward (for example, a coupon).

[0108] The producer or provider of the content corresponding to the input audio signal can obtain enlightenment about the criteria for producing or providing the content from the notified object information. In addition, the object information notified to the corresponding stakeholder PC2 can also be anonymized. In addition, the patient can freely choose whether to provide the object information in exchange for a reward (such as a coupon).

[0109] Other patients belonging to the same community can check the actual behavior of patients who are their opponents or companions by browsing the notified object information, and review their own actual behavior, so it is easy to maintain the motivation to continue listening to the voice for improving cognitive dysfunction. In addition, the patient is aware that his actual behavior can be seen by other patients belonging to the same community, so it is easy to maintain the motivation to continue listening to the voice for improving cognitive dysfunction.

[0110] (3) Database

[0111] The database of this embodiment is described. The following database is stored in the storage device 11. Alternatively, the database may be stored in a storage device possessed by an external device (e.g., a server) that can be accessed by the signal processing device 10. In this case, information related to a plurality of different signal processing devices 10 may be collected into one database.

[0112] (3-1) Listening History Database

[0113] The listening history database of this embodiment is described. Figure 5 2 is a diagram showing the data structure of the listening history database according to the present embodiment.

[0114] The listening history records are stored in a listening history record database. The listening history records are log information related to the patient's listening behavior of listening to sounds for improving cognitive function.

[0115] like Figure 5 As shown, the listening history database includes an "ID" field, a "start date and time" field, an "end date and time" field, a "modulation method" field, a "feature amount" field, and a "playing volume" field. Each field is associated with each other.

[0116] The listening history ID is stored in the “ID” field. The listening history ID is information for identifying the listening history.

[0117] The start date and time information is stored in the "start date and time" field. The start date and time information is information related to the date and time when the output audio signal starts to be played in the corresponding listening history record.

[0118] The end date and time information is stored in the "end date and time" field. The end date and time information is information related to the date and time when the playback of the output audio signal ends in the corresponding listening history record. That is, the period during which the patient listened to the sound based on the output audio signal can be determined based on the start date and time information and the end date and time information.

[0119] The modulation method information is stored in the "Modulation Method" field. The modulation method information is information related to the processing performed to generate the output audio signal according to the input audio signal in the corresponding listening history record. For example, the modulation method information can include information related to at least one of the following:

[0120] Modulation function used in amplitude modulation

[0121] Modulation

[0122] Audio signal to which amplitude modulation is applied

[0123] Audio signal without amplitude modulation applied

[0124] Here, when an intermediate audio signal including a plurality of audio signals (described in detail later) is generated based on an input audio signal and whether amplitude modulation is applied to each of the plurality of audio signals is determined individually, each of the plurality of audio signals corresponds to “an audio signal to which amplitude modulation is applied” or “an audio signal to which amplitude modulation is not applied”. Similarly, when an input audio signal is composed of a plurality of audio signals and whether amplitude modulation is applied to each of the plurality of audio signals is determined individually, each of the plurality of audio signals corresponds to “an audio signal to which amplitude modulation is applied” or “an audio signal to which amplitude modulation is not applied”. In these cases, the function or modulation degree used in amplitude modulation may be different for each audio signal to which amplitude modulation is applied, and therefore, the modulation method information may include information on “the function used in amplitude modulation” or “the modulation degree” of each audio signal.

[0125] The feature quantity information is stored in the "feature quantity" field. The feature quantity information is information related to the feature quantity of the audio signal to which amplitude modulation is applied in the corresponding listening history record. For example, the feature quantity information can include information related to at least one of the following.

[0126] The distribution of amplitude, energy, or effective value in the frequency domain (e.g., the result of a fast Fourier transform (FFT))

[0127] Distribution of amplitude, energy or effective value in the time domain

[0128] Type of sound source (e.g., vocalization, instrument, object, music, conversational speech, natural sound, or electronic sound)

[0129] Content attributes (e.g., program information for a TV program that includes an audio signal)

[0130] The playback volume information is stored in the "Play Volume" field. The playback volume information is information related to the playback volume of the output audio signal in the corresponding listening history record. The signal processing device 10 can control the playback volume information by specifying the playback volume to the audio output device 30, or can obtain the playback volume information from the audio output device 30.

[0131] (4) Information processing

[0132] The information processing in this embodiment is described.

[0133] (4-1) Audio Signal Processing

[0134] The audio signal processing according to this embodiment will be described. Figure 6This is a flowchart of the audio signal processing according to this embodiment.

[0135] Figure 6 The audio signal processing is realized by the processor 12 of the signal processing device 10 reading out and executing the program stored in the storage device 11.

[0136] Start in response to any of the following start conditions being met Figure 6 Audio signal processing.

[0137] Called by other processes or external instructions Figure 6 Audio signal processing.

[0138] · The relevant person has carried out the Figure 6 Audio signal processing operations.

[0139] The signal processing device 10 is changed to a predetermined state (for example, the power is turned on).

[0140] ·The specified date and time has arrived.

[0141] · From a specified event (e.g., when the signal processing device 10 is started or last executed Figure 6 A specified time has passed since the audio signal processing of the

[0142] like Figure 6 As shown, the signal processing apparatus 10 performs acquisition of an input audio signal ( S110 ).

[0143] Specifically, the signal processing device 10 receives an input audio signal transmitted from the sound source device 50 .

[0144] In step S110 , the signal processing device 10 may further perform A / D conversion on the input audio signal.

[0145] The input audio signal corresponds to, for example, at least one of the following.

[0146] Music content (e.g., singing, playing, or a combination of these (i.e., music). It can also include voice content accompanying moving image content.)

[0147] Voice content (e.g., voice of reading, commentary, broadcast, radio drama, personal performance, conversation, monologue, or a combination thereof. Voice content accompanying moving image content may be included.)

[0148] Other audio content (e.g., electronic, ambient, or mechanical sounds)

[0149] The singing or speech content is not limited to the speech produced by human vocal organs, but can include speech generated by speech synthesis technology.

[0150] After step S110 , the signal processing device 10 performs generation of an output audio signal ( S111 ).

[0151] Specifically, the signal processing device 10 generates an output audio signal by performing amplitude modulation on at least a portion of the input audio signal acquired in step S110 .

[0152] As a first example of generating an output audio signal (S111), the signal processing device 10 generates an intermediate audio signal including a plurality of audio signals having different features based on an input audio signal. Here, the feature may be determined based on an input operation performed by a relevant person or an instruction from an external source, or may be determined by an algorithm. For example, the signal processing device 10 may also determine the feature used to generate the intermediate audio signal based on the result obtained by analyzing the input audio signal.

[0153] The feature may be, for example, at least one of the following.

[0154] Characteristics of the sound (especially quality characteristics)

[0155] Frequency characteristics

[0156] Characteristics of time

[0157] Amplitude characteristics

[0158] Output characteristics

[0159] The signal processing device 10 selects one or more audio signals (hereinafter referred to as "object signals") to which amplitude modulation is to be applied from a plurality of audio signals included in the intermediate audio signal. The selection of which audio signal to be used as the object signal may be determined based on an input operation performed by a relevant person or an instruction from the outside, or may be determined by an algorithm. For example, the signal processing device 10 may also determine the object signal based on the characteristics of the audio signal (balance between speech and music, volume changes, type of music, timbre or other characteristics). Thus, the signal processing device 10 can select the object signal in a manner that makes the improvement effect of cognitive function based on modulation higher, or select the object signal in a manner that makes the discomfort caused to the patient less. Among them, the signal processing device 10 may also process all of the plurality of audio signals included in the intermediate audio signal as object signals.

[0160] The signal processing device 10 performs amplitude modulation on the selected object signal. As an example, the signal processing device 10 performs amplitude modulation on the object signal using a modulation function having a frequency corresponding to a gamma wave (e.g., a frequency of 35 Hz to 45 Hz). Specifically, when a modulation function having a periodicity of 35 Hz to 45 Hz is set to A(t), a function representing the waveform of the audio signal before modulation is set to X(t), and a function representing the waveform of the modulated audio signal is set to Y(t),

[0161] Y(t)=A(t)·X(t).

[0162] Thus, a change in amplitude corresponding to the above-mentioned frequency is added to the object signal. Here, the signal processing device 10 may also determine a modulation function used in the amplitude modulation of the object signal or a modulation degree of the amplitude modulation. The modulation function or the modulation degree may be determined based on an input operation performed by a relevant person or an instruction from the outside, or may be determined by an algorithm. In addition, the signal processing device 10 may determine a common modulation function or modulation degree for a plurality of object signals, or may determine a modulation function or modulation degree for each of the plurality of object signals separately.

[0163] The signal processing device 10 generates an output audio signal based on an audio signal (hereinafter referred to as a "non-object signal") that is not selected as an object signal among a plurality of audio signals included in the intermediate audio signal and a modulated object signal. That is, the signal processing device 10 converts the non-object signal and the modulated object signal into an output audio signal. Specifically, the signal processing device 10 synthesizes two or more audio signals among the non-object signal and the modulated object signal, or extracts or separates the audio signal from at least one of the non-object signal and the modulated object signal. There is no limitation on the synthesis method of the audio signal, and for example, it can include signal merging processing, HRTF (Head Related Transfer Function) convolution processing, convolution processing of a transfer function that assigns position information of a sound source, or merging processing after performing these convolution processing. In addition, the signal processing device 10 can also perform at least one of amplification, volume adjustment, and D / A conversion of the output audio signal. On the other hand, when the non-object signal and the modulated object signal match the output format of the audio output device 30 (for example, when the non-object signal and the modulated object signal correspond to a multi-channel audio signal associated with each speaker constituting a surround-sound system as the audio output device 30), such conversion is not necessary. In this case, the non-object signal and the modulated object signal are processed as output audio signals.

[0164] In the second example of generating an output audio signal (S111), the input audio signal includes a plurality of audio signals having different characteristics. The signal processing device 10 selects one or more audio signals to which amplitude modulation is applied from the plurality of audio signals included in the input audio signal. The second example of generating an output audio signal (S111) can be understood by appropriately referring to the "intermediate audio signal" in the description of the first example as the "input audio signal".

[0165] In the third example of generating an output audio signal (S111), the signal processing device 10 performs amplitude modulation on the input audio signal. The amplitude modulation on the input audio signal is the same as the amplitude modulation on the object signal described in the first example of generating an output audio signal (S111). Thus, a change in amplitude corresponding to the frequency of the gamma wave is added to the input audio signal. Here, the signal processing device 10 may also determine a modulation function used in the amplitude modulation of the input audio signal or a modulation degree of the amplitude modulation. The modulation function or the modulation degree may be determined based on an input operation performed by a relevant person or an instruction from the outside, or may be determined by an algorithm.

[0166] The signal processing device 10 generates an output audio signal based on the modulated input audio signal. That is, the signal processing device 10 converts the modulated input audio signal into an output audio signal. Specifically, in the case where the modulated input audio signal is composed of multiple audio signals, the signal processing device 10 synthesizes two or more audio signals among the multiple audio signals, or extracts or separates the audio signal from the modulated input audio signal. The details of the audio signal synthesis method are as described in the first example of generating the output audio signal (S111). On the other hand, in the case where the modulated input audio signal matches the output form of the audio output device 30 (for example, the modulated input audio signal is equivalent to a multi-channel audio signal associated with each speaker constituting the surround sound system of the audio output device 30), the above-mentioned conversion is not required. In this case, the modulated input audio signal is processed as the output audio signal.

[0167] After step S111 , the signal processing device 10 performs transmission of an output audio signal ( S112 ).

[0168] Specifically, the signal processing device 10 transmits the output audio signal generated in step S111 to the audio output device 30. The audio output device 30 generates a sound corresponding to the output audio signal.

[0169] After step S112 , the signal processing device 10 performs storage of the listening history ( S113 ).

[0170] Specifically, the signal processing device 10 stores the listening history records including at least one of the following in the listening history database ( Figure 5 ).

[0171] Information on the feature amount of the audio signal to which amplitude modulation is applied in step S111

[0172] Information on the modulation method of the amplitude modulation performed in step S111

[0173] Information about the date and time when the audio output device 30 starts playing the output audio signal sent in step S112

[0174] Information about the playback volume that has been set for the audio output device 30 when playing the output audio signal sent in step S112

[0175] Information about the date and time when the audio output device 30 ends playing the output audio signal sent in step S112

[0176] The signal processing device 10 ends with step S113. Figure 6 Audio signal processing.

[0177] In addition, the signal processing device 10 can perform centralized processing on an input audio signal with a fixed playback period (such as a piece of music content). Figure 6 The audio signal processing may also be repeated at each specified playback interval (e.g., every 100ms) of the input audio signal. Figure 6 Alternatively, the signal processing device 10 may continuously modulate the input audio signal and output the modulated audio signal, for example, as in the case of modulation based on analog signal processing. The processing may also be terminated in response to a specific termination condition (for example, a certain period of time has passed, an operation has been performed by a relevant person, or the output history of the modulated sound has reached a specified state). Figure 6 The audio signal processing shown.

[0178] (4-2) Information Notification Processing

[0179] The information notification process according to the present embodiment will be described. Figure 7 This is a flowchart of the information notification process according to the present embodiment. Figure 8 : is a diagram showing an example of a score function used in the information notification process of the present embodiment. Fig. 9 This is a diagram showing an example of presentation of target information in the information notification process according to the present embodiment. Fig.10 This is a diagram showing an example of presentation of target information in the information notification process according to the present embodiment. Fig.11This is a diagram showing an example of presentation of target information in the information notification process according to the present embodiment.

[0180] Starts in response to any of the following start conditions being met Figure 7 Information notification processing.

[0181] ·Called out due to other processing or external instructions Figure 7 Information notification processing.

[0182] · The relevant person has carried out the Figure 7 Information notification processing operation.

[0183] The signal processing device 10 is changed to a predetermined state (for example, the power is turned on).

[0184] ·The specified date and time has arrived.

[0185] From a specified event (e.g., the signal processing device 10 is started, a new listening history is saved, or the last execution Figure 7 The specified time has passed since the information notification processing).

[0186] like Figure 7 As shown, the signal processing device 10 performs acquisition of listening history (S120).

[0187] Specifically, the signal processing device 10 retrieves the listening history database ( Figure 5 ) obtains the patient's listening history. As an example, the signal processing device 10 extracts the listening history belonging to the period (hereinafter referred to as "target period") during which at least one of the start date and time information and the end date and time information is the object of object information generation from the listening history database. Here, the target period can also be any of the following.

[0188] ·Executed Figure 7 The day, week or month to which the information notification was processed

[0189] From execution Figure 7 The prescribed period from the time of information notification processing to the past

[0190] ・The period specified by the relevant party

[0191] ·The period specified by the algorithm

[0192] After step S120 , the signal processing device 10 performs generation of object information ( S121 ).

[0193] Specifically, the signal processing device 10 generates the object information based on the listening history acquired in step S120. The object information may include at least one of the following, for example.

[0194] Listening history (e.g., the period during which the patient listened to the sounds used to improve cognitive function)

[0195] Information obtained by processing listening history (e.g., statistical information)

[0196] · A score related to the magnitude of the improvement in cognitive function corresponding to the sound heard by the patient

[0197] Regarding the calculation of the score, as an example, the signal processing device 10 can refer to a score function for deriving a score corresponding to an element included in the listening history (e.g., a listening period of the sound, a playback volume of the output audio signal, a modulation method of amplitude modulation applied to the audio signal, a feature amount of the audio signal to which amplitude modulation is applied, or a combination thereof). The score function can be stored in the storage device 11 or in a storage device of an external device (e.g., a server) accessible to the signal processing device 10.

[0198] Here, the score is defined as: the degree of improvement of cognitive function, and is related to the estimated result of the amount of brain waves generated by sound stimulation (i.e., to what extent gamma waves are induced), the index calculated according to the synchronization of brain waves at each moment (i.e., to what extent the synchronization of gamma waves is achieved) or the index calculated according to the ratio of brain waves generated at each moment (i.e., the historical record of to what extent gamma waves are induced compared to the frequency band of other brain waves). As an example, the score is determined in a manner that increases (e.g., increases linearly) relative to the increase in the amount of brain waves generated. As another example, the score is determined in the following manner: within the range where the amount of brain waves generated is less than the first threshold, the score hardly increases, and after the amount of brain waves generated exceeds the first threshold, the score increases relative to the increase in the amount of brain waves generated. As another example, the score is determined in the following manner: within the range where the amount of brain waves generated is less than the second threshold, the score increases relative to the increase in the amount of brain waves generated, and after the amount of brain waves generated exceeds the second threshold, the score hardly increases (i.e., saturation). By using an electroencephalogram to measure the amount of brain waves generated by a subject when the subject listens to a variety of audio signals under a variety of conditions, it is possible to describe the relationship between the elements contained in the listening history record and the amount of brain waves generated. Based on the relationship between the elements contained in the listening history record and the amount of brain waves generated, and the relationship between the score and the amount of brain waves generated, a score function can be created to derive the score corresponding to the elements contained in the listening history record. The score function can be defined as a formula with the elements of the listening history record as independent variables, or as a lookup table.

[0199] As an example, the signal processing device 10 uses Figure 8The score function shown is used to calculate the score corresponding to the listening history. Figure 8 In the score function of , the score is defined as increasing or maintaining (in other words, not decreasing) relative to the increase in the listening period. Figure 8 In the score function, the score in the case of playback volume C1 is defined as being higher than the score in the case of playback volume C2 (smaller than C1).

[0200] When the signal processing device 10 acquires a plurality of listening history records in step S120 , the signal processing device 10 may calculate the transition of the cumulative score during the target period (ie, the change in the cumulative score over time) by accumulating the scores corresponding to the respective listening history records.

[0201] When the patient's target score for the target period has been determined, the signal processing device 10 may also calculate the ratio of the cumulative score to the target score as the patient's achievement. The target score is equivalent to the benchmark of the score (that is, the sound that the patient should listen to) that the patient should obtain during the target period in order to improve cognitive function. The signal processing device 10 may calculate the achievement at a point in time or the change in the achievement during the target period (that is, the change in time-achievement).

[0202] The signal processing device 10 may store the score calculation results in a database (not shown). In this way, the scores calculated in the past can be acquired in a short time while suppressing the consumption of calculation resources.

[0203] After step S121, the signal processing device 10 performs notification of object information (S122).

[0204] Specifically, the signal processing device 10 sends the information required to present the object information generated in step S121 to the stakeholder terminal 70 (for example, the object information itself, or information related to the actions of light, images, sounds, text, equipment, etc. that can be used to present the object information).

[0205] Information related to the notification destination of the target information (hereinafter referred to as "notification destination information") is defined in a manner associated with information for identifying a patient (e.g., patient ID). The notification destination information is an account name, email address, or telephone number, etc., of a stakeholder to be notified. By using the notification destination information, it is possible to identify the stakeholder terminal 70 to which the information required to present the target information is to be sent.

[0206] The signal processing device 10 can notify the object information in at least one of the following forms, for example.

[0207] App notifications

[0208] SNS (Social Networking Service) messages

[0209] ·e-mail

[0210] SMS (Short Service Message) messages

[0211] Information (e.g., messages) that can be viewed after logging into a specific application or SaaS (Software as a Service)

[0212] The stakeholder terminal 70 presents the object information to the stakeholder via the output device. The signal processing device 10 can express the object information using at least any one of light, sound, text, image, and the operation of a device.

[0213] As described above, the object information can include a score. The score included in the object information can also be at least one of the following.

[0214] · Score for each listening history

[0215] ·Cumulative points at any point in time during the target period

[0216] ・Changes in accumulated points during the applicable period

[0217] · Achievement degree at any point in time during the target period

[0218] ・Changes in the degree of achievement during the target period

[0219] The degree of non-achievement relative to the target score

[0220] As an example, Fig. 9 , Fig.10 and Fig.11 As shown, the stakeholder terminal 70 can present the object information to the stakeholder through an image displayed on the display 81 .

[0221] When the stakeholder has the authority to view the object information of multiple patients (typically, the stakeholder corresponds to a medical stakeholder or a nursing stakeholder, etc.), the stakeholder terminal 70 Fig. 9 The screen P80 is displayed on the display 81. On the other hand, when the stakeholder has the authority to view only the object information of one patient (typically, the stakeholder is the patient himself or a family member of the patient, etc.), the stakeholder terminal 70 will Fig. 9 Screen P81 is displayed on display 81.

[0222] Screen P80 displays a list of patients whose information can be viewed. Fig. 9As shown, the screen P80 includes an object A801.

[0223] The object A801 ​​receives a user instruction for designating a patient whose object information is to be browsed. When the object A801 ​​is selected (or the name is clicked), the stakeholder terminal 70 displays a screen P81 on the display 81.

[0224] Screen P81 is a menu screen for selecting which information of the patient to view. Fig. 9 As shown, the screen P81 includes objects A811 to A812. In addition, the selectable objects or displayed contents may be limited in the menu screen according to the authority of the relevant person (for example, whether the person is a doctor).

[0225] The object A811 receives a user instruction to display the basic information of the patient. When the object A811 is selected, the stakeholder terminal 70 displays a screen (not shown) for displaying the basic information of the patient on the display 81 .

[0226] The object A812 accepts a user instruction to display the patient's score, etc. When the object A812 is selected, the stakeholder terminal 70 Fig.10 The screen P82 shown is displayed on the display 81.

[0227] The object A813 receives a user instruction to display the results of the cognitive function test of the patient. When the object A811 is selected, the stakeholder terminal 70 displays a screen (not shown) showing the results of the cognitive function test of the patient on the display 81 .

[0228] The object A814 receives a user instruction to display the medical records of the patient. When the object A814 is selected, the stakeholder terminal 70 displays a screen (not shown) showing the result of displaying the medical records of the patient on the display 81 .

[0229] Screen P82 is a dashboard screen related to the patient's listening status. Fig.10 As shown, the screen P82 includes objects A821 to A827. In addition, selectable objects or display contents may be limited on the dashboard screen according to the authority of the relevant person (for example, whether the person is a doctor).

[0230] The object A821 displays the patient's achievement level today (that is, the browsing day of the screen P82). The object A821 expresses the patient's achievement level using a pie chart and numbers.

[0231] The object A822 accepts a user instruction to display the detailed listening history of the patient. When the object A822 is selected, the stakeholder terminal 70 displays a screen (not shown) showing the detailed listening history of the patient on the display 81 .

[0232] The object A823 displays the patient's past achievement history in a bar graph. The achievement history displayed in the object A823 can be switched in units of years, months, or weeks. When a bar representing the achievement of a certain day in the object A823 is selected, the stakeholder terminal 70 will Fig.11 Screen P83 is displayed on display 81.

[0233] The object A824 receives a user instruction to display the history of doctor's comments. When the object A824 is selected, the stakeholder terminal 70 displays a screen (not shown) displaying the history of doctor's comments on the display 81.

[0234] The object A825 receives a user instruction to display the results of the cognitive function test of the patient. When the object A825 is selected, the stakeholder terminal 70 displays a screen (not shown) showing the results of the cognitive function test of the patient on the display 81 .

[0235] The object A826 accepts a user instruction for displaying statistical information. When the object A826 is selected, the stakeholder terminal 70 displays a screen (not shown) showing the statistical information on the display 81. Here, the statistical information is equivalent to the result obtained by statistically processing the listening history records or scores collected about a plurality of patients. The listening history records or scores of a plurality of patients can be collected from the signal processing device 10 and other signal processing devices having the same function as the signal processing device 10 by an external device (e.g., a server not shown). The stakeholder terminal 70 can also display the statistical information in a manner of comparing it with the patient's listening history records or scores.

[0236] The object A827 receives a user instruction to display an editing screen of a message for transmission. When the object A827 is selected, the stakeholder terminal 70 displays an editing screen (not shown) of the message for transmission on the display 81.

[0237] Screen P83 is a screen showing the listening status of the patient on a specific date selected by the stakeholders. Fig.11 As shown, screen P83 includes objects A831 to A834.

[0238] The object A831 displays the achievement level of the patient on the selected specific date. The object A831 expresses the achievement level of the patient using a pie chart and numbers.

[0239] Object A832 uses a line graph to show the changes in the achievement of patients on a specific date selected. In object A832, the time axis clearly shows the period when the increase rate of the achievement per unit time is high ("high efficiency"), the period when the increase rate of the achievement per unit time is low ("low efficiency"), the period when the processing (amplitude modulation) of the input audio signal is invalid ("processing off"), and the period when the playback of the output audio signal is stopped ("voice off"). In addition, the classification of efficiency can also be three or more levels instead of two levels. In addition, the current time can also be displayed.

[0240] The object A833 accepts a user instruction to move the selected date to the previous day. When the object A833 is selected, the stakeholder terminal 70 updates the contents of the objects A831 and A832.

[0241] The object A834 accepts a user instruction to shift the selected date to a date one day after the selected date. When the object A834 is selected, the stakeholder terminal 70 updates the contents of the objects A831 to A832.

[0242] exist Figures 9 to 11 , an example of displaying object information related to the listening status of a selected patient is shown. However, the present invention is not limited thereto, and the signal processing device 10 may send object information related to multiple patients to the stakeholder terminal 70, and the stakeholder terminal 70 may simultaneously display the object information related to multiple patients. For example, the stakeholder terminal 70 may also display the information of the screen P82 related to multiple patients in a row on the display 81. In addition, the number of items of information displayed on the display 81, the display size, or the amount of information may be changed according to the number of patients who are the display objects of the information. In this way, the object information related to multiple patients can be efficiently displayed on one screen.

[0243] In addition, the object information may also be presented on an application screen or a website screen, or may be presented using a browsing screen of an email or other message (such as an SNS message, an SMS message, etc.). Alternatively, the object information may also be displayed on a screen of a smart speaker, a television, a smart watch, a car navigation system, or the like.

[0244] Alternatively, the stakeholder terminal 70 may express the score by sound output from a speaker (an example of an output device).

[0245] The signal processing device 10 ends with step S122. Figure 7 Information notification processing.

[0246] (5) Summary

[0247] As described above, the signal processing device 10 of this embodiment outputs an audio signal (that is, an output audio signal) having a change in amplitude corresponding to the frequency of the gamma wave generated by amplitude modulation, and stores information (for example, listening history) of the output audio signal. The signal processing device 10 sends a notification based on the stored information to a device (that is, a stakeholder terminal 70) outside the signal processing device 10. As a result, the stakeholder can understand the patient's listening status of the sound for improving cognitive function.

[0248] The signal processing device 10 may also send a notification to at least one of a device held by the patient (that is, the user of the audio output device 30), a device held by the patient's family, a device held by the patient's caregiver, and a device held by a medical practitioner. Thus, at least one of the patient, the patient's family, the patient's caregiver, and the medical practitioner can understand the patient's listening status.

[0249] The information of the audio signal stored by the signal processing device 10 may also include information related to at least one of the period of outputting the audio signal, the modulation method applied to the audio signal, the size (volume) of the sound emitted according to the audio signal, and the feature quantity of the audio signal. Thus, the details of the patient's listening condition can be notified to relevant parties.

[0250] The notification may include information indicating at least one of the listening history and the score determined based on the listening history. Thus, stakeholders can understand the details of the patient's listening status.

[0251] The output audio signal may have a change in amplitude corresponding to a frequency of 35 Hz or more and 45 Hz or less. This can induce gamma waves in the patient's brain and achieve an improvement effect on cognitive function.

[0252] (6) Modification

[0253] Modifications of this embodiment will be described.

[0254] (6-1) Modification 1

[0255] Modification 1 will be described. Modification 1 is an example in which notification of object information is performed conditionally.

[0256] (6-1-1) Information notification processing

[0257] The information notification process according to Modification 1 is described. Fig.12 This is a flowchart of the information notification process of Modification Example 1.

[0258] The information notification process of Modification 1 is started in response to the same start condition as that of the information notification process of the present embodiment being satisfied.

[0259] like Fig.12 As shown, the signal processing device 10 executes acquisition of listening history (S120) and generation of object information (S121). The details of acquisition of listening history (S120) and generation of object information (S121) are as described in this embodiment.

[0260] After step S121, the signal processing device 10 performs determination of a notification condition (S223).

[0261] Specifically, the signal processing device 10 acquires information corresponding to the determination material, and determines whether a predetermined notification condition is satisfied based on the acquired information. The notification condition may be editable by a stakeholder. The notification condition may be defined for each stakeholder.

[0262] As a first example of determining the notification condition (S223), the signal processing device 10 may also obtain the listening history records collected about the patient and determine whether the acquired listening history records meet the specified notification condition. For example, if no listening history records were recorded in the last week, or if the listening history records were recorded every day in the last three days, the signal processing device 10 may determine that the notification condition is met.

[0263] As a second example of determining the notification condition (S223), the signal processing device 10 may also obtain the score calculated for the patient and determine whether the obtained score satisfies the specified notification condition. For example, if the average of the achievement degree for each day in the past week is lower than or higher than the threshold, the signal processing device 10 may determine that the notification condition is satisfied.

[0264] As a third example of determining the notification condition (S223), the signal processing device 10 may also obtain the result of a cognitive function test on the patient, and determine whether the obtained result satisfies the specified notification condition. For example, when the result of the cognitive function test indicates that the level of the patient's cognitive function has changed (improved or deteriorated), the signal processing device 10 determines that the notification condition is met. In addition, the signal processing device 10 may also obtain the result of the cognitive function test from an external device, or obtain the result by performing a cognitive function test on the patient by the signal processing device 10 itself. In other words, the signal processing device 10 may also be equipped with a function of performing a cognitive function test (for example, a function of measuring cognitive function through voice dialogue with the patient).

[0265] In step S223, when it is determined that the notification condition is satisfied, the signal processing device 10 executes the notification of the target information (S122). The details of the notification of the target information (S122) are as described in the present embodiment.

[0266] The signal processing device 10 ends with step S122. Fig.12 Information notification processing.

[0267] In addition, in step S223, if it is determined that the notification condition is satisfied, the signal processing device 10 ends Fig.12 Information notification processing.

[0268] (6-1-2) Summary

[0269] As described above, the signal processing device 10 of the modification example 1 conditionally notifies the object information. For example, the signal processing device 10 may also send a notification in response to the listening history record or score satisfying a prescribed condition. Thus, the patient's signs of giving up listening to the sound or the patient's efforts are fed back to the stakeholders in a timely manner, so that the stakeholders can easily encourage the patient to continue listening to the sound.

[0270] Alternatively, the signal processing device 10 may obtain the results of the patient's cognitive function test and send a notification in response to the result of the cognitive function test satisfying a specified condition. This can encourage stakeholders to use the results of the cognitive function test as an opportunity to study the patient's future actions (e.g., setting goals related to listening to sounds for improving cognitive function).

[0271] (6-2) Modification 2

[0272] Modification 2 is described. Modification 2 is an example in which object information is generated in consideration of the patient's listening environment.

[0273] (6-2-1) Audio system structure

[0274] The structure of the audio system is described. Fig.13 This is a block diagram showing the structure of an audio system according to Modification Example 2.

[0275] like Fig.13 As shown, the audio system 2 includes a signal processing device 10 , an audio output device 30 , a sound source device 50 , a stakeholder terminal 70 , and a sensor 90 .

[0276] The signal processing device 10 and the sensor 90 may be connected to each other via a network (for example, the Internet or an intranet), or may be connected to each other via a predetermined interface (for example, a wireless interface using Bluetooth or the like).

[0277] The sensor 90 performs sensing related to the patient's listening environment. Specifically, the sensor 90 measures a physical quantity related to at least one of the following.

[0278] The patient's listening position (particularly relative position to the audio output device 30)

[0279] The sound pressure generated by the sound emitted from the audio output device 30 at the patient's listening position

[0280] The patient's behavior when listening to the sound emitted from the audio output device 30 (e.g., whether he is concentrating on listening or paying attention to other situations)

[0281] The sensor 90 can include, for example, at least one of the following.

[0282] ・Equipment (e.g., equipment that sends or receives wireless signals) that constitutes a positioning system based on radio waves (e.g., Bluetooth's AoA (Angle of Arrival))

[0283] Distance measuring sensors (e.g., ultrasonic sensors or LIDAR (Light Detection And Ranging))

[0284] For example, pressure sensors embedded in chairs, beds, or floors

[0285] Camera (visible light camera or infrared camera)

[0286] ·microphone

[0287] (6-2-2) Information Processing

[0288] The information processing of Modification 2 is described.

[0289] (6-2-2-1) Audio Signal Processing

[0290] The audio signal processing according to Modification 2 is described. Fig.14 This is a flowchart of audio signal processing according to Modification Example 2.

[0291] The audio signal processing of Modification 2 is started in response to the same start condition as that of the audio signal processing of the present embodiment being satisfied.

[0292] like Fig.14 As shown, the signal processing device 10 performs acquisition of an input audio signal (S110), generation of an output audio signal (S111), and transmission of the output audio signal (S112). The details of acquisition of an input audio signal (S110), generation of an output audio signal (S111), and transmission of an output audio signal (S112) are as described in this embodiment.

[0293] After step S212 , the signal processing device 10 performs determination of the listening environment ( S214 ).

[0294] Specifically, the signal processing device 10 acquires the sensing result of the sensor 90. The signal processing device 10 determines the patient's listening environment based on the sensing result. The patient's listening environment can include, for example, at least one of the following.

[0295] The distance from the audio output device 30 to the patient's listening position or the classification result thereof

[0296] The direction from the audio output device 30 to the patient's listening position or its classification result

[0297] The sound pressure generated by the sound emitted from the audio output device 30 at the patient's listening position (that is, the degree to which the sound emitted from the audio output device 30 is attenuated) or the classification result thereof

[0298] The patient's behavior when listening to the sound emitted from the audio output device 30 (e.g., type of activity such as sitting, lying down, exercising, operating a smartphone, etc.)

[0299] After step S214 , the signal processing device 10 performs storage of the listening history ( S113 ).

[0300] Specifically, the signal processing device 10 stores the listening history including information on the determination result in step S214 (hereinafter referred to as "listening environment information") in the listening history database. Other elements that can be included in the listening history are as described in the present embodiment.

[0301] The signal processing device 10 ends with step S113. Fig.14 Audio signal processing.

[0302] In addition, the signal processing device 10 can perform centralized processing on an input audio signal with a fixed playback period (such as a piece of music content). Fig.14 The audio signal processing may also be repeated at each specified playback interval (e.g., every 100ms) of the input audio signal. Fig.14 Alternatively, the signal processing device 10 may continuously modulate the input audio signal and output the modulated audio signal, for example, as in the case of modulation based on analog signal processing. The processing may also be terminated in response to a specific termination condition (for example, a certain period of time has passed, an operation has been performed by a relevant person, or the output history of the modulated sound has reached a specified state). Fig.14 The audio signal processing shown.

[0303] (6-2-2-2) Information Notification Processing

[0304] The information notification process of Modification 2 is described.

[0305] The information notification process of the second modification is started in response to the same start condition as that of the information notification process of the present embodiment being satisfied.

[0306] The signal processing device 10 acquires the listening history ( S120 ). The details of acquiring the listening history ( S120 ) are as described in this embodiment.

[0307] After step S120 , the signal processing device 10 performs generation of object information ( S121 ).

[0308] Specifically, the signal processing device 10 generates object information based on the listening history record acquired in step S120. For example, the generated object information may include listening environment information or information obtained by processing the listening environment information (e.g., statistical information). In addition, the signal processing device 10 may also use a score function to calculate the score corresponding to the listening history record in the same manner as in the present embodiment. In variant 2, the signal processing device 10 may also correct the score corresponding to the listening history record based on the listening environment information contained in the listening history record.

[0309] As a first example of generating object information (S121), the signal processing device 10 performs correction in a manner that reduces or maintains the score relative to the increase in the distance from the audio output device 30 to the patient's listening position. As an example, the score before correction is the same, and the signal processing device 10 performs correction when the distance is a first value so that the score becomes smaller than when the distance is a second value (the second value is smaller than the first value). The sound pressure reaching the patient is reduced at the listening position away from the audio output device 30, and the modulation degree is relatively reduced due to the echo in the room or external sound, so that the effect of improving cognitive function based on listening may be reduced, but a more reasonable score can be obtained by performing the above correction.

[0310] As a second example of generating object information (S121), the signal processing device 10 performs correction in a manner that reduces or maintains the score relative to the increase in the size of the difference between the direction from the audio output device 30 to the patient's listening position and the direction of the directivity of the audio output device 30 (hereinafter referred to as "direction difference"). As an example, the score before correction is the same, and the signal processing device 10 performs correction when the direction difference is a first value, so that the score becomes smaller than the case where the direction difference is a second value (the second value is smaller than the first value). The correction amount can also be determined based on the characteristics (gain characteristics) of the directivity of the audio output device 30. When the listening position is located in a direction away from the direction of the directivity of the audio output device 30, a decrease in the sound pressure reaching the patient or a decrease in the modulation degree occurs, so that the effect of improving cognitive function based on listening may be reduced, but a more reasonable score can be obtained by performing the above correction.

[0311] As a third example of generating object information (S121), the signal processing device 10 performs correction in a manner that reduces or maintains the score relative to the reduction in sound pressure or modulation generated by the sound emitted from the audio output device 30 at the patient's listening position. As an example, the score before correction is the same, and the signal processing device 10 performs correction when the sound pressure is a first value so that the score becomes smaller than when the sound pressure is a second value (the second value is greater than the first value). When a reduction in sound pressure or a reduction in modulation that reaches the patient occurs, the improvement effect of cognitive function based on listening may be reduced, but a more reasonable score can be obtained by performing the above correction.

[0312] As a fourth example of generating the object information (S121), a correction amount is determined in advance for each behavior (e.g., listening condition, awake state, or stress level) of the patient when listening to the sound emitted from the audio output device 30, and the signal processing device 10 corrects the score according to the correction amount. The cognitive function improvement effect based on listening may vary depending on the behavior of the patient when listening to the sound emitted from the audio output device 30, but a more reasonable score can be obtained by performing the above correction.

[0313] After step S120, the signal processing device 10 performs notification of the object information (S122). The details of the notification of the object information (S122) are as described in the present embodiment.

[0314] (6-2-3) Summary

[0315] As described above, the signal processing device 10 of the second modification can also determine the listening environment of the patient listening to the sound corresponding to the output audio signal, and generate object information based on the determination result. Thus, it is possible to notify the relevant parties of information related to the patient's listening environment, or to notify the relevant parties of the score calculated by considering the influence of the patient's listening environment on the improvement effect of cognitive function.

[0316] (7) Other Modifications

[0317] The storage device 11 may be connected to the signal processing device 10 via a network. The display 21 may be integrated with the signal processing device 10. The storage device 71 may be connected to the stakeholder terminal 70 via a network. The display 81 may be integrated with the stakeholder terminal 70.

[0318] Each step of the above-mentioned information processing can be performed by either the signal processing device 10 or the stakeholder terminal 70. A part of the above-mentioned information processing steps may be performed by the stakeholder terminal 70 or an external device (e.g., a server) not shown. For example, the signal processing device 10 may store the listening history in an external device, the external device may generate object information based on the listening history, and the signal processing device 10 may obtain and notify the object information generated by the external device. In this case, the signal processing device 10 may also omit the process of storing the listening history in an external device. Figure 7 In the process of S120, a score is acquired from an external device in S121.

[0319] In the above description, an example is shown in which the signal processing device 10 is connected to one audio output device 30. The signal processing device 10 may be connected to a plurality of audio output devices 30, and in this case, the signal processing device 10 may be configured to select to which audio output device 30 the output audio signal is transmitted.

[0320] In the above description, an example is shown in which the signal processing device 10 is connected to one sound source device 50. The signal processing device 10 may be connected to a plurality of sound source devices 50, and in this case, the signal processing device 10 may be configured to select which sound source device 50 to acquire the input audio signal from.

[0321] In the above description, the case where one signal processing device 10 corresponds to one patient is described as the center. However, it is not limited to this, and multiple patients can also correspond to one signal processing device 10. For example, consider the case where there are multiple patients (users who listen to the sound output by the audio output device 30) in a room where the signal processing device 10 and the speaker as the audio output device 30 are set. In this case, the signal processing device 10 identifies the multiple patients in the room, and updates the listening history database corresponding to each patient based on the output result of the audio signal output by the signal processing device 10. Thus, even in the case where multiple patients listen to the sound based on the audio signal output from one signal processing device 10, it is possible to easily grasp the listening status of each patient. As a method for the signal processing device 10 to identify the patient who listens to the sound emitted by the audio output device 30 (for example, a patient in the same room as the audio output device 30), there is, for example, a method of obtaining the ID information registered by the patient when entering the room. In addition, for example, there is a method of determining the patient who is near the audio output device 30 by performing wireless communication such as Bluetooth between the terminal device held by the patient and the audio output device 30. However, the present invention is not limited thereto, and the signal processing device 10 can use various existing recognition methods.

[0322] In addition, multiple signal processing devices 10 can also be associated with one patient. For example, consider a case where signal processing devices 10 and speakers as audio output devices 30 are respectively installed in multiple rooms in a facility where a patient lives. In this case, multiple signal processing devices 10 in the facility access a common listening history database. Moreover, each signal processing device 10 identifies a patient who is in the same room as the audio output device 30 connected to the signal processing device 10, and updates the listening history database corresponding to the patient based on the output result of the audio signal output by the signal processing device 10. Thus, even in the case where a patient listens to sounds based on audio signals output from multiple signal processing devices 10, the patient's listening status can be easily grasped. As a method for the signal processing device 10 to identify a patient, the various methods described above can be used.

[0323] Similarly, a plurality of signal processing devices 10 may be associated with a plurality of patients. That is, each of the plurality of signal processing devices 10 may update a listening history database associated with each of a plurality of patients who listen to sounds based on the audio signals output from the signal processing device 10. Furthermore, the signal processing device 10 may send a notification based on the information acquired from the listening history database to the stakeholder terminal 70 associated with each patient.

[0324] The signal processing device 10 may be equipped with a function of performing a cognitive function test (for example, a function of measuring cognitive function by voice conversation with the patient). In this case, the signal processing device 10 may also notify the stakeholder terminal 70 of object information including information indicating at least one of the content of the patient's answer to the cognitive function test and the result of the patient's cognitive function test. In this way, the stakeholder can timely grasp the patient's cognitive function in addition to the patient's voice listening status.

[0325] In Modification 2, an example in which the score is calculated in consideration of the listening environment information is shown. The signal processing device 10 may calculate the score in consideration of other information.

[0326] As a first example, the signal processing device 10 may also obtain information related to the attributes of the patient who listens to the sound corresponding to the output audio signal (hereinafter referred to as "attribute information"), and further calculate a score based on the attribute information. The attribute information may include, for example, at least one of the following: the patient's age, gender, hearing (which may include hearing or speech listening ability in different frequency bands), risk of cognitive dysfunction, past listening history (which may include analysis results of past listening history), or preferences or willingness (involvement) for sound listening, past results of electroencephalogram measurements or psychological experiments, etc., or feature quantities based on the results. Thus, a score obtained by considering the influence of the patient's attributes on the improvement effect of cognitive function can be obtained. The signal processing device 10 may also use a sensor 90 (particularly a device or camera constituting a positioning system based on radio waves) to identify the patient. In order to measure the patient's hearing, a hearing test function may also be installed in the signal processing device 10 or the relevant party terminal 70.

[0327] As a second example, the signal processing device 10 may also obtain the result of a cognitive function test of a patient who listens to a sound corresponding to the output audio signal (that is, the patient's current cognitive function), and also calculate a score based on the result of the test. Thus, a score can be obtained that takes into account the effect of the patient's current cognitive function on the improvement effect of cognitive function. The signal processing device 10 or the stakeholder terminal 70 may also be equipped with a function for performing a cognitive function test (for example, a function for measuring cognitive function through voice dialogue with the patient).

[0328] The signal processing device 10 may also change the method of generating the output audio signal (e.g., modulation method, synthesis ratio of two or more audio signals, determination method of audio signal to which amplitude modulation is applied, etc.) or the playback volume of the sound corresponding to the output audio signal according to the patient's score. As a first example, the signal processing device 10 may generate an output audio signal corresponding to a sound that has a higher score that can be obtained by listening as the remaining time until the end of the period in which the target score has been determined is shorter, or the playback volume may be increased as the remaining time until the end of the period in which the target score has been determined is shorter. As a second example, the signal processing device 10 may generate an output audio signal corresponding to a sound that has a higher score that can be obtained by listening as the patient's current achievement degree is lower, or the playback volume may be increased as the patient's current achievement degree is lower. In addition, when the achievement degree of the score does not meet the prescribed benchmark, the signal processing device 10 may automatically play an audio signal that has been amplitude modulated, or may play a prescribed audio signal with a large amount of brain wave generation (e.g., a buzzer or chirp with a frequency corresponding to a gamma wave, which is a sound unrelated to the input audio signal). Thus, even if the patient is not consciously paying attention to the status of the score being obtained, support can be provided in such a way that the target score can be achieved.

[0329] In the above description, an example of calculating a score based on a listening history record saved by the signal processing device 10 is shown. However, in addition to the above-mentioned listening history record, or instead of the above-mentioned listening history record, the score can also be calculated based on a voice signal received by a microphone (for example, the intensity, duration, waveform or a combination thereof of the voice signal). Such a microphone can be set at a position closer to the patient's listening position than the audio output device 30, or it can be, for example, a microphone provided by a terminal used by the patient or a microphone that can be worn on the patient's body (for example, near the ear).

[0330] The algorithm for determining the method for generating the output audio signal (e.g., the modulation method, the synthesis ratio of two or more audio signals, the method for determining the audio signal to which amplitude modulation is applied, etc.) or the default method for generating the output audio signal may also be changed by updating the firmware of the signal processing device 10. For example, the firmware may be updated by the signal processing device 10 communicating with an external device (e.g., a server not shown). Similarly, the method for generating object information (which may include a method for calculating a score), the method for notifying object information, the method for presenting object information, etc. may also be changed by updating the firmware of the signal processing device 10. The update content of the firmware of the signal processing device 10 may be notified to the relevant parties in advance, and the signal processing device 10 may provide an action mode for the relevant parties to listen to the sounds before and after the update in a comparative manner.

[0331] Although an example of acquiring the input audio signal from the sound source device 50 is shown, the input audio signal may be generated by receiving the ambient sound from a microphone. The input audio signal may also be read from the storage device 11 or a removable medium connected to the signal processing device 10. The sound source device 50 (including the sound source device 50 built into the signal processing device 10) may also generate the input audio signal at any time.

[0332] In the above description, an example of modulating an audio signal derived from an input audio signal is shown. However, an output audio signal may be generated based on the input audio signal and a modulated audio signal that is not derived from the input audio signal.

[0333] In the above description, the example in which the modulation function has a frequency of 35 Hz or more and 45 Hz or less is mainly used for explanation. However, the modulation function used by the signal processing device 10 is not limited to this, as long as it is a modulation function that affects the induction of gamma waves in the brain of the listener. For example, the modulation function may also have a frequency of 25 Hz or more and 140 Hz or less. In addition, for example, the frequency of the modulation function may change over time, and the modulation function may partially have a frequency less than 35 Hz or a frequency higher than 45 Hz.

[0334] In the above description, the case where the output audio signal generated by the signal processing device 10 is output to the audio output device 30 that emits sound and allows the patient to hear it is described. However, the output destination of the output audio signal of the signal processing device 10 is not limited to this. For example, the signal processing device 10 may also output the output audio signal to an external storage device or information processing device via a communication network or by broadcasting. At this time, the signal processing device 10 may also output the input audio signal that has not been modulated together with the output audio signal generated by the modulation process to an external device. Thus, the external device can arbitrarily select one of the audio signal that has not been modulated and the audio signal that has been modulated to play.

[0335] In addition, the signal processing device 10 may output information indicating the content of the modulation process to an external device together with the output audio signal. The information indicating the content of the modulation process includes, for example, any of the following.

[0336] Information indicating the sound source corresponding to the modulated audio signal

[0337] Information indicating the channel corresponding to the modulated audio signal

[0338] Information indicating the characteristics of the modulated audio signal

[0339] Information indicating the modulation function

[0340] ・Indicates the modulation information

[0341] ・Indicates volume information

[0342] This allows the external device to change the method of reproducing the audio signal according to the content of the modulation process.

[0343] In the above description, an example is shown in which the signal processing device 10 performs modulation processing including amplitude modulation to generate a modulated audio signal. However, the signal processing device 10 may also obtain a modulated audio signal generated by an external device and transmit an output audio signal based on the modulated audio signal. In this case, the signal processing device 10 may also obtain information (e.g., modulation method information or feature quantity information) required for generating a listening history record from the external device.

[0344] The stakeholder terminal 70 may also collect other information (eg, TV viewing status (program type, viewing time, etc.), vital data of the patient obtained from the wearable terminal), and present the collected information together with the object information.

[0345] Furthermore, when the signal processing device 10 acquires the input audio signal and additional information (eg, an ID3 tag in an MP3 file), it may also modify the additional information and output it to an external device together with the output audio signal.

[0346] The embodiments of the present invention have been described in detail above, but the scope of the present invention is not limited to the above embodiments. In addition, the above embodiments can be variously improved and changed within the scope of the gist of the present invention. In addition, the above embodiments and modifications can be combined.

[0347] Description of Reference Numerals

[0348] 1: audio system; 2: audio system; 10: signal processing device; 11: storage device; 12: processor; 13: input and output interface; 14: communication interface; 21: display; 30: audio output device; 50: sound source device; 70: relevant party terminal; 71: storage device; 72: processor; 73: input and output interface; 74: communication interface; 81: display; 90: sensor.

Claims

1. A signal processing device, comprising: an output unit that outputs an audio signal generated by amplitude modulation, the audio signal having a change in amplitude corresponding to a frequency of the gamma wave; a storage unit for storing information of the audio signal output by the output unit; as well as A transmitting unit transmits a notification based on the information stored in the storage unit to a device outside the signal processing device.

2. The signal processing device according to claim 1, wherein: The transmission unit transmits the notification to at least any one of a device held by a user who listens to the audio signal output by the output unit, a device held by a family member of the user, a device held by a caregiver of the user, and a device held by a medical practitioner.

3. The signal processing device according to claim 1, wherein: The audio signal information stored in the storage unit includes information indicating at least any one of a period during which the audio signal is output by the output unit, a modulation method of the output audio signal, a size of the output audio signal, and a feature amount of the output audio signal.

4. The signal processing device according to claim 1, wherein: The notification transmitted by the transmitting unit includes information indicating at least any one of an output history of the audio signal output by the output unit and a score determined based on the output history of the audio signal.

5. The signal processing device according to claim 1, wherein: The transmission unit transmits the notification in response to at least either one of an output history of the audio signal output by the output unit and a score determined based on the output history of the audio signal satisfying a prescribed condition.

6. The signal processing device according to claim 1, wherein: The device also has an acquisition unit, which acquires the result of the cognitive function test. The sending unit sends the notification in response to the result of the cognitive function test acquired by the acquiring unit satisfying a prescribed condition.

7. The signal processing device according to claim 1, wherein: The transmitting unit transmits information indicating at least one of a content of an answer to a cognitive function test and a result of the cognitive function test together with the notification.

8. The signal processing device according to claim 1, wherein: The audio signal output by the output unit has a change in amplitude corresponding to a frequency of 35 Hz or more and 45 Hz or less.

9. A cognitive function improvement system comprising: The signal processing device according to any one of claims 1 to 8; and A unit for causing a user to listen to a sound corresponding to the audio signal output by the signal processing device.

10. A signal processing method is a signal processing method performed by a signal processing device, the signal processing method comprising the following steps: outputting an audio signal generated by amplitude modulation, the audio signal having a change in amplitude corresponding to the frequency of the gamma wave; storing information of the output audio signal; as well as A notification based on the stored information is sent to a device external to the signal processing device. 11 . A program for causing a computer to function as each unit of the signal processing device according to claim 1 .

Images