BEHAVIOR RECOGNITION METHOD, BEHAVIOR RECOGNITION DEVICE, AND BEHAVIOR RECOGNITION PROGRAM

Abstract

An action identification device (1) acquires sound data from a microphone (2), calculates the feature amount of the sound data, determines whether or not a user is present in a space in which the microphone (2) is installed, if the user is not present in the space, calculates a noise feature amount indicating the feature amount of noise on the basis of the calculated feature amount and stores the calculated noise feature amount in a noise feature amount storage unit (108), and if the user is present in the space, subtracts the noise feature amount stored in the noise feature amount storage unit (108) from the calculated feature amount to extract an action sound feature amount indicating the feature amount of action sound generated by the user taking action, and identifies the action of the user using the action sound feature amount.

Description

[Technical field]

[0001] The present disclosure relates to an activity identification method, an activity identification device, and an activity identification program for identifying a user's activity. [Background technology]

[0002] In recent years, monitoring services, home appliance control services, and information presentation services based on human behavior in living spaces have been considered. From the viewpoint of protecting privacy, technology has been developed to estimate human behavior not from images of people but from the sounds that are generated when people behave.

[0003] In order to estimate a person's actions from the action sounds they emit, it is necessary to identify the action sounds they emit. However, in living spaces, various noises occur in addition to the action sounds. When noise is mixed into the action sounds, the signal-to-noise ratio decreases, and there is a risk that the accuracy of action identification will decrease.

[0004] Therefore, for example, Patent Document 1 discloses a technology for reducing noise. The noise reduction device of Patent Document 1 calculates a plurality of feature amounts for a speech-noise mixed signal, analyzes information on speech and noise using the plurality of feature amounts and the input speech-noise mixed signal, calculates reduction variables corresponding to a plurality of noise reduction processes using the analyzed information and the input speech-noise mixed signal, and reduces noise in the input speech-noise mixed signal through the plurality of noise reduction processes using the calculated reduction variables.

[0005] However, in the above-mentioned conventional technology, there is a risk that the action sounds to be classified may also be reduced, making it difficult to classify actions with high accuracy, and further improvement is required. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 4456504 specification Summary of the Invention

[0007] The present disclosure has been made to solve the above problems, and aims to provide a technology that can identify user behavior with higher accuracy.

[0008] A behavior identification method according to one aspect of the present disclosure is a behavior identification method for identifying a user's behavior, in which a computer acquires sound data from a microphone, calculates features of the sound data, determines whether the user is present in a space in which the microphone is installed, and if the user is not present in the space, calculates noise features indicating features of noise based on the calculated features and stores the calculated noise features in a storage unit, and if the user is present in the space, subtracts the noise features stored in the storage unit from the calculated features to extract behavior sound features indicating features of behavior sounds generated by the user's behavior, and identifies the user's behavior using the behavior sound features.

[0009] According to the present disclosure, user behavior can be identified with higher accuracy. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a behavior identification system according to an embodiment of the present disclosure. [Diagram 2] 2 is a diagram for explaining the arrangement of a behavior recognition device, a microphone, and a human presence sensor according to an embodiment of the present disclosure. FIG. [Diagram 3] 2 is a diagram illustrating a configuration of a noise characteristic calculation unit illustrated in FIG. [Figure 4] 1 is a diagram for explaining a noise suppression method according to an embodiment of the present invention; [Diagram 5] 4 is a first flowchart illustrating a behavior identification process in the present embodiment. [Figure 6] 13 is a second flowchart illustrating the behavior identification process in the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] (Findings on which this disclosure is based) In the above conventional technology, noise is reduced from a speech-noise mixed signal in which a human voice and noise are mixed. However, in the above conventional technology, when noise is reduced from a signal in which non-speech action sounds and noise are mixed, there is a risk that the action sounds to be identified will also be reduced, making it difficult to accurately identify actions.

[0012] In order to solve the above problems, an action identification method according to one aspect of the present disclosure is an action identification method for identifying a user's action, in which a computer acquires sound data from a microphone, calculates features of the sound data, determines whether the user is present in a space in which the microphone is installed, and if the user is not present in the space, calculates noise features indicating features of noise based on the calculated features and stores the calculated noise features in a storage unit, and if the user is present in the space, subtracts the noise features stored in the storage unit from the calculated features to extract action sound features indicating features of action sounds generated by the user's action, and identifies the user's action using the action sound features.

[0013] In a space where a user does not exist, only noise other than action sounds generated by the user's actions is detected. Therefore, when a user does not exist in the space, a noise feature indicating the feature of noise is calculated based on the feature of sound data acquired from a microphone arranged in the space, and the calculated noise feature is stored in a storage unit. Then, when a user exists in the space, the noise feature stored in the storage unit is subtracted from the feature of sound data acquired from a microphone arranged in the space. In this way, it is possible to extract only the action sound feature indicating the feature of action sounds in which noise is suppressed in the space. Then, since the user's actions are identified using the feature of action sounds in which noise is suppressed, the user's actions can be identified with higher accuracy even in a space where action sounds and noise are mixed.

[0014] Furthermore, when a user is not present in the space, a noise feature indicating a feature of noise is stored in the storage unit, so that when a user is present in the space, the noise feature stored in the storage unit can be used to acquire the user's action sound in real time, thereby making it possible to identify the user's action in real time.

[0015] In addition, in the above behavior identification method, it is also possible to further obtain identification information for identifying the microphone, and in calculating the features, divide the sound data into frames of a certain interval and calculate the features for each of the frames, and in storing the noise features, determine the number of frames based on the identification information, and calculate an average of the features of each of a plurality of frames of the determined number as the noise feature.

[0016] It can be said that action sounds and noise depend on the space that the microphone is installed in. Therefore, by determining the number of frames based on the identification information for identifying the microphone, it is possible to calculate noise features from noise of an optimal length according to the type of noise generated in the space where the microphone is installed.

[0017] Furthermore, in the above-mentioned behavior identification method, the number of frames determined based on the identification information of the microphone installed in a space where stationary noise with little time variation exists as the noise may be greater than the number of frames determined based on the identification information of the microphone installed in a space where non-stationary noise with large time variation exists as the noise.

[0018] According to this configuration, when stationary noise with little time fluctuation exists as noise, noise features can be calculated with higher accuracy by using noise of a relatively long duration. On the other hand, when non-stationary noise with large time fluctuation exists as noise, long duration noise is not necessary, and noise features can be calculated with higher accuracy by using noise of a relatively short duration.

[0019] In addition, in the above behavior identification method, it is also possible to further obtain identification information for identifying the microphone, and in calculating the feature amount, divide the sound data into frames of a certain interval and calculate the feature amount for each of the frames, and further, when the identification information is a predetermined identification information, calculate an average of the feature amounts of a plurality of frames past the current frame as the noise feature amount, and further subtract the calculated noise feature amount from the calculated feature amount of the current frame to extract the action sound feature amount.

[0020] For example, the echo sound generated by the sound of a person walking echoing off the surrounding walls can be suppressed in real time by using the sound data of the most recent frame. Therefore, if the acquired identification information is the identification information of a microphone installed in a space where the echo sound is generated, the noise can be suppressed in real time by subtracting the average of the feature amounts of each of a plurality of frames prior to the current frame from the feature amount of the current frame.

[0021] In the behavior identification method, the predetermined identification information may be the identification information of the microphone installed in a space where reverberation exists as the noise. With this configuration, it is possible to suppress reverberation in real time.

[0022] In the above behavior identification method, the feature amount may be a cepstrum. With this configuration, the behavior of the user can be identified using the cepstrum of the behavior sound in which noise is suppressed.

[0023] A behavior identification device according to another aspect of the present disclosure is a behavior identification device that identifies a user's behavior, and includes: a sound data acquisition unit that acquires sound data from a microphone; a feature calculation unit that calculates a feature of the sound data; a determination unit that determines whether the user is present in a space in which the microphone is installed; a noise calculation unit that calculates a noise feature indicating a feature of noise based on the calculated feature and stores the calculated noise feature in a storage unit when the user is not present in the space; an action sound extraction unit that extracts an action sound feature indicating a feature of an action sound generated by the user's action by subtracting the noise feature stored in the storage unit from the calculated feature when the user is present in the space; and a behavior identification unit that identifies the user's behavior using the action sound feature.

[0024] In a space where a user does not exist, only noise other than action sounds generated by the user's actions is detected. Therefore, when a user does not exist in the space, a noise feature indicating the feature of noise is calculated based on the feature of sound data acquired from a microphone arranged in the space, and the calculated noise feature is stored in a storage unit. Then, when a user exists in the space, the noise feature stored in the storage unit is subtracted from the feature of sound data acquired from a microphone arranged in the space. In this way, it is possible to extract only the action sound feature indicating the feature of action sounds in which noise is suppressed in the space. Then, since the user's actions are identified using the feature of action sounds in which noise is suppressed, the user's actions can be identified with higher accuracy even in a space where action sounds and noise are mixed.

[0025] Furthermore, when a user is not present in the space, a noise feature indicating a feature of noise is stored in the storage unit, so that when a user is present in the space, the noise feature stored in the storage unit can be used to acquire the user's action sound in real time, thereby making it possible to identify the user's action in real time.

[0026] A behavior identification program according to another aspect of the present disclosure is a behavior identification program for identifying a user's behavior, which acquires sound data from a microphone, calculates features of the sound data, determines whether the user is present in a space in which the microphone is installed, and if the user is not present in the space, calculates a noise feature indicating a feature of noise based on the calculated features and stores the calculated noise feature in a memory unit, and if the user is present in the space, extracts an action sound feature indicating a feature of an action sound generated by the user's action by subtracting the noise feature stored in the memory unit from the calculated feature, and causes a computer to function as follows:

[0027] In a space where a user does not exist, only noise other than action sounds generated by the user's actions is detected. Therefore, when a user does not exist in the space, a noise feature indicating the feature of noise is calculated based on the feature of sound data acquired from a microphone arranged in the space, and the calculated noise feature is stored in a storage unit. Then, when a user exists in the space, the noise feature stored in the storage unit is subtracted from the feature of sound data acquired from a microphone arranged in the space. In this way, it is possible to extract only the action sound feature indicating the feature of action sounds in which noise is suppressed in the space. Then, since the user's actions are identified using the feature of action sounds in which noise is suppressed, the user's actions can be identified with higher accuracy even in a space where action sounds and noise are mixed.

[0028] Furthermore, when a user is not present in the space, a noise feature indicating a feature of noise is stored in the storage unit, so that when a user is present in the space, the noise feature stored in the storage unit can be used to acquire the user's action sound in real time, thereby making it possible to identify the user's action in real time.

[0029] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that the following embodiments are examples of the present disclosure and do not limit the technical scope of the present disclosure.

[0030] (Embodiment) 1 is a diagram illustrating an example of a configuration of a behavior recognition system according to an embodiment of the present disclosure. The behavior recognition system illustrated in FIG.

[0031] The microphone 2 collects surrounding sounds and outputs the collected sound data and a microphone ID for identifying the microphone 2 to the behavior recognition device 1.

[0032] The human presence sensor 3 detects a user present in the vicinity. The human presence sensor 3 outputs, to the behavior recognition device 1, presence information indicating whether or not a user has been detected, and a sensor ID for identifying the human presence sensor 3.

[0033] The behavior recognition system is installed in a house where a user lives. The microphone 2 and the human presence sensor 3 are arranged in each room in the house.

[0034] FIG. 2 is a diagram for explaining the arrangement of the behavior recognition device, the microphone, and the human presence sensor according to the embodiment of the present disclosure.

[0035] The microphone 2 and the motion sensor 3 are arranged, for example, in the living room 301, the kitchen 302, the bedroom 303, the bathroom 304, and the hallway 305, respectively. The microphone 2 and the motion sensor 3 may be provided in one housing, or in different housings. Some home appliances, such as smart speakers, have a built-in microphone. Some home appliances, such as air conditioners, have a built-in motion sensor. Therefore, the microphone 2 and the motion sensor 3 may be built into the home appliances.

[0036] The behavior recognition device 1 recognizes the behavior of a user. The behavior recognition device 1 is installed in a residence where a user lives. The behavior recognition device 1 is arranged in a predetermined room in the residence. For example, the behavior recognition device 1 is arranged in a living room 301. Note that the room in which the behavior recognition device 1 is arranged is not particularly limited. The behavior recognition device 1 is connected to each of a microphone 2 and a human presence sensor 3, for example, via a wireless LAN (Local Area Network).

[0037] The behavior recognition device 1 includes a sound data acquisition unit 101, a feature calculation unit 102, a microphone ID acquisition unit 103, a microphone ID determination unit 104, an occupancy information acquisition unit 105, an occupancy determination unit 106, a noise characteristic calculation unit 107, a noise feature storage unit 108, a noise suppression unit 109, a behavior recognition unit 110, and a behavior label output unit 111.

[0038] The sound data acquisition unit 101, the feature amount calculation unit 102, the microphone ID acquisition unit 103, the microphone ID determination unit 104, the occupancy information acquisition unit 105, the occupancy determination unit 106, the noise characteristic calculation unit 107, the noise suppression unit 109, the behavior identification unit 110, and the behavior label output unit 111 are realized by a processor. The processor is composed of, for example, a CPU (Central Processing Unit) or the like.

[0039] The noise feature storage unit 108 is realized by a memory. The memory is, for example, a read only memory (ROM) or an electrically erasable programmable read only memory (EEPROM).

[0040] The sound data acquisition unit 101 acquires sound data from the microphone 2. The sound data acquisition unit 101 receives the sound data transmitted by the microphone 2.

[0041] The feature calculation unit 102 calculates the feature of the sound data. The feature calculation unit 102 divides the sound data into frames of a certain interval, and calculates the feature for each frame. The feature in this embodiment is a cepstrum. The cepstrum is obtained by logarithmically expressing spectrum information obtained by Fourier transforming the sound data, and further Fourier transforming the logarithmically expressed information. The feature calculation unit 102 outputs the calculated feature to the noise characteristic calculation unit 107 and the noise suppression unit 109.

[0042] The microphone ID acquisition unit 103 acquires a microphone ID (identification information) for identifying the microphone 2. The microphone ID acquisition unit 103 receives the microphone ID transmitted by the microphone 2. The microphone ID is transmitted together with the sound data. The microphone ID makes it possible to identify in which room the sound data was collected. The microphone ID acquisition unit 103 outputs the acquired microphone ID to the microphone ID determination unit 104 and the noise characteristic calculation unit 107.

[0043] The microphone ID determination unit 104 determines whether the microphone 2 corresponding to the microphone ID acquired by the microphone ID acquisition unit 103 is placed in a first room in which noise is suppressed by a first noise suppression method, or in a second room in which noise is suppressed by a second noise suppression method different from the first noise suppression method. A memory (not shown) previously stores a table that associates microphone IDs with rooms in which the microphone 2 corresponding to the microphone IDs is placed.

[0044] In the first noise suppression method, when the user is absent, an average of features for a predetermined number of frames is calculated, and the calculated average feature is stored as a noise feature in the noise feature storage unit 108, and, when the user is present, the noise feature stored in the noise feature storage unit 108 is subtracted from the feature of the current frame. In the second noise suppression method, an average of features for each of a plurality of frames prior to the current frame is calculated as a noise feature, and the calculated noise feature is subtracted from the calculated feature of the current frame.

[0045] The second room is a room (space) where reverberation exists as noise, such as a hallway, and the first room is a room (space) where noise other than reverberation exists, such as a bathroom, washroom, toilet, kitchen, bedroom, and living room.

[0046] The microphone ID determination unit 104 outputs the determination result as to whether the microphone 2 corresponding to the microphone ID acquired by the microphone ID acquisition unit 103 is located in the first room or the second room to the noise characteristic calculation unit 107 and the noise suppression unit 109.

[0047] The presence information acquisition unit 105 acquires, from the human presence sensor 3, presence information indicating whether or not a user is present in the room (space) in which the microphone 2 is installed. The presence information acquisition unit 105 receives the presence information transmitted by the human presence sensor 3.

[0048] The occupancy information acquisition unit 105 acquires, together with the occupancy information, a sensor ID for identifying the human presence sensor 3 from the human presence sensor 3. A memory (not shown) stores in advance a table that associates the sensor ID with the room in which the human presence sensor 3 corresponding to the sensor ID is located. The occupancy information acquisition unit 105 can specify which room the acquired occupancy information pertains to by referring to the table.

[0049] The presence determination unit 106 determines whether or not a user is present in the room (space) in which the microphone 2 is installed. The presence determination unit 106 determines whether or not a user is present in the room in which the microphone 2 that collected the sound data is installed, based on the presence information acquired by the presence information acquisition unit 105. The presence determination unit 106 outputs the determination result as to whether or not a user is present in the room in which the microphone 2 is installed to the noise characteristic calculation unit 107 and the noise suppression unit 109.

[0050] When no user is present in the space, the noise characteristic calculation unit 107 calculates a noise feature indicating the feature of noise based on the calculated feature, and stores the calculated noise feature in the noise feature storage unit 108. When the presence determination unit 106 determines that no user is present in the room, the noise characteristic calculation unit 107 calculates the noise feature based on the calculated feature.

[0051] The noise feature amount storage unit 108 stores the noise feature amount calculated by the noise characteristic calculation unit 107. The noise feature amount storage unit 108 stores the noise feature amount in association with the microphone ID.

[0052] FIG. 3 is a diagram illustrating a configuration of the noise characteristic calculation unit illustrated in FIG.

[0053] The noise characteristic calculation unit 107 includes a past frame feature amount storage unit 201 , a consecutive frame number determination unit 202 , and a noise feature amount calculation unit 203 .

[0054] The past frame feature amount storage unit 201 stores the feature amount for each past frame calculated by the feature amount calculation unit 102. The feature amount calculation unit 102 stores the calculated feature amount for each frame in the past frame feature amount storage unit 201.

[0055] The consecutive frame number determination unit 202 determines the number of frames based on the microphone ID (identification information). When calculating noise features, features of multiple consecutive frames are used. The number of consecutive frames differs depending on the type of noise. The number of frames determined based on the microphone ID (identification information) of a microphone 2 installed in a space where stationary noise with little time fluctuation exists as noise is greater than the number of frames determined based on the microphone ID (identification information) of a microphone 2 installed in a space where non-stationary noise with much time fluctuation exists as noise.

[0056] An example of stationary noise is the sound of a ventilation fan. The sound of a ventilation fan is mainly found in kitchens, bathrooms, washrooms, and toilets. Examples of non-stationary noise are outdoor noise, television sounds, and reverberation. Outdoor noise and television sounds are mainly found in living rooms and bedrooms. Reverberation sounds are mainly found in corridors.

[0057] Therefore, when the microphone ID of the microphone 2 installed in the kitchen, bathroom, washroom, or toilet is acquired, the consecutive frame number determination unit 202 determines the first consecutive frame number. The first consecutive frame number is, for example, 100. Since the length of one frame is, for example, 20 msec, the length of the first consecutive frame number is 2.0 sec. Also, when the microphone ID of the microphone 2 installed in the living room, bedroom, or hallway is acquired, the consecutive frame number determination unit 202 determines the second consecutive frame number that is less than the first consecutive frame number. The second consecutive frame number is, for example, 10. Since the length of one frame is, for example, 20 msec, the length of the second consecutive frame number is 200 msec. Note that the length of one frame, the length of the first consecutive frame number, and the length of the second consecutive frame number are not limited to the above.

[0058] Furthermore, in this embodiment, the number of frames is determined in advance for each microphone ID or room, but the number of frames may be changed depending on the type of noise.

[0059] When no user is present in the room (space) in which the microphone 2 is installed, the noise feature calculation unit 203 calculates the average of the features of each of a number of frames determined by the number of consecutive frames determination unit 202 as the noise feature.

[0060] Here, when the microphone ID determination unit 104 determines that the room in which the microphone 2 that collected the sound data is installed is the first room, the presence determination unit 106 determines that no user is present in the room in which the microphone 2 that collected the sound data is installed, and the consecutive frame number determination unit 202 determines the first consecutive frame number, the noise feature calculation unit 203 calculates the average of the features of each frame of the first consecutive frame number as the noise feature. At this time, the noise feature calculation unit 203 reads out the features of each frame of the first consecutive frame number from the past frame feature storage unit 201, and calculates the average of the features of each frame of the first consecutive frame number as the noise feature.

[0061] Furthermore, when the microphone ID determination unit 104 determines that the room in which the microphone 2 that collected the sound data is installed is the first room, the presence determination unit 106 determines that no user is present in the room in which the microphone 2 that collected the sound data is installed, and the consecutive frame number determination unit 202 determines the second consecutive frame number, the noise feature calculation unit 203 calculates the average of the features of each frame of the second consecutive frame number as the noise feature. At this time, the noise feature calculation unit 203 reads out the features of each frame of the second consecutive frame number from the past frame feature storage unit 201, and calculates the average of the features of each frame of the second consecutive frame number as the noise feature.

[0062] Furthermore, when the microphone ID (identification information) is a predetermined microphone ID (identification information), the noise feature calculation unit 203 calculates the average of the features of a plurality of frames past the current frame as the noise feature. The predetermined microphone ID (identification information) is the microphone ID (identification information) of the microphone 2 installed in a room (space) where reverberation exists as noise. That is, when the microphone ID determination unit 104 determines that the room in which the microphone 2 that collected the sound data is installed is the second room and the consecutive frame number determination unit 202 determines the second consecutive frame number, the noise feature calculation unit 203 calculates the average of the features of each frame of the second consecutive frame number past the current frame as the noise feature. At this time, the noise feature calculation unit 203 reads out the features of each frame of the second consecutive frame number from the frame immediately before the current frame from the past frame feature storage unit 201, and calculates the average of the features of each frame of the second consecutive frame number as the noise feature.

[0063] In addition, the echo of the user's walking sound occurs when the user is present in the room. This echo needs to be suppressed in real time from the acquired sound data. Therefore, when the room in which the microphone 2 that collected the sound data is installed is the second room, the noise feature amount calculation unit 203 calculates the average of the features of each of multiple frames past the current frame as the noise feature amount, regardless of whether the user is present in the second room or not.

[0064] In addition, when the microphone ID determination unit 104 determines that the room in which the microphone 2 that collected the sound data is installed is the second room, and the presence determination unit 106 determines that a user is present in the room in which the microphone 2 that collected the sound data is installed, the noise feature calculation unit 203 may calculate the average of the features of each of multiple frames prior to the current frame as the noise feature.

[0065] When the microphone ID determination unit 104 determines that the room in which the microphone 2 that collected the sound data is installed is the first room and the presence determination unit 106 determines that no user is present in the room in which the microphone 2 that collected the sound data is installed is the first room, the noise feature calculation unit 203 stores the calculated noise feature in the noise feature storage unit 108. On the other hand, when the microphone ID determination unit 104 determines that the room in which the microphone 2 that collected the sound data is installed is the second room, the noise feature calculation unit 203 outputs the calculated noise feature to the noise suppression unit 109.

[0066] When a user is present in the room (space) in which the microphone 2 is installed, the noise suppression unit 109 extracts an action sound feature indicating the feature of an action sound generated by the user's action by subtracting the noise feature stored in the noise feature storage unit 108 from the feature calculated by the feature calculation unit 102.

[0067] Here, when the microphone ID determination unit 104 determines that the room in which the microphone 2 that collected the sound data is installed is the first room, and the presence determination unit 106 determines that a user is present in the room in which the microphone 2 that collected the sound data is installed, the noise suppression unit 109 subtracts the noise feature stored in the noise feature memory unit 108 from the feature of the current frame calculated by the feature calculation unit 102.

[0068] In addition, when the microphone ID determination unit 104 determines that the room in which the microphone 2 that collected the sound data is installed is the second room, the noise suppression unit 109 extracts action sound features by subtracting the noise features calculated by the noise characteristic calculation unit 107 from the features of the current frame calculated by the feature calculation unit 102.

[0069] Here, action sounds will be described. Action sounds are sounds that are generated by a user taking an active action. Action sounds do not include the user's speech. Action sounds in a bathroom and washroom are, for example, the sound of a shower, the sound of brushing teeth, the sound of washing hands, and the sound of a hair dryer. Action sounds in a kitchen are, for example, the sound of washing hands. Action sounds in a bedroom are, for example, the sound of a door opening and closing. Action sounds in a hallway are, for example, the sound of walking and the sound of a door opening and closing.

[0070] The behavior identification unit 110 identifies the behavior of the user using the behavior sound feature extracted by the noise suppression unit 109. The behavior identification unit 110 inputs the behavior sound feature to a classification model, and obtains a behavior label output from the classification model. The classification model is stored in advance in a memory (not shown). For example, when an behavior sound feature indicating a shower sound is input to the classification model, a behavior label indicating that the user is taking a shower is output from the classification model.

[0071] The discrimination model may be generated by machine learning. Examples of machine learning include supervised learning, which learns the relationship between input and output using teacher data to which a label (output information) is assigned to the input information, unsupervised learning, which builds a data structure only from unlabeled input, semi-supervised learning, which handles both labeled and unlabeled data, and reinforcement learning, which learns actions that maximize rewards by trial and error. Specific examples of machine learning include neural networks (including deep learning using multi-layered neural networks), genetic programming, decision trees, Bayesian networks, and support vector machines (SVMs). In the machine learning of the present disclosure, any of the specific examples listed above may be used.

[0072] The discrimination model may be trained using only the feature quantities of the action sounds that do not include noise, or may be trained using the feature quantities of the action sounds that include noise.

[0073] The activity label output unit 111 outputs the result of the user's activity identification by the activity identification unit 110. At this time, the activity label output unit 111 outputs an activity label indicating the identified user's activity.

[0074] FIG. 4 is a diagram for explaining the noise suppression method according to the present embodiment.

[0075] The table shown in FIG. 4 indicates the relationship between the installation location of the microphone 2, the action sound generated at the installation location, the noise generated at the installation location, the number of consecutive frames according to the installation location, and the noise suppression method.

[0076] In the bathroom or washroom, the action sounds are, for example, the sound of a shower, the sound of brushing teeth, the sound of washing hands, and the sound of a hair dryer, and the noise is the sound of a ventilation fan. When the microphone ID of the microphone 2 installed in the bathroom or washroom is acquired, the noise is suppressed by the first noise suppression method. In the first noise suppression method, when the presence determination unit 106 determines that the user is not in the bathroom or washroom, the noise feature calculation unit 203 calculates an average of the features of each frame of the first continuous frame number, and stores the calculated average feature in the noise feature storage unit 108 as a noise feature. In addition, in the first noise suppression method, when the presence determination unit 106 determines that the user is in the bathroom or washroom, the noise suppression unit 109 subtracts the noise feature stored in the noise feature storage unit 108 from the feature of the current frame. As a result, only the action sounds are extracted.

[0077] In addition, in a kitchen, the action sound is, for example, the sound of washing hands, and the noise is the sound of a ventilation fan, etc. When the microphone ID of the microphone 2 installed in the kitchen is acquired, the noise is suppressed by the first noise suppression method.

[0078] In the bedroom, the action sound is, for example, the sound of a door opening and closing, and the noise is, for example, outdoor noise or the sound of a television. When the microphone ID of the microphone 2 installed in the bedroom is acquired, the noise is suppressed by a first noise suppression method. In the first noise suppression method, when the presence determination unit 106 determines that the user is absent from the bedroom, the noise feature calculation unit 203 calculates an average of the features of each frame of a second number of consecutive frames that is less than the first number of consecutive frames, and stores the calculated average feature in the noise feature storage unit 108 as a noise feature.

[0079] It should be noted that television sounds are sounds that are generated when a user turns on the power of a television, and therefore television sounds may be classified as action sounds rather than as noise.

[0080] In addition, in a corridor, the action sounds are, for example, walking sounds or door opening and closing sounds, and the noise is reverberation sounds. When the microphone ID of the microphone 2 installed in the corridor is acquired, the noise is suppressed by the second noise suppression method. In the second noise suppression method, the noise feature calculation unit 203 calculates an average of the features of each frame of a second number of consecutive frames before the current frame, and outputs the calculated average feature to the noise suppression unit 109 as the noise feature. The noise suppression unit 109 subtracts the noise feature calculated by the noise feature calculation unit 203 from the feature of the current frame. As a result, only the action sounds are extracted.

[0081] Next, the behavior identification process in this embodiment will be described with reference to FIGS.

[0082] Fig. 5 is a first flowchart for explaining the behavior identification process in the present embodiment, and Fig. 6 is a second flowchart for explaining the behavior identification process in the present embodiment. In the following explanation of the flowcharts, the cepstrum is used as a feature.

[0083] First, in step S 1 , the sound data acquisition unit 101 acquires sound data from the microphone 2 .

[0084] Next, in step S2, the feature calculation unit 102 divides the sound data into frames for each fixed interval, and calculates a cepstrum for each frame.

[0085] Next, in step S3, the feature calculation unit 102 stores the calculated cepstrum for each frame in the past frame feature storage unit 201.

[0086] Next, in step S 4 , the microphone ID acquisition unit 103 acquires the microphone ID from the microphone 2 .

[0087] Next, in step S5, the microphone ID determination unit 104 determines whether or not the microphone 2 is installed in a first room based on the acquired microphone ID. The first room is a room in which noise other than reverberation exists, such as a bathroom, washroom, toilet, kitchen, bedroom, or living room.

[0088] Here, if it is determined that the microphone 2 is installed in the first room (YES in step S5), in step S6, the presence information acquisition unit 105 acquires presence information indicating whether or not a user is present in the first room in which the microphone 2 is installed from the human presence sensor 3. Note that the presence information acquisition unit 105 may acquire the presence information transmitted from the human presence sensor 3 at the same timing as the sound data, or may transmit a request signal requesting the presence information to the human presence sensor 3 and acquire the presence information transmitted in response to the request signal.

[0089] Next, in step S7, the presence determining unit 106 determines whether or not the user is absent from the first room.

[0090] If it is determined that the user is not in the first room (YES in step S7), in step S8, the presence determination unit 106 determines whether the current time is a predetermined timing. The predetermined timing is, for example, a time when a predetermined time has elapsed since the previous noise cepstrum was stored in the noise feature storage unit 108. The predetermined time is, for example, one hour.

[0091] If it is determined that the current time is not the predetermined timing (NO in step S8), the process returns to step S1.

[0092] On the other hand, if it is determined that the current time is the predetermined timing (YES in step S8), in step S9, the consecutive frame number determination unit 202 determines the number of frames based on the microphone ID. At this time, if the microphone ID is the microphone ID of microphone 2 installed in a room where stationary noise exists as noise, the consecutive frame number determination unit 202 determines the number of frames to be the first consecutive frame number. On the other hand, if the microphone ID is the microphone ID of microphone 2 installed in a room where non-stationary noise exists as noise, the consecutive frame number determination unit 202 determines the number of frames to be the second consecutive frame number that is smaller than the first consecutive frame number.

[0093] Next, in step S 10 , the noise feature calculation unit 203 reads out the cepstrum of each of a number of consecutive frames determined by the consecutive frame number determination unit 202 from the past frame feature storage unit 201 .

[0094] Next, in step S11, the noise feature amount calculation unit 203 calculates the average of the cepstrum of each of the multiple consecutive frames read out from the past frame feature amount storage unit 201 as the noise cepstrum.

[0095] Next, in step S12, the noise feature amount calculation unit 203 stores the calculated noise cepstrum in the noise feature amount storage unit 108. Then, after the process of step S12 is performed, the process returns to step S1.

[0096] On the other hand, if it is determined that the user is present in the first room (NO in step S7), the noise suppressor 109 reads out the noise cepstrum stored in the noise feature amount storage unit 108 in step S13.

[0097] Next, in step S14, the noise suppression unit 109 subtracts the noise cepstrum read from the noise feature storage unit 108 from the cepstrum of the current frame calculated by the feature calculation unit 102. In this way, the noise suppression unit 109 extracts an action sound cepstrum indicating the cepstrum of the action sound.

[0098] Next, in step S15, the behavior identifying unit 110 identifies the user's behavior using the behavior sound cepstrum extracted by the noise suppressing unit 109.

[0099] Next, in step S16, the behavior identification unit 110 outputs a behavior label indicating the user's behavior, which is the identification result. 16 After the process is performed, the process returns to step S1. It is preferable that the behavior label is output together with information indicating the microphone ID or the room identified by the microphone ID. This makes it possible to identify the behavior performed by the user and the room in which the user performed the behavior.

[0100] On the other hand, if it is determined that the microphone 2 is not installed in the first room, that is, if it is determined that the microphone 2 is installed in the second room (NO in step S5), in step S17, the consecutive frame number determination unit 202 determines the number of frames based on the microphone ID. At this time, if the microphone ID is the microphone ID of the microphone 2 installed in a room where stationary noise exists as noise, the consecutive frame number determination unit 202 determines the number of frames to be the first consecutive frame number. On the other hand, if the microphone ID is the microphone ID of the microphone 2 installed in a room where non-stationary noise exists as noise, the consecutive frame number determination unit 202 determines the number of frames to be the second consecutive frame number that is smaller than the first consecutive frame number.

[0101] Next, in step S18, the noise feature calculation unit 203 reads out from the past frame feature storage unit 201 the cepstrum of each of a number of consecutive frames determined by the number of consecutive frames in the past from the current frame determination unit 202.

[0102] Next, in step S19, the noise feature amount calculation unit 203 calculates, as a noise cepstrum, an average of the cepstrum of each of the multiple consecutive frames read out from the past frame feature amount storage unit 201. The noise feature amount calculation unit 203 outputs the calculated noise cepstrum to the noise suppression unit 109.

[0103] Next, in step S20, the noise suppression unit 109 subtracts the noise cepstrum calculated by the noise feature calculation unit 203 from the cepstrum of the current frame calculated by the feature calculation unit 102. In this way, the noise suppression unit 109 extracts an action sound cepstrum indicating the cepstrum of the action sound.

[0104] The processes in steps S21 and S22 are the same as those in steps S15 and S16, and therefore will not be described.

[0105] In a space where a user does not exist, only noise other than action sounds generated by the user's actions is detected. Therefore, when a user does not exist in the space, a noise feature indicating the feature of noise is calculated based on the feature of sound data acquired from the microphone 2 arranged in the space, and the calculated noise feature is stored in the storage unit. Then, when a user exists in the space, the noise feature stored in the noise feature storage unit 108 is subtracted from the feature of sound data acquired from the microphone 2 arranged in the space. This makes it possible to extract only the action sound feature indicating the feature of action sounds in which noise is suppressed in the space. Then, since the user's actions are identified using the feature of action sounds in which noise is suppressed, the user's actions can be identified with higher accuracy even in a space where action sounds and noise are mixed.

[0106] Furthermore, when no user is present in the space, a noise feature indicating a feature of noise is stored in the noise feature storage unit 108, so that when a user is present in the space, the action sound of the user can be acquired in real time using the noise feature stored in the noise feature storage unit 108. As a result, the action of the user can be identified in real time.

[0107] In this embodiment, the cepstrum is used as the feature, but the present disclosure is not particularly limited thereto. The feature may be the logarithmic energy for each frequency band (Mel-filterbank log energy) or the Mel-frequency cepstral coefficient (MFCC). Even if the feature is the logarithmic energy for each frequency band or the Mel-frequency cepstral coefficient, it is possible to suppress noise and to identify behavior with high accuracy, as in this embodiment.

[0108] In addition, in the present embodiment, the behavior recognition system includes one behavior recognition device 1, and the behavior recognition device 1 is placed in a predetermined room in a residence, but the present disclosure is not particularly limited thereto. The behavior recognition system may include multiple behavior recognition devices 1. The multiple behavior recognition devices 1 may be placed in each room in the residence together with a microphone 2 and a human presence sensor 3. Each of the multiple behavior recognition devices 1 may identify the user's behavior in each room. Furthermore, the behavior recognition device 1 may be a server placed outside the residence. In this case, the behavior recognition device 1 is communicably connected to the microphone 2 and the human presence sensor 3 via a network such as the Internet.

[0109] In each of the above embodiments, each component may be implemented by dedicated hardware or by executing a software program suitable for each component. Each component may be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0110] A part or all of the functions of the device according to the embodiment of the present disclosure are typically realized as an LSI (Large Scale Integration), which is an integrated circuit. These may be individually integrated into one chip, or may be integrated into one chip that includes some or all of the functions. The integrated circuit is not limited to an LSI, and may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells inside the LSI, may be used.

[0111] Furthermore, some or all of the functions of the device according to the embodiment of the present disclosure may be realized by a processor such as a CPU executing a program.

[0112] Furthermore, all the numbers used above are merely examples for the purpose of specifically explaining the present disclosure, and the present disclosure is not limited to the numbers exemplified.

[0113] In addition, the order in which each step is performed shown in the above flowchart is merely an example for specifically explaining the present disclosure, and an order other than the above may be used as long as the same effect is obtained. In addition, some of the steps may be performed simultaneously (in parallel) with other steps. [Industrial Applicability]

[0114] INDUSTRIAL APPLICABILITY The technology disclosed herein is capable of identifying user behavior with higher accuracy and is therefore useful as a technology for identifying user behavior.

Claims

1. 1. An activity identification method for identifying an activity of a user, comprising: The computer Acquire sound data from a microphone, Obtaining identification information of the microphone; Calculating a feature amount of the sound data; determining whether the user is present in a space in which the microphone is installed; When the user is not present in the space, a noise feature indicating a feature of noise is calculated based on the calculated feature and the identification information, and the calculated noise feature is stored in a storage unit; when the user is present in the space, extracting an action sound feature indicating a feature of an action sound generated by an action of the user by subtracting the noise feature stored in the storage unit from the calculated feature; identifying an action of the user using the action sound feature; Behavioral identification methods.

2. In the calculation of the feature amount, the sound data is divided into frames each having a fixed interval, and the feature amount is calculated for each of the frames; In storing the noise feature, the number of frames is determined based on the identification information, and an average of the feature amounts of each of the determined number of frames is calculated as the noise feature amount. The activity identification method according to claim 1 .

3. the number of frames determined based on the identification information of the microphone installed in a space where stationary noise with little time fluctuation exists as the noise is greater than the number of frames determined based on the identification information of the microphone installed in a space where non-stationary noise with large time fluctuation exists as the noise. The activity identification method according to claim 2.

4. In the calculation of the feature amount, the sound data is divided into frames of a certain interval, and the feature amount is calculated for each of the frames; Furthermore, when the identification information is a predetermined identification information, an average of the feature amounts of a plurality of frames past the current frame is calculated as the noise feature amount; and extracting the action sound feature by subtracting the calculated noise feature from the calculated feature of the current frame. The activity identification method according to claim 1 .

5. The predetermined identification information is the identification information of the microphone installed in a space where a reverberation sound exists as the noise. The activity identification method according to claim 4.

6. The feature is a cepstrum. The method for identifying an action according to any one of claims 1 to 5.

7. An activity identification device that identifies an activity of a user, a sound data acquisition unit that acquires sound data from a microphone; an identification information acquisition unit that acquires identification information of the microphone; a feature amount calculation unit that calculates a feature amount of the sound data; a determination unit that determines whether the user is present in a space in which the microphone is installed; a noise calculation unit that calculates a noise feature indicating a feature of noise based on the calculated feature and the identification information when the user is not present in the space, and stores the calculated noise feature in a storage unit; an action sound extraction unit that extracts, when the user is present in the space, an action sound feature amount indicating a feature amount of an action sound generated by an action of the user by subtracting the noise feature amount stored in the storage unit from the calculated feature amount; and an action identification unit that identifies an action of the user by using the action sound feature; A behavior recognition device comprising:

8. An activity identification program for identifying an activity of a user, Acquire sound data from a microphone, Obtaining identification information of the microphone; Calculating a feature amount of the sound data; determining whether the user is present in a space in which the microphone is installed; When the user is not present in the space, a noise feature indicating a feature of noise is calculated based on the calculated feature and the identification information, and the calculated noise feature is stored in a storage unit; when the user is present in the space, extracting an action sound feature indicating a feature of an action sound generated by an action of the user by subtracting the noise feature stored in the storage unit from the calculated feature; An action identification program that causes a computer to function so as to identify the action of the user using the action sound feature.

9. A method for identifying a user's behavior, comprising: The computer Acquire sound data from a microphone, Calculating a feature amount of the sound data; determining whether the user is present in a space in which the microphone is installed; When the user is not present in the space, a noise feature indicating a feature of noise is calculated based on the calculated feature, and the calculated noise feature is stored in a storage unit; when the user is present in the space, extracting an action sound feature indicating a feature of an action sound generated by an action of the user by subtracting the noise feature stored in the storage unit from the calculated feature; Identifying an action of the user using the action sound feature; acquiring identification information for identifying the microphone; In the calculation of the feature amount, the sound data is divided into frames each having a certain interval, and the feature amount is calculated for each of the frames; In storing the noise feature, the number of frames is determined based on the identification information, and an average of the feature amounts of each of the determined number of frames is calculated as the noise feature amount; the number of frames determined based on the identification information of the microphone installed in a space where stationary noise with little time fluctuation exists as the noise is greater than the number of frames determined based on the identification information of the microphone installed in a space where non-stationary noise with large time fluctuation exists as the noise. Behavioral identification methods.

10. A method for identifying a user's behavior, comprising: The computer Acquire sound data from a microphone, Calculating a feature amount of the sound data; determining whether the user is present in a space in which the microphone is installed; When the user is not present in the space, a noise feature indicating a feature of noise is calculated based on the calculated feature, and the calculated noise feature is stored in a storage unit; when the user is present in the space, extracting an action sound feature indicating a feature of an action sound generated by an action of the user by subtracting the noise feature stored in the storage unit from the calculated feature; acquiring identification information for identifying the microphone; In the calculation of the feature amount, the sound data is divided into frames each having a certain interval, and the feature amount is calculated for each of the frames; If the identification information is a predetermined identification information, an average of feature amounts of a plurality of frames past the current frame is calculated as the noise feature amount; extracting the action sound feature by subtracting the calculated noise feature from the calculated feature of the current frame; identifying an action of the user using the action sound feature; Behavioral identification methods.

11. A behavior identification device for identifying a user's behavior, comprising: a sound data acquisition unit that acquires sound data from a microphone; a feature amount calculation unit that calculates a feature amount of the sound data; a determination unit that determines whether the user is present in a space in which the microphone is installed; a noise calculation unit that calculates a noise feature indicating a feature of noise based on the calculated feature when the user is not present in the space, and stores the calculated noise feature in a storage unit; an action sound extraction unit that extracts, when the user is present in the space, an action sound feature amount indicating a feature amount of an action sound generated by an action of the user by subtracting the noise feature amount stored in the storage unit from the calculated feature amount; and an action identification unit that identifies an action of the user by using the action sound feature; an identification information acquisition unit that acquires identification information for identifying the microphone; Equipped with The feature amount calculation unit divides the sound data into frames each having a certain section, and calculates the feature amount for each of the frames; the noise calculation unit determines the number of frames based on the identification information, and calculates an average of features of the determined number of frames as the noise feature; the number of frames determined based on the identification information of the microphone installed in a space where stationary noise with little time fluctuation exists as the noise is greater than the number of frames determined based on the identification information of the microphone installed in a space where non-stationary noise with large time fluctuation exists as the noise. Behavioral Identification Device.

12. A behavior identification device for identifying a user's behavior, comprising: a sound data acquisition unit that acquires sound data from a microphone; a feature amount calculation unit that calculates a feature amount of the sound data; a determination unit that determines whether the user is present in a space in which the microphone is installed; a noise calculation unit that calculates a noise feature indicating a feature of noise based on the calculated feature when the user is not present in the space, and stores the calculated noise feature in a storage unit; an action sound extraction unit that extracts, when the user is present in the space, an action sound feature amount indicating a feature amount of an action sound generated by an action of the user by subtracting the noise feature amount stored in the storage unit from the calculated feature amount; and an action identification unit that identifies an action of the user by using the action sound feature; an identification information acquisition unit that acquires identification information for identifying the microphone; Equipped with The feature amount calculation unit divides the sound data into frames each having a certain section, and calculates the feature amount for each of the frames; when the identification information is predetermined identification information, the noise calculation unit calculates an average of feature amounts of a plurality of frames past a current frame as the noise feature amount; the action sound extraction unit extracts the action sound feature by subtracting the calculated noise feature from the calculated feature of the current frame. Behavioral Identification Device.

13. A behavior identification program for identifying a user's behavior, comprising: Acquire sound data from a microphone, Calculating a feature amount of the sound data; determining whether the user is present in a space in which the microphone is installed; When the user is not present in the space, a noise feature indicating a feature of noise is calculated based on the calculated feature, and the calculated noise feature is stored in a storage unit; when the user is present in the space, extracting an action sound feature indicating a feature of an action sound generated by an action of the user by subtracting the noise feature stored in the storage unit from the calculated feature; Identifying an action of the user using the action sound feature; Activating a computer to obtain identification information for identifying the microphone; In the calculation of the feature amount, the sound data is divided into frames each having a certain interval, and the feature amount is calculated for each of the frames; In storing the noise feature, the number of frames is determined based on the identification information, and an average of the feature amounts of each of the determined number of frames is calculated as the noise feature amount; the number of frames determined based on the identification information of the microphone installed in a space where stationary noise with little time fluctuation exists as the noise is greater than the number of frames determined based on the identification information of the microphone installed in a space where non-stationary noise with large time fluctuation exists as the noise. Behavioral Identification Program.

14. A behavior identification program for identifying a user's behavior, comprising: Acquire sound data from a microphone, Calculating a feature amount of the sound data; determining whether the user is present in a space in which the microphone is installed; When the user is not present in the space, a noise feature indicating a feature of noise is calculated based on the calculated feature, and the calculated noise feature is stored in a storage unit; when the user is present in the space, extracting an action sound feature indicating a feature of an action sound generated by an action of the user by subtracting the noise feature stored in the storage unit from the calculated feature; acquiring identification information for identifying the microphone; In the calculation of the feature amount, the sound data is divided into frames each having a certain interval, and the feature amount is calculated for each of the frames; If the identification information is a predetermined identification information, an average of feature amounts of a plurality of frames past the current frame is calculated as the noise feature amount; extracting the action sound feature by subtracting the calculated noise feature from the calculated feature of the current frame; An action identification program that causes a computer to function so as to identify the action of the user using the action sound feature.

Images