Estimation program, learning program, estimation device, learning device, estimation method, learning method, and learning model

Abstract

To improve accuracy of estimation of communication activity.SOLUTION: An estimation program is provided for causing a computer to execute: an acquisition step to acquire an acoustic signal and a non-acoustic signal related to a speech source; an acoustic feature calculation step to calculate acoustic features based on the acoustic signal; a non-acoustic feature calculation step to calculate non-acoustic features based on the non-acoustic signal; an activity feature degree calculation step to calculate an activity feature degree based on the acoustic features and non-acoustic features; and an activity degree estimation step for estimating a communication activity degree based on a comparison of the activity feature degree to an evaluation scale.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to an estimation program, a learning program, an estimation device, a learning device, an estimation method, a learning method, and a learning model. [Background technology]

[0002] The degree of enthusiasm when two or more people are having a conversation or exchanging opinions is generally called the communication activity level. Conventionally, the communication activity level has been estimated.

[0003] For example, Patent Document 1 proposes a meeting function calculation device that evaluates meeting activity level by considering the relationships between meeting participants, such as the number of times a meeting induces other meeting participants to speak. This device acquires time-series data of voice and image that represents actions or phenomena between meeting participants, and calculates correlation data between two or more meeting participants. Then, from the time-series data and the correlation data, meeting support information is generated and displayed in accordance with a predetermined evaluation scale related to the meeting.

[0004] Patent Document 2 proposes a conference support system that quantitatively evaluates the atmosphere in a conference and assists in smoothly advancing discussions. This system acquires the voices of the conference participants, extracts frequently occurring keywords from the voice text data, generates conference support information such as the degree of enthusiasm of the discussion, and provides the conference support information to the conference participants. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2009-163431 A [Patent Document 2] JP 2017-112545 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the conventional technology has a problem in that the accuracy of estimating the communication activity level drops significantly when a person cannot wear a microphone or when the person's face is not facing the camera. In other words, the conventional technology has a problem in that the accuracy of estimating the communication activity level drops significantly.

[0007] The problem to be solved by the present invention is to provide an estimation program, a learning program, an estimation device, a learning device, an estimation method, a learning method, and a learning model that can improve the accuracy of estimating communication activity level. [Means for solving the problem]

[0008] The estimation program according to the embodiment is an estimation program for causing a computer to execute an acquisition step of acquiring an acoustic signal and a non-acoustic signal related to a sound source, an acoustic feature calculation step of calculating an acoustic feature based on the acoustic signal, a non-acoustic feature calculation step of calculating a non-acoustic feature based on the non-acoustic signal, an activity feature calculation step of calculating an activity feature level based on the acoustic feature and the non-acoustic feature, and an activity level estimation step of estimating a communication activity level based on a comparison of the activity feature level with an evaluation scale. [Brief description of the drawings]

[0009] [Figure 1] Schematic diagram of an estimation device. [Diagram 2] 1 is an explanatory diagram of an input signal, an acoustic signal, and a non-acoustic signal. [Diagram 3] FIG. 11 is a diagram illustrating a flow of a communication activity level estimation process. [Figure 4] 13 is a flowchart showing the flow of a communication activity level estimation process. [Diagram 5] Schematic diagram of the learning device. [Figure 6] FIG. 1 is a diagram illustrating an example of the configuration of an integrated neural network. [Figure 7] 13 is a flowchart showing the flow of a learning process. [Figure 8]A table showing the results of the validation. [Figure 9] Hardware configuration diagram. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an estimation program, a learning program, an estimation device, a learning device, an estimation method, a learning method, and a learning model according to the present embodiment will be described in detail with reference to the drawings.

[0011] (First embodiment) 1 is a schematic diagram of an example of an estimation device 10 according to the present embodiment. The estimation device 10 is a computer that estimates a communication activity level.

[0012] The estimation device 10 includes a processing unit 11, a storage unit 12, an input unit 13, a communication unit 14, a display unit 15, and an audio output unit 16. The processing unit 11, the storage unit 12, the input unit 13, the communication unit 14, the display unit 15, and the audio output unit 16 are connected to each other so as to be able to communicate with each other via a bus 17 or the like.

[0013] The storage unit 12 stores various calculation results by the processing unit 11, an estimation program executed by the processing unit 11, and the like. The processing unit 11 will be described in detail later. The storage unit 12 is an example of a computer-readable recording medium. The storage unit 12 is configured, for example, by a ROM (Read Only Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), an integrated circuit storage device, and the like.

[0014] The input unit 13 inputs various commands from a user. For example, a keyboard, a mouse, various switches, a touch pad, a touch panel display, etc. can be used as the input unit 13. Signals corresponding to various commands input by the user operating the input unit 13 are supplied to the processing unit 11. The input unit 13 may be a computer connected to the processing unit 11 via a wired or wireless connection.

[0015] The communication unit 14 is an interface for performing information communication with an external device connected to the estimation device 10 via a network. For example, the communication unit 14 receives an acoustic signal and a non-acoustic signal from a device that collects the acoustic signal and the non-acoustic signal, and receives a first trained model 21, a second trained model 22, a third trained model 23, a fourth trained model 24, a fifth trained model 25, and a sixth trained model 26 from a learning device described later. The acoustic signal, the non-acoustic signal, the first trained model 21, the second trained model 22, the third trained model 23, the fourth trained model 24, the fifth trained model 25, and the sixth trained model 26 will be described later.

[0016] The display unit 15 displays various information. As the display unit 15, a CRT (Cathode-Ray Tube) display, a liquid crystal display, an organic EL (Electro Luminescence) display, an LED (Light-Emitting Diode) display, a plasma display, or any other display known in the art may be appropriately used. The display unit 15 may be a projector.

[0017] The acoustic output unit 16 converts an electric signal into sound and emits it. As the acoustic output unit 16, a magnetic speaker, a dynamic speaker, a condenser speaker, or any other speaker known in the art can be appropriately used.

[0018] The processing unit 11 executes information processing in the estimation device 10.

[0019] The processing unit 11 includes an acquisition unit 11A, an acoustic feature calculation unit 11B, a non-acoustic feature calculation unit 11C, an integrated feature calculation unit 11D, a voice presence calculation unit 11E, a laughter presence calculation unit 11F, an emotion feature calculation unit 11G, an activity feature calculation unit 11H, an activity estimation unit 11I, a situation information generation unit 11J, and an output control unit 11K.

[0020] The processing unit 11 has a processor such as a CPU (Central Processing Unit) and a memory such as a RAM (Random Access Memory). The processing unit 11 executes an estimation program stored in the storage unit 12 to execute a communication activity estimation process for estimating a communication activity of an input signal. The estimation program is recorded in a non-transitory computer-readable recording medium. The processing unit 11 reads out the estimation program from the recording medium and executes it to realize an acquisition unit 11A, an acoustic feature calculation unit 11B, a non-acoustic feature calculation unit 11C, an integrated feature calculation unit 11D, a voice presence calculation unit 11E, a laughter presence calculation unit 11F, an emotion feature calculation unit 11G, an activity feature calculation unit 11H, an activity estimation unit 11I, a situation information generation unit 11J, and an output control unit 11K. The estimation program may have a plurality of modules in which the functions of each unit (the acquisition unit 11A to the output control unit 11K) are divided and implemented.

[0021] The hardware implementation of the processing unit 11 is not limited to the above-mentioned embodiment. For example, the processing unit 11 may be configured by a circuit such as an application specific integrated circuit (ASIC) that realizes at least one of the acquisition unit 11A, the acoustic feature calculation unit 11B, the non-acoustic feature calculation unit 11C, the integrated feature calculation unit 11D, the voice presence calculation unit 11E, the laughter presence calculation unit 11F, the emotion feature calculation unit 11G, the activity feature calculation unit 11H, the activity estimation unit 11I, the situation information generation unit 11J, and the output control unit 11K. At least one of the acquisition unit 11A, the acoustic feature calculation unit 11B, the non-acoustic feature calculation unit 11C, the integrated feature calculation unit 11D, the voice presence calculation unit 11E, the laughter presence calculation unit 11F, the emotion feature calculation unit 11G, the activity feature calculation unit 11H, the activity estimation unit 11I, the situation information generation unit 11J, and the output control unit 11K may be implemented in a single integrated circuit or individually in multiple integrated circuits.

[0022] The acquisition unit 11A acquires an input signal including an acoustic signal related to a sound source and a non-acoustic signal. The input signal is a video signal including an acoustic signal related to the same sound source and an image signal. The acoustic signal and the non-acoustic signal are time-series signals. In detail, the acoustic signal and the non-acoustic signal are time-series signals made up of a plurality of frames successive in time series. The acoustic signal and the non-acoustic signal are synchronized in time on a frame-by-frame basis.

[0023] In this embodiment, it is assumed that the acoustic signal and the non-acoustic signal originate from the same sound source. However, it is not necessary for the acoustic signal and the non-acoustic signal to originate from the same sound source as long as there is a correlation between the acoustic signal and the non-acoustic signal.

[0024] FIG. 2 is a diagram illustrating an example of an input signal, an acoustic signal, and a non-acoustic signal.

[0025] 2 shows an example in which the input signal is a video signal. The video signal is a time-series signal including a time-series acoustic signal and a time-series image signal that are synchronized in time. The length of the time period of the video signal is not particularly limited, but is assumed to be, for example, about 30 seconds. The video signal is collected, for example, by a video camera device including a microphone and an imaging device.

[0026] The acoustic signal is a signal related to speech from a speech source. The speech source is, for example, a speaker. In detail, the acoustic signal includes a speech signal derived from the speaker's speech and a noise signal derived from noise. In other words, the acoustic signal is a signal related to speech from the speaker, which is the speech source, and noise.

[0027] The acoustic signal is collected by a microphone. The microphone collects the sound of the speaker's speech, converts the sound pressure of the collected sound into an analog electric signal (acoustic signal), and converts the acoustic signal into a digital time-domain electric signal (acoustic signal) by A / D conversion. The time-domain acoustic signal is acquired by the acquisition unit 11A and converted into a frequency-domain acoustic signal by short-time Fourier transform or the like.

[0028] The non-acoustic signal is a signal other than the acoustic signal related to the speaker, collected at approximately the same time as the acoustic signal. In detail, the non-acoustic signal is a signal other than the voice of the speaker, which is the voice source, and is an image of the speaker or a sensor signal of the speaker detected by a sensor. Specifically, for example, the non-acoustic signal is an image signal related to the speaker who is speaking, or a sensor signal related to the physiological response of the speaker's lips and facial muscles due to the speech, brain waves, etc. In this embodiment, a case will be described assuming that the non-acoustic signal is an image signal related to the speaker. That is, in this embodiment, a form in which the non-acoustic signal is an image signal synchronized in time with the acoustic signal will be described as an example.

[0029] The image signal is collected by an imaging device including a plurality of imaging elements such as CCDs (Charge Coupled Devices). The imaging device optically captures a speaker who is speaking and generates a digital spatial domain image signal (image data) relating to the speaker on a frame-by-frame basis. The image signal is required to correlate with the speaker's speech. The image frame needs to include at least the lip area, the shape of which changes in response to speech, as the subject of the image capture, and may also include the speaker's entire face area. The image signal is acquired on a frame-by-frame basis by the acquisition unit 11A.

[0030] The time-series acoustic signal and the time-series image signal are defined according to the following equation (1).

[0031]

number

[0032] In equation (1), A represents a time-series acoustic signal. In the equation for A, which represents the acoustic signal, T represents the number of dimensions in the time domain of the frame to be processed. F represents the number of dimensions in the frequency domain of the frame to be processed. V represents a time-series image signal. In the equation for V, which represents the image signal, T represents time, H represents the vertical width of the image, W represents the horizontal width of the image, and C represents the number of dimensions of the color channels. In other words, the time-series acoustic signal A is an acoustic signal that has T dimensions in the time domain and F dimensions in the frequency domain of the frame to be processed. The image signal V is an image signal that has dimensions of time T, vertical width H, horizontal width W, and color channel C.

[0033] Returning to Figure 1, we will continue with the explanation.

[0034] The acoustic feature calculation unit 11B calculates an acoustic feature based on the acoustic signal. The acoustic feature is a feature amount of the acoustic signal. The acoustic feature has a value based on the acoustic signal and has a value correlated with the voice by the speaker.

[0035] The acoustic feature calculation unit 11B calculates an acoustic feature for each frame of the acoustic signal. Therefore, the acoustic feature calculation unit 11B calculates time-series data made up of acoustic features that are continuous in time series by calculating an acoustic feature for each frame from a time-series acoustic signal made up of a plurality of frames.

[0036] In detail, the acoustic feature calculation unit 11B calculates an acoustic feature according to a change in the sound represented by the acoustic signal. The change in the sound represented by the acoustic signal is represented by, for example, the pitch, volume, peak value, etc.

[0037] In this embodiment, the acoustic feature calculation unit 11B calculates an acoustic feature from an acoustic signal using a first trained model 21.

[0038] The first trained model 21 is a neural network trained to input an acoustic signal and output an acoustic feature. For example, an encoder network trained to convert an acoustic signal into an acoustic feature is used as the neural network. The first trained model 21 is stored in the storage unit 12 or the like, and is used for calculating the acoustic feature by the acoustic feature calculation unit 11B. The first trained model 21 is generated by a learning device described later.

[0039] The relationship between the acoustic signal and the acoustic feature is as follows. The acoustic signal is waveform data of a time series of sound pressure values ​​of the voice uttered by the speaker. The acoustic signal correlates with the voice uttered by the speaker. For example, the peak value of the acoustic signal has a relatively high value when the speaker is speaking, and a relatively low value when the speaker is not speaking. The acoustic feature is designed so that its value correlates with the peak value of the acoustic signal, in other words, so as to distinguish between speech components and silent components contained in the acoustic signal. For example, the higher the peak value of the acoustic signal, the higher the value of the acoustic feature, and the lower the peak value of the acoustic signal, the lower the value of the acoustic feature.

[0040] The non-acoustic feature calculation unit 11C calculates non-acoustic features based on the non-acoustic signal. The non-acoustic features are feature quantities of the non-acoustic signal. The non-acoustic features have values ​​based on the non-acoustic signal and have characteristic values ​​that correlate with the speech of the speaker.

[0041] The non-acoustic feature calculation unit 11C calculates a non-acoustic feature for each frame. Therefore, the non-acoustic feature calculation unit 11C calculates a non-acoustic feature for each frame from a time-series non-acoustic signal consisting of a plurality of frames, thereby calculating time-series data consisting of non-acoustic features that are continuous in time series.

[0042] In detail, the non-acoustic feature calculation unit 11C calculates a non-acoustic feature according to a change in a non-acoustic signal, which is represented, for example, by a movement of at least a part of a speaker.

[0043] In this embodiment, the non-acoustic feature calculation unit 11C uses the second trained model 22 to calculate the non-acoustic feature from the non-acoustic signal.

[0044] The second trained model 22 is a neural network trained to input a non-acoustic signal and output a non-acoustic feature. For example, the neural network is a neural network that trains an image signal V to output an image feature E V The second trained model 22 is stored in the storage unit 12 or the like, and is used for calculating non-acoustic features by the non-acoustic feature calculation unit 11C. The second trained model 22 is generated by a learning device described later.

[0045] The relationship between the image signal and the image feature is as follows: the image signal correlates with the shape of the facial region and the movement of the body parts when the speaker is speaking. The image feature is designed to calculate the communication feature contained in the image signal. Specifically, the image signal has a different shape when the speaker is making a sound and when he is not making a sound. The image feature is designed so that its value correlates with the speaker's co-communication feature. For example, the more the speaker's body parts move, the higher the value of the image feature, and the less the speaker's body parts move, the lower the value of the image feature.

[0046] The integrated feature calculation unit 11D calculates an integrated feature by integrating the acoustic feature and the non-acoustic feature. The integrated feature calculation unit 11D calculates an integrated feature based on the acoustic feature and the non-acoustic feature. The integrated feature calculation unit 11D calculates an integrated feature for each frame.

[0047] The integrated feature is an integrated feature that combines an acoustic feature and a non-acoustic feature, and is represented by, for example, a multi-dimensional vector that combines the acoustic feature and the non-acoustic feature.

[0048] Specifically, the integrated feature is expressed by the following formula (2).

[0049]

number

[0050] In formula (2), Z AV represents the integration feature. E A represents the acoustic feature. E V represents the image feature. t represents the frame time. That is, the integrated feature Z AV is the acoustic feature E A and image feature E V is an integrated feature of the vectors, which is calculated for each frame time t.

[0051] In formula (2), E A (t、d) E is an acoustic feature vector at frame time t ∈ {1, 2, ..., T} and coordinate d ∈ {1, 2, ..., D} compressed to D dimensions, and is an example of an acoustic feature. V (t、e) Z is an image feature vector at frame time t ∈ {1, 2, ..., T} and coordinate e ∈ {1, 2, ..., E} compressed into D dimensions, and is an example of an image feature. AV (t、d+e) is an integrated feature vector at frame time t ∈ {1, 2, ..., T} and coordinate d ∈ {1, 2, ..., d+e} compressed into D dimensions, and is an example of an integrated feature.

[0052] The voice presence calculation unit 11E calculates the voice presence based on the acoustic feature and the non-acoustic feature. The voice presence is a feature value related to the presence or absence of a voice signal. The voice signal means an acoustic signal derived from an utterance by a speaker among acoustic signals. The voice presence is a feature value indicating the presence or absence of a voice signal derived from an utterance by a speaker among acoustic signals including noise.

[0053] The voice presence calculation unit 11E calculates the voice presence for each frame. In this embodiment, the voice presence calculation unit 11E calculates the voice presence based on the integrated feature. Therefore, the voice presence calculation unit 11E calculates the voice presence for each frame from the integrated feature of a time series consisting of a plurality of frames, thereby calculating time series data consisting of voice presences that are continuous in time series.

[0054] In detail, the voice presence calculation unit 11E calculates the voice presence according to the voice features derived from the sound signal represented by the sound features and the non-sound signal represented by the non-sound features.

[0055] In this embodiment, the voice presence calculation unit 11E uses the third trained model 23 to calculate the voice presence from the integrated features.

[0056] The third trained model 23 is a neural network trained to input the integrated features and output the voice presence. In detail, the third trained model 23 is a neural network trained to reduce a third loss related to the difference between the voice presence and the correct voice presence label related to the voice presence. For example, an estimation network trained to estimate the voice presence from the integrated features is used as the neural network. The third trained model 23 is stored in the storage unit 12 or the like, and is used for the calculation of the voice presence by the voice presence calculation unit 11E. The third trained model 23 is generated by a learning device described later.

[0057] The laughter presence calculation unit 11F calculates the laughter presence based on the acoustic features and non-acoustic features. The laughter presence is a feature value related to the presence or absence of a laughter signal. The laughter signal means an acoustic signal derived from laughter by a speaker among acoustic signals. The laughter presence has a feature value indicating the presence or absence of a laughter signal by a speaker in an acoustic signal including noise.

[0058] The laughter presence calculation unit 11F calculates the laughter presence for each frame. In this embodiment, the laughter presence calculation unit 11F calculates the laughter presence based on the integrated feature. Therefore, the laughter presence calculation unit 11F calculates the laughter presence for each frame from the integrated feature of a time series consisting of a plurality of frames, thereby calculating time series data consisting of chronologically consecutive laughter presences.

[0059] In detail, the laughter presence calculation section 11F calculates the laughter presence according to the features of laughter derived from the audio signal represented by the audio features and the non-audio signal represented by the non-audio features.

[0060] In the present embodiment, the laughter presence calculation unit 11F uses the fourth trained model 24 to calculate the laughter presence.

[0061] The fourth trained model 24 is a neural network trained to input the integrated features and output the laughter presence. In detail, the fourth trained model 24 is a neural network trained to reduce a fourth loss related to the difference between the laughter presence and the correct laughter presence label related to the laughter presence. For example, an estimation network trained to estimate the laughter presence from the integrated features is used as the neural network. The fourth trained model 24 is stored in the storage unit 12 or the like, and is used for calculating the laughter presence by the laughter presence calculation unit 11F. The fourth trained model 24 is generated by a learning device described later.

[0062] The emotion feature calculation unit 11G calculates the emotion feature based on the acoustic feature and the non-acoustic feature. The emotion feature is a feature value related to an emotion signal. The emotion signal means an audio signal derived from the emotion of a speaker's voice among the audio signals. The emotion feature has a value representing an emotion feature by a speaker included in the audio signal. In other words, the emotion feature has a feature value representing the emotion of a voice signal derived from the speaker's utterance among the audio signals.

[0063] Emotional characteristic degree calculation unit 11G calculates the emotional characteristic degree for each frame. In this embodiment, emotional characteristic degree calculation unit 11G calculates the emotional characteristic degree based on the integrated feature. Therefore, emotional characteristic degree calculation unit 11G calculates the emotional characteristic degree for each frame from the integrated feature of a time series consisting of a plurality of frames, thereby calculating time series data made up of emotional characteristic degrees that continue in time series.

[0064] In detail, the emotion feature degree calculation unit 11G calculates an emotion feature degree according to emotion-related features contained in an audio signal represented by audio features and a non-audio signal represented by non-audio features, the emotion feature degree being derived from the audio signal represented by audio features and the non-audio signal represented by non-audio features.

[0065] In this embodiment, emotion feature degree calculation unit 11G uses fifth trained model 25 to calculate emotion feature degrees from integrated features.

[0066] The fifth trained model 25 is a neural network trained to input the integrated features and output an emotional feature score. The fifth trained model 25 is a neural network trained to reduce a fifth loss related to the difference between the emotional feature score and a correct emotional feature score label related to the emotional feature. As the neural network, for example, an estimation network trained to estimate the emotional feature score from the integrated features is used. The fifth trained model 25 is stored in the storage unit 12 or the like, and is used for calculating the emotional feature score by the emotional feature score calculation unit 11G. The fifth trained model 25 is generated by a learning device described later.

[0067] The activity feature calculation unit 11H calculates the activity feature based on the acoustic feature and the non-acoustic feature. The activity feature is a feature value related to a communication activity signal. The communication activity signal means an acoustic signal derived from the excitement of communication by a speaker. The activity feature has a feature value representing a communication activity feature by a speaker from an acoustic signal including noise.

[0068] The activity feature degree calculation unit 11H calculates the activity feature degree for each frame. In this embodiment, the activity feature degree calculation unit 11H calculates the activity feature degree based on the integrated feature. Therefore, the activity feature degree calculation unit 11H calculates the activity feature degree for each frame from the integrated feature of a time series consisting of a plurality of frames, thereby calculating time series data consisting of activity feature degrees that are continuous in time series.

[0069] In detail, the activity feature degree calculation unit 11H calculates an activity feature degree that indicates an activity feature derived from an acoustic signal represented by an acoustic feature and a non-acoustic signal represented by a non-acoustic feature.

[0070] In this embodiment, the activity feature degree calculation unit 11H calculates the activity feature degree from the integrated feature by using the sixth trained model 26.

[0071] The sixth trained model 26 is a neural network trained to input the integrated feature and output the emotion feature. In detail, the sixth trained model 26 is a neural network trained to reduce a sixth loss related to the difference between the correct activity feature label related to the activity feature and the activity feature. As the neural network, for example, an estimation network trained to estimate the activity feature from the integrated feature is used. The sixth trained model 26 is generated by a learning device described later. The sixth trained model 26 is stored in the storage unit 12 or the like, and is used for calculating the activity feature by the activity feature calculation unit 11H. The sixth trained model 26 is generated by a learning device described later.

[0072] The activity level estimation unit 11I estimates the communication activity level based on a comparison of the activity characteristic level with an evaluation scale. The evaluation scale is set as a boundary with a value corresponding to the communication activity level. For example, assume that the activity characteristic level takes values ​​from "-2" to "3". In this case, the evaluation scale is preset as follows, for example: "-2": no conversation, "-1": thinking, "0": stagnant, "1": slightly active, "2": active, and "3": very. Through the estimation process by the activity level estimation unit 11I, the communication activity level for the input signal including the acoustic signal and the non-acoustic signal is estimated for each frame.

[0073] The situation information generating unit 11J generates situation information representing a communication situation based on at least the voice presence degree. In this embodiment, the situation information generating unit 11J generates situation information based on the voice presence degree, the laughter presence degree, and the emotion characteristic degree.

[0074] The situation information is information about a communication situation, and includes, for example, information about the amount of speech, the number of laughs, and the rate of emotional change related to the communication situation by the speaker's speech.

[0075] That is, the situation information generating unit 11J generates situation information including the amount of speech by the sound source represented by the sound presence, the number of occurrences of laughter represented by the laughter presence, and the emotion change rate represented by the emotion characteristic degree. For example, the situation information generating unit 11J stores in advance in the storage unit 12 or the like a table associating the sound presence with the amount of speech, a table associating the laughter presence with the number of occurrences of laughter, and a table associating the emotion characteristic degree with the emotion change rate. Then, the situation information generating unit 11J may use these tables to specify the amount of speech, the number of occurrences of laughter, and the emotion change rate corresponding to the sound presence, the laughter presence, and the emotion characteristic degree calculated by the sound presence calculating unit 11E, the laughter presence calculating unit 11F, and the emotion characteristic degree calculating unit 11G, respectively, thereby calculating situation information including these.

[0076] The output control unit 11K outputs various information to the display unit 15 and the audio output unit 16. In this embodiment, the output control unit 11K outputs the communication activity level estimated by the activity level estimation unit 11I. The output control unit 11K may also output the communication activity level estimated by the activity level estimation unit 11I and the situation information generated by the situation information generation unit 11J.

[0077] For example, the output control unit 11K displays the communication activity level and the situation information on the display unit 15. At this time, it is preferable that the output control unit 11K visually associates the communication activity level and the situation information with the audio signal and / or the image signal and displays them on the display unit 15. In addition, the output control unit 11K may output the communication activity level and the situation information to the audio output unit 16.

[0078] In the present embodiment, a form in which the processing unit 11 of the estimation device 10 includes an integrated feature calculation unit 11D will be described as an example. However, the processing unit 11 may be configured not to include the integrated feature calculation unit 11D.

[0079] That is, the voice presence calculation unit 11E does not necessarily need to calculate the voice presence from an integrated feature based on the acoustic feature and the non-acoustic feature. The voice presence calculation unit 11E may calculate the voice presence directly from the acoustic feature and the non-acoustic feature, or from another intermediate output based on the acoustic feature and the non-acoustic feature. As an example, the voice presence calculation unit 11E may calculate the voice presence from the acoustic feature and the non-acoustic feature to be processed using a neural network trained to input the acoustic feature and the non-acoustic feature and output the voice presence.

[0080] Similarly, the laughter presence calculation unit 11F does not necessarily have to calculate the laughter presence from an integrated feature based on the acoustic features and non-acoustic features. The laughter presence calculation unit 11F may calculate the laughter presence directly from the acoustic features and non-acoustic features, or from another intermediate output based on the acoustic features and non-acoustic features. As an example, the laughter presence calculation unit 11F may calculate the laughter presence from the acoustic features and non-acoustic features of the processing target using a neural network trained to input the acoustic features and non-acoustic features and output the laughter presence.

[0081] Similarly, the emotion feature calculation unit 11G does not necessarily have to calculate the emotion feature from an integrated feature based on the acoustic feature and the non-acoustic feature. The emotion feature calculation unit 11G may calculate the emotion feature directly from the acoustic feature and the non-acoustic feature, or from another intermediate output based on the acoustic feature and the non-acoustic feature. As an example, the emotion feature calculation unit 11G may calculate the emotion feature from the acoustic feature and the non-acoustic feature of the processing target using a neural network trained to input the acoustic feature and the non-acoustic feature and output the emotion feature.

[0082] Similarly, the activity feature degree calculation unit 11H does not necessarily have to calculate the activity feature degree from an integrated feature based on the acoustic feature and the non-acoustic feature. The activity feature degree calculation unit 11H may calculate the activity feature degree directly from the acoustic feature and the non-acoustic feature, or from another intermediate output based on the acoustic feature and the non-acoustic feature. As an example, the activity feature degree calculation unit 11H may calculate the activity feature degree from the acoustic feature and the non-acoustic feature to be processed using a neural network trained to input the acoustic feature and the non-acoustic feature and output the activity feature degree.

[0083] Furthermore, the processing unit 11 of the estimation device 10 may be configured not to include at least one of the laughter presence calculation unit 11F and the emotion characteristic degree calculation unit 11G. If the laughter presence and / or the emotion characteristic degree is not used in the estimation device 10, the processing unit 11 may be configured not to include the laughter presence calculation unit 11F and / or the emotion characteristic degree calculation unit 11G.

[0084] Next, an example of the flow of a communication activity level estimation process executed by the processing unit 11 of the estimation device 10 of this embodiment will be described.

[0085] 3 and 4 are explanatory diagrams of an example of the flow of the communication activeness estimation process executed by the processing unit 11. FIG. 3 is a diagram that shows a schematic flow of the communication activeness estimation process by the processing unit 11. FIG. 4 is a flowchart that shows an example of the flow of the communication activeness estimation process by the processing unit 11. In the explanation of FIGS. 3 and 4, for the sake of concrete explanation, the non-acoustic signal will be explained as an image signal. The communication activeness estimation process is executed by the processing unit 11 operating in accordance with an estimation program stored in the storage unit 12 or the like.

[0086] The acquiring unit 11A acquires an input signal including an acoustic signal and an image signal (steps S1 and S100).

[0087] The acoustic feature calculation unit 11B uses the first trained model 21 to derive an acoustic feature E from the acoustic signal A acquired in steps S1 and S100.A is calculated (steps S2, S102).

[0088] The non-acoustic feature calculation unit 11C uses the second trained model 22 to derive an image feature E from the image signal V acquired in steps S1 and S100. V is calculated (step S3, step S103).

[0089] The order of steps S2, S102, S3, and S103 is not particularly limited. For example, steps S2 and S102 may be executed after steps S3 and S103. Also, steps S2 and S102 and steps S3 and S103 may be executed in parallel.

[0090] The integrated feature calculation unit 11D calculates an integrated feature Z AV is calculated (step S4, step S104).

[0091] The voice presence calculation unit 11E uses the third trained model 23 to calculate the integrated feature Z AV From the voice presence calculation Y VAD is calculated (step S5, step S105).

[0092] The laughter presence calculation unit 11F uses the fourth trained model 24 to calculate the integrated feature Z AV Laughter presence degree Y^ LD is calculated (step S6, step S106).

[0093] The emotion feature degree calculation unit 11G uses the fifth trained model 25 to calculate the integrated feature Z AV From the emotion characteristic score Y^ ER is calculated (step S7, step S107).

[0094] The activity feature degree calculation unit 11H uses the sixth trained model 26 to calculate the integrated feature Z AV From the activity characteristic degree Y^ COMM is calculated (step S8, step S108).

[0095] The processing order of steps S5 to S8 and steps S105 to S108 is not limited to the above order. For example, steps S7 to S5 and steps S107 to S105 may be executed after steps S8 and S108. Furthermore, steps S5 to S8 and steps S105 to S108 may be executed in parallel.

[0096] The activity degree estimation unit 11I calculates the activity characteristic degree Y^ calculated in steps S8 and S108. COMM Based on the comparison with the evaluation scale, the communication activity level is estimated (step S9, step S109).

[0097] The situation information generating unit 11J uses the voice presence degree calculation Y^ calculated in steps S5 and S105. VAD The laughter presence degree calculation Y^ calculated in steps S6 and S106 LD and the emotion characteristic degree Y^ calculated in steps S7 and S107. ER Based on the above, situation information is generated (steps S10 and S110).

[0098] The output control unit 11K outputs the communication activity levels estimated in steps S9 and S109 and the situation information generated in steps S10 and S110 (steps S11 and S110), and then ends the communication activity level estimation process.

[0099] Note that the communication activeness estimation process shown in Fig. 3 and Fig. 4 is an example, and the communication activeness estimation process according to the present embodiment is not limited to the procedure shown in Fig. 3 and Fig. 4. As described above, the processing unit 11 may be configured not to include the integrated feature calculation unit 11D. In this case, steps S4 and S104 may not be executed. Furthermore, the processing unit 11 may be configured not to include the laughter presence calculation unit 11F and / or the emotion feature calculation unit 11G. In this case, steps S6, S106, and / or S7 and S107 may not be executed.

[0100] As described above, the estimation device 10 of this embodiment includes an acquisition unit 11A, an acoustic feature calculation unit 11B, a non-acoustic feature calculation unit 11C, an activity feature calculation unit 11H, and an activity estimation unit 11I. The acquisition unit 11A acquires an acoustic signal and a non-acoustic signal related to a sound source. The acoustic feature calculation unit 11B calculates an acoustic feature based on the acoustic signal. The non-acoustic feature calculation unit 11C calculates a non-acoustic feature based on the non-acoustic signal. The activity feature calculation unit 11H calculates an activity feature based on the acoustic feature and the non-acoustic feature. The activity estimation unit 11I estimates a communication activity feature based on a comparison of the activity feature to an evaluation scale.

[0101] Here, in conventional technologies such as Patent Document 1 and Patent Document 2 that estimate communication activity using voice and images, there is a problem that the accuracy of estimating communication activity drops significantly when a person cannot wear a microphone or when the person's face is not facing the camera. Also, there are still limitations for practical applications such as when estimating communication activity from a microphone and camera installed on the ceiling, and there is a need to improve the accuracy of estimating communication activity.

[0102] On the other hand, the estimation device 10 of the present embodiment uses an acoustic signal and a non-audio signal, calculates an activity feature degree based on the acoustic feature of the speech signal and the non-audio feature of the non-audio signal, and estimates a communication activity degree from the activity feature degree.

[0103] In this way, the estimation device 10 of this embodiment estimates communication activity level based on acoustic signals and non-acoustic signals, and therefore can estimate communication activity level with high accuracy even when it is not possible to pick up a person's voice at close range or when it is not possible to distinguish a person's face from an image.

[0104] Therefore, the estimation device 10 of the present embodiment can improve the accuracy of estimating the communication activity level.

[0105] Furthermore, the estimation device 10 of this embodiment calculates an activity feature measure using an acoustic feature based on an acoustic signal and a non-acoustic feature based on a non-acoustic signal. Therefore, in addition to the above-mentioned effects, the estimation device 10 of this embodiment can improve the efficiency of mutual feature extraction for the activity feature measure.

[0106] Furthermore, the estimation device 10 of the present embodiment generates situation information based on the voice presence, the laughter presence, and the emotion feature degree. Therefore, the estimation device 10 of the present embodiment generates situation information in addition to the communication activity degree, thereby providing information that enables detection of the communication activity degree with higher accuracy in addition to the above-mentioned effects.

[0107] Second embodiment 5 is a schematic diagram of an example of the learning device 30 of this embodiment. The learning device 30 is a computer that learns the trained model used in the estimation device 10.

[0108] The learning device 30 has a processing unit 31, a memory unit 32, an input unit 33, a communication unit 34, a display unit 35, and an audio output unit 36. The processing unit 31, the memory unit 32, the input unit 33, the communication unit 34, the display unit 35, and the audio output unit 36 ​​are connected to each other so as to be able to communicate with each other via a bus 37 or the like.

[0109] The memory unit 32, the input unit 33, the communication unit 34, the display unit 35, and the audio output unit 36 ​​are similar to the memory unit 12, the input unit 13, the communication unit 14, the display unit 15, and the audio output unit 16 in the above embodiment, respectively.

[0110] The processing unit 31 executes information processing in the learning device 30 .

[0111] The processing unit 31 has an acquisition unit 31A, an acoustic feature calculation unit 31B, a non-acoustic feature calculation unit 31C, an integrated feature calculation unit 31D, a voice presence calculation unit 31E, a laughter presence calculation unit 31F, an emotion feature calculation unit 31G, an activity feature calculation unit 31H, an update unit 31I, an update end determination unit 31J, and an output control unit 31K.

[0112] The processing unit 31 has a processor such as a CPU and a memory such as an RA. The processing unit 31 executes a learning program stored in the storage unit 32 to execute a learning process for generating the first trained model 21 to the sixth trained model 26 by learning. The learning program is recorded in a non-transitory computer-readable recording medium. The processing unit 31 reads out the learning program from the recording medium and executes it to realize an acquisition unit 31A, an acoustic feature calculation unit 31B, a non-acoustic feature calculation unit 31C, an integrated feature calculation unit 31D, a voice presence calculation unit 31E, a laughter presence calculation unit 31F, an emotion feature calculation unit 31G, an activity feature calculation unit 31H, an update unit 31I, an update end determination unit 31J, and an output control unit 31K. The learning program may have multiple modules in which the functions of each unit (the acquisition unit 31A to the output control unit 31K) are divided and implemented.

[0113] The hardware implementation of the processing unit 31 is not limited to the above-mentioned embodiment. For example, the processing unit 31 may be configured by a circuit such as an application specific integrated circuit (ASIC) that realizes at least one of the acquisition unit 31A, the acoustic feature calculation unit 31B, the non-acoustic feature calculation unit 31C, the integrated feature calculation unit 31D, the voice presence calculation unit 31E, the laughter presence calculation unit 31F, the emotion feature calculation unit 31G, the activity feature calculation unit 31H, the update unit 31I, the update end determination unit 31J, and the output control unit 31K. At least one of the acquisition unit 31A, the acoustic feature calculation unit 31B, the non-acoustic feature calculation unit 31C, the integrated feature calculation unit 31D, the voice presence calculation unit 31E, the laughter presence calculation unit 31F, the emotion feature calculation unit 31G, the activity feature calculation unit 31H, the update unit 31I, the update end determination unit 31J, and the output control unit 31K may be implemented in a single integrated circuit or individually in multiple integrated circuits.

[0114] The acquisition unit 31A acquires an input signal including an acoustic signal and a non-acoustic signal related to a voice source as training data. The input signal is the same as described above. That is, the input signal is a time-series signal including a time-series acoustic signal and a time-series non-acoustic signal. As described above, the non-acoustic signal is an image signal related to a speaker who is speaking, a sensor signal related to a physiological response of the speaker's lips and facial muscles due to the speaking, an electroencephalogram, etc.

[0115] The acoustic feature calculation unit 31B calculates acoustic features from the acoustic signal using a first neural network 41. The acoustic features calculated by the acoustic feature calculation unit 31B are similar to the acoustic features calculated by the acoustic feature calculation unit 11B. The first neural network 41 is a neural network that inputs an acoustic signal and outputs acoustic features. The first trained model 21 is generated by training the first neural network 41.

[0116] The non-acoustic feature calculation unit 31C calculates non-acoustic features from the non-acoustic signal using a second neural network 42. The non-acoustic features calculated by the non-acoustic feature calculation unit 31C are similar to the non-acoustic features calculated by the non-acoustic feature calculation unit 11C. The second neural network 42 is a neural network that inputs a non-acoustic signal and outputs non-acoustic features. The second trained model 22 is generated by training the second neural network 42.

[0117] The integrated feature calculation unit 31D calculates an integrated feature based on the acoustic feature and the non-acoustic feature. The integrated feature calculated by the integrated feature calculation unit 31D is similar to the integrated feature calculated by the integrated feature calculation unit 11D.

[0118] The voice presence calculation unit 31E calculates the voice presence from the integrated features using the third neural network 43. The voice presence calculated by the voice presence calculation unit 31E is similar to the voice presence calculated by the voice presence calculation unit 11E. The third trained model 23 is generated by training the third neural network 43.

[0119] The third neural network 43 is a neural network trained to reduce a third loss related to the difference between the correct voice presence label and the voice presence. The correct voice presence label is a correct label indicating a voice section or a correct label indicating a non-voice section, which is assigned to each frame of the audio signal. The correct voice presence label may be created manually based on the audio signal, or may be created automatically using a voice recognition technology or the like, or may be created by combining these manual and automatic methods.

[0120] The laughter presence calculation unit 31F calculates the laughter presence from the integrated features using the fourth neural network 44. The laughter presence calculated by the laughter presence calculation unit 31F is similar to the laughter presence calculated by the laughter presence calculation unit 11F. A fourth trained model 24 is generated by training the fourth neural network 44.

[0121] The fourth neural network 44 is a neural network trained to reduce a fourth loss related to the difference between the correct laughter presence label and the laughter presence. The correct laughter presence label is a correct label indicating a laughter section or a correct label indicating a non-laughter section, which is assigned to each frame of the audio signal. The correct laughter presence label may be created manually based on the audio signal, or may be created automatically using a voice recognition technology or the like, or may be created by combining these manual and automatic methods.

[0122] Emotion characteristic degree calculation unit 31G calculates the emotion characteristic degree from the integrated features using a fifth neural network 45. The emotion characteristic degree calculated by emotion characteristic degree calculation unit 31G is similar to the emotion characteristic degree calculated by emotion characteristic degree calculation unit 11G. A fifth trained model 25 is generated by training the fifth neural network 45.

[0123] The fifth neural network 45 is a neural network trained to reduce a fifth loss related to the difference between a correct emotion salience label and an emotion salience related to an emotion feature. The correct emotion salience label is a correct label indicating an emotional state, which is assigned to each frame of an audio signal. The correct emotion salience label may be created manually based on an audio signal or a non-audio signal, may be created automatically using emotion recognition technology, or may be created by combining these manual and automatic methods.

[0124] The activity characteristic degree calculation unit 31H calculates the activity characteristic degree from the integrated characteristic using the sixth neural network 46. The activity characteristic degree calculated by the activity characteristic degree calculation unit 31H is the same as the activity characteristic degree calculated by the activity characteristic degree calculation unit 11H. A sixth trained model 26 is generated by training the sixth neural network 46.

[0125] The sixth neural network 46 is a neural network trained to reduce a sixth loss related to the difference between the correct liveness feature degree label and the liveness feature degree calculation for the liveness feature. The correct liveness feature degree label is a correct label indicating the liveness state of communication, which is assigned to each frame of the audio signal. The correct liveness feature degree label may be created manually based on the audio signal or non-audio signal, may be created automatically using emotion recognition technology, or may be created by combining these manual and automatic methods.

[0126] The update unit 31I updates the first neural network 41, the second neural network 42, the third neural network 43, the fourth neural network 44, the fifth neural network 45, and the sixth neural network 46 through learning that reduces the integrated losses of the third loss, the fourth loss, the fifth loss, and the sixth loss.

[0127] As described above, the third loss is a loss related to the difference between the correct label for voice presence (correct voice presence label) and the voice presence. The fourth loss is a loss related to the difference between the correct label for laughter presence (correct laughter presence label) and the laughter presence. The fifth loss is a loss related to the difference between the correct label for emotion salience (correct emotion salience label) and the emotion salience. The sixth loss is a loss related to the difference between the correct label for activity salience (correct activity salience) and the activity salience.

[0128] Updating a neural network means updating the learning parameters of the neural network.

[0129] The update end determination unit 31J determines whether a stop condition for the learning process is satisfied. The stop condition is, for example, that the number of updates of the learning parameters has reached a predetermined number, or that the update amount of the learning parameters is less than a threshold value.

[0130] Until the update termination determination unit 31J determines that the stop condition is satisfied, a series of processes are repeatedly executed, including calculation of acoustic features by acoustic feature calculation unit 31B, calculation of non-acoustic features by non-acoustic feature calculation unit 31C, calculation of integrated features by integrated feature calculation unit 31D, calculation of audio presence by audio presence calculation unit 31E, calculation of laughter presence by laughter presence calculation unit 31F, calculation of emotion feature calculation unit 31G, calculation of activity feature calculation unit 31H, and updating of the first neural network 41, the second neural network 42, the third neural network 43, the fourth neural network 44, the fifth neural network 45 and the sixth neural network 46 by update unit 31I.

[0131] When the update termination determination unit 31J determines that the stop condition is satisfied, the output control unit 31K outputs the first neural network 41 at the time the determination was made as the first trained model 21, the second neural network 42 as the second trained model 22, the third neural network 43 as the third trained model 23, the fourth neural network 44 as the fourth trained model 24, the fifth neural network 45 as the fifth trained model 25, and the sixth neural network 46 as the sixth trained model 26.

[0132] For example, the output control unit 31K outputs the first trained model 21, the second trained model 22, the third trained model 23, the fourth trained model 24, the fifth trained model 25, and the sixth trained model 26 to the estimation device 10. The estimation device 10 may use these trained models trained by the learning device 30 to execute the above-mentioned processing.

[0133] That is, the learning model integrating the third trained model 23, the fourth trained model 24, the fifth trained model 25, and the sixth trained model 26 is a learning model that includes a neural network trained to reduce the integration losses of the third loss related to the difference between the correct label for the voice presence and the voice presence, the fourth loss related to the difference between the correct label for the laughter presence and the laughter presence, the fifth loss related to the difference between the correct label for the emotion feature and the emotion feature, and the sixth loss related to the difference between the correct label for the activity feature and the activity feature, and when integrated features of an acoustic signal and a non-acoustic signal are input, performs calculations using the neural network and causes a computer to function to output the voice presence, the laughter presence, the emotion feature, and the activity feature.

[0134] Next, a configuration example of an integrated neural network will be described.

[0135] FIG. 6 is a diagram illustrating an example of the configuration of an integrated neural network.

[0136] The integrated neural network includes a first neural network 41, a second neural network 42, an integrated feature calculation module 51D, a third neural network 43, a fourth neural network 44, a fifth neural network 45, and a sixth neural network 46.

[0137] The first neural network 41 receives an acoustic signal A and generates an acoustic feature E A The second neural network 42 receives the image signal V and outputs the image feature E V The integrated feature calculation module 51D outputs the acoustic feature E output from the first neural network 41. A and the image feature E output from the second neural network 42 V Enter and integrate the features Z AV The integrated feature calculation module 51D is a module corresponding to the function of the integrated feature calculation unit 31D.

[0138] The third neural network 43 uses the integrated feature Z AV Enter the sound presence Y^ VAD The fourth neural network 44 uses the integrated feature Z AV Enter the laughter presence level Y^ LD The fifth neural network 45 outputs the integrated feature Z AV Enter the emotion feature score Y^ ER The sixth neural network 46 outputs the integrated feature Z AV Enter the activity characteristic score Y^ COMM Output.

[0139] It is assumed that initial values ​​of learning parameters and the like are assigned to the first neural network 41, the second neural network 42, the third neural network 43, the fourth neural network 44, the fifth neural network 45, and the sixth neural network 46. The learning parameters are weights, biases, and the like. The learning parameters may include any hyperparameters.

[0140] The first neural network 41 derives acoustic features E from the acoustic signal A. A The first neural network 41 has an encoder network architecture capable of calculating the image features E from the image signal V. As an example, the first neural network 41 includes four one-dimensional convolution layers. The second neural network 42 calculates the image features E from the image signal V. V The second neural network 42 has an encoder network architecture capable of calculating the following: As an example, the second neural network 42 includes three 3D convolution layers. At least one of the three 3D convolution layers may be connected to a maximum pooling layer and / or a global average pooling layer.

[0141] The third neural network 43 is the integrated feature ZAV from the sound presence Y^ VAD The architecture of the detection network is capable of calculating the integrated feature Z. As an example, the third neural network 43 includes two dense layers. The fourth neural network 44 calculates the integrated feature Z AV Laughter presence degree Y^ LD The fourth neural network 44 has an architecture for a detection network capable of calculating:

[0142] The fifth neural network 45 is the integrated feature Z AV From the emotion characteristic score Y^ ER The fifth neural network 45 has an architecture capable of calculating the integrated feature Z AV From the activity characteristic degree Y^ COMM The sixth neural network 46 has an architecture for a detection network capable of calculating:

[0143] All convolutional and dense layers of the first neural network 41, the second neural network 42, the third neural network 43, the fourth neural network 44, the fifth neural network 45, and the sixth neural network 46 are followed by a normalized linear function unit, which applies a normalized linear function to the output of the layer. Note that the final layers of the fifth neural network 45 and the sixth neural network 46 are followed by a sigmoid activation function unit or a softmax activation function unit, rather than a normalized linear function unit, which applies a sigmoid function or a softmax function to the output of the final layer.

[0144] The integrated neural network inputs an audio signal A and an image signal V and calculates the sound presence Y^ VAD and laughter presence Y^ LD and emotion characteristic score Y^ER and the activity characteristic degree Y^ COMM and output.

[0145] Audio presence Y^ VAD and the correct speech presence (ground truth) Y VAD The third loss function for the difference between the laughter presence Y^ LD and the correct laughter presence (ground truth) Y LD The fourth loss function for the difference between the ER and the ground truth emotion feature score Y ER The fifth loss function for the difference between the COMM and the ground truth Y COMM and a sixth loss function relating to the difference between the first neural network 41 and the sixth neural network 46. The learning parameters of the first neural network 41, the second neural network 42, the third neural network 43, the fourth neural network 44, the fifth neural network 45 and the sixth neural network 46 are updated so as to minimize an integrated loss function including the sixth loss function relating to the difference between the first neural network 41 and the sixth loss function relating to the difference between the first neural network 41 and the sixth loss function relating to the difference between the first neural network 42 and the sixth loss function relating to the difference between the first neural network 42 and the sixth loss function relating to the difference between the first neural network 43 ...4 and the sixth loss function relating to the difference between the first neural network 42 and the sixth loss function relating to the difference between the first neural network

[0146] Next, an example of the flow of the learning process by the processing unit 31 of the learning device 30 of this embodiment will be described.

[0147] Fig. 7 is a flowchart showing an example of the flow of the learning process executed by the processing unit 31 of the learning device 30 of this embodiment. The flow of the learning process will be described with reference to Fig. 6 and Fig. 7. For the sake of specificity, the following description will be given assuming that the non-acoustic signal is an image signal.

[0148] The learning process is executed by the processing unit 31 operating in accordance with a learning program stored in the storage unit 32 or the like. In the learning process, the processing unit 31 parallelly learns the first neural network 41, the second neural network 42, the third neural network 43, the fourth neural network 44, the fifth neural network 45, and the sixth neural network 46. More specifically, the processing unit 31 performs supervised learning of an integrated neural network including the first neural network 41, the second neural network 42, the third neural network 43, the fourth neural network 44, the fifth neural network 45, and the sixth neural network 46 based on training data.

[0149] The acquisition unit 31A acquires an input signal including an acoustic signal A and an image signal V (steps S11 and S201). The acquisition unit 31A acquires an input signal that is one learning sample. The frame length of the time period of the input signal is not particularly limited, but is, for example, about 30 seconds. The acoustic signal A and the image signal V included in the input signal are synchronized in time.

[0150] The acoustic feature calculation unit 31B uses the first neural network 41 to calculate an acoustic feature E from the acoustic signal A included in the input signal acquired in steps S11 and S201. A (steps S12 and S202). It is assumed that the learning of the first neural network 41 used in steps S12 and S202 is not completed until a positive determination is made in step S210, which will be described later. It is assumed that the acoustic signal A input to the first neural network 41 has been transformed from the time domain to the frequency domain.

[0151] The non-acoustic feature calculation unit 31C uses the second neural network 42 to calculate image features Ev, which are non-acoustic features, from the image signal V, which is a non-acoustic signal included in the input signal acquired in steps S1 and S201 (steps S13 and S203). It is assumed that the learning of the second neural network 42 used in steps S203 and S13 has not been completed until a positive determination is made in step S210, which will be described later.

[0152] The order of steps S12, S202, S13, and S203 is not particularly limited. For example, steps S12 and S202 may be executed after steps S13 and S203. Also, steps S12 and S202 and steps S13 and S203 may be executed in parallel.

[0153] The integrated feature calculation unit 31D calculates the acoustic feature E by using the integrated feature calculation module 51D. A and image feature E V Integration features with Z AV (Step S14, Step S204). AV is calculated for each frame time t.

[0154] The voice presence calculation unit 31E uses the third neural network 43 to calculate the integrated feature Z AV from the sound presence Y^ VAD (Steps S15 and S205). It is assumed that the learning of the third neural network 43 used in steps S15 and S205 is not completed until a positive determination is made in step S210, which will be described later.

[0155] The laughter presence calculation unit 31F uses the fourth neural network 44 to calculate the integrated feature Z AV Laughter presence degree Y^ LD(Steps S16 and S206). It is assumed that the learning of the fourth neural network 44 used in steps S16 and S206 is not completed until a positive determination is made in step S210, which will be described later.

[0156] The emotion characteristic degree calculation unit 215 uses the third neural network 43 to calculate the integrated characteristic Z AV From the emotion characteristic score Y^ ER (Steps S17 and S207). It is assumed that the learning of the third neural network 43 used in steps S17 and S207 is not completed until a positive determination is made in step S210, which will be described later.

[0157] The activity characteristic calculation unit 216 uses the fourth neural network 44 to calculate the integrated characteristic Z AV From the activity characteristic degree Y^ COMM (Steps S18 and S208). It is assumed that the learning of the fourth neural network 44 used in steps S18 and S208 is not completed until a positive determination is made in step S210, which will be described later.

[0158] The processing order of steps S205 to S208 and steps S15 to S18 is not particularly limited. The processing may be performed in the order of steps S208 to S205 and steps S18 to S15. Moreover, the processing of steps S205 to S208 and steps S15 to S18 may be performed in parallel.

[0159] The update unit 31I updates the first neural network 41, the second neural network 42, the third neural network 43, the fourth neural network 44, the fifth neural network 45, and the sixth neural network 46 using an integrated loss including the third loss, the fourth loss, the fifth loss, and the sixth loss (steps S19 and S209).

[0160] As shown in the following equation (3), the integrated loss L at frame time t and coordinate d is TOTAL is the third loss L VAD and the fourth loss L LD and the fifth loss L ER and the sixth loss L COMM It is defined as the sum of

[0161]

number

[0162] The third loss L VAD is a loss for penalizing the discrepancy between the ground-truth speech presence label and the speech presence. As shown in the following equation (4), the third loss L VAD is defined as binary cross entropy (BCE). The binary cross entropy is the probability that the correct speech presence label Y VAD and sound presence Y^ VAD It is used as a measure to evaluate the difference between

[0163] The fourth loss L LD is a loss for penalizing the difference between the ground truth laughter presence label and the laughter presence. As shown in the following formula (5), the fourth loss L LD is defined as the binary cross entropy. The binary cross entropy is the probability that the correct laughter presence label Y LD and laughter presence Y^ LD It is used as a measure to evaluate the difference between

[0164] The fifth loss L ER is a loss for penalizing the difference between the correct emotion feature label and the emotion feature. As shown in the following formula (6), the fifth loss L ERis defined as SoftMax Cross Entropy (SCE). SoftMax Cross Entropy is the correct emotion feature label Y ER and emotion characteristic score Y^ ER It is used as a measure to evaluate the difference between

[0165] The sixth loss L COMM is a loss for penalizing the difference between the correct active feature label and the active feature score for the active feature. As shown in the following formula (7), the sixth loss L COMM is defined as the mean squared error (MSE). The mean squared error is the correct activation feature label Y COMM and the activity characteristic degree Y^ COMM It is used as a measure to evaluate the difference between

[0166]

number

[0167] The update unit 31I updates the integrated loss L according to an arbitrary optimization method. TOTAL The learning parameters of the first neural network 41, the second neural network 42, the third neural network 43, the fourth neural network 44, the fifth neural network 45, and the sixth neural network 46 are updated so as to minimize the correct speech presence label Y VAD and sound presence Y^ VAD and the correct laughter presence label Y LD and laughter presence Y^ LD and the correct emotion feature label Y ER and emotion characteristic score Y^ ER and the correct activation feature label Y COMM and the activity characteristic degree Y^ COMMThe learning parameters of the first neural network 41, the second neural network 42, the third neural network 43, the fourth neural network 44, the fifth neural network 45, and the sixth neural network 46 are updated so as to comprehensively minimize the difference between and As the optimization method, any method such as stochastic gradient descent or Adam (adaptive moment estimation) may be used.

[0168] The update end determination unit 31J determines whether or not the stop condition is satisfied (steps S210 and S20). The stop condition may be set, for example, that the number of updates of the learning parameters has reached a predetermined number, that the update amount of the learning parameters is less than a threshold, etc. If it is determined that the stop condition is not satisfied (step S210: No), the process returns to steps S201 and S11.

[0169] The update end determination unit 31J makes a negative determination until it determines that the stop condition is satisfied (step S210: Yes). Therefore, the series of processes in step S11, step S201 to step S19, and step S209 are repeatedly executed until it is determined that the stop condition is satisfied.

[0170] Note that a series of processes from step S11 to step S19 and step S201 to step S209 may be repeated for one learning sample (batch learning), or a series of processes from step S11 to step S19 and step S201 to step S209 may be repeated for multiple learning samples (mini-batch learning).

[0171] Then, if it is determined in step S210 that the stopping condition is satisfied (step S10: Yes), the update termination determination unit 31J outputs the first neural network 41 at the time the stopping condition is satisfied as the first trained model 21, the second neural network 42 as the second trained model 22, the third neural network 43 as the third trained model 23, the fourth neural network 44 as the fourth trained model 24, the fifth neural network 45 as the fifth trained model 25, and the sixth neural network 46 as the sixth trained model 26 (steps S21, S211).

[0172] By the processing of steps S21 and S210, the first trained model 21, the second trained model 22, the third trained model 23, the fourth trained model 24, the fifth trained model 25, and the sixth trained model 26 are transmitted to the estimation device 10 via the communication unit 34 and the like, and are stored in the storage unit 12 of the estimation device 10. The update termination determination unit 31J may output an integrated neural network including the first trained model 21, the second trained model 22, the third trained model 23, the fourth trained model 24, the fifth trained model 25, the sixth trained model 26, and the integrated feature calculation module 51D. Then, this routine is terminated.

[0173] Through the processes of steps S11 to S21 and steps S201 to S211, the learning process by the processing unit 31 is completed.

[0174] As described above, the estimation device 10 estimates a communication activity level from an acoustic signal and an image signal by using the first trained model 21, the second trained model 22, the third trained model 23, the fourth trained model 24, and the sixth trained model 26 generated by the learning device 30. The estimation device 10 may estimate a communication activity level by inputting the acoustic signal and the image signal to the integrated neural network.

[0175] As described above, the learning device 30 of the present embodiment includes an acquisition unit 31A, an audio feature calculation unit 31B, a non-audio feature calculation unit 31C, an integrated feature calculation unit 31D, a voice presence calculation unit 31E, a laughter presence calculation unit 31F, an emotion feature calculation unit 31G, an activity feature calculation unit 31H, and an update unit 31I. The acquisition unit 31A acquires an audio signal and a non-audio signal related to a sound source. The audio feature calculation unit 31B calculates an audio feature from the audio signal using a first neural network 41. The non-audio feature calculation unit 31C calculates a non-audio feature from the non-audio signal using a second neural network 42. The integrated feature calculation unit 31D calculates an integrated feature based on the audio feature and the non-audio feature. The audio presence calculation unit 31E calculates audio presence from the integrated feature using a third neural network 43. The laughter presence calculation unit 31F calculates the laughter presence from the integrated features using a fourth neural network 44. The emotion feature calculation unit 31G calculates the emotion feature from the integrated features using a fifth neural network 45. The activity feature calculation unit 31H calculates the activity feature from the integrated features using a sixth neural network 46. The update unit 31I updates the first neural network 41, the second neural network 42, the third neural network 43, the fourth neural network 44, the fifth neural network 45, and the sixth neural network 46 by learning to reduce the integrated losses of the third loss related to the difference between the correct label for the voice presence and the voice presence, the fourth loss related to the difference between the correct label for the laughter presence and the laughter presence, the fifth loss related to the difference between the correct label for the emotion feature and the emotion feature, and the sixth loss related to the difference between the correct label for the activity feature and the activity feature.

[0176] Therefore, in the learning device 30 of this embodiment, by performing multimodal learning of acoustic signals and non-acoustic signals, and multitask learning of a voice presence estimation task, a laughter presence estimation task, an emotion feature estimation task, and an activity feature estimation task, it becomes possible to generate an integrated neural network of the first trained model 21, the second trained model 22, the third trained model 23, the fourth trained model 24, the fifth trained model 25, and the sixth trained model 26, which are highly accurate communication activity estimation models.

[0177] In the estimation device 10, the activity feature calculated using the sixth trained model 26 can be more clearly and accurately calculated from the voice feature calculated using the first trained model 21, the non-voice feature calculated using the second trained model 22, the voice presence calculated using the third trained model 23, the laughter presence calculated using the fourth trained model 24, and the emotion feature calculated using the fifth trained model 25. As a result, it is possible to detect the communication activity level even when it is not possible to collect a person's voice at a close distance or when it is not possible to distinguish a person's face from an image. That is, the learning device 30 of this embodiment can learn a learning model that can improve the estimation accuracy of the communication activity level.

[0178] (Example) Verification of the performance of the integrated neural network of this embodiment (hereinafter referred to as configuration example 3 of the present invention) including the first neural network 41, the second neural network 42, the third neural network 43, the fourth neural network 44, the fifth neural network 45, and the sixth neural network 46 generated by the above embodiment was performed using simulated dialogue video data.

[0179] The simulated dialogue video data was collected from one-on-one simulated dialogues held in an office conference room using a 360-degree omnidirectional fisheye camera mounted on the ceiling. A total of 12 pairs of subjects, each consisting of two people, who agreed to data collection, were filmed for 10 minutes each for three dialogue tasks (Task B-1, Task B-2, Task C) (total of 6 hours = 12 pairs x 3 tasks x 10 minutes). In addition, assuming multimodal processing of audio and image information, the footage was re-encoded as a monaural sound source with a resolution of 960 x 960 pixels, a frame rate of 25, and a sampling rate of 16 kHz.

[0180] The topics of the simulated dialogue given to each dialogue task were as follows: In Task B-1, subjects were asked to freely talk about what they would like to do after the COVID-19 pandemic is over, simulating a very active communication situation from the subjects. In Task B-2, subjects were asked to freely talk about plans to invite friends or guests from overseas to Japan and entertain them, simulating a somewhat active communication situation from the subjects. In Task C, subjects were asked to jointly discuss test questions from the Synthetic Personality Inventory (SPI) ability test, simulating a somewhat inactive communication situation from the subjects.

[0181] The simulated dialogue video data was allocated as a training dataset, a validation dataset, and a test dataset, respectively, and used for deep learning and accuracy verification of configuration example 3 of the present invention. The division ratio was 7:1:2, and was adjusted based on the correct label distribution of communication activity levels so that the features of each dataset were not biased as much as possible. The validation dataset was used to identify hyperparameters suitable for each model and each experimental condition.

[0182] 8 is a table showing the results of the validation using a test dataset of simulated dialogue video data. The validation results are shown for each model. Configuration example 3 of the present invention significantly outperforms the configuration examples of the prior art in all experimental conditions, providing a maximum relative improvement of approximately 1.32 reduction in average error value and approximately 26.37 percentage point improvement in estimation processing accuracy.

[0183] The above embodiment is merely an example, and various modifications are possible. For example, the acoustic signal is a voice signal, but may be a vocal cord signal or vocal tract signal obtained by decomposing the voice signal. Also, the acoustic signal is waveform data of sound pressure values ​​in the time domain or frequency domain, but may be data obtained by converting the waveform data into an arbitrary space.

[0184] Thus, according to some of the above embodiments, it becomes possible to detect communication activity even when it is not possible to pick up a person's voice at close range or when it is not possible to identify a person's face from an image.

[0185] Next, an example of the hardware configuration of the estimation device 10 and the learning device 30 of the above embodiment will be described.

[0186] FIG. 9 is a diagram illustrating an example of a hardware configuration of the estimation device 10 and the learning device 30 according to the above embodiment.

[0187] The estimation device 10 and learning device 30 of the above embodiment include a control device such as a CPU (Central Processing Unit) 90D, storage devices such as a ROM (Read Only Memory) 90E, a RAM (Random Access Memory) 90F, and a HDD (Hard Disk Drive) 90G, an I / F section 90B that interfaces with various devices, an output section 90A that outputs various information such as output information, an input section 90C that accepts operations by the user, and a bus 90H that connects each section, and has a hardware configuration that utilizes a normal computer.

[0188] In the estimation device 10 and the learning device 30 of the above embodiment, the CPU 90D reads out an information processing program from the ROM 90E onto the RAM 90F and executes it, thereby realizing each of the above units on the computer.

[0189] The information processing program for executing each of the above processes executed by the estimation device 10 and the learning device 30 of the above embodiment may be stored in the HDD 90G. Also, the information processing program for executing each of the above processes executed by the estimation device 10 and the learning device 30 of the above embodiment may be provided by being pre-installed in the ROM 90E.

[0190] The information processing program for executing the above-mentioned processes executed by the estimation device 10 and the learning device 30 of the above-mentioned embodiment may be stored in a computer-readable storage medium such as a CD-ROM, a CD-R, a memory card, a DVD (Digital Versatile Disc), or a flexible disk (FD) in an installable or executable format file and provided as a computer program product. The program for executing the above-mentioned processes executed by the estimation device 10 and the learning device 30 of the above-mentioned embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading the program via the network. The program for executing the above-mentioned processes executed by the estimation device 10 and the learning device 30 of the above-mentioned embodiment may be provided or distributed via a network such as the Internet.

[0191] Although the embodiment of the present invention has been described above, the embodiment is presented as an example and is not intended to limit the scope of the invention. This new embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0192] 10 Estimation device 11A Acquisition Department 11B Acoustic feature calculation unit 11C Non-acoustic feature calculation unit 11D Integrated feature calculation unit 11E Voice Presence Calculation Unit 11F Laughter presence calculation section 11G Emotion Feature Calculation Unit 11H Activity characteristic degree calculation section 11I Activity Estimator 11J Situation Information Generation Unit 11K Output control section 21 First trained model 22 Second trained model 23 Third trained model 24 Fourth trained model 25 Fifth trained model 26 6th trained model 30 Learning Device 31A Acquisition Department 31B Acoustic feature calculation unit 31C Non-acoustic feature calculation unit 31D Integrated feature calculation unit 31E Voice Presence Calculation Unit 31F Laughter presence calculation section 31G Emotion characteristic calculation unit 31H Activity characteristic calculation section 31I Update Section 41 The First Neural Network 42 The Second Neural Network 43 The third neural network 44 The Fourth Neural Network 45 The Fifth Neural Network 46 The 6th Neural Network

Claims

1. acquiring acoustic and non-acoustic signals relating to a sound source; an acoustic feature calculation step of calculating an acoustic feature based on the acoustic signal; a non-acoustic feature calculation step of calculating a non-acoustic feature based on the non-acoustic signal; an activity feature degree calculation step of calculating an activity feature degree based on the acoustic feature and the non-acoustic feature; an activity level estimating step of estimating a communication activity level based on a comparison of the activity feature level with an evaluation scale; An estimation program for causing a computer to execute the above.

2. The acoustic feature calculation step includes: Calculating the acoustic features according to changes in the sound represented by the acoustic signal; The estimation program according to claim 1 .

3. The non-acoustic feature calculation step includes: calculating the non-acoustic features responsive to non-acoustic changes represented by the non-acoustic signal; The estimation program according to claim 1 .

4. The activity characteristic degree calculation step includes: calculating the activity feature measure representing an activity feature derived from the acoustic signal represented by the acoustic features and the non-acoustic signal represented by the non-acoustic features; The estimation program according to claim 1 .

5. a voice presence calculation step of calculating a voice presence based on the acoustic features and the non-acoustic features; a situation information generating step of generating situation information representing a communication situation based on the voice presence; The estimation program according to claim 1 , further comprising:

6. a laughter presence calculation step of calculating a laughter presence based on the acoustic features and the non-acoustic features; an emotion feature degree calculation step of calculating an emotion feature degree based on the acoustic features and the non-acoustic features; Further comprising: The situation information generating step includes: generating the situation information based on the voice presence, the laughter presence, and the emotion characteristic degree; The estimation program according to claim 5 .

7. The voice presence calculation step includes: calculating the voice presence according to voice features derived from the acoustic signal represented by the acoustic features and the non-acoustic signal represented by the non-acoustic features; The estimation program according to claim 5 .

8. The laughter presence calculation step includes: calculating the laughter presence according to laughter features derived from the acoustic signal represented by the acoustic features and the non-acoustic signal represented by the non-acoustic features; The estimation program according to claim 6.

9. The emotion feature degree calculation step includes: calculating the emotion feature degree according to emotion-related features derived from the acoustic signal represented by the acoustic features and the non-acoustic signal represented by the non-acoustic features; The estimation program according to claim 6.

10. The situation information generating step includes: the amount of speech by the speech source represented by the voice presence; and the number of occurrences of laughter represented by the laughter presence degree; and An emotion change rate represented by the emotion characteristic degree; and generating said status information, The estimation program according to claim 6.

11. The method further includes an integrated feature calculation step of calculating an integrated feature by integrating the acoustic feature and the non-acoustic feature, The voice presence calculation step calculates the voice presence based on the integrated features, the laughter presence calculation step calculates the laughter presence based on the integrated feature; the emotion feature degree calculation step calculates the emotion feature degree based on the integrated feature; The activity characteristic degree calculation step calculates the activity characteristic degree based on the integrated characteristic. The estimation program according to claim 6.

12. an output control step of outputting the communication activity level; The estimation program according to claim 1 , further comprising:

13. The output control step includes: outputting situation information representing the communication activity level and the communication situation; The estimation program according to claim 12.

14. The acoustic feature calculation step calculates the acoustic feature from the acoustic signal using a first trained model; the non-acoustic feature calculation step calculates the non-acoustic feature from the non-acoustic signal using a second trained model; the active feature calculation step calculates the active feature from integrated features of the acoustic signal and the non-acoustic signal by using a sixth trained model; The estimation program according to claim 1 .

15. the voice presence calculation step calculates the voice presence from integrated features of the acoustic signal and the non-acoustic signal by using a third trained model; the laughter presence calculation step calculates the laughter presence from the integrated features by using a fourth trained model; the emotion feature degree calculation step calculates the emotion feature degree from the integrated features by using a fifth trained model; The estimation program according to claim 6.

16. The estimation program according to claim 1 , wherein the non-acoustic signal is an image signal that is time-synchronized with the acoustic signal.

17. acquiring acoustic and non-acoustic signals relating to a sound source; an acoustic feature calculation step of calculating an acoustic feature from the acoustic signal using a first neural network; a non-acoustic feature calculation step of calculating a non-acoustic feature from the non-acoustic signal using a second neural network; an integrated feature calculation step of calculating an integrated feature based on the acoustic feature and the non-acoustic feature; a voice presence calculation step of calculating a voice presence from the integrated features using a third neural network; a laughter presence calculation step of calculating a laughter presence from the integrated features using a fourth neural network; an emotion feature degree calculation step of calculating an emotion feature degree from the integrated features by using a fifth neural network; an activity feature calculation step of calculating an activity feature from the integrated features by using a sixth neural network; an updating step of updating the first neural network, the second neural network, the third neural network, the fourth neural network, the fifth neural network, and the sixth neural network by learning that reduces an integrated loss of a third loss related to a difference between a correct label related to the voice presence and the voice presence, a fourth loss related to a difference between a correct label related to the laughter presence and the laughter presence, a fifth loss related to a difference between a correct label related to the emotion feature measure and the emotion feature measure, and a sixth loss related to a difference between a correct label related to the activity feature measure and the activity feature measure; A learning program that allows a computer to execute the above.

18. an acquisition unit that acquires an acoustic signal and a non-acoustic signal related to a sound source; an acoustic feature calculation unit that calculates an acoustic feature based on the acoustic signal; a non-acoustic feature calculation unit that calculates a non-acoustic feature based on the non-acoustic signal; an activity feature measure calculation unit that calculates an activity feature measure based on the acoustic feature and the non-acoustic feature; an activity level estimation unit that estimates a communication activity level based on a comparison of the activity feature level with an evaluation scale; An estimation device comprising:

19. an acquisition unit that acquires an acoustic signal and a non-acoustic signal related to a sound source; an acoustic feature calculation unit that calculates an acoustic feature from the acoustic signal by using a first neural network; a non-acoustic feature calculation unit that calculates non-acoustic features from the non-acoustic signal using a second neural network; an integrated feature calculation unit that calculates an integrated feature based on the acoustic feature and the non-acoustic feature; a voice presence calculation unit that calculates a voice presence from the integrated features by using a third neural network; a laughter presence calculation unit that calculates a laughter presence from the integrated features by using a fourth neural network; an emotion feature degree calculation unit that calculates an emotion feature degree from the integrated features by using a fifth neural network; an activity feature degree calculation unit that calculates an activity feature degree from the integrated feature by using a sixth neural network; an update unit that updates the first neural network, the second neural network, the third neural network, the fourth neural network, the fifth neural network, and the sixth neural network by learning that reduces an integrated loss of a third loss related to a difference between a correct label related to the voice presence and the voice presence, a fourth loss related to a difference between a correct label related to the laughter presence and the laughter presence, a fifth loss related to a difference between a correct label related to the emotion feature measure and the emotion feature measure, and a sixth loss related to a difference between a correct label related to the activity feature measure and the activity feature measure; A learning device comprising:

20. Executed by a computer, acquiring acoustic and non-acoustic signals relating to a sound source; an acoustic feature calculation step of calculating an acoustic feature based on the acoustic signal; a non-acoustic feature calculation step of calculating a non-acoustic feature based on the non-acoustic signal; an activity feature degree calculation step of calculating an activity feature degree based on the acoustic feature and the non-acoustic feature; an activity level estimating step of estimating a communication activity level based on a comparison of the activity feature level with an evaluation scale; An estimation method including:

21. Executed by a computer, acquiring acoustic and non-acoustic signals relating to a sound source; an acoustic feature calculation step of calculating an acoustic feature from the acoustic signal using a first neural network; a non-acoustic feature calculation step of calculating a non-acoustic feature from the non-acoustic signal using a second neural network; an integrated feature calculation step of calculating an integrated feature based on the acoustic feature and the non-acoustic feature; a voice presence calculation step of calculating a voice presence from the integrated features using a third neural network; a laughter presence calculation step of calculating a laughter presence from the integrated features using a fourth neural network; an emotion feature degree calculation step of calculating an emotion feature degree from the integrated features by using a fifth neural network; an activity feature calculation step of calculating an activity feature from the integrated features by using a sixth neural network; an updating step of updating the first neural network, the second neural network, the third neural network, the fourth neural network, the fifth neural network, and the sixth neural network by learning that reduces an integrated loss of a third loss related to a difference between a correct label related to the voice presence and the voice presence, a fourth loss related to a difference between a correct label related to the laughter presence and the laughter presence, a fifth loss related to a difference between a correct label related to the emotion feature measure and the emotion feature measure, and a sixth loss related to a difference between a correct label related to the activity feature measure and the activity feature measure; Learning methods including:

22. a neural network trained to reduce a third loss related to a difference between a correct label for a voice presence and the voice presence, a fourth loss related to a difference between a correct label for a laughter presence and the laughter presence, a fifth loss related to a difference between a correct label for an emotion feature measure and the emotion feature measure, and a sixth loss related to a difference between a correct label for an activity feature measure and the activity feature measure; causing a computer to function in such a manner that, when an integrated feature of an audio signal and a non-audio signal is input, the computer executes a calculation using the neural network, and outputs the voice presence, the laughter presence, the emotion feature, and the activity feature. Learning model.

Images