Atmosphere estimation method, information processing device, and program
The method enhances atmosphere estimation in group conversations by analyzing sound data for features like unity, magnitude, and pitch, converting them into color scores, and using pattern matching, thereby improving the accuracy of social atmosphere assessment.
Patent Information
- Application Number
- JP2023085679
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-12-05
AI Technical Summary
Existing technologies fail to effectively estimate the atmosphere of a place where multiple people are conversing, lacking consideration for factors beyond individual emotion analysis.
An atmosphere estimation method that includes acquiring sound data, calculating feature amounts such as unity, magnitude, and pitch of sounds, converting these into color scores, and estimating atmosphere based on pattern matching of histograms, optionally incorporating voice recognition and image analysis.
Enables accurate estimation of conversation atmosphere by considering multiple factors, improving understanding of social dynamics and emotional states in group interactions.
Smart Images

Figure 2024168753000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an atmosphere estimation method, an information processing device, and a program. [Background technology]
[0002] There are known techniques for estimating the emotion of a target person by analyzing voice data. For example, Patent Literature 1 discloses a store clerk management system that estimates the emotion of a store clerk based on a conversation between the store clerk and a conversation partner, and calculates the employee satisfaction level of the store clerk. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent Publication 2011-237957 Summary of the Invention [Problem to be solved by the invention]
[0004] The conventional technology focuses on estimating individual emotions by analyzing voice data. In addition, in order to grasp the emotions and employee satisfaction of store clerks, it is necessary to recognize the speech intervals of the store clerks. On the other hand, the conventional technology does not consider at all the estimation of the atmosphere in a place where multiple people are having a conversation. Thus, there is room for improvement in the technology for estimating the atmosphere of a place where a conversation is taking place.
[0005] In view of the above circumstances, an object of the present disclosure is to improve the technology for estimating the atmosphere of a place where a conversation takes place. [Means for solving the problem]
[0006] (1) An atmosphere estimation method according to an embodiment of the present disclosure is an atmosphere estimation method executed by an information processing device, comprising: acquiring sound data of a place where a conversation takes place; calculating a feature amount related to the sound data based on the sound data; A step of estimating an atmosphere of a place where the conversation takes place based on the feature amount; Including, The feature amount includes unity of sound.
[0007] (2) An atmosphere estimation method according to an embodiment of the present disclosure is the atmosphere estimation method according to (1), The sound data includes conversational voice, sound effects, environmental sounds and emotional sounds.
[0008] (3) An atmosphere estimation method according to an embodiment of the present disclosure is the atmosphere estimation method according to (1) or (2), The features include loudness and pitch of a sound.
[0009] (4) An atmosphere estimation method according to an embodiment of the present disclosure is the atmosphere estimation method according to (2), further comprising: converting the features into color scores by associating the loudness, pitch, and uniformity of the sound with RGB values, respectively; outputting a histogram of the color scores for a given period of the conversation; Includes.
[0010] (5) An atmosphere estimation method according to an embodiment of the present disclosure is the atmosphere estimation method according to (4), further comprising: The estimating step estimates the atmosphere of the place where the conversation takes place based on pattern matching of the histogram.
[0011] (6) An atmosphere estimation method according to an embodiment of the present disclosure is the atmosphere estimation method according to any one of (1) to (5), The feature amount includes turn-taking.
[0012] (7) An atmosphere estimation method according to an embodiment of the present disclosure is the atmosphere estimation method according to any one of (1) to (6), The feature amount includes the presence or absence of a silent period.
[0013] (8) An atmosphere estimation method according to an embodiment of the present disclosure is the atmosphere estimation method according to any one of (1) to (7), acquiring the content of the sound data by voice recognition; In the estimating step, the atmosphere is estimated based on the feature amount and a result of the speech recognition.
[0014] (9) An atmosphere estimation method according to an embodiment of the present disclosure is the atmosphere estimation method according to any one of (1) to (7), further comprising: acquiring an image of at least one of the plurality of people having a conversation or a position of at least one of the plurality of people; In the estimating step, the atmosphere is estimated based on the feature amount and the video or the position.
[0015] (10) An atmosphere estimation method according to an embodiment of the present disclosure is the atmosphere estimation method according to any one of (1) to (7), further comprising: acquiring, using a microphone array, at least one location of the people engaged in the conversation; The atmosphere is estimated based on the feature amount and the position. By using a microphone array in this manner, the position of a sound source may be identified from sound data, and the atmosphere of the place where a conversation is taking place may be estimated based on the sound source position and the feature amount.
[0016] (11) An information processing device according to an embodiment of the present disclosure is an information processing device that includes a control unit and estimates an atmosphere, The control unit is Acquire sound data from conversations, Calculating a feature amount related to the sound data based on the sound data; An information processing device that estimates an atmosphere of a place where the conversation takes place based on the feature amount, The feature amount includes unity of sound.
[0017] (12) A program according to an embodiment of the present disclosure is a program for estimating an atmosphere executed by an information processing device, the program comprising: Acquiring sound data in a conversation; Calculating a feature amount related to the sound data based on the sound data; Estimating the atmosphere of the place where the conversation takes place based on the feature amount; and Run the command, The feature amount includes unity of sound. Effect of the Invention
[0018] According to an atmosphere estimation method, an information processing device, and a program according to an embodiment of the present disclosure, it is possible to improve the technique for estimating the atmosphere of a place where a conversation takes place. [Brief description of the drawings]
[0019] [Figure 1] 1 is a block diagram showing a schematic configuration of an information processing device that executes an atmosphere estimation technique according to an embodiment of the present disclosure. [Diagram 2] 1 is a flowchart illustrating an atmosphere estimation method according to an embodiment of the present disclosure. [Diagram 3] This is a table showing the correspondence between RGB values and color scores. [Figure 4] This is a table showing the correspondence between RGB values and color scores. [Diagram 5] 1 is an example of a histogram of color scores. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0020] Hereinafter, a technique for estimating an atmosphere of a conversation location according to an embodiment of the present disclosure will be described with reference to the drawings.
[0021] In each drawing, the same or corresponding parts are denoted by the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.
[0022] First, an overview of the present embodiment will be described. The conversation atmosphere estimation technology according to the embodiment of the present disclosure is executed by an information processing device 10. The conversation atmosphere estimation technology according to the embodiment of the present disclosure can be used, for example, in the field of care, nursing, medical care, etc. Specifically, the conversation atmosphere estimation technology according to the embodiment of the present disclosure can estimate the atmosphere of the conversation from a conversation between a caregiver and a care recipient. This can be used to measure the happiness of the caregiver and the care recipient.
[0023] More specifically, the information processing device 10 acquires sound data in a place where a conversation is taking place. The sound data includes not only the voice of the conversation, but also, for example, sound effects, environmental sounds, and emotional sounds. The sound effects include sounds such as clapping. The environmental sounds include air conditioning sounds, etc. The emotional sounds include laughter, etc. It is preferable that such sound data includes the voice of the conversation. Based on the acquired sound data, the information processing device 10 calculates a feature amount related to the sound data, for example, by performing a Fourier transform on the sound data. Such a feature amount includes the unity of the sound. The unity of the sound is a quantitative index determined based on the temporal fluctuation of each frequency element of the sound within a predetermined time. The predetermined time can be set arbitrarily according to the length of the conversation. For example, the predetermined time is, for example, one minute. The predetermined time is not limited to this, and may be less than one minute or longer than one minute. Each element of the sound may include, for example, the pitch of the sound, the loudness of the sound, etc. The pitch of the sound is an index indicating the frequency pattern contained in the sound based on a predetermined frequency, that is, the pitch pattern of the sound. The loudness of a sound is an index value that indicates the loudness of a sound compared to a predetermined reference volume. When the variation of each element of a sound over a predetermined time is small, the unity of the sound is high, and when the variation of each element of a sound over a predetermined time is large, the unity of the sound is low. Note that the elements of a sound used to evaluate unity are not limited to those mentioned above, and may include, for example, the tone of a sound. The tone of a sound is an index that represents the timbre and texture. The information processing device 10 then estimates the atmosphere of the place where the conversation takes place based on the feature amount including unity.
[0024] In this way, according to the atmosphere estimation technique of the conversation place according to the present embodiment, a feature is calculated from sound data in the conversation place, preferably from sound data during the conversation, and the atmosphere of the conversation place is estimated based on the feature. In particular, the feature includes the unity of sound. By using the unity of sound for estimation, the atmosphere of the conversation place is estimated with high accuracy. Specifically, for example, when one person continues to talk in a conversation, the unity may be high, but in this case, it is difficult to say that the atmosphere of the conversation place is good. In this way, according to the atmosphere estimation technique of the conversation place according to the present embodiment, the estimation technique of the atmosphere of the conversation place is improved by considering the unity of sound.
[0025] Here, the feature amount may include the loudness and pitch of a sound in addition to the uniformity of the sound. In the following embodiment, a case will be described in which the feature amount includes three items, namely, the uniformity of the sound, the loudness and pitch of a sound.
[0026] (Configuration of information processing device) Next, a detailed description will be given of each component of the information processing device 10. The information processing device 10 is any device used by a user. For example, a personal computer, a server computer, a general-purpose electronic device, or a dedicated electronic device can be adopted as the information processing device 10.
[0027] As shown in FIG. 1, the information processing device 10 includes a control unit 11, a storage unit 12, an input unit 13, an output unit 14, and a communication unit 15.
[0028] The control unit 11 includes at least one processor, at least one dedicated circuit, or a combination of these. The processor is a general-purpose processor such as a central processing unit (CPU) or a graphics processing unit (GPU), or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, a field-programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The control unit 11 executes processes related to the operation of the information processing device 10 while controlling each part of the information processing device 10.
[0029] The storage unit 12 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a random access memory (RAM) or a read only memory (ROM). The RAM is, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The ROM is, for example, an electrically erasable programmable read only memory (EEPROM). The storage unit 12 functions as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 12 stores data used in the operation of the information processing device 10 and data obtained by the operation of the information processing device 10.
[0030] The input unit 13 includes at least one input interface. The input interface is, for example, a physical key, a capacitive key, a pointing device, or a touch screen integrally provided with a display. The input interface may also be, for example, a sound sensor that accepts voice input, or a camera that accepts gesture input. The input unit 13 accepts an operation to input data used for the operation of the information processing device 10. The input unit 13 may be connected to the information processing device 10 as an external input device instead of being provided in the information processing device 10. As a connection method, any method such as USB (Universal Serial Bus), HDMI (registered trademark) (High-Definition Multimedia Interface), or Bluetooth (registered trademark) can be used.
[0031] The output unit 14 includes at least one output interface. The output interface is, for example, a display that outputs information as a video. The display is, for example, a liquid crystal display (LCD) or an organic EL (electro luminescence) display. The output unit 14 displays and outputs data obtained by the operation of the information processing device 10. The output unit 14 may be connected to the information processing device 10 as an external output device instead of being provided in the information processing device 10. As a connection method, any method such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used.
[0032] The communication unit 15 includes at least one external communication interface. The communication interface may be either a wired communication interface or a wireless communication interface. In the case of wired communication, the communication interface is, for example, a LAN (Local Area Network) interface or a USB (Universal Serial Bus). In the case of wireless communication, the communication interface is, for example, an interface compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th generation), or 5G (5th generation), or an interface compatible with short-range wireless communication such as Bluetooth (registered trademark). The communication unit 15 receives data used in the operation of the information processing device 10, and transmits data obtained by the operation of the information processing device 10.
[0033] The functions of the information processing device 10 are realized by executing a program according to this embodiment on a processor equivalent to the information processing device 10. That is, the functions of the information processing device 10 are realized by software. The program causes a computer to execute the operations of the information processing device 10, thereby causing the computer to function as the information processing device 10. That is, the computer functions as the information processing device 10 by executing the operations of the information processing device 10 in accordance with the program.
[0034] In this embodiment, the program can be recorded on a computer-readable recording medium. The computer-readable recording medium includes a non-transitory computer-readable medium, such as a magnetic recording device, an optical disc, a magneto-optical recording medium, or a semiconductor memory. The program is distributed, for example, by selling, transferring, or lending a portable recording medium such as a DVD (digital versatile disc) or a CD-ROM (compact disc read only memory) on which the program is recorded. The program may also be distributed by storing the program in the storage of an external server and transmitting the program from the external server to another computer. The program may also be provided as a program product.
[0035] A part or all of the functions of the information processing device 10 may be realized by a dedicated circuit corresponding to the control unit 11. In other words, a part or all of the functions of the information processing device 10 may be realized by hardware.
[0036] Referring to the flowchart of FIG. 2, a method for estimating an atmosphere of a conversation scene according to an embodiment of the present disclosure is illustrated.
[0037] Step S101: The control unit 11 of the information processing device 10 acquires sound data in a place where a conversation is taking place, preferably during the conversation. Any method can be adopted for acquiring the sound data. For example, the control unit 11 may acquire the sound data by the input unit 13. Also, for example, the control unit 11 may acquire the sound data by receiving sound data collected by an external microphone from the microphone via the communication unit 15.
[0038] The control unit 11 may perform a filter process on the sound data and use the sound data after the filter process in the following process. For example, a band-pass filter may be used in the filter process. This makes it possible to remove sounds that are not related to conversation, noise, etc. In other words, the control unit 11 may obtain a frequency domain limited signal as the sound data.
[0039] Step S102: The control unit 11 calculates a feature amount related to the sound data based on the sound data. For example, the control unit 11 calculates a feature amount related to the sound data in the frequency domain by performing a short-time Fourier transform (STFT) on the sound data. The feature amount includes sound unity. By using sound unity for estimation, the atmosphere of the place where the conversation takes place can be estimated with high accuracy.
[0040] Step S103: The control unit 11 estimates the atmosphere of the place where the conversation takes place based on the feature amount calculated in step S102.
[0041] In this way, according to the conversation atmosphere estimation technique of the present embodiment, a feature amount is calculated from sound data in the conversation, preferably during the conversation, and the atmosphere of the conversation is estimated based on the feature amount. In particular, the feature amount includes sound unity. By using sound unity for estimation, the atmosphere of the conversation is estimated with high accuracy. In this way, according to the conversation atmosphere estimation technique of the present embodiment, the estimation technique of the conversation atmosphere is improved.
[0042] Here, the loudness, pitch, and uniformity of the sound may be associated with RGB values, respectively. In other words, these three feature quantities may be assigned to the RGB values of the three primary colors, and the sound pattern may be expressed by the colors. For example, the control unit 11 may associate the loudness, pitch, and uniformity of the sound with RGB values, respectively, and convert the feature quantities into color scores (Color-Scale Values). The color scores may be appropriately determined by the RGB values. In this embodiment, the loudness, pitch, and uniformity of the sound may determine, for example, an impression of a conversation by a color-scored value. Examples of the impression of a conversation include smooth-awkward (affinity), lively-lively (arousal), serious-unserious (atmosphere), open-secret (openness), fun-boring (evaluation), one-sided-interactive (speaker relationship), calm-flustered (context), etc. Each of these impressions can be scored using RGB values. In this embodiment, the color-scored values based on the loudness, pitch, and unity of the sound are classified into seven types, but this is not limiting, and each of these index values may be evaluated in any number of types.
[0043] 3 and 4 show a correspondence table between RGB values and color scores (hereinafter, also referred to as a color feature conversion table). The feature is converted into a color by referring to the color feature conversion table, and the feature of the sound is expressed by a color. Here, in the color feature conversion table, the values of the R, G, and B columns are determined based on the values of the three feature of the sound: loudness, pitch, and unity. The values of the Color and Color-scale columns are determined based on the values of the R, G, and B columns. Therefore, by referring to the color feature conversion table, the three feature are converted into the values of the Color and Color-scale columns. The control unit 11 may also display and output a histogram as a frequency distribution of the color score in a predetermined period of time by the output unit 14. The predetermined period is appropriately determined as 5 minutes, etc. Note that the predetermined period may be less than 5 minutes, or may be greater than 5 minutes. FIG. 5 shows an example of a histogram of the color score. In FIG. 5, the horizontal axis is the color score, and the vertical axis is the frequency. The color of each bar in the histogram is displayed by the color of the corresponding color score. By displaying and outputting the histogram in this way, the distribution of sound features during a given period of conversation can be visually shown. Also, since loudness of sound is indicated by R (red), high pitch by G (green), and uniformity of sound by B (blue), the colors of the histogram allow the user to intuitively understand the distribution of sound features and the atmosphere of the place where the conversation is taking place, which is indicated by the distribution.
[0044] The histogram of the color scores may be used to estimate the atmosphere of the place where the conversation takes place. For example, in the estimation step of the atmosphere estimation technique of the place where the conversation takes place according to the present embodiment, the control unit 11 may estimate the atmosphere of the place where the conversation takes place based on pattern matching of the histogram. In the pattern matching process, the histogram may be converted into a feature vector, and the feature vector may be pattern-classified to perform the estimation process. In such estimation process, a learning model created by machine learning using a machine learning algorithm may be used. The learning model may be, for example, a machine learning model built based on a decision tree. The machine learning model built based on a decision tree is, for example, Light GBM, XGBoost, etc., but is not limited to these. Alternatively, the learning model may be a model generated based on a machine learning algorithm such as Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), or other deep learning.
[0045] In the present embodiment, the feature amount includes three items: sound unity, loudness, and high / low, but is not limited to these. The feature amount may include, for example, turn-taking. Turn-taking is an index showing the number of times a speaker changes to another speaker. For example, the larger the turn-taking, the more likely it is that a conversation is being carried out by multiple speakers. Note that in the present embodiment, turn-taking also includes a case where one speaker responds to what the other speaker said with an emotional sound such as laughter. Turn-taking is considered to have a certain degree of correlation with the atmosphere of the place where the conversation takes place. Therefore, by including turn-taking in the feature amount, the estimation accuracy of the atmosphere of the place where the conversation takes place can be improved.
[0046] The feature may also include, for example, the presence or absence of a silent period. A silent period in a conversation is an index that indicates the length of a period during which neither speaker speaks. The presence or absence of a silent period is considered to have a certain degree of correlation with the atmosphere of the place where the conversation takes place. Thus, by including the presence or absence of a silent period in the feature and detecting the silent period, it is possible to improve the accuracy of estimating the atmosphere of the place where the conversation takes place.
[0047] Here, in the atmosphere estimation technique of the conversation place according to the present embodiment, an example of estimating the atmosphere of the conversation place based on the feature amount has been shown, but the present invention is not limited to this. For example, the control unit 11 may acquire sound data by a sound sensor, and further acquire the contents of the sound data by voice recognition. The control unit 11 may estimate the atmosphere of the conversation place using the result of the voice recognition. In other words, the control unit 11 may include a step of acquiring the contents of the voice by voice recognition, and may estimate the atmosphere of the conversation place based on the feature amount and the result of the voice recognition. In this way, it is possible to estimate the atmosphere of the conversation place taking into account the contents of the conversation.
[0048] Furthermore, a video of the conversation situation may be used to estimate the atmosphere. Specifically, in this case, the control unit 11 acquires a video of at least one of the multiple people having a conversation. Then, the atmosphere of the place where the conversation takes place may be estimated based on the feature amount and the video. In this way, the atmosphere of the place where the conversation takes place can be estimated taking into account video information such as the facial expressions of the people having the conversation.
[0049] Also, for example, a microphone array may be used as a sound sensor to identify the location of a sound source from sound data, and the atmosphere of the place where the conversation takes place may be estimated based on the location of the sound source and the feature amount. For example, when the sound data includes the voice of a conversation, the control unit 11 identifies the location where the conversation took place. Also, the control unit 11 identifies the speaker based on the location where the conversation took place and the locations of the conversation participants known in advance. This allows the control unit 11 to identify the speech situation of each speaker. The control unit 11 may estimate the atmosphere of the place where the conversation takes place based on the speech situation and the feature amount. In this way, for example, it is also possible to estimate the "excitement" of a meeting such as a small-group conference.
[0050] The control unit 11 may estimate the atmosphere of the place where the conversation takes place based only on the feature amount. In other words, the control unit 11 may estimate the atmosphere of the place where the conversation takes place without using other elements such as the content of the voice, the video, and the speech state of the conversation participants. In this case, the information processing device 10 can estimate the atmosphere of the place where the conversation takes place with a simpler configuration. In addition, since voice recognition is not performed, problems regarding privacy and personal information do not occur. In addition, when collecting sound, the quality of pronunciation does not become an issue.
[0051] Although the present disclosure has been described based on the drawings and embodiments, it should be noted that those skilled in the art can easily make various modifications and corrections based on the present disclosure. Therefore, it should be noted that these modifications and corrections are included in the scope of the present disclosure. For example, the functions included in each means or step can be rearranged so as not to be logically inconsistent, and multiple means or steps can be combined or divided into one. [Explanation of symbols]
[0052] 10. Information processing device 11 Control section 12 Storage section 13 Input section 14 Output section 15 Communications Department
Claims
1. An atmosphere estimation method executed by an information processing device, comprising: acquiring sound data of a place where a conversation takes place; calculating a feature amount related to the sound data based on the sound data; A step of estimating an atmosphere of a place where the conversation takes place based on the feature amount; Including, The method for estimating atmosphere, wherein the feature amount includes unity of sound.
2. The atmosphere estimation method according to claim 1, The sound data includes conversational voices, sound effects, environmental sounds, and emotional sounds.
3. The atmosphere estimation method according to claim 1, The feature amount includes loudness and pitch of a sound.
4. The atmosphere estimation method according to claim 2, further comprising: converting the features into color scores by associating the loudness, pitch, and uniformity of the sound with RGB values; outputting a histogram of the color scores for a given period of the conversation; The atmosphere estimation method includes:
5. The atmosphere estimation method according to claim 4, The atmosphere estimation method, wherein the estimating step estimates the atmosphere of the place where the conversation takes place based on pattern matching of the histogram.
6. The atmosphere estimation method according to claim 1, The method for estimating an atmosphere, wherein the feature amount includes turn-taking.
7. The atmosphere estimation method according to claim 1, The atmosphere estimation method, wherein the feature amount includes the presence or absence of a silent period.
8. The atmosphere estimation method according to any one of claims 1 to 7, further comprising: acquiring the content of the sound data by voice recognition; In the estimating step, the atmosphere is estimated based on the feature amount and a result of the speech recognition.
9. The atmosphere estimation method according to any one of claims 1 to 7, further comprising: acquiring an image of at least one of the plurality of people having a conversation or a position of at least one of the plurality of people; In the estimating step, the atmosphere is estimated based on the feature amount and the video or the position.
10. The atmosphere estimation method according to any one of claims 1 to 7, further comprising: acquiring, using a microphone array, at least one location of the people engaged in the conversation; The atmosphere estimation method estimates the atmosphere based on the feature amount and the position.
11. An information processing device that includes a control unit and estimates an atmosphere, The control unit is Acquire sound data from the place where the conversation is taking place, Calculating a feature amount related to the sound data based on the sound data; An information processing device that estimates an atmosphere of a place where the conversation takes place based on the feature amount, The information processing device, wherein the feature includes unity of sound.
12. A program for estimating an atmosphere executed by an information processing device, the program comprising: Acquiring sound data of a place where a conversation takes place; Calculating a feature amount related to the sound data based on the sound data; Estimating the atmosphere of the place where the conversation takes place based on the feature amount; and Run the command, The feature includes unity of sound.
Citation Information
Patent Citations
Satisfaction calculation device, satisfaction calculation method and program
JP2011237957A
