Information processing equipment, methods, programs and systems

Abstract

To provide a system capable of quantifying excitement in a prescribed space while maintaining security.SOLUTION: A system includes means installed in a prescribed space and collecting sounds generated in the space; means installed in the space, classifying types of the sounds collected by the means for collecting the sounds, and determining a state relating to excitement and a degree of the state on the basis of the classified types and volumes; means for aggregating the state and the degree and calculating time transition of the excitement in the space; and means for presenting information based on the time transition of the excitement in the space to a user.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, method, program, and system.

Background Art

[0002] As an index as to whether a meaningful meeting has been held or not, the fact that the meeting has become lively or that the meeting has been lively is an important factor. In recent years, techniques for measuring the degree of liveliness of a meeting have been proposed (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, for example, a meeting analysis apparatus that calculates the degree of liveliness of a meeting from meeting data including voice and / or video is described. In the meeting analysis apparatus, feature amounts as a time series along the progress of the meeting are extracted from the meeting data, and based on the feature amounts, the degree of liveliness as a time series along the progress of the meeting is calculated as an absolute evaluation value.

[0005] However, in Patent Document 1, the meeting data is transmitted to the meeting analysis apparatus, and the information spoken at the meeting is stored in the meeting analysis apparatus. Therefore, there is not zero possibility that information that is desired to be retained only within the meeting leaks to the outside.

[0006] An object of the present disclosure is to quantify the liveliness in a predetermined space while maintaining security.

Means for Solving the Problems

[0007] The system comprises: means installed in a predetermined space to collect sounds generated in that space; means installed in the space to classify the types of sounds collected by the sound-collecting means, and to determine the state of excitement and the degree of that state based on the classified type and volume; means to aggregate the state and degree and calculate the time progression of excitement in the space; and means to present information based on the time progression of excitement in the space to the user. [Effects of the Invention]

[0008] According to this disclosure, it is possible to quantify the level of excitement in a given space while maintaining security. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing the overall configuration of System 1. [Figure 2] This is a block diagram showing an example of the functional configuration of the terminal device 10. [Figure 3] This block shows a functional configuration example for server 20. [Figure 4] This is a block diagram showing an example of the functional configuration of the analysis device 30. [Figure 5] This diagram shows the data structure of a table. [Figure 6] This diagram shows the data structure of a table. [Figure 7] This diagram shows the data structure of a table. [Figure 8] This diagram shows the data structure of a table. [Figure 9] This figure shows an example of the processing flow in System 1. [Figure 10] This figure shows an example of the processing flow in System 1. [Figure 11] This figure shows an example of the screen in this disclosure. [Figure 12] This figure shows an example of the screen in this disclosure. [Figure 13] This figure shows an example of the screen in this disclosure. [Figure 14]This is a diagram showing an example screen of the present disclosure. [Figure 15] This is a block diagram showing the basic hardware configuration of the computer 90.

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In all the drawings for describing the embodiments, the same reference numerals are given to common components, and repeated descriptions are omitted. Note that the following embodiments do not unduly limit the content of the present disclosure described in the claims. Also, not all the components shown in the embodiments are essential components of the present disclosure. Also, each figure is a schematic diagram and is not necessarily drawn precisely.

[0011] Also, in the following description, a "processor" is one or more processors. At least one processor is typically a microprocessor such as a CPU (Central Processing Unit), but may also be another type of processor such as a GPU (Graphics Processing Unit). At least one processor may be single-core or multi-core.

[0012] Also, at least one processor may be a processor in a broad sense such as a hardware circuit (for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)) that performs part or all of the processing.

[0013] Also, in the following description, an expression such as "xxx table" may be used to describe information from which an output is obtained for an input, but this information may be data of any structure or a learning model such as a neural network that generates an output for an input. Therefore, "xxx table" can be referred to as "xxx information".

[0014] In the following description, the configuration of each table is an example. One table may be divided into two or more tables, or all or part of two or more tables may be combined into one table.

[0015] In the following description, the "program" may be used as the subject to describe the process. However, since the program performs the defined process by being executed by a processor while appropriately using a storage unit and / or an interface unit, etc., the subject of the process may be the processor (or a device such as a controller having the processor).

[0016] The program may be installed in a device such as a computer, or may be in, for example, a program distribution server or a computer-readable (e.g., non-transitory) recording medium. Also, in the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0017] In the following description, an identification number is used as the identification information for various objects, but other types of identification information (e.g., an identifier including letters or symbols) may be adopted.

[0018] In the following description, when describing elements of the same type without distinction, reference signs (or common signs among the reference signs) are used, and when describing elements of the same type while distinguishing them, the identification numbers (or reference signs) of the elements may be used.

[0019] In the following description, the control lines and information lines show those considered necessary for the description, and not necessarily all control lines and information lines on the product are shown. All components may be interconnected.

[0020] Each information processing device consists of a computer equipped with an arithmetic unit and a memory device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by said hardware configuration will be described later. For each of the terminal device 10 and the server 20, explanations that overlap with the basic hardware configuration and basic functional configuration of the computer described later will be omitted.

[0021] <Overview> The system according to this embodiment uses edge AI to determine the type of sound, and based on the determined type and volume, determines a numerical value related to the level of excitement. Using the determined numerical value, it estimates the level of excitement in the space.

[0022] The following explanation uses the example of estimating the level of excitement in a conference room. However, the space in which this embodiment is implemented is not limited to a closed space such as a conference room, but may also be an open space within an office. An open space within an office may be, for example, an area within the office such as a booth, lounge, or cafeteria. Hereafter, an open space within an office will be referred to as an area.

[0023] <1 System Configuration Diagram> Figure 1 is a block diagram showing an example of the overall configuration of System 1. System 1 shown in Figure 1 includes, for example, a terminal device 10, a server 20, an analysis device 30, a microphone 31, a camera 32, and a speaker 33. The terminal device 10, server 20, analysis device 30, and speaker 33 are connected to each other via, for example, a network 40.

[0024] Figure 1 shows an example where System 1 includes two terminal devices 10, but the number of terminal devices 10 included in System 1 is not limited to two. System 1 may include fewer than three terminal devices 10, or three or more.

[0025] In this embodiment, a collection of multiple devices may be treated as a single server. The method of allocating the multiple functions required to implement the server 20 according to this embodiment to one or more hardware can be appropriately determined in view of the processing capacity of each hardware and / or the specifications required for the server 20.

[0026] The terminal device 10 shown in Figure 1 is, for example, an information processing device used by a user to grasp the elevation of a space. The terminal device 10 may be, for example, a mobile device such as a smartphone or tablet, or a stationary PC (Personal Computer) or laptop PC. It may also be a wearable device such as an HMD (Head Mount Display) or a smartwatch. The terminal device 10 may, for example, be owned and carried by an individual, or it may be shared among employees and fixed to the wall of a conference room.

[0027] The terminal device 10 includes a communication interface 12, an input device 13, an output device 14, a memory 15, storage 16, and a processor 19.

[0028] The communication interface 12 is an interface for inputting and outputting signals so that the terminal device 10 can communicate with a device within system 1, such as a server 20.

[0029] The input device 13 is a device for receiving input operations from the user (for example, a touch panel, touchpad, pointing device such as a mouse, keyboard, etc.).

[0030] The output device 14 is a device (such as a display or speaker) for presenting information to the user.

[0031] Memory 15 is for temporarily storing programs and data processed by programs, etc., and is a volatile memory such as DRAM (Dynamic Random Access Memory).

[0032] Storage 16 is for storing data, and can be, for example, flash memory or an HDD (Hard Disk Drive).

[0033] The processor 19 is hardware for executing the instruction set described in the program, and consists of an arithmetic unit, registers, peripheral circuits, etc.

[0034] Server 20 is an information processing device that analyzes the rise based on information acquired from, for example, the analysis device 30. Server 20 is implemented by, for example, a computer connected to the network 40. As shown in Figure 1, Server 20 includes a communication IF 22, an input / output IF 23, a memory 25, a storage 26, and a processor 29.

[0035] Communication IF22 is an interface for inputting and outputting signals so that the server 20 can communicate with devices within system 1, such as terminal device 10.

[0036] Input / Output IF23 functions as an interface to an input device for receiving input operations from the user, and to an output device for presenting information to the user.

[0037] Memory 25 is for temporarily storing programs and data processed by programs, etc., and is a volatile memory such as DRAM.

[0038] Storage 26 is for storing data, and can be, for example, flash memory or an HDD.

[0039] The processor 29 is hardware for executing the instruction set described in the program, and consists of an arithmetic unit, registers, peripheral circuits, etc.

[0040] The analysis device 30 is an information processing device that analyzes audio and images from the perspective of excitement, for example.

[0041] One analysis device 30 may cover one space or multiple spaces. Specifically, one analysis device 30 may analyze audio and images obtained in one conference room or multiple conference rooms. The analysis device 30 is installed in a location where information analysis is possible. For example, the analysis device 30 is placed in a corner of the conference room. More specifically, one analysis device 30 may analyze audio and images obtained in one area or multiple areas. The analysis device 30 is installed in a location where information analysis is possible. For example, the analysis device 30 is placed in a corner of the area.

[0042] The analysis device 30 receives information transmitted from the microphone 31 and the imaging device 32, analyzes the received information, and transmits the analyzed information to the server 20. The analysis device 30 also transmits information acquired from the server 20 to the microphone 31 and the imaging device 32. The information acquired from the server 20 includes, for example, information for updating the settings of the microphone 31 or the imaging device 32.

[0043] Microphone 31 is installed to collect sound within a space. For example, microphone 31 is installed to collect sound from the entire conference room. Specifically, microphone 31 is, for example, a disc-shaped sound-collecting microphone installed on a table in the center of the conference room. Alternatively, microphone 31 is installed to collect sound from the entire area. Specifically, microphone 31 is, for example, a disc-shaped sound-collecting microphone installed on the ceiling in the center of the area. Microphone 31 outputs the acquired audio signal to the analysis device 30. If one microphone 31 cannot capture sound from the entire space, multiple microphones 31 are installed. In this case, microphone 31 may be, for example, a microphone with a pre-set directivity.

[0044] The imaging device 32 is installed to capture the situation within the space. For example, the imaging device 32 is installed to capture the situation of the entire conference room. Specifically, the imaging device 32 is, for example, an omnidirectional camera installed on a table in the center of the conference room. Alternatively, for example, the imaging device 32 is installed to capture the situation of the entire area. Specifically, the imaging device 32 is, for example, an omnidirectional camera installed on the ceiling in the center of the area. The imaging device 32 outputs the acquired image signal to the analysis device 30. If one imaging device 32 cannot capture the situation of the entire space, multiple devices are installed. In this case, the imaging device 32 may be, for example, a camera with a preset shooting direction.

[0045] Speaker 33 is a speaker installed in a space to play music within that space. For example, speaker 33 is installed to deliver music throughout the entire conference room. Specifically, speaker 33 is, for example, an omnidirectional speaker installed on a table in the center of the conference room. Alternatively, speaker 33 is installed to deliver music to an entire area. Specifically, speaker 33 is, for example, an omnidirectional speaker installed on the ceiling in the center of the area. Speaker 33 converts the audio signal provided by server 20 into music and outputs the music into the space. If one speaker 33 is insufficient to deliver music to the entire space, multiple speakers 33 are installed.

[0046] <2 Functional Configuration of Terminal Devices> Figure 2 is a block diagram showing an example of the functional configuration of the terminal device 10 shown in Figure 1. The terminal device 10 shown in Figure 2 can be implemented as a mobile terminal, a PC, or a wearable terminal. As shown in Figure 2, the terminal device 10 includes a communication unit 120, an input device 13, an output device 14, an audio processing unit 17, a microphone 171, a speaker 172, a camera 161, a location information sensor 150, a storage unit 180, and a control unit 190. Each block included in the terminal device 10 is electrically connected, for example, by a bus.

[0047] The communication unit 120 performs modulation and demodulation processing for the terminal device 10 to communicate with other devices. The communication unit 120 performs transmission processing on the signal generated by the control unit 190 and transmits it to an external source (for example, the server 20). The communication unit 120 performs reception processing on the signal received from the external source and outputs it to the control unit 190.

[0048] The input device 13 is a device for a user operating the terminal device 10 to input instructions or information. The input device 13 can be implemented, for example, by a touch-sensitive device 131 on which instructions are input by touching the operating surface. If the terminal device 10 is a PC, the input device 13 may be implemented by a reader, keyboard, mouse, etc. The input device 13 converts the instructions input by the user into electrical signals and outputs the electrical signals to the control unit 190. The input device 13 may also include, for example, a receiving port that accepts electrical signals input from an external input device.

[0049] The output device 14 is a device for presenting information to the user operating the terminal device 10. The output device 14 is implemented, for example, by a display 141. The display 141 displays data according to the control of the control unit 190. The display 141 is implemented, for example, by an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0050] The audio processing unit 17 performs, for example, digital-to-analog conversion processing of the audio signal. The audio processing unit 17 converts the signal received from the microphone 171 into a digital signal and provides the converted signal to the control unit 190. The audio processing unit 17 also provides the audio signal to the speaker 172. The audio processing unit 17 is implemented, for example, by an audio processing processor. The microphone 171 receives an audio input and provides the audio signal corresponding to that audio input to the audio processing unit 17. The speaker 172 converts the audio signal received from the audio processing unit 17 into audio and outputs the audio to the outside of the terminal device 10.

[0051] Camera 161 is a device that receives light using a photodetector and outputs it as an image signal.

[0052] The location information sensor 150 is a sensor that detects the position of the terminal device 10, and is, for example, a GPS (Global Positioning System) module. A GPS module is a receiving device used in a satellite positioning system. In a satellite positioning system, signals are received from at least three or four satellites, and the current position of the terminal device 10, which is equipped with a GPS module, is detected based on the received signals. The location information sensor 150 may also detect the current position of the terminal device 10 from the position of the wireless base station to which the terminal device 10 is connected.

[0053] The storage unit 180 is implemented, for example, by memory 15 and storage 16, and stores data and programs used by the terminal device 10.

[0054] The control unit 190 is realized when the processor 19 reads a program stored in the memory unit 180 and executes instructions contained in the program. The control unit 190 controls the operation of the terminal device 10. By operating according to the program, the control unit 190 performs the functions of an operation reception unit 191, a transmission / reception unit 192, and a presentation control unit 193.

[0055] The operation reception unit 191 performs processing to receive instructions or information input from the input device 13. Specifically, for example, the operation reception unit 191 receives information based on instructions input from a touch-sensitive device 131, etc. Instructions input via the touch-sensitive device 131, etc. are, for example, editing instructions from the user.

[0056] Furthermore, the operation reception unit 191 receives voice instructions input from the microphone 171. Specifically, for example, the operation reception unit 191 receives voice signals input from the microphone 171 and converted into digital signals by the voice processing unit 17. The operation reception unit 191 obtains instructions from the user by, for example, analyzing the received voice signals and extracting predetermined nouns.

[0057] The transmitting / receiving unit 192 performs processing to enable the terminal device 10 to send and receive data with external devices such as the server 20 in accordance with a communication protocol. Specifically, for example, the transmitting / receiving unit 192 sends editing instructions entered by the user to the server 20. The transmitting / receiving unit 192 also receives information about the user from the server 20.

[0058] The presentation control unit 193 controls the output device 14 in order to present information provided by the server 20 to the user. Specifically, for example, the presentation control unit 193 displays the information transmitted from the server 20 on the display 141. The presentation control unit 193 also outputs the information transmitted from the server 20 through the speaker 172.

[0059] <3. Functional Configuration of the Server> Figure 3 is a block diagram showing an example of the functional configuration of server 20. As shown in Figure 3, server 20 functions as a communication unit 201, a storage unit 202, and a control unit 203.

[0060] The communications unit 201 performs processing to enable the server 20 to communicate with external devices.

[0061] The storage unit 202 includes, for example, a voice analysis log database (DB) 2021, an image analysis log database (DB) 2022, etc. The tables stored in the storage unit 202 are not limited to these.

[0062] The voice analysis log DB2021 aggregates the contents of the voice analysis log DB3021 from each analysis device 30. Further details will be described later.

[0063] Image analysis log DB2022 aggregates the contents of image analysis log DB3022 for each analysis device 30. Further details will be provided later.

[0064] The control unit 203 is realized when the processor 29 reads a program stored in the memory unit 202 and executes instructions contained in the program. By operating according to the program, the control unit 203 performs the functions shown as the receive control module 2031, the transmit control module 2032, the analysis control module 2033, and the presentation control module 2034.

[0065] The receive control module 2031 controls the process by which the server 20 receives signals from external devices according to a communication protocol.

[0066] The transmission control module 2032 controls the process by which the server 20 transmits signals to external devices according to a communication protocol.

[0067] The analysis control module 2033 controls the process of analyzing information received from the analysis device 30. Specifically, the analysis control module 2033 controls the process of analyzing information related to spatial elevation, for example.

[0068] The presentation control module 2034 controls the process of presenting information to the user.

[0069] <4 Functional Configuration of the Analysis Device> Figure 4 is a block diagram showing an example of the functional configuration of the analysis device 30. As shown in Figure 4, the analysis device 30 functions as a communication unit 301, a storage unit 302, and a control unit 303.

[0070] The communication unit 301 performs processing to enable the analysis device 30 to communicate with an external device.

[0071] The memory unit 302 includes, for example, a speech analysis log database (DB) 3021, an image analysis log database (DB) 3022, a first trained model 3023, a second trained model 3024, and the like. The tables stored in the memory unit 302 are not limited to these.

[0072] The voice analysis log DB3021 is a database for storing information about the sounds in the space collected by microphone 31. Further details will be provided later.

[0073] The image analysis log DB3022 is a database for storing information about images of the spatial situation captured by the imaging device 32. Further details will be described later.

[0074] The first pre-trained model 3023 is a model that, upon receiving audio input, learns to output information for evaluating the level of excitement based on the input audio. For example, the first pre-trained model 3023 learns to distinguish the type of sound in a given space upon receiving audio input. Examples of sound types include conversation, cheers, laughter, snorting, shouting, silence, screams, crying, groans, sneezes, humming, typing, etc.

[0075] In this case, the training data would, for example, use audio as input data, and the correct output data would be the judgment about the type of sound contained in this input data.

[0076] The second pre-trained model 3024 is a model that, when an image is input, learns to output information for evaluating the level of excitement based on the input image. For example, when an image is input, the second pre-trained model 3024 learns to determine the number of people in a space, their facial expressions, etc. Facial expressions include, for example, smiling, neutral, crying, etc. Facial expressions also include gestures such as nodding, shaking head, frowning, looking down, etc.

[0077] In this case, the training data would be, for example, human images as input data, and the judgments made about the facial expressions contained in this input data would be used as the correct output data.

[0078] The first pre-trained model 3023 and the second pre-trained model 3024 are models generated by having a machine learning model perform machine learning according to a model training program. The first pre-trained model 3023 and the second pre-trained model 3024 are, for example, parameterized composite functions composed of multiple functions that perform predetermined inference based on input data. A parameterized composite function is defined by a combination of multiple tunable functions and parameters. The pre-trained models according to this embodiment may be any parameterized composite function that satisfies the above requirements.

[0079] For example, when the first trained model 3023 and the second trained model 3024 are generated using a feedforward multilayer network, the parameterized composite function is defined as a combination of, for example, linear relationships between layers using weight matrices, nonlinear (or linear) relationships using activation functions in each layer, and biases. The weight matrices and biases are called the parameters of the multilayer network. The parameterized composite function changes its form as a function depending on how the parameters are chosen. In a multilayer network, by appropriately setting the constituent parameters, it is possible to define a function that can output desirable results from the output layer.

[0080] As the multilayer network according to this embodiment, for example, a deep neural network (DNN), which is a multilayer neural network targeted by deep learning, may be used. As the DNN, for example, a convolutional neural network (CNN) that targets images may be used.

[0081] The control unit 303 is realized when the processor in the analysis device 30 reads a program stored in the memory unit 302 and executes the instructions contained in the program. By operating according to the program, the control unit 303 performs the functions indicated as the receiving control module 3031, the transmitting control module 3032, the analysis control module 2033, and the presentation control module 3034.

[0082] The reception control module 3031 controls the process by which the analysis device 30 receives signals from an external device according to a communication protocol.

[0083] The transmission control module 3032 controls the process by which the analysis device 30 transmits signals to an external device according to a communication protocol.

[0084] The analysis control module 3033 controls the process of analyzing the information received from the microphone 31 and the imaging device 32. Specifically, the analysis control module 3033 analyzes the audio and images from the perspective of excitement. More specifically, for example, the analysis control module 3033 analyzes the audio using the first trained model 3023. Also, for example, the analysis control module 3033 analyzes the images using the second trained model 3024.

[0085] The presentation control module 3034 controls the process of presenting information to the user.

[0086] <5 Data Structure> This section describes the data structure of a table. Note that the data structure described is just one example, and does not exclude data not listed. Furthermore, even data within the same table may be stored in separate memory areas.

[0087] Figure 5 shows the data structure of the voice analysis log DB 3021. As shown in Figure 5, each record in the voice analysis log DB 3021 includes, for example, the items "Location", "Microphone ID", "Time", "Voice", "Volume", "State", and "Value". When voice is output by the first trained model 3023, a new record is created in the voice analysis log DB 3021, and the information is stored in the corresponding items. The voice analysis log DB 3021 of each analysis device 30 records information about the space covered by that analysis device 30.

[0088] The item "Location" indicates the space covered by the analysis device 30.

[0089] The item "Microphone ID" indicates identification information for identifying the microphone 31 installed in the space.

[0090] The item "Time" indicates the time when audio was collected by microphone 31.

[0091] The item "Speech" indicates the type of speech. For example, the item "Speech" indicates the type of speech detected by the first trained model 3023.

[0092] The "Volume" item indicates the volume of the "Sound" item. The "Volume" item can be expressed in, for example, three levels. If the intensity of the sound acquired by the microphone 31 exceeds a set threshold, the server 20 sets a value corresponding to the exceeded threshold. Note that the "Volume" item is not limited to three levels.

[0093] The "State" category indicates the state related to excitement. This category includes, for example, "Excitement," "Downturn," and "Neutral." "Excitement" includes positive sounds such as laughter. "Downturn" includes negative sounds such as shouting. "Neutral" means a state that is neither "Excitement" nor "Downturn." The states related to excitement are not limited to these three types. For example, there could be just two types: "Excitement" and "Downturn."

[0094] The "Value" item indicates the degree of excitement or excitement. The "Value" item can be expressed, for example, on a four-point scale. However, the "Value" item is not limited to four points.

[0095] Figure 7 shows the data structure of the voice analysis log DB2021. As shown in Figure 7, each record in the voice analysis log DB2021 includes, for example, the fields "Location", "Microphone ID", "Time", "Voice", "Volume", "Status", and "Value". As will be described later, the voice analysis log DB2021 aggregates the contents of the voice analysis log DB3021 of each analysis device 30.

[0096] Since each item in the voice analysis log DB2021 represents the same thing as each item in the voice analysis log DB3021, the explanation of each item in the voice analysis log DB2021 is omitted.

[0097] Figure 6 shows the data structure of the image analysis log DB 3022. As shown in Figure 6, each record in the image analysis log DB 3022 includes, for example, the items "Location", "Camera ID", "Time", "Facial Expression", "Number of People", "State", and "Value". When a facial expression is output by the second trained model 3024, a new record is created in the image analysis log DB 3022, and the information is stored in the corresponding items. The image analysis log DB 3022 of each analysis device 30 records information about the space covered by that analysis device 30.

[0098] The item "Location" indicates the space covered by the analysis device 30.

[0099] The item "Camera ID" indicates identification information for identifying the imaging device 32 installed in the space.

[0100] The item "Time" indicates the time during which the space was photographed by the imaging device 32.

[0101] The item "Facial Expression" indicates the type of facial expression. For example, the item "Facial Expression" indicates the type of facial expression detected by the second trained model 3024. The item "Facial Expression" may also include information related to gestures such as nodding.

[0102] The item "Number of people" indicates the number of people who made facial expressions. For example, the item "Number of people" indicates the number of people who made facial expressions detected by the second trained model 3024.

[0103] The "State" item indicates the state related to excitement. The "State" item includes, for example, "Excitement," "Downturn," and "Neutral." "Excitement" includes positive expressions such as smiles. "Downturn" includes negative expressions such as frowning. "Neutral" means a state that is neither "Excitement" nor "Downturn." The states related to excitement are not limited to three types. For example, there could be two types: "Excitement" and "Downturn."

[0104] The "Value" item indicates the degree of excitement or excitement. The "Value" item can be expressed, for example, on a four-point scale. However, the "Value" item is not limited to four points.

[0105] Figure 8 shows the data structure of the image analysis log DB2022. As shown in Figure 8, each record in the image analysis log DB2022 includes, for example, the items "Location", "Camera ID", "Time", "Facial Expression", "Number of People", "State", and "Value". As will be described later, the image analysis log DB2022 aggregates the contents of the image analysis log DB3022 of each analysis device 30.

[0106] Since each item in Image Analysis Log DB2022 represents the same thing as each item in Image Analysis Log DB3022, the explanation of each item in Image Analysis Log DB2022 is omitted.

[0107] <6 operations> An example of the processing flow in System 1 is described below.

[0108] Figure 9 is a flowchart illustrating an example of how the analysis device 30 operates when analyzing audio and images.

[0109] In step S701, the analysis device 30 receives audio and images. Specifically, for example, the receiving control module 3031 receives audio data of the space collected by the microphone 31 from the microphone 31, and image data of the space captured by the imaging device 32 from the imaging device 32.

[0110] In step S702, the control unit 303 analyzes the audio data and the image data.

[0111] Specifically, the analysis control module 3033 periodically inputs received audio data, i.e., audio data collected by the microphone 31, to the first trained model 3023. The first trained model 3023 outputs information for evaluating the level of excitement based on the input audio. Specifically, the first trained model 3023 outputs the type of audio when audio is input. Specifically, the first trained model 3023 outputs laughter from audio recorded in the second conference room at 10:30:10 on April 1, 2024.

[0112] The input cycle is based on time, for example. This time can be set based on, for example, the approximate time it takes for a person to perform a single action. Specifically, it can be set based on the time it takes for a person to perform a single action, such as laughing out loud. For example, the analysis control module 3033 inputs audio data to the first trained model 3023 every 2 to 5 seconds.

[0113] Specifically, the analysis control module 3033 periodically inputs received image data, i.e., image data captured by the imaging device 32, to the second trained model 3024. The second trained model 3024 outputs information for evaluating excitement based on the input images. Specifically, the second trained model 3024 outputs the type of facial expression when an image is input. The second trained model 3024 also outputs the number of people with that type of facial expression when an image is input. For example, the second trained model 3024 outputs two smiling faces from an image taken in the second conference room at 10:30:10 on April 1, 2024.

[0114] The input cycle is based on time, for example. This time can be set based on, for example, the approximate time it takes for a person to perform a single action. Specifically, it can be set based on the time it takes for a person to perform a single nodding motion, for example. For example, the analysis control module 3033 inputs image data to the second trained model 3024 every 2 to 5 seconds. The image data input to the second trained model 3024 may be a single still image or multiple still images from multiple frames.

[0115] Specifically, the analysis control module 3033 evaluates the level of excitement in the space based on the outputted information. More specifically, the analysis control module 3033 determines the state and degree of excitement based on the type and volume of the sound. For example, the analysis control module 3033 evaluates laughter at volume 2 as a state of "excitement" with a value of "3". Furthermore, more specifically, the analysis control module 3033 determines the state and degree of excitement based on facial expressions and the number of people making those expressions. For example, the analysis control module 3033 evaluates two people frowning as a state of "decline" with a value of "2".

[0116] In this embodiment, for example, the type of sound and the state related to excitement are pre-associated. Specifically, for example, positive types of sounds such as cheers, laughter, and belly laughs are associated with the state of excitement as "excitement." Also, for example, negative types of sounds such as shouting, silence, screaming, crying, groaning, and humming are associated with the state of excitement as "decline." Furthermore, for example, commonly occurring types of sounds such as conversation, sneezing, and typing are associated with the state of excitement as "neutral." Note that the relationship between the type of sound and the state related to excitement is not limited to these. Also, for example, volume and the degree of excitement (decline) are pre-associated. Specifically, for example, volume 1 is associated with the degree of excitement (decline) 1, volume 2 is associated with the degree of excitement (decline) 2, and volume 3 is associated with the degree of excitement (decline) 3.

[0117] In this embodiment, for example, facial expressions are pre-associated with states related to excitement. Specifically, for example, positive facial expressions such as smiling and nodding are associated with a state related to excitement of "excitement." Also, for example, negative facial expressions such as crying, frowning, and shaking of the head are associated with a state related to excitement of "decline." Furthermore, for example, commonly occurring facial expressions such as a neutral face are associated with a state related to excitement of "neutral." Note that the relationship between facial expressions and states related to excitement is not limited to these. Also, for example, the number of people is pre-associated with the degree of excitement (decline). Specifically, for example, the number of people 1 is associated with a degree of excitement (decline) 1, the number of people 2 is associated with a degree of excitement (decline) 2, and the number of people 3 is associated with a degree of excitement (decline) 3.

[0118] In step S703, the control unit 303 stores the information related to the analysis in step S702. Specifically, for example, the analysis control module 3033 records information corresponding to the items "microphone ID", "location", "time", "voice", "volume", "state", and "value" in the voice analysis log DB 3021. Also, specifically, for example, the analysis control module 3033 records information corresponding to the items "camera ID", "location", "time", "facial expression", "number of people", "state", and "value" in the image analysis log DB 3022.

[0119] In step S704, the control unit 303 transmits information related to the analysis in step S702. Specifically, for example, the transmission control module 3032 transmits the contents of the audio analysis log DB 3021 and the image analysis log DB 3022 recorded in step S703 to the server 20.

[0120] The transmission control module 3032 continuously transmits the information recorded in step S703 to the server 20. Alternatively, the transmission control module 3032 may transmit at regular intervals based on the volume of untransmitted data. Specifically, for example, the transmission control module 3032 transmits untransmitted data to the server 20 whenever the volume of untransmitted data reaches a predetermined volume. Alternatively, the transmission control module 3032 may transmit at regular intervals based on the time elapsed since the last transmission. Specifically, for example, the transmission control module 3032 transmits to the server 20 every time a predetermined time has elapsed since the most recent transmission.

[0121] Figure 10 is a flowchart illustrating an example of how server 20 operates when analyzing spatial elevation.

[0122] In step S801, the control unit 203 receives information related to spatial elevation from each analysis device 30. Specifically, for example, the receiving control module 2031 receives the contents of the audio analysis log DB3021 and the image analysis log DB3022 transmitted in step S704 from each analysis device 30.

[0123] The control unit 203 records, for example, the contents of the voice analysis log DB 3021 in the voice analysis log DB 2021, and the contents of the image analysis log DB 3022 in the image analysis log DB 2022. In other words, the voice analysis log DB 2021 stores the contents of the voice analysis log DB 3021, and the image analysis log DB 2022 stores the contents of the image analysis log DB 3022.

[0124] In step S802, the control unit 203 analyzes the received information regarding the elevation in the space.

[0125] (Step S802A: Evaluate the ongoing set of 1) Specifically, for example, the analysis control module 2033 evaluates the excitement of a gathering in progress within a space. For example, based on information relating to the excitement of a gathering in space, the analysis control module 2033 calculates an index value indicating the excitement of a gathering, either from a vocal perspective or from a facial expression perspective. Specifically, the larger the "value" for the "state" being "excited," the larger the index value, and the larger the "value" for the "state" being "decreased," the smaller the index value. Alternatively, the analysis control module 2033 may calculate an index value comprehensively from both vocal and facial expression perspectives. Comprehensive calculation is, for example, the sum of the vocal value and the facial expression value. For example, the analysis control module 2033 calculates an index value "4" based on a vocal state "excited" value of "3" and a facial state "excited" value of "1." For example, the analysis control module 2033 calculates an index value "2" based on a vocal state "excited" value of "3" and a facial state "decreased" value of "1." Instead of simply adding them up, it would be better to weight at least one of the values ​​before adding them up.

[0126] Furthermore, the analysis and control module 2033 calculates the temporal trend of the index value of the rise of a group by continuously evaluating the rise of the group.

[0127] (Step S802B: Obtain trends related to the surge) Specifically, for example, the analysis control module 2033 acquires trends related to the level of excitement. For example, the analysis control module 2033 acquires trends related to excitement by comparing the temporal progression of excitement in multiple past gatherings. Specifically, for example, the analysis control module 2033 acquires that a gathering held on Monday from 10 to 11 a.m. in the third meeting room, with the sales manager as the organizer and sales department managers as participants, and sales reports are elements that contribute to the excitement of the gathering, by comparing multiple gatherings in terms of speech. Furthermore, specifically, the analysis control module 2033 acquires that icebreakers and closing talks at the beginning of a gathering are elements that contribute to the excitement of the gathering, by comparing multiple gatherings in terms of speech.

[0128] The groups being compared may be groups in the same time and different spaces, groups in different times and the same space, or groups in different times and different spaces.

[0129] (Step S802C: Obtain and utilize exemplary transitions) Specifically, for example, the analysis and control module 2033 obtains a model trend of the surge. For example, the analysis and control module 2033 obtains a predetermined model trend trajectory by referring to past collections that generally showed high surges. A predetermined model trend trajectory is, for example, a trajectory that generally slopes upward while reaching periodic peaks in the surge.

[0130] Specifically, for example, the analysis and control module 2033 utilizes exemplary progressions of excitement. For instance, the analysis and control module 2033 scores the excitement of an ongoing gathering by comparing its progression, which is evaluated in step S802A, with an exemplary progression. The more closely the progression of the excitement of the ongoing gathering mirrors the exemplary progression, the higher the score calculated for the progression of the excitement of the ongoing gathering. The gathering may also be scored after completion. The scoring may be a numerical evaluation out of 100 points, or an alphabetical evaluation from A to F.

[0131] (Step S802D: Evaluate the business establishment) Specifically, for example, the analysis and control module 2033 evaluates the activity of the entire business site. For example, based on the index values ​​indicating the level of activity of the group in each space calculated in step 802A, the analysis and control module 2033 calculates an index representing the level of activity of the group in all spaces that make up one business site. For example, the index is the sum of the index values ​​in each space. Note that the index may be calculated using a predetermined statistical method, not limited to a sum. The index may also be the sum of the index values ​​that have been weighted according to the size of each space. For example, even if the original index values ​​of two spaces are the same, if the size of one space is twice that of the other space, the index value of one space will be twice the index value of the other space.

[0132] Specifically, for example, the analysis and control module 2033 calculates statistical data regarding the activity of the business establishment. For example, the analysis and control module 2033 calculates the average value of the index over defined time periods. These defined time periods may be, for example, morning, day of the week, week, month, year, etc. Furthermore, for example, the analysis and control module 2033 may also calculate the median, maximum, and minimum values ​​of the index over defined time periods.

[0133] (Verify the effect of step S802E action) Specifically, for example, the analysis and control module 2033 verifies the effect of an action on generating excitement. For example, the analysis and control module 2033 verifies the effect of an action on generating excitement by observing the trend of the excitement index value calculated in step S802A for a certain period of time from the time the action was performed. If the index value rises for a certain period of time from the time the action was performed, the analysis and control module 2033 determines that the action has increased excitement and that the action is effective in generating excitement.

[0134] Actions designed to generate enthusiasm are those that involve human engagement. For example, actions to generate enthusiasm include morning exercises, serving coffee, and project kickoff speeches.

[0135] (Step S802F: Music and volume selection) Specifically, for example, the analysis control module 2033 selects the music and volume to play in a space according to the index value of the excitement level of the gathering in that space, calculated in step 802A. For example, in spaces where the current index value is low, the analysis control module 2033 selects an up-tempo song and a high volume to encourage excitement. Also, for example, in spaces where the current index value is high, the analysis control module 2033 lowers the volume to maintain excitement.

[0136] The songs may, for example, be downloaded from the internet each time a selection is made, or they may be stored in the memory unit 202.

[0137] In step S803, the control unit 203 provides feedback to the space based on the analysis in step S802.

[0138] (Step S803A Displayed on the terminal in the space) Specifically, for example, the transmission control module 3032 transmits the analysis results regarding the space to the terminal device 10. For example, the transmission control module 3032 transmits to the terminal device 10 installed in the space the trend of the index value of the crowd's excitement in space, calculated in step S802A, and the exemplary trend of the excitement and the score for the excitement, obtained in step S802C. The transmission control module 3032 may transmit continuously in real time while the crowd is in progress, or it may transmit once when the crowd is completed.

[0139] (Step S803B: Displayed on the terminal at the business office) Specifically, for example, the transmission control module 3032 transmits the analysis results regarding the business establishment to the terminal device 10. For example, the transmission control module 3032 transmits the index representing the overall activity of one business establishment, calculated in step S802D, to the terminal device 10 installed at that business establishment.

[0140] (Step S803C speaker) Specifically, for example, the transmission control module 3032 transmits the music to be played in the space to the speaker 33. For example, the transmission control module 3032 transmits the music and volume selected in step S802F to the speaker 33 installed in the space.

[0141] <7 Screen Examples> An example of the screen shown in this disclosure is explained below.

[0142] Figure 11 is a schematic diagram showing an example of the display screen of the terminal device 10's display 141. Figure 11 illustrates the case where the excitement level of an ongoing meeting in conference room 201 is displayed.

[0143] Text 9001 is a text that describes the space where the gathering is taking place.

[0144] Text 9002 is text indicating the current time.

[0145] Graph 9003 is a graph used to represent the trend of an indicator value of the level of excitement at a gathering. Graph 9003 consists of a vertical axis showing the indicator value of excitement and a horizontal axis showing the trend over time. A larger value on the vertical axis indicates a higher indicator value of excitement. The origin of the horizontal axis may, for example, represent the start time of the ongoing gathering or the start time of the business establishment.

[0146] The solid line 9004 is a solid line that shows the trend of the ongoing surge index value calculated in step S802A. The solid line 9004 may be drawn using real-time processing, batch processing, or all at once after the collection is complete. Batch processing is performed, for example, every 30 seconds.

[0147] The dashed line 9005 is a dashed line for comparison with the solid line 9004. Specifically, for example, the dashed line 9005 depicts the trend of the average value of the index for the same time period in the past. Alternatively, for example, the dashed line 9005 may depict the exemplary trend obtained in step S802C.

[0148] The dashed line 9005 may depict only the progression up to the current time, or it may depict the progression beyond the current time.

[0149] Figure 12 is a schematic diagram showing an example of the display screen of the display 141 of the terminal device 10. Figure 12 illustrates the case where the scores of completed meetings in conference room 201 are displayed.

[0150] Text 1001 is a text that describes the space where the gathering took place.

[0151] Text 1002 is text indicating the time the gathering took place.

[0152] Icon 1003 is an icon that describes the level of excitement at a gathering. The analysis control module 2033 divides the current gathering into time segments and compares the progress of each segment with the average progress of past gatherings. The analysis control module 2033 determines that the gathering is exciting if it generally exceeds the average progress, and that it is not exciting if it generally falls below the average progress. For example, the analysis control module 2033 outputs a sunny then cloudy icon for a gathering that was exciting in the first half but not so exciting in the second half. The presentation control module 2034 presents the output icons to the user.

[0153] Text 1004 represents the score given for the level of enthusiasm at the gathering. For example, Text 1004 represents the score calculated in step S802C.

[0154] Text 1005 is text that represents advice on how to make the gathering more engaging. The analysis control module 2033 compares the progress of the current gathering to an ideal progress and analyzes how much the gathering's engagement falls short of the ideal progress. The analysis control module 2033 outputs advice based on the progress of the gathering. For example, for a gathering that was engaging in the first half but became less engaging in the second half compared to the ideal progress, the analysis control module 2033 outputs the advice, "Next time, try to make it more engaging with the closing talk." The presentation control module 2034 presents the outputted advice to the user.

[0155] Figure 13 is a schematic diagram showing an example of the display screen of the display 141 of the terminal device 10. Figure 13 explains the case where activity levels at the Azabudai business site are displayed as an example.

[0156] Text 1101 is a text that shows the business location and the number of people reporting to that business location today.

[0157] Graph 1102 is a graph that represents emotional beat, or excitement, within a workplace. Graph 1102 consists of a vertical axis showing an index of excitement and a horizontal axis showing its progression over time. A higher value on the vertical axis indicates a higher index of excitement. The horizontal axis represents time periods such as one day, one week, one month, three months, six months, or one year. The time period on the horizontal axis may be switched by the user through interface operations.

[0158] Graph 1102 may be accompanied by icons representing emotions. For example, scenes in Graph 1102 that exceed a predetermined index may be accompanied by an icon representing a smile.

[0159] The solid line 1103 is a solid line that shows the trend of the index of activity for one business establishment, calculated in step S802D. The solid line may be drawn using real-time processing of the index, or it may be drawn using batch processing. Batch processing is performed, for example, every 30 seconds.

[0160] The dashed line 1104 is a dashed line used for comparison with the solid line 1103. Specifically, for example, the dashed line 1104 shows the trend of the average index for the same time period in the past.

[0161] The dashed line 1104 may depict only the progression up to the current time, or it may depict the progression beyond the current time.

[0162] Pie chart 1105 shows the emotional rate, that is, the proportion of facial expressions related to the workplace. Specifically, for example, pie chart 1105 is a pie chart that shows the proportion of people with a specific facial expression among the number of people present at work. Specifically, the analysis control module 2033 calculates the sum of the number of people whose "State" is "Excited", the sum of the number of people whose "State" is "Depressed", and the sum of the number of people whose "State" is "Neutral", from records in the image analysis log DB 2022 where the item "Time" is the current time. The analysis control module 2033 also calculates the number of "Other" people by subtracting the sum of the above three sums from the current number of people present at work. Then, specifically, the analysis control module 2033 calculates the proportion of people with "Excited" expressions relative to the number of people present at work, the proportion of people with "Depressed" expressions relative to the number of people present at work, the proportion of people with "Neutral" expressions relative to the number of people present at work, and the proportion of people with "Other" expressions relative to the number of people present at work. Specifically, the analysis control module 2033 outputs the percentage of people who are "excited" with a smiling face icon, the percentage of people who are "disappointed" with an anxious face icon, the percentage of people who are "neutral" with a serious face icon, and the percentage of people who are "other" with a document icon. The presentation control module 2034 then presents the output percentages and icons to the user.

[0163] List 1106 is a list of actions to encourage enthusiasm within the workplace and the people who should take part.

[0164] Figure 14 is a schematic diagram showing an example of the display screen of the terminal device 10's display 141. Figure 14 illustrates the case where the real-time excitement level of gatherings in various spaces within a business premises is displayed.

[0165] Map 1201 is a map showing the entire business premises and the spaces that comprise them. Each space is a place where gatherings take place and where activity is evaluated, and it is labeled with text indicating its name. Examples of space names include Conference Room A, Salon Table 1, etc. Map 1201 may be drawn in two dimensions or in three dimensions.

[0166] The volume meter 1202, as exemplified by volume meters 1202-1 to 1202-3, is a volume meter that shows the real-time rise of a gathering in space based on an index value indicating real-time rise calculated in step S802A. For example, the volume meter 1202 divides the range from the highest to the lowest possible index value into five stages, and represents the real-time index value with five bars corresponding to the five stages. The higher the index value, that is, the more the gathering rises, the more the bars of the volume meter 1202 stack up.

[0167] The volume meter 1202 may be marked with an icon indicating a rising trend. For example, a volume meter in a space where the rise has been on an upward trend over the past five minutes may be marked with an upward arrow.

[0168] <8 Summary> As described above, in the above embodiment, System 1 includes means for collecting sounds generated in a predetermined space (a space within an office or a conference room) that is installed in the predetermined space; means for classifying the types of sounds collected by the means for collecting sounds in the predetermined space and determining the state of excitement and the degree of that state based on the classified type and volume; means for aggregating the states and degrees and calculating the time progression of excitement in the space; and means for presenting information based on the time progression of excitement in the space to the user. As a result, it is possible to calculate the excitement in the space without transmitting raw audio and image data to the cloud. Therefore, according to the program, method, information processing device, and system of this embodiment, it is possible to automatically quantify the excitement in a predetermined space from an audio perspective while maintaining security.

[0169] Furthermore, in the above embodiment, System 1 is installed in a predetermined space and includes means for capturing images within the space. The determination means classifies the types of facial expressions contained in the images collected by the capturing means, and determines the state of excitement and the degree of that state based on the classified types and the number of people with each type of facial expression. As a result, raw audio and image data are not transmitted to the cloud, and the level of excitement in a predetermined space can be automatically quantified from the perspective of participants' facial expressions while maintaining security.

[0170] Furthermore, in the above embodiment, System 1 includes means for comparing the progress of excitement in multiple gatherings in a space and acquiring predetermined trends regarding excitement. This enables Server 20 to acquire measures to promote excitement while comparing multiple gatherings. In addition, in the above embodiment, System 1 includes means for presenting improvement suggestions regarding excitement based on the acquired trends. This enables users to learn about measures to promote excitement and improve the vibrancy of the gatherings they participate in.

[0171] Furthermore, in the above embodiment, System 1 includes means for verifying the effect of an action on the level of excitement, based on an index value indicating the level of excitement achieved by the action. This makes it possible for users to recognize what kinds of actions contribute to the level of excitement. Conventionally, it was difficult for users to quantitatively verify measures to promote excitement, but this allows users to learn about measures that have been supported by quantitative verification.

[0172] Furthermore, in the above embodiment, System 1 includes means for statistically calculating an index representing the activity of the entire business entity based on indicator values ​​showing the level of activity in multiple spaces. This makes it possible to automatically quantify the activity of the entire business entity. Conventionally, it was difficult to visualize the activity level of a business entity, but this embodiment makes it possible to visualize the activity level of a business entity. As a result, users can estimate the state of the business entity based on the visualized activity level. For example, if the activity level of a business entity is shown to be low, the business entity's manager can make a decision to increase the activity level of the business entity by launching a new project.

[0173] Furthermore, in the above embodiment, System 1 includes means for playing music in the space according to the spatial excitement index value. This makes it possible to promote and maintain a high level of excitement in the space.

[0174] <9 Variations> In the above embodiment, we described an example where a microphone is used to collect sound within the space and a camera is used to capture images of the space in order to measure the level of excitement. However, beacons may also be used to obtain the number of people and their locations within the space in order to measure the level of excitement.

[0175] For example, a user terminal with a dedicated beacon app installed receives a beacon signal transmitted from an analysis device 30 installed in the space. Then, for example, the user terminal transmits its location information to a nearby analysis device 30, and the analysis device 30 transmits the user terminal's location information to the server 20. Alternatively, the user terminal may transmit its location information directly to the server 20. As a result, the server 20 obtains the number of people in the space and their locations. The user may have a device that transmits beacon signals, and the analysis device 30 may be configured to respond to beacon signals transmitted from that device.

[0176] The analysis device 30 may have a function to analyze the number, position, etc., in space. Based on the detected number, the analysis device 30 determines the state related to the surge and the degree of that state. Specifically, the analysis device 30 determines the state related to the surge and the degree of that state based on the number of detected users, density, amount of change, or a combination of at least two of these. For example, if the number of detected users increases by a predetermined number or more in a short period of time, the analysis device 30 evaluates the state "surge" to a value of "3". The analysis device 30 transmits the analysis results to the server 20.

[0177] In this modified version, System 1 includes, for example, a means for detecting the user's position within a predetermined space, and the determination means determines the state of elevation and the degree of that state based on the number detected. This makes it possible to automatically quantify the elevation in a predetermined space from the perspective of the user's position.

[0178] Alternatively, individuals within a space may be identified using identification numbers assigned to each person via beacons.

[0179] For example, a user terminal with a dedicated beacon app installed receives a beacon signal transmitted from an analysis device 30 installed in the space. Then, for example, the user terminal transmits its location information to a nearby analysis device 30. The analysis device 30 transmits the user terminal's location information to the server 20. Alternatively, the user terminal may transmit its location information directly to the server 20. The transmitted and received information includes an identification number assigned to each person's account in the app. The server 20 identifies the person in the space based on the identification number. The user may have a device that transmits beacon signals, and the analysis device 30 may respond to beacon signals transmitted from that device.

[0180] Server 20 analyzes the relationship between the level of activity in a space and individuals identified based on their identification numbers. This allows for the acquisition of correlations between the level of activity in a space and individuals within that space, such as determining that the space becomes more active when a specific person is present.

[0181] Furthermore, the above embodiment was described using the example of a case where the analysis device 30 does not have a display. However, the analysis device 30 may have a display for showing the user information related to the elevation of the space in which the analysis device 30 is installed. For example, the display control module 3034 causes the analysis device 30's display to show the trend of the index value of the elevation of the cluster in space 1 calculated in step S802A, and the exemplary trend of the elevation and the score for the elevation obtained in step S802C.

[0182] <10 Basic Computer Hardware Configuration> Figure 15 is a block diagram showing the basic hardware configuration of computer 90. Computer 90 includes at least a processor 901, main memory 902, auxiliary storage 903, and a communication interface 991. These are electrically connected to each other by a communication bus 921.

[0183] The processor 901 is hardware for executing the instruction set written in a program. The processor 901 consists of an arithmetic unit, registers, peripheral circuits, etc.

[0184] Main memory 902 is used to temporarily store programs and data processed by programs, etc. For example, it is a volatile memory such as DRAM (Dynamic Random Access Memory).

[0185] Auxiliary storage device 903 refers to a storage device for saving data and programs. Examples include flash memory, HDD (Hard Disc Drive), magneto-optical disk, CD-ROM, DVD-ROM, and semiconductor memory.

[0186] The IF991 communication interface is an interface for inputting and outputting signals for communication with other computers via a network using wired or wireless communication standards. A network consists of various mobile communication systems, such as the internet, LANs, and wireless base stations. For example, a network includes 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks that can connect to the internet via designated access points (e.g., Wi-Fi®). When connecting wirelessly, communication protocols include, for example, Z-Wave®, ZigBee®, and Bluetooth®. When connecting via a wired connection, the network also includes connections made directly via USB (Universal Serial Bus) cables, etc.

[0187] Furthermore, by distributing all or part of each hardware configuration across multiple computers 90 and connecting them to each other via a network, a computer 90 can be virtually realized. Thus, the concept of computer 90 includes not only a computer 90 housed in a single enclosure or case, but also a virtualized computer system.

[0188] <Basic Functional Configuration of Computer 90> The functional configuration of the computer realized by the basic hardware configuration of computer 90 is described below. The computer comprises at least one functional unit: a control unit, a memory unit, and a communication unit.

[0189] Furthermore, the functional units of computer 90 can also be realized by distributing all or part of each functional unit across multiple computers 90 interconnected via a network. The concept of computer 90 includes not only a single computer 90 but also a virtualized computer system.

[0190] The control unit is realized when the processor 901 reads various programs stored in the auxiliary storage device 903, loads them into the main memory device 902, and executes processing according to those programs. The control unit can realize various functional units that perform information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0191] The memory unit is implemented by the main memory 902 and the auxiliary memory 903. The memory unit stores data, various programs, and various databases. The processor 901 can also reserve memory areas corresponding to the memory unit in the main memory 902 or the auxiliary memory 903 according to the program. The control unit can also cause the processor 901 to perform operations such as adding, updating, and deleting data stored in the memory unit according to the various programs.

[0192] A database, specifically a relational database, is used to manage and associate data sets called masters, which are structured in a tabular format defined by rows and columns. In a database, tables are called tables, masters are called masters, the columns of tables are called columns, and the rows of tables are called records. In a relational database, relationships can be established and linked between tables and masters. Typically, each table and master has a primary key column to uniquely identify records, but setting a primary key column is not mandatory. The control unit can instruct the processor 901 to add, delete, or update records in specific tables and masters stored in the memory unit, according to various programs. Furthermore, by storing data, various programs, and various databases in the memory unit, the information processing device and information processing system related to this disclosure can be considered to have been manufactured.

[0193] Furthermore, the databases and masters in this disclosure may include any data structures (lists, dictionaries, associative arrays, objects, etc.) in which information is structurally defined. Data structures also include data that can be considered as data structures by combining data with functions, classes, methods, etc., written in any programming language.

[0194] The communication unit is implemented by the communication IF991. The communication unit provides the functionality to communicate with other computers 90 via the network. The communication unit can receive information transmitted from other computers 90 and input it to the control unit. The control unit can cause the processor 901 to perform information processing on the received information according to various programs. The communication unit can also transmit information output from the control unit to other computers 90.

[0195] Furthermore, each of the above-mentioned configurations, functions, processing units, processing means, etc., may be implemented in hardware, either partially or entirely, by designing them as integrated circuits, for example. The present invention can also be implemented by software program code that realizes the functions of the embodiment. In this case, a storage medium on which the program code is recorded is provided to a computer, and the processor of that computer reads the program code stored in the storage medium. In this case, the program code read from the storage medium itself realizes the functions of the embodiment described above, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media used to supply such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, and the like.

[0196] Furthermore, the program code that implements the functions described in this embodiment can be implemented in a wide range of programming or scripting languages, such as assembler, C / C++, Perl, Shell, PHP, and Java (registered trademark).

[0197] Furthermore, the program code for the software that implements the functions of the embodiment may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the computer's processor may read and execute the program code stored in the storage means or storage medium.

[0198] The functions realized by the components described herein may be implemented in a circuit or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to realize the functions described herein. A processor is considered to be a circuit or processing circuitry, including transistors and other circuits. A processor may be a programmed processor that executes a program stored in memory. In this specification, circuitry, unit, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein. If the hardware is a processor that is considered to be a type of circuitry, then the circuitry, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0199] While several embodiments of this disclosure have been described above, these embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications are permitted without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0200] <Note> The details described in each of the above embodiments are noted below. (Note 1) A system comprising: means installed in a predetermined space to collect sounds generated in that space; means installed in the space to classify the types of sounds collected by the sound-collecting means, and determining the state of excitement and the degree of that state based on the classified type and volume; means to aggregate the state and degree and calculate the time progression of excitement in the space; and means to present information based on the time progression of excitement in the space to the user. (Note 2) The system described in (Appendix 1), which is installed in a space and includes means for capturing images within the space, wherein the determining means classifies the types of facial expressions contained in the images collected by the capturing means, and determines the state of excitement and the degree of that state based on the classified types and the number of people with that type of facial expression. (Note 3) The system described in (Appendix 1) or (Appendix 2), which is installed in space and includes means for detecting the position of a user in the space, and means for determining the position, which determines the state of elevation and the degree of said state based on the number of detected positions. (Note 4) The designated space is a space within the office, and the system is one of the systems described in (Appendix 1) to (Appendix 3). (Note 5) The designated space is a conference room, and the system is as described in any of (Appendix 1) to (Appendix 3). (Note 6) A system according to any one of (Appendix 1) to (Appendix 5), comprising means for comparing the progression of excitement of multiple gatherings in space and obtaining a predetermined trend regarding the excitement. (Note 7) The means for acquiring trends is the system described in (Appendix 6), which presents improvement suggestions regarding the excitement based on the acquired trends. (Note 8) A system as described in any of (Appendix 1) to (Appendix 7), comprising means for verifying the effect of an action on the level of excitement, based on an index value indicating the level of excitement resulting from an action taken to promote excitement. (Note 9) A system described in any of (Appendix 1) to (Appendix 8), comprising means for statistically calculating an index representing the activity of the entire business entity based on indicator values ​​showing the level of activity in multiple spaces. (Note 10) A system described in any of (Appendix 1) to (Appendix 9), which includes a means for playing music in a space according to an index value of the space's excitement. (Note 11) A method for performing operations on a system comprising a microphone, an analysis device, and a server, the method comprising: the steps of: the microphone being installed in a predetermined space and collecting sounds occurring in the space; the analysis device being installed in the space and classifying the types of sounds collected in the sound collection step, and determining the state of excitement and the degree of said state based on the classified type and volume; the server aggregating the states and degrees and calculating the time progression of excitement in the space; and the server presenting information based on the time progression of excitement in the space to the user. (Note 12) A program for operating a computer installed in a predetermined space, comprising a processor and memory, the program causing the processor to perform the steps of: acquiring sound generated in the space, collected by a microphone installed in the space; classifying the type of sound acquired; and determining a state related to excitement and the degree of said state based on the classified type and volume. (Note 13) An information processing device installed in a predetermined space, comprising a control unit and a storage unit, wherein the control unit performs the steps of: acquiring sound generated in the space, collected by a microphone installed in the space; classifying the type of sound acquired; and determining a state related to excitement and the degree of said state based on the classified type and volume. [Explanation of symbols]

[0201] 1... System 10…Terminal device 12…Communication IF 13…Input device 14…Output device 15…Memory 16…Storage 19… Processor 20... Server 22...Communication IF 23…Input / Output Interface 25…Memory 2 hours… storage 29… Processor 30…Information Processing Equipment 31... Mike 32... Imaging device 33…Speaker 40…Network

Claims

1. A system comprising a microphone, an analysis device, and a server, The aforementioned microphone is It is installed in a predetermined space and collects sound generated in the said space. The aforementioned analysis device is The aforementioned space is installed A means for classifying the type of sound by inputting the collected sound into a trained model that has been trained to output the type of sound, A means for determining the state of excitement and the degree of said state based on the classified type and the volume of the collected sound, A means for transmitting information regarding the determined state of the rise and the degree of said state to the server, Equipped with, The aforementioned server, A means for aggregating the state and degree and calculating the time progression of the rise in the space, A means for presenting information to the user based on the time progression of the rise in the aforementioned space, A system equipped with these features.

2. The space is equipped with a camera that is installed in the space and captures images of the space, The system according to claim 1, wherein the means for determining the above-mentioned means classifies the types of facial expressions contained in the image captured by the imaging device, and determines the state of excitement and the degree of that state based on the classified types and the number of people with that type of facial expression.

3. Installed in the aforementioned space and equipped with means for detecting the position of a user within the aforementioned space, The system according to claim 1, wherein the means for determining the above-mentioned state of elevation and the degree of said state are determined based on the number detected.

4. The system according to claim 1, wherein the predetermined space is a space within an office.

5. The system according to claim 1, wherein the predetermined space is a conference room.

6. The system according to claim 1, wherein the server comprises means for comparing the progression of the rise of a plurality of gatherings in the space and obtaining a predetermined trend regarding the rise.

7. The system according to claim 6, wherein the means for acquiring the trend presents improvement suggestions regarding the excitement based on the acquired trend.

8. The system according to claim 1, wherein the server comprises means for verifying the effect of an action on the level of excitement, based on an index value indicating the level of excitement after an action has been taken to promote excitement.

9. The system according to claim 1, wherein the server comprises means for statistically calculating an index representing the activity of the entire business entity based on index values ​​indicating the level of activity in a plurality of spaces.

10. The system according to claim 1, further comprising a speaker that plays music into the space according to an index value of the excitement level of the space.

11. A method performed on a system comprising a microphone, an analysis device, and a server, The steps include: installing the microphone in a predetermined space and collecting sound generated in the space; The analysis device is installed in the space, and the steps include: inputting the collected audio into a trained model that has been trained to output types of sounds, thereby classifying the types of sounds collected; The analysis device performs the steps of determining the state of excitement and the degree of that state based on the classified type and the volume of the collected sound, The analysis device transmits information regarding the determined state of the elevation and the degree of that state to the server. The server performs the steps of aggregating the state and degree and calculating the time progression of the rise in the space, The server provides the user with information based on the time progression of the rise in the space, How to do it.

12. A program for operating a computer installed in a predetermined space, comprising a processor and memory, wherein the program is configured to operate the processor, The steps include acquiring sound generated in the space, collected by a microphone installed in the space, The steps include: inputting the acquired audio into a trained model that has been trained to output the type of sound, thereby classifying the type of sound acquired; A step of determining the state related to excitement and the degree of said state based on the classified type and the volume of the acquired sound, The steps include transmitting information regarding the determined state of the elevation and the degree of that state to an external device that aggregates and analyzes the information, A program that executes the command.

13. An information processing device installed in a predetermined space, comprising a control unit and a storage unit, wherein the control unit is The steps include acquiring sound generated in the space, collected by a microphone installed in the space, The steps include: inputting the acquired audio into a trained model that has been trained to output the type of sound, thereby classifying the type of sound acquired; A step of determining the state related to excitement and the degree of said state based on the classified type and the volume of the acquired sound, The steps include transmitting information regarding the determined state of the elevation and the degree of that state to an external device that aggregates and analyzes the information, An information processing device that performs the following actions.

Images