Sensor device, program, estimation method, and communication system

Abstract

To provide a technique of estimating a behavior.SOLUTION: A sensor device has a first inertial sensor that contacts a torso of a user, and a second inertial sensor that contacts a neck of the user. The sensor device estimates a behavior of the user on the basis of a first signal detected by the first inertial sensor and a second signal detected by the second inertial sensor.SELECTED DRAWING: Figure 18

Description

[Technical field]

[0001] The present invention relates to a sensor device, a program, an estimation method, and a communication system. [Background technology]

[0002] It is said that trust between employees, a sense of efficacy, and psychological safety are important for creative activities and effective communication in offices, etc. For example, the way of listening is important for effective communication, and behavior such as nodding to a speaker gives the speaker a sense of security and indicates the listener's level of understanding.

[0003] There is known a sensor device that detects the posture of a user by a sensor worn by the user (see, for example, Patent Document 1). Patent Document 1 discloses a detection device that includes a main body that is fixed to the upper body of the user, a posture detection unit that is disposed at a predetermined position and detects the posture of the user, and a connection unit that connects the main body and the posture detection unit and has elasticity in a direction toward the user. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional techniques have limitations in the user behavior that can be estimated. For example, when a user nods, it is difficult to estimate that the user has nodded based on the acceleration of the torso alone.

[0005] In view of the above problems, the present invention provides a technique capable of estimating behavior. [Means for solving the problem]

[0006] The present invention provides a sensor device having a first inertial sensor in contact with a user's torso and a second inertial sensor in contact with the neck, and a behavior estimation unit that estimates the user's behavior based on a first signal detected by the first inertial sensor and a second signal detected by the second inertial sensor. Effect of the Invention

[0007] The present invention can provide a technique for estimating behavior. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a perspective view of an example of an activity sensor. [Diagram 2] FIG. 1 is a configuration diagram of an example of a communication system. [Diagram 3] FIG. 1 is a diagram for explaining an example of a conference room. [Figure 4] FIG. 2 is a diagram illustrating a hardware configuration of an example of a computer. [Diagram 5] FIG. 2 is a diagram illustrating a hardware configuration of an example of an information processing terminal. [Figure 6] FIG. 2 is a hardware configuration diagram of an example of an activity sensor. [Figure 7] FIG. 2 is an external view of an example of an activity sensor 9A. [Figure 8] FIG. 13 is an external view of an example of an activity sensor 9B. [Figure 9] 1 is a top view of an example showing an activity sensor worn by a user from above; [Figure 10] FIG. 2 is a functional configuration diagram of an example of an information processing system and an activity sensor. [Figure 11] FIG. 13 is a diagram illustrating an example of activity information. [Figure 12] FIG. 13 is a diagram illustrating an example of behavior speech information. [Figure 13] FIG. 1 is a diagram illustrating a user's direction. [Figure 14] FIG. 13 is a diagram illustrating grouping of users. [Figure 15] 13 is a diagram illustrating a method for determining whether users are facing each other when another user is present. FIG. [Figure 16] 1A to 1C are diagrams illustrating the body movements of a user when nodding. [Figure 17] 17 is a diagram showing an example of a change in acceleration detected when the body moves as shown in FIG. 16. FIG. [Figure 18]11 is a flowchart illustrating an example of a process in which a behavior estimation unit estimates that a behavior of a nodding has occurred. FIG. [Figure 19] 11 is a sequence diagram illustrating an example of a process in which the information processing system notifies a speaker that a listener has nodded. FIG. [Figure 20] 1A to 1C are diagrams illustrating the body movements of a user when nodding. [Figure 21] 21 is a diagram showing an example of changes in acceleration and angular velocity detected when the body moves as shown in FIG. 20. FIG. [Figure 22] 11 is a flowchart illustrating an example of a process in which a behavior estimation unit estimates that a behavior of a nodding has occurred. FIG. [Diagram 23] 11 is a flowchart illustrating an example of a process in which a behavior estimation unit estimates that a behavior of a nodding has occurred based on a frequency spectrum. FIG. [Figure 24] 1A and 1B are diagrams illustrating an example of a leaning back and a leaning forward posture. [Diagram 25] 13 is a flowchart illustrating an example of a process in which a behavior estimating unit estimates that a behavior of leaning backward has occurred. FIG. [Figure 26] 13 is a flowchart illustrating an example of a process in which a behavior estimation unit estimates that a forward leaning motion has occurred. FIG. [Figure 27] FIG. 13 illustrates an example situation where the user is leaning to the left or right side. [Figure 28] FIG. 1 is a diagram showing an example of a hunched posture. [Figure 29] 13 is a flowchart illustrating an example of a process performed by a behavior estimation unit to estimate that the posture is hunched. FIG. [Diagram 30] 1A to 1C are diagrams illustrating the movement of the body when a user shakes his / her head. [Diagram 31] 11 is a flowchart illustrating an example of a process in which a behavior estimation unit estimates that a head shaking motion has occurred. FIG. [Diagram 32] 1A to 1C are diagrams illustrating the movement of the user's body when tilting his / her head. [Diagram 33]11 is a flowchart illustrating an example of a process in which a behavior estimation unit estimates that a head tilting behavior has occurred. FIG. [Diagram 34] FIG. 11 is a diagram illustrating a method for estimating a face direction. [Diagram 35] 11 is a flowchart illustrating an example of a process in which a listening level estimation unit estimates a listening level based on a face direction. FIG. [Diagram 36] FIG. 13 is a diagram illustrating a user nodding and saying "yes." [Figure 37] FIG. 11 is a flowchart illustrating an example of behavior reinforcement using speech data. [Figure 38] FIG. 13 is a diagram illustrating that the activity sensor can detect vital data. [Figure 39] 13 is a flowchart illustrating an example of a process in which a behavior estimation unit estimates psychological safety based on vital data. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, as an example of an embodiment of the present invention, an activity sensor and an estimation method performed by the activity sensor will be described with reference to the drawings.

[0010] <Comparative techniques for motion detection> As mentioned above, the way of attentive listening is important in communication, and actions such as nodding along to what is being said give the speaker a positive feeling and make communication even smoother. The same goes for looking at the speaker. Conversely, attentive listeners can also convey their level of understanding to the speaker by tilting or shaking their head. If activity sensors or other devices detect these behaviors and record the actions of employees and provide feedback to each other, more detailed communication can be promoted, and improvements in work efficiency and creativity can be expected.

[0011] There are known technologies that capture images of people's movements during communication with a camera and use image processing technology to estimate the movements, or estimate emotions from facial expressions. However, methods that use cameras have spatial constraints, in that the person must fit within the camera's field of view. Another known method of detecting human movements is to attach inertial sensors to the body. Although this method has fewer spatial constraints than camera-based methods, it requires multiple sensors to be attached in different locations and the sensors to be tied down, making it unsuitable for long hours of office work.

[0012] <Outline of the activity sensor according to this embodiment> Therefore, in this embodiment, various behaviors (movements and postures) of the user are detected by a wearable activity sensor 9 that can be easily worn by the user.

[0013] FIG. 1 shows a perspective view of the activity sensor 9 of this embodiment. FIG. 1(a) shows an activity sensor 9A (an example of a first inertial sensor) that contacts the chest side of the user, and FIG. 1(b) shows an activity sensor 9B (an example of a second inertial sensor) that contacts the neck (nape) side of the user. In the following, when the activity sensor 9A and the activity sensor 9B are collectively referred to, they are called "activity sensor 9". The activity sensor 9A and the activity sensor 9B are connected in a circular shape by a string to form a necklace-shaped activity sensor 9. When worn, the activity sensor 9A contacts the torso. More specifically, the activity sensor 9A is in a state of hanging in contact with the chest of the user with the neck as a fulcrum. When worn, the activity sensor 9B is in close contact with the nape of the user's neck, which is the fulcrum. The user only needs to hang the activity sensor 9 around the neck, and there is no need to tie or attach it.

[0014] Activity sensor 9A and activity sensor 9B each have an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, etc., and activity sensor 9A acquires signals (an example of a first signal) from the acceleration sensor, the angular velocity sensor, the geomagnetic sensor, etc. Activity sensor 9B acquires signals (an example of a second signal) from the acceleration sensor, the angular velocity sensor, the geomagnetic sensor, etc. Activity sensor 9 can accurately detect behaviors such as nodding based on signals detected by two sensors (an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, etc.) arranged at different locations on the user's body.

[0015] In this way, the activity sensor 9 of this embodiment is easy for the user to wear as it only needs to be worn around the neck, and since it detects acceleration and angular velocity signals at different parts of the body, it can detect various movements such as nodding, shaking the head, tilting the head, and facial direction, as well as various postures such as leaning back or leaning forward.

[0016] <Terminology> Behavior refers to the way a person carries themselves. Behavior includes any kind of movement or posture. In this embodiment, the movement of the user's body is referred to as movement, and the posture that the user maintains for a certain period of time is referred to as posture. The distinction between these does not need to be strict.

[0017] A sensor device is a device having one or more sensors for estimating a user's behavior. The sensor device detects, for example, signals of acceleration or angular velocity at different locations on the body. In this embodiment, the sensor device is described using the term activity sensor. Since the sensor device can be worn by a user, it may be called a wearable sensor, a life log sensor, or the like.

[0018] Position information is information indicating the position of a user in space. The position may be, for example, coordinates based on a reference point in space. In this embodiment, the position information is detected by UWB (Ultra Wide Band). The position may also be latitude and longitude detected by IMES (Indoor MEssaging System) or GNSS (Global Navigation Satellite System). In this embodiment, the term may simply be "position".

[0019] An inertial sensor is a sensor that can detect acceleration and angular velocity. The acceleration sensor and the angular velocity sensor may be separate sensors.

[0020] <System configuration> Fig. 2 is a configuration diagram of an example of a communication system 100 according to the present embodiment. Fig. 3 is a diagram for explaining an example of a conference room according to the present embodiment. The communication system 100 in Fig. 2 includes an information processing system 10, a video display device 12, a sensor device 14, a speaker 16, a camera 18, a microphone 20, and an information processing terminal 22, which are connected by wire or wirelessly so as to be able to communicate with each other via a network N such as the Internet or a LAN.

[0021] Each conference room is provided with one or more video display devices 12, sensor devices 14, speakers 16, cameras 18, microphones 20, and information processing terminals 22. The conference room may also be provided with a temperature sensor, a humidity sensor, an illuminance sensor, etc. that notify the information processing system 10. Although an example in which the information processing system 10 is provided outside the conference room is shown in FIG. 2, the information processing system 10 may also be provided inside the conference room.

[0022] For example, a user entering a conference room carries an activity sensor 9 that emits radio waves, such as a beacon. Sensor device 14 in the conference room receives the radio waves emitted from activity sensor 9 of the user in the conference room as a signal for detecting the user's location information, and notifies information processing system 10. Sensor device 14 may be any sensor of a positioning system that can output a signal for detecting the user's location information.

[0023] The activity sensor 9 on the measurement subject side has activity sensors 9A and 9B, each of which has an acceleration / angular velocity sensor, a geomagnetic sensor, a vital sensor, etc., and is shaped so that it can be worn around the neck like a necklace. Activity sensor 9A has a microphone and can pick up the voice of the person wearing it. In addition, the vital sensor can acquire vital data of the user. Activity sensor 9 can also estimate the user's behavior from the signals detected by the acceleration / angular velocity sensors and geomagnetic sensors of activity sensors 9A and 9B and their relative values.

[0024] The activity sensor 9 may be a dedicated sensor, a smart watch, a smartphone, or various BLE (Bluetooth Low Energy, registered trademark) sensors. The information processing system 10 detects the location information of each user in the conference room based on a signal for detecting the user's location information notified from one or more sensor devices 14. Note that the activity sensor 9 described above is an example of a transmitting device, and the device does not have to be in the form of an activity sensor 9 as long as it transmits a signal for detecting the user's location information.

[0025] The information processing terminal 22 is a device operated by a user in the conference room. For example, the information processing terminal 22 is a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a game console, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, a desktop PC, a device dedicated to the conference room, etc. The information processing terminal 22 may be one brought into the conference room by the user, or may be one installed in the conference room.

[0026] Furthermore, the information processing terminal 22 may be a measurement target by a positioning system. For example, the sensor device 14 in the conference room may receive radio waves transmitted from an activity sensor 9 included in the information processing terminal 22 and notify the information processing system 10. The sensor device 14 may notify the information processing system 10 of a signal for detecting position information in the conference room of the user operating the information processing terminal 22, as shown in FIG. 3, for example. The activity sensor 9 may be built into the information processing terminal 22 or may be in another form. Furthermore, the information processing terminal 22 may be provided with a sensor for measuring the user's heart rate, and the measured heart rate of the user may be notified to the information processing system 10.

[0027] The camera 18 in the conference room captures images of the conference room and transmits the captured video data as an output signal to the information processing system 10. For example, a Kinect (registered trademark) video camera may be used as the camera 18. The Kinect (registered trademark) video camera is an example of a video camera having a distance image sensor, an infrared sensor, and an array microphone. When a video camera having a distance image sensor, an infrared sensor, and an array microphone is used, the movement and posture of the user can be recognized.

[0028] The microphone 20 in the conference room converts the user's voice into an electrical signal. The microphone 20 transmits the electrical signal converted from the user's voice as an output signal to the information processing system 10. Note that a microphone of the information processing terminal 22 may be used instead of or in addition to the microphone 20 in the conference room.

[0029] The conference room speaker 16 converts an electrical signal into a physical signal and outputs sounds such as environmental sounds. The speaker 16 outputs sounds such as environmental sounds under the control of the information processing system 10. Note that the speaker of the information processing terminal 22 may be used in place of or in addition to the conference room speaker 16. The conference room microphone 20 and the microphone of the information processing terminal 22 are examples of input devices. The conference room speaker 16 and the speaker of the information processing terminal 22 are examples of output devices.

[0030] In addition to those shown in Fig. 2, a vibration generator that generates vibrations by a motor or the like, an odor device that generates an odor, etc. may be arranged. The vibration generator has a vibration motor, and a weight called an oscillator is connected to a rotating shaft. The oscillator is designed so that the connection part with the rotating shaft and the center of gravity are offset, and when it rotates, it vibrates due to the eccentricity, and this vibration propagates to the surroundings, becoming a vibration source. The odor device can be of an air cannon type or a blower type, but the type is not important.

[0031] Furthermore, the image display device 12, the sensor device 14, the speaker 16, the camera 18, the microphone 20, the information processing terminal 22, the activity sensor 9, the vibration generating device, and the odor device may be a device in which two or more of these components are combined.

[0032] An example of the multiple image display devices 12 in the conference room is a projector, which can display images on the surfaces that partition the conference room as shown in FIG. 3 under the control of the information processing system 10. The surfaces that partition the conference room are, for example, the front wall, the back wall, the right wall, the left wall, the floor, and the ceiling. The position of each wall is specified by coordinates that use the junction of the end of the wall and the end of the wall as the reference (origin), similar to the position information. The projector can detect the user's operation on the wall (handwriting, etc.) by a well-known method. Also, an example of the image display device 12 is a flat panel display such as a liquid crystal or organic EL. The flat panel display is preferably a touch panel, and preferably has a large screen. Also, the image display device 12 may be an electronic whiteboard having a flat panel display. The electronic whiteboard may be embedded in the wall or placed in front of the wall. Note that the image display device 12 is an example of a display device that displays an image, and any display device that has at least a function of displaying an image can be applied.

[0033] The shape of the conference room in FIG. 3 is an example, and other shapes may be used. As described above, the conference room does not necessarily need to have all surfaces such as walls, floors, and ceilings partitioned, and may be an open conference room with some surfaces not partitioned. The conference room is an example of the same space in which multiple users are present, and includes various spaces such as a room for seminars or lectures, a meeting space, an event space, a children's room, a children's center, a classroom, a hospital, a game center, and an amusement park. Thus, the space described in this embodiment is a concept that includes a place or room where multiple users are present.

[0034] The information processing system 10 is one or more information processing devices. Based on the user's position information detected by a signal notified from the sensor device 14, the output signal from the camera 18, and the output signal from the microphone 20, the information processing system 10 outputs environmental sounds and images suitable for interactions (interactions such as conversations and meetings) between users in a conference room, as described below.

[0035] 2 is an example. The information processing system 10 may be realized by a single computer or multiple computers, or may be realized by using a cloud service. The information processing system 10 may be, for example, a projector, a display device having an electronic whiteboard function, an output device such as digital signage, a HUD (Head Up Display) device, industrial machinery, an imaging device, a sound collection device, a medical device, a network home appliance, an automobile (Connected Car), a notebook PC, a mobile phone, a smartphone, a tablet terminal, a game console, a PDA, a digital camera, a wearable PC, or a desktop PC.

[0036] <Hardware configuration> <<Computer>> The information processing system 10 is realized, for example, by a computer 500 having the hardware configuration shown in Fig. 4. Furthermore, when the information processing terminal 22 is a PC, it is realized, for example, by the computer 500 having the hardware configuration shown in Fig. 4.

[0037] Fig. 4 is a hardware configuration diagram of an example of a computer according to this embodiment. As shown in Fig. 4, the computer 500 includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, a HD 504, a HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a data bus 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 516.

[0038] Of these, the CPU 501 controls the overall operation of the computer 500. The ROM 502 stores programs used to drive the CPU 501, such as the IPL. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501.

[0039] The display 506 displays various information such as a cursor, a menu, a window, characters, or an image. The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 509 is an interface for data communication using the network N. The data bus 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501.

[0040] The keyboard 511 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 514 controls reading and writing of various data from a DVD-RW 513 as an example of a removable recording medium. The DVD-RW may be a DVD-R or the like. The media I / F 516 controls reading and writing (storing) of data from a recording medium 515 such as a flash memory.

[0041] <<Smartphone>> The information processing terminal 22 may be realized, for example, by a smartphone 600 having the hardware configuration shown in Fig. 5. Even if the information processing terminal 22 is a notebook PC, a mobile phone, a smartphone, a tablet terminal, a game machine, a PDA, a digital camera, a wearable PC, a desktop PC, a dedicated device in a conference room, or the like, it may be realized with a hardware configuration similar to that shown in Fig. 5. Also, some of the hardware configuration shown in Fig. 5 may not be included, and some configuration may be added to the hardware configuration shown in Fig. 5.

[0042] Fig. 5 is a diagram showing an example of the hardware configuration of a smartphone according to this embodiment. As shown in Fig. 5, the smartphone 600 includes a CPU 601, a ROM 602, a RAM 603, an EEPROM 604, a CMOS sensor 605, an image sensor I / F 606, an acceleration / direction sensor 607, a media I / F 609, and a GPS receiver 611.

[0043] Of these, the CPU 601 controls the overall operation of the smartphone 600. The ROM 602 stores the CPU 601 and programs used to drive the CPU 601, such as the IPL. The RAM 603 is used as a work area for the CPU 601. The EEPROM 604 reads and writes various data, such as smartphone programs, under the control of the CPU 601.

[0044] The CMOS (Complementary Metal Oxide Semiconductor) sensor 605 is a type of built-in imaging means that captures an image of a subject (mainly a self-portrait) under the control of the CPU 601 to obtain image data. The CMOS sensor 605 may be an imaging means such as a CCD (Charge Coupled Device) sensor. The imaging element I / F 606 is a circuit that controls the driving of the CMOS sensor 605. The acceleration / direction sensor 607 is one of various sensors such as an electronic magnetic compass that detects geomagnetism, a gyrocompass, and an acceleration sensor.

[0045] A media I / F 609 controls reading and writing (storing) of data from and to a recording medium 608 such as a flash memory. A GPS receiving unit 611 receives GPS signals from GPS satellites.

[0046] The smartphone 600 also includes a long-distance communication circuit 612, a CMOS sensor 613, an image sensor I / F 614, a microphone 615, a speaker 616, an audio input / output I / F 617, a display 618, an external device connection I / F 619, a short-distance communication circuit 620, an antenna 620a of the short-distance communication circuit 620, and a touch panel 621.

[0047] Of these, the long-distance communication circuit 612 is a circuit that communicates with other devices via the network N. The CMOS sensor 613 is a type of built-in imaging means that captures an image of a subject and obtains image data under the control of the CPU 601. The image sensor I / F 614 is a circuit that controls the drive of the CMOS sensor 613. The microphone 615 is a built-in circuit that converts voice or sound into an electrical signal. The speaker 616 is a built-in circuit that converts the electrical signal into physical vibrations to generate sounds such as environmental sounds, music, or voices.

[0048] The audio input / output I / F 617 is a circuit that processes input and output of audio signals between the microphone 615 and the speaker 616 under the control of the CPU 601. The display 618 is a type of display means such as a liquid crystal or organic EL (Electro Luminescence) that displays an image of a subject, various icons, and the like.

[0049] The external device connection I / F 619 is an interface for connecting various external devices. The short-range communication circuit 620 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The touch panel 621 is a type of input means that allows the user to operate the smartphone 600 by pressing the display 618.

[0050] The smartphone 600 also includes a bus line 610. The bus line 610 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 601 shown in FIG.

[0051] <<Activity Sensor 9>> Fig. 6 is a hardware configuration diagram of an example of the activity sensor 9 according to this embodiment. As shown in Fig. 6, the activity sensor 9 has an activity sensor 9A and an activity sensor 9B. The activity sensor 9A has a microcomputer 701, a microphone 702, a UWB module 703, a vital sensor 704, an acceleration and angular velocity sensor 705, a geomagnetic sensor 706, a communication I / F 707, and an LED light 708.

[0052] Of these, the microcomputer 701 has a series of functions constituting a general computer, such as a CPU, ROM, RAM, and EEPROM. The microphone 702 is a built-in circuit that converts voice and sound into electrical signals. The UWB (Ultra Wide Band) module 703 periodically transmits short and sharp rectangular wave (pulse) radio waves. Although the communication range is short distances of about 10 meters, it is capable of high-speed communication with low power consumption. In addition, the sensor device 14 can detect the direction and distance of the activity sensor 9 with a high degree of accuracy, with an error of about several centimeters, by receiving the rectangular wave (pulse) radio waves.

[0053] The vital sensor 704 detects the vital data of the user. Vital data refers to indicators such as heart rate, pulse, and blood pressure that indicate that a human being is alive. The acceleration and angular velocity sensor 705 is also called an inertial sensor, and detects the acceleration in three axial directions that the user experiences, and the angular velocity of rotation around the three axes. The microcomputer 701 can estimate the user's direction mainly by integrating the angular velocity. The geomagnetic sensor 706 is a sensor that detects the magnetic force of the earth, and detects the direction (more specifically, an angle of 0 to 360 degrees) in which the user is facing, east, west, north, south. The geomagnetic sensor 706 is also called an electronic compass.

[0054] The communication I / F 707 transmits sensor signals detected by the activity sensor 9 to the sensor device 14 or the information processing system 10 by connecting to a network via wireless LAN, Bluetooth (registered trademark), or the like. The LED light 708 can be turned on with different combinations of light color and blinking pattern, and outputs information notified by the activity sensor 9A from the information processing system 10, detected behavior, and the like.

[0055] The activity sensor 9B has a vital sensor 711, an acceleration / angular velocity sensor 712, a geomagnetic sensor 713, and a vibration motor 714. These functions may be the same as those of the activity sensor 9A. The vibration motor 714 vibrates when energized and transmits information to the user by touch. In addition, the activity sensors 9A and 9B may have a speaker, an odor generating device, a camera, etc.

[0056] Since the activity sensor 9B is connected to the activity sensor 9A via a serial cable 715, signals detected by these sensors are transmitted in real time to the activity sensor 9A. In addition, the activity sensor 9B is supplied with power by a battery 709 included in the activity sensor 9A.

[0057] Microcomputer 701 can estimate the user's behavior based on two accelerations detected by acceleration and angular velocity sensor 705 of activity sensor 9A and acceleration and angular velocity sensor 712 of activity sensor 9B. Microcomputer 701 can estimate the user's behavior based on the acceleration detected by acceleration and angular velocity sensor 705 of activity sensor 9A and the angular velocity detected by acceleration and angular velocity sensor 705 of activity sensor 9B.

[0058] <External appearance of Activity Sensor 9> FIG. 7 shows an external view of activity sensor 9A. FIG. 8 shows an external view of activity sensor 9B. Activity sensor 9A and activity sensor 9B are electrically connected by a serial cable such as a USB cable. The exterior of the serial cable is covered with a string, so the user is unlikely to be aware of the serial cable. There is a serial cable socket at the end of activity sensor 9B, and the user can wear activity sensor 9 by connecting the end of the string to the socket in the same way as a normal necklace.

[0059] Fig. 7(a) shows the front side of the activity sensor 9A, and Fig. 7(b) shows the back side. The front side of the activity sensor 9A is curved like a natural stone, and the back side is flat, so that the user will not get lost when wearing it. In addition, the shape of each activity sensor 9A varies from one to another, preventing the loss of individuality of the user who wears it.

[0060] FIG. 8(a) shows the front surface of the activity sensor 9B, and FIG. 8(b) shows the back surface. FIG. 8(c) is a side view. As shown in FIG. 8(c), the surface with the opening through which the string passes is inclined toward the back surface so that the exit of the string is difficult to see. Also, the activity sensor 9B is placed a few centimeters outward from the center of the back of the neck so that the back surface of the activity sensor 9B is in close contact with the nape of the neck. Also, dummies of the same shape are placed symmetrically on the left and right sides of the center of the back of the neck of the user. By placing dummies of the same shape as the activity sensor 9B symmetrically on the left and right sides, even if the contact part of the activity sensor 9B changes as the user moves, the string between the activity sensor 9B and the dummy is curved, and the activity sensor 9B can maintain a state of being in close contact with the neck. Also, the activity sensor 9B and the dummy have an excellent aesthetic appearance.

[0061] Fig. 9(a) is a top view showing the state in which activity sensors 9A and 9B are attached to a user. As shown in Fig. 9(a), a dummy 9C having the same shape as activity sensor 9B is placed symmetrically with respect to the center of the back of the user's neck. In Fig. 9(a), dummy 9C is on the left, but dummy 9C may be on the right. Dummy 9C does not have a sensor.

[0062] The position where the activity sensor 9A and the dummy 9C are in close contact changes depending on the movement of the head. For example, in Fig. 9(b), the activity sensor 9A and the dummy 9C move to the narrow part of the neck, but the curve of the string between the activity sensor 9A and the dummy 9C ensures that the sensor always fits the neck. This ensures a stable contact area with the skin, improving the measurement stability of the vital sensor.

[0063] On the other hand, as shown in Fig. 9(c) and (d), the dummy 9C may be omitted. In this case, the activity sensor 9B is placed in the center of the back of the neck. An adhesive material may be used on the back surface of the activity sensor 9B.

[0064] For the sake of convenience in the following explanation, The axis passing through the user (activity sensors 9A and 9B) in the left-right direction is axis 1, The axis passing through the user (activity sensors 9A and 9B) in the front-rear direction is axis 2, The axis passing through the user (activity sensors 9A and 9B) in the vertical direction is axis 3, Acceleration and angular velocity sensors 705 and 712 detect the angular velocities of rotation about these axes 1 to 3.

[0065] <Functional configuration> The communication system 100 according to this embodiment is realized by, for example, a functional configuration as shown in Fig. 10. Fig. 10 is a functional configuration diagram of an example of the information processing system 10 and the activity sensor 9 according to this embodiment. In the functional configuration of Fig. 10, configurations unnecessary for the description of this embodiment are appropriately omitted.

[0066] <<Activity Sensor 9>> First, the activity sensor 9A has a signal transmission unit 61, an acceleration acquisition unit 62, an angular velocity acquisition unit 63, a direction acquisition unit 64, a voice data acquisition unit 65, a vital data acquisition unit 66, a behavior estimation unit 67, a communication unit 68, an interface unit 69, a psychological safety estimation unit 41, and a listening level estimation unit 42. These functional units of the activity sensor 9A are functions or means realized by the microcomputer 701 shown in Fig. 6 executing instructions included in one or more programs installed in the activity sensor 9A. Alternatively, some or all of the functions may be realized by a hardware circuit such as an ASIC.

[0067] The signal transmission unit 61 periodically transmits a signal to the sensor device 14 using radio waves known as UWB. The signal includes a user ID, and the sensor device 14 can detect the position of the user while identifying the user. The user's position information is two-dimensional coordinates (x, y) based on a specific position in the conference room. The user's position information is specified, for example, by coordinates based on the junction of the ends of walls that make up the conference room, or the center of the conference room.

[0068] The acceleration acquisition unit 62 acquires the three-axis acceleration occurring in the activity sensor 9A detected by the acceleration and angular velocity sensor 705. The angular velocity acquisition unit 63 acquires the angular velocity when the activity sensor 9A rotates around the above-mentioned three axes detected by the acceleration and angular velocity sensor 705. The direction acquisition unit 64 acquires the direction in which the activity sensor 9A is facing within a range of 0 to 360° with the north as the reference direction detected by the geomagnetic sensor 706, for example.

[0069] The voice data acquisition unit 65 acquires voice data by converting the voice acquired by the microphone into a digital signal by PCM conversion or the like. The voice data is transmitted to the information processing system 10, which can then perform voice recognition on the voice data and convert it into speech data (text data). The information processing system 10 transmits the speech data to the activity sensor 9A. The conversion from voice data to speech data may be performed internally by the activity sensor 9A.

[0070] The vital data acquiring unit 66 acquires vital data detected by the vital sensor 704. The vital data is, for example, heart rate, pulse rate, blood pressure (diastolic and diastolic), body temperature, saturated oxygen concentration, amount of sweating, subjective symptoms (level of consciousness), etc., and any information that can be detected from the user may be included in the vital data.

[0071] The behavior estimation unit 67 estimates the behavior (movement, posture) of the user based on the acceleration and angular velocity detected by the activity sensor 9A and the acceleration and angular velocity detected by the activity sensor 9B. Examples of behavior include nodding, tilting the head, shaking the head, etc., and examples of posture include leaning back, leaning forward, hunched back, etc. A method of detecting behavior will be described later. The behavior estimation unit 67 can also estimate the face direction based on the direction. The behavior estimation unit 67 may also be included in the information processing system 10. The information processing system 10 estimates the behavior based on the acceleration, angular velocity, direction, etc. of the chest and neck transmitted from the activity sensor 9A.

[0072] The behavior estimation unit 67 may estimate the behavior by machine learning. In this case, it is assumed that the correspondence between the behavior and signals such as acceleration, angular velocity, and direction is obtained in advance by machine learning or the like. In other words, the behavior estimation unit 67 outputs the behavior corresponding to one or more of the acceleration, angular velocity, and direction.

[0073] Machine learning is a technology for making a computer acquire a human-like learning ability, in which the computer autonomously generates an algorithm required for judgment such as data identification from learning data previously acquired, and applies this to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods. There is no restriction on the learning method for machine learning. Machine learning methods include perceptron, deep learning, support vector machine, logistic regression, naive Bayes, decision tree, random forest, etc., and are not limited to the methods described in this disclosure.

[0074] For example, deep learning is an algorithm that predicts an output value based on input data, and then adjusts the weights between neural networks using the backpropagation method to reduce errors with the training data.In addition, the gradient boosting decision tree is an algorithm that trains multiple weak classifiers independently using the gradient method, integrates the prediction results of the multiple weak classifiers using majority voting or averaging, and outputs the overall (strong classifier) ​​prediction result.

[0075] The communication unit 68 transmits location information, direction, behavior information regarding the behavior estimated by the behavior estimation unit 67, vital data acquired by the vital data acquisition unit 66, and voice data acquired by the voice data acquisition unit 65 to the sensor device 14 or the information processing system 10.

[0076] The interface unit 69 enables communication with the interface unit 111 on the activity sensor 9B side, so that the activity sensor 9A can receive a signal from the activity sensor 9B. The interface unit 69 also supplies power to the activity sensor 9B.

[0077] The listening level estimation unit 42 estimates the listening level of a plurality of users determined to be a group according to the facial orientation of the users. The listening level estimation unit 42 may be included in the information processing system 10. Whether or not the group is determined to be a group is notified to the activity sensor 9A from the information processing system 10.

[0078] The psychological safety estimation unit 41 estimates a psychological safety level according to vital data of the plurality of users determined to be a group. The psychological safety estimation unit 41 may also take into consideration the time that the plurality of users face each other. The psychological safety estimation unit 41 may be included in the information processing system 10.

[0079] On the other hand, the activity sensor 9B has an interface unit 111, an acceleration acquisition unit 112, a direction acquisition unit 113, an angular velocity acquisition unit 114, and a vital data acquisition unit 115. These functions may be similar to those of the activity sensor 9A.

[0080] <<Input devices, output devices>> The sensor device 14 has an output signal transmitting unit 70. The camera 18 has an output signal transmitting unit 80. The microphone 20 has an output signal transmitting unit 90. The information processing terminal 22 has an output signal transmitting unit 91 and an output unit 92. The speaker 16 has an output unit 93.

[0081] The output signal transmitting unit 70 of the sensor device 14 transmits the information received from the activity sensor 9 to the information processing system 10. That is, the output signal transmitting unit 70 transmits the position information of multiple users in the conference room together with their user IDs to the information processing system 10. In addition, the sensor device 14 transmits the direction, behavior information, speech data converted from voice data, and vital data received from the activity sensor 9 to the information processing system 10.

[0082] An output signal transmitting unit 80 of the camera 18 transmits the imaging results of imaging the inside of the conference room to the information processing system 10. An output signal transmitting unit 90 of the microphone 20 transmits to the information processing system 10 electrical signals converted from the voices of multiple users in the conference room.

[0083] Furthermore, the output signal transmitting section 91 of the information processing terminal 22 transmits to the information processing system 10 an electrical signal that is converted by the microphone 615 from the voice of the user operating the information processing terminal 22 .

[0084] The output unit 92 of the information processing terminal 22 outputs sounds such as environmental sounds in accordance with the sound data received from the information processing system 10. The output unit 93 of the speaker 16 outputs sounds such as environmental sounds in accordance with the sound data received from the information processing system 10.

[0085] 10 are examples of input devices, and the output devices 92 and 93 are examples of output devices.

[0086] <<Information Processing System>> The information processing system 10 has a communication unit 31, a storage unit 32, a notification unit 33, a user group estimation unit 34, and a memory unit 50. The memory unit 50 stores activity information 51 and behavior and speech information 52. These functional units of the information processing system 10 are functions or means realized by a CPU 501 shown in Fig. 4 executing instructions included in one or more programs installed in the information processing system 10. The memory unit 50 is constructed in a RAM 503, HD04, etc. shown in Fig. 4.

[0087] The communication unit 31 of the information processing system 10 receives location information, the direction detected by the activity sensor 9, behavior information, speech data converted from voice data, vital data, and the like from the output signal transmission unit 70 of the sensor device 14. The communication unit 31 receives the image capturing result of capturing an image of the inside of the conference room from the output signal transmission unit 80 of the camera 18. The communication unit 31 receives an electrical signal converted from the voices of multiple users in the conference room from the output signal transmission unit 90 of the microphone 20. The communication unit 31 receives an electrical signal converted by the microphone 615 from the voice of the user operating the information processing terminal 22 from the output signal transmission unit 91 of the information processing terminal 22. The communication unit 31 also receives an operation signal accepted by the information processing terminal 22 from the user.

[0088] The storage unit 32 stores, as activity information 51, the location information received by the communication unit 31, the direction detected by the activity sensor 9, behavior information, speech data converted from voice data, vital data, and the like.

[0089] The notification unit 33 notifies the activity sensor 9A that the user is estimated to belong to the user group. Furthermore, the notification unit 33 notifies one user of the behavior of the other user who is estimated to belong to the user group, for example, notifying the other user that the one user has nodded.

[0090] The user group estimation unit 34 estimates a group of two or more users based on the location information and direction of the users. The user group estimation unit 34 identifies two or more users who are facing each other or talking face to face, and estimates that these users are a group.

[0091] The storage unit 50 stores activity information 51 and behavior speech information 52 in a table format, for example, as shown in Figures 11 and 12. Note that the activity information 51 and behavior speech information 52 do not necessarily need to be in the table format shown in Figures 11 and 12, and it is sufficient if similar information can be stored and managed.

[0092] Fig. 11 is a diagram showing an example of the activity information 51. The activity information 51 in Fig. 11 has items such as time, location information, direction, behavior, posture, heart rate, speech data, reinforcement, psychological safety, and listening level. The time item is Japan Standard Time on the relevant date. The activity sensor 9A transmits behavior information, etc., for example, every second, so it is recorded every second. One second is just an example, and may be automatically changed depending on the remaining battery power, etc. In the location information item, the location (coordinates) of the user detected by communication between the UWB module 703 and the sensor device 14 is stored. The direction item stores the direction the user is facing. The direction in Figure 11 is the horizontal angle, but the latitudinal (up and down) angle calculated by integrating the angular velocity of the neck rotation around axis 1 may also be detected. The behavior item is the behavior taken by the user, and behavior information estimated by the behavior estimation unit 67 is stored. The heart rate item is one of the vital data and is the user's heart rate. The heart rate may be, for example, a value detected in the past 10 seconds or so converted into a value per minute. The speech data item stores text data converted from the voice data spoken by the user. The item "Reinforcement" indicates whether the detected behavior was reinforced by speech data. Details will be explained in Figure 12. The psychological safety item is, for example, psychological safety detected based on vital data (heart rate) during communication within a group. The listening level item is the level of listening detected based on the direction of a user's face, for example, during group communication.

[0093] 12 shows an example of the configuration of the behavior speech information 52. The behavior speech information 52 is information that associates the type of behavior with speech data that reinforces the behavior. The activity sensor 9A has a microphone 702, and since the activity sensor 9A is located on the chest, it can acquire speech with high sensitivity. For example, if the user mutters "yes" while nodding, the activity sensor 9A can also acquire a soft voice at the same time because the microphone 702 is located near the mouth.

[0094] It can be said that positive speech data such as "yes" reinforces the positive behavior of nodding. The behavior speech information 52 is downloaded to the activity sensor 9A, and when the behavior and speech data are associated with the behavior speech information 52, the behavior estimation unit 67 records in the activity information 51 that the behavior of nodding has been reinforced (high probability). It is preferable that the behavior estimation unit 67 records the degree of reinforcement taking into consideration not only the speech data but also the volume. Depending on whether the volume is greater than the threshold, the behavior estimation unit 67 not only records "reinforced" (Y), but also records the strengths 1, 2, and 3 of the behavior. For example, strength 1 is no reinforcement, strength 2 is reinforced but the volume is below the threshold, and strength 3 is reinforced and the volume is above the threshold.

[0095] Similarly, in the behavior speech information 52, negative behaviors such as "tilting one's head" are associated with negative speech data such as "Hmm." Behavior expressing a question such as "shaking one's head" is associated with questioning speech data such as "Is that so."

[0096] The determination of whether or not a behavior has been reinforced and the degree of reinforcement can also be performed by the information processing system 10. Since the information processing system 10 receives speech data together with the behavior from the activity sensor 9A, the storage unit 32 can determine whether or not a behavior has been reinforced and the degree of reinforcement by referring to the behavior speech information 52.

[0097] <Direction, group decision> The user's direction will be described with reference to Fig. 13. Fig. 13 is a diagram for explaining the user's direction. In Fig. 13, the direction is shown as an angle clockwise, with the north direction as the reference, as an example. Therefore, the direction 204 is 90 degrees, and the direction 205 is 270 degrees. When the direction reaches 360 degrees, it returns to 0 degrees. The direction changes within the range from 0 degrees to 360 degrees. Fig. 13 shows the horizontal direction, but the user's direction in the elevation angle direction is also detected.

[0098] Next, the grouping of users will be described with reference to Fig. 14. Fig. 14 is a diagram for explaining the grouping of users. A group refers to two or more users facing each other, or two or more users having a conversation while facing each other. Furthermore, facing each other refers to a state in which the distance between users is within a threshold and the users are facing each other in the same direction.

[0099] Fig. 14(a) shows a state in which two users A and B face each other. As shown in Fig. 14(a), when sectors 211 and 212 set for each user overlap, they are facing each other. The threshold distance at which it is determined that they are facing each other is determined appropriately (for example, about 1 meter).

[0100] The user group estimation unit 34 can estimate whether two or more users facing each other, or two or more users having a conversation while facing each other, correspond to close contacts of an infectious disease, in the same way as in the case of determining a group. A close contact is, for example, a person who touches with his / her hands without taking necessary infection prevention measures, or who has been in contact with each other for 15 minutes or more within reach of each other (within about 1 meter).

[0101] The centers of sectors 211 and 212 are directions 213 and 214. The angles that define the extent of sectors 211 and 212 are pre-determined. The angles that define the extent may be such that they are presumed to face each other. An example of determining whether or not they face each other will be described assuming that the angles that define the extent are 30 degrees forward and backward from directions 213 and 214 as the center. If direction 213 is 90 degrees, the direction that is completely opposite to it is 180 degrees opposite, so 90+180=270 degrees. Therefore, if direction 214 is 240 (=270-30) to 300 (=270+30) degrees, it is determined that users A and B are facing each other.

[0102] Fig. 14(b) shows three users A to C facing each other. The method of determining whether users are facing each other is the same when there are three or more users, and they are facing each other when sectors 215 to 217 set and displayed for each user overlap. Sector 215 of user A and sector 216 of user B, sector 215 of user A and sector 217 of user C, and sector 216 of user B and sector 217 of user C all overlap, so users A to C are facing each other.

[0103] Although the case of three users has been described in Fig. 14, the same applies when there are four or more users. As the number of users who converse increases, the distance between the users also increases, so the threshold value for the distance between the users may also be increased according to the number of users.

[0104] It is not necessary to determine whether the sectors of the three people overlap each other. For example, as shown in Fig. 14(c), there may be a case where sector 215 of user A and sector 216 of user B overlap, and sector 215 of user A and sector 217 of user C overlap, but sector 216 of user B and sector 217 of user C do not overlap. In this case, since users A and B and users A and C face each other, the user group estimation unit 34 may also consider users B and C to face each other.

[0105] Also, as shown in Fig. 15, if another user D exists between the two users, it is not determined that users A and B are facing each other. Fig. 15 is a diagram explaining a method for determining whether users are facing each other when another user exists. In this case, it is determined that users A and D, or users A and B, are facing each other based on the direction of user D.

[0106] <Nodding inference> Hereinafter, a method for estimating a nod using one or more of acceleration and angular velocity will be described with reference to FIGS.

[0107] Fig. 16 shows a schematic diagram of the body movement when the user nods. Fig. 17 shows the change in acceleration detected when the body moves as shown in Fig. 16. The upper part of Fig. 17 shows the acceleration detected by the activity sensor 9A (an example of the first acceleration), and the lower part shows the acceleration detected by the activity sensor 9B (an example of the second acceleration).

[0108] FIG. 16(a) shows an upright state before nodding. In this embodiment, the acceleration and angular velocity detected by the activity sensors 9A and 9B in the upright state are used as reference values. In the upright state, gravitational acceleration (converted to a reference value) acts downward on the activity sensor 9A, and acceleration of a reference value (almost zero but converted to a reference value) acts forward on the activity sensor 9B. Note that the vertical acceleration is positive downward, the front-back acceleration is positive in front of the user, and the left-right acceleration is positive to the right as seen from the user.

[0109] Figure 16(b) shows the state where the user nods most strongly. When changing from the state of Figure 16(a) to the state of Figure 16(b), the neck moves forward, so that activity sensor 9A, which is suspended in mid-air, moves downward due to gravity. Activity sensor 9B moves forward in conjunction with the movement of the neck. Therefore, downward acceleration due to the nod is added to activity sensor 9A, and forward acceleration acts on activity sensor 9B. Signal of activity sensor 9A: Downward (downward acceleration) minimum value P1 Signal from activity sensor 9B: Maximum value P2 upward (forward acceleration) Note that the minimum values ​​P1, P4 and the maximum values ​​P2, P3 are merely examples, as they differ depending on the signal. For convenience of explanation, the following explanation will use the same signals as in Fig. 16 to show the minimum values ​​P1, P4 and the maximum values ​​P2, P3, but the signals themselves may differ depending on the sensor.

[0110] Fig. 16(c) shows the state where the user returns to an upright state after nodding. When changing from Fig. 16(b) to Fig. 16(c), the neck moves backward, so activity sensor 9A, which is suspended in mid-air, moves upward. Activity sensor 9B moves backward in conjunction with the movement of the neck. Therefore, an acceleration acts on activity sensor 9A in the opposite direction to the gravitational acceleration, and an acceleration acts backward on activity sensor 9B. Signal of activity sensor 9A: Upward (upward acceleration) maximum value P3 Signal of activity sensor 9B: Downward (acceleration backward) minimum value P4 Therefore, by having the behavior estimating unit 67 monitor the change in acceleration occurring in the activity sensors 9A and 9B as shown in FIGS. 16(a) to (c), it is possible to estimate that a motion such as a nodding has occurred.

[0111] FIG. 18 is a flowchart illustrating a process in which the behavior estimation unit 67 estimates that a behavior of a nodding has occurred.

[0112] The acceleration acquisition units 62, 112 of the activity sensors 9A, 9B repeatedly acquire the acceleration in the vertical direction and the front-rear direction, respectively (S11).

[0113] The behavior estimation unit 67 determines whether the amount of change in acceleration of the activity sensor 9A exceeds a threshold a1 and whether the amount of change in acceleration of the activity sensor 9B exceeds a threshold a2 (S12). This determination is for detecting whether the user has started to nod. If the determination in step S12 is No, the process returns to step S11.

[0114] If the determination in step S12 is Yes, the behavior estimation unit 67 determines whether the acceleration of the activity sensors 9A, 9B exceeds the thresholds b1, b2 within a certain period of time after the determination in step S12 is Yes (S13). Therefore, it is determined whether the minimum value P1 is equal to or smaller than the threshold b1, and it is determined whether the maximum value P2 exceeds the threshold b2. If the determination in step S13 is No, the process returns to step S11.

[0115] If the determination in step S13 is Yes, the behavior estimation unit 67 determines whether the acceleration of the activity sensors 9A, 9B returns to the reference value within a certain period of time after the determination in step S13 is Yes (S14). This is because, due to the characteristics of the nodding action, the position of the neck returns to the original position, so that it is possible to detect that the user has nodded, rather than keeping their head facing down. In other words, it is determined whether the signal has returned to the reference value via the maximum value P3 and the minimum value P4. The maximum value P3 and the minimum value P4 may be compared with threshold values. If the determination in step S14 is No, the process returns to step S11.

[0116] If the determination in step S14 is Yes, the behavior estimation unit 67 determines whether the acceleration of the activity sensors 9A, 9B has not changed for a certain period of time or more (S15). This is a determination to more reliably detect the end of nodding, that is, that nodding has occurred. The determination in step S15 is a determination to reliably detect nodding, so it may not be necessary. If the determination in step S15 is No, the process returns to step S11.

[0117] If the determination in step S15 is Yes, the behavior estimation unit 67 estimates that the user has performed a nodding behavior (S16).

[0118] In this way, the behavior estimation unit 67 can estimate that the user has nodded, based on the acceleration detected by the two activity sensors 9A and 9B.

[0119] The user's activity sensor 9A may output that the user has nodded. When the activity sensor 9A indicates that the user has nodded by blinking an LED light or the like, surrounding users can understand that consent has been obtained.

[0120] Also, as shown in Fig. 19, when a user group is formed, the information processing system 10 can notify the speaker that the attentive listener has nodded. Fig. 19 is a sequence diagram for explaining a process in which the information processing system 10 notifies the speaker that the attentive listener has nodded. For example, in a certain group, it is assumed that user A is a speaker and user B is a listener.

[0121] S201: The user group estimation unit 34 of the information processing system 10 detects the user group of the speaker and the listener.

[0122] S202, 203: The notification unit 33 notifies, for example, the activity sensor 9A of each user in the group to light up the LED light 708 in the same color indicating that the users belong to the same group.

[0123] S204, 205: The activity sensors 9A of the users A and B light up in the same color.

[0124] S206: User B nods. The communication unit 68 transmits the behavior (nod) to the information processing system 10 in a communication process repeated, for example, every second. The communication unit 68 transmits information that the activity sensors 9A and 9B can detect (position information, direction, behavior, posture, heart rate, etc.) and user identification information, etc. to the information processing system 10.

[0125] S207: When user B nods, first, the activity sensor 9A of user B blinks.

[0126] S208: Since the information that user B has nodded has been transmitted to the information processing system 10, the storage unit 32 stores the time, location information, direction, behavior, posture, heart rate, etc. in the activity information 51. The notification unit 33 of the information processing system 10 notifies the speaker that there is a user who has nodded. The notification unit 33 notifies the activity sensor 9A (an example of a second sensor device) of user A that a user in the group has nodded. Since the identification information of user B is also notified to the information processing system 10, the notification unit 33 can also notify user A which user has nodded.

[0127] S209: User A can know that user B has nodded by the vibration of user A's activity sensor 9B. In other words, user A can know that user B has nodded by both the lighting of user B's activity sensor 9A and the vibration of the activity sensor 9B worn by user A. User A's activity sensor 9B may also output a message or a melody to that effect from a speaker. This improves the trust and sense of efficacy between employees.

[0128] <<Nodding Estimation Using Angular Velocity>> A method of estimating a nod using the angular velocity detected by the activity sensor 9B is described in Fig. 20. Fig. 20 shows a schematic diagram of the body movement when the user nods. In the description of Fig. 20, the differences from Fig. 16 are mainly described.

[0129] Fig. 21 shows the changes in acceleration and angular velocity detected when the body moves as shown in Fig. 20. The upper part of Fig. 21 shows the acceleration detected by activity sensor 9A (an example of a first acceleration), and the lower part shows the angular velocity detected by activity sensor 9B. Note that since the angular velocity signal of activity sensor 9B in Fig. 21 shows the same tendency as the acceleration of activity sensor 9B (even though the absolute values ​​are different), the angular velocity signal of activity sensor 9B in Fig. 21 is the same as the acceleration signal of activity sensor 9B in Fig. 17.

[0130] FIG. 20(a) may be the same as FIG. 16(a). FIG. 20(b) shows the state where the user nods the most. When changing from FIG. 20(a) to the state shown in FIG. 20(b), the neck moves forward, so that activity sensor 9A, which is suspended in mid-air, moves downward due to gravity. On the other hand, when the user nods, the neck rotates around axis 1, so activity sensor 9B detects the angular velocity in the direction of rotation around axis 1. Therefore, the downward acceleration of the nod is added to activity sensor 9A, and activity sensor 9B detects the angular velocity (assumed to be positive) of rotation around axis 1. Signal of activity sensor 9A: Downward (downward acceleration) minimum value P1 Signal of activity sensor 9B: Maximum P2 upward (positive angular velocity around axis 1) Figure 20(c) shows the state where the user returns to an upright position after nodding. When changing from the state of Figure 20(b) to the state of Figure 20(c), the neck moves backward, so activity sensor 9A, which is suspended in mid-air, moves upward. As the neck rotates in the opposite direction around axis 1, activity sensor 9B detects the angular velocity of rotation around axis 1 in the opposite direction. Therefore, an acceleration acts on activity sensor 9A in the opposite direction to the gravitational acceleration, and activity sensor 9B detects the negative angular velocity of rotation around axis 1. Signal of activity sensor 9A: Upward (upward acceleration) maximum value P3 Signal of activity sensor 9B: minimum value P4 downward (negative angular velocity around axis 1) Therefore, as shown in FIGS. 20(a) to (c), by monitoring the changes in acceleration and angular velocity occurring in the activity sensors 9A and 9B, the behavior estimation unit 67 can estimate that a motion such as a nodding has occurred.

[0131] FIG. 22 is a flowchart illustrating a process in which the behavior estimation unit 67 estimates that a behavior of a nodding has occurred.

[0132] The acceleration acquisition section 62 of the activity sensor 9A repeatedly acquires the acceleration in the vertical direction, and the angular velocity acquisition section 114 of the activity sensor 9B repeatedly acquires the angular velocity of the first axis (S21).

[0133] The behavior estimation unit 67 determines whether the amount of change in the acceleration of the activity sensor 9A exceeds a threshold a1 and whether the amount of change in the angular velocity of the activity sensor 9B exceeds a threshold a2 (S22). This determination is for detecting whether the user has started to nod. If the determination in step S22 is No, the process returns to step S21.

[0134] If the determination in step S22 is Yes, the behavior estimation unit 67 determines whether the acceleration of the activity sensor 9A exceeds a threshold b1 within a certain period of time after the determination in step S22 is Yes, and whether the angular velocity of the activity sensor 9B exceeds a threshold b2 within a certain period of time after the determination in step S22 is Yes (S23). Therefore, it is determined whether the minimum value P1 is equal to or smaller than the threshold b1, and it is determined whether the maximum value P2 exceeds the threshold b2. If the determination in step S23 is No, the process returns to step S21.

[0135] If the determination in step S23 is Yes, the behavior estimation unit 67 determines whether the acceleration of the activity sensor 9A and the angular velocity of the activity sensor 9B return to the reference value within a certain period of time after the determination in step S23 is Yes (S24). This is because, due to the action of nodding, the position of the neck returns to the original position, so that it is possible to detect that the user has nodded, rather than keeping their head facing down. In other words, it is determined whether the signal has returned to the reference value via the maximum value P3 and the minimum value P4. The maximum value P3 and the minimum value P4 may be compared with a threshold value. If the determination in step S24 is No, the process returns to step S21.

[0136] If the determination in step S24 is Yes, the behavior estimation unit 67 determines whether the acceleration of the activity sensor 9A and the angular velocity of the activity sensor 9B have not changed for a certain period or more (S25). This is a determination to more reliably detect the end of the nodding, that is, the nodding. The determination in step S25 is a determination to reliably detect the nodding, so it is not necessary. If the determination in step S25 is No, the process returns to step S21.

[0137] If the determination in step S25 is Yes, the behavior estimation unit 67 estimates that the user has performed a nodding behavior (S26).

[0138] In this way, the behavior estimation unit 67 can estimate that the user has nodded, based on the acceleration and angular velocity detected by the two activity sensors 9A and 9B.

[0139] <<Estimating nodding using signal frequency components>> The signal waveform shown in FIG. 17 is a characteristic waveform that occurs when the user nods. Therefore, the activity sensor 9A can estimate that the user has nodded by analyzing the frequency components of such a waveform. The behavior estimation unit 67 calculates a frequency spectrum by, for example, Fourier transforming the signal. Since it is known that a peak appears at a specific frequency when the user nods, the behavior estimation unit 67 can estimate that the user has nodded when the frequency spectrum exceeds a threshold value in a certain frequency range (range of frequencies A to B).

[0140] FIG. 23 is a flowchart illustrating a process in which the behavior estimation unit 67 estimates that a behavior such as a nodding has occurred based on a frequency spectrum.

[0141] The acceleration acquisition units 62, 112 of the activity sensors 9A, 9B repeatedly acquire the acceleration in the vertical direction and the front-rear direction, respectively (S31).

[0142] The behavior estimation unit 67 determines whether the amount of change in the acceleration of the activity sensor 9A exceeds a threshold a1 and whether the amount of change in the angular velocity of the activity sensor 9B exceeds a threshold a2 (S32). This determination is for detecting whether the user has started to nod. If the determination in step S32 is No, the process returns to step S31.

[0143] If the determination in step S32 is Yes, the behavior estimation unit 67 samples the acceleration of the activity sensors 9A and 9B for a certain period of time, and creates a frequency spectrum by Fourier transform (S33).

[0144] The behavior estimation unit 67 determines whether or not both of the frequency spectra of the acceleration signals detected by the activity sensors 9A and 9B in a certain frequency range (range of frequencies A to B) exceed a threshold value (S34). If the determination in step S34 is No, the process returns to step S31.

[0145] If the determination in step S34 is Yes, the behavior estimation unit 67 estimates that the user has nodded (S35).

[0146] In this way, the behavior estimation unit 67 can estimate that the user has nodded by performing frequency analysis on the acceleration signals detected by the two activity sensors 9A and 9B.

[0147] Note that behavior estimation unit 67 may perform nodding estimation based on the frequency spectrum of the angular velocity of rotation about axis 1, instead of the frequency spectrum of the acceleration. Also, nodding estimation may be performed based on the frequency spectrum of the acceleration for activity sensor 9A, and based on the frequency spectrum of the angular velocity for activity sensor 9B.

[0148] <Posture Estimation> Next, a method for estimating a user's posture will be described with reference to Fig. 24 to Fig. 29. Fig. 24 shows a leaning back posture and a leaning forward posture. Fig. 24 describes a method for estimating the posture of the upper body based on the acceleration detected by the activity sensors 9A and 9B.

[0149] FIG. 24(a) shows activity sensors 9A and 9B worn by a user in an upright position. FIG. 24(b) shows a user leaning back on a chair. This position is often a posture in which the user is relaxed or engaged in deep thought. Because the posture is tilted, the downward acceleration detected by activity sensor 9A is smaller than the acceleration of gravity. On the other hand, because the neck is at approximately the same angle as in an upright position, the downward acceleration detected by activity sensor 9B remains the same as the acceleration of gravity. Therefore, when the difference between the downward acceleration detected by activity sensor 9A and the downward acceleration detected by activity sensor 9B exceeds threshold c, behavior estimation unit 67 detects a leaning back posture.

[0150] FIG. 24(c) shows a state in which the user is sitting in a chair and leaning forward. This state is often a posture in which the user is concentrating (absorbed) on desk work. Although the posture is tilted, the downward acceleration detected by activity sensor 9A is approximately the same as the acceleration of gravity because the user is suspended in mid-air. On the other hand, because the neck is tilted forward, the downward acceleration detected by activity sensor 9B is smaller than the acceleration of gravity. Therefore, when the difference between the downward acceleration detected by activity sensor 9A and the downward acceleration detected by activity sensor 9B exceeds threshold d, behavior estimation unit 67 detects a posture of leaning forward.

[0151] FIG. 25 is a flowchart illustrating the process performed by the behavior estimating unit 67 to estimate that a motion of leaning backward has occurred.

[0152] The acceleration acquisition units 62, 112 of the activity sensors 9A, 9B repeatedly acquire their respective vertical accelerations (S41).

[0153] The behavior estimation unit 67 determines whether the amount of change in the acceleration of the activity sensor 9A exceeds the threshold a1 (S42). It may also determine whether the acceleration of the activity sensor 9B has changed. This determination is for detecting whether the user has started to lean backward. If the determination in step S42 is No, the process returns to step S41.

[0154] If the determination in step S42 is Yes, the behavior estimation unit 67 determines whether the difference between the acceleration of the activity sensor 9A and the acceleration of the activity sensor 9B exceeds the threshold c within a certain period of time after the determination in step S42 is Yes (S43). Since the acceleration of the activity sensor 9A is less than the acceleration of the activity sensor 9B, it is determined whether the "acceleration of the activity sensor 9B - the acceleration of the activity sensor 9A" exceeds the threshold c. If the determination in step S43 is No, the process returns to step S41.

[0155] If the determination in step S43 is Yes, the behavior estimation unit 67 determines whether the acceleration of the activity sensors 9A and 9B has not changed for a certain period of time or more (S44). This is a determination to detect that the backward leaning state has continued. If the determination in step S44 is No, the process returns to step S41.

[0156] If the determination in step S44 is Yes, the behavior estimation unit 67 estimates that the user has performed a leaning-back behavior (S45). The communication unit 68 notifies the information processing system 10 in the repeated communication process that a leaning-back posture has been detected.

[0157] In this way, the behavior estimation unit 67 can estimate that the user has assumed a leaning backward posture, based on the acceleration detected by the two activity sensors 9A and 9B.

[0158] The user's activity sensor 9A may output the fact that the user has taken the leaning-back posture by itself. When the activity sensor 9A indicates that the user has taken the leaning-back posture by, for example, vibration or sound, the user can know that he or she is relaxed.

[0159] FIG. 26 is a flowchart illustrating the process performed by the behavior estimation unit 67 to estimate that a forward leaning motion has occurred.

[0160] The acceleration acquisition units 62, 112 of the activity sensors 9A, 9B repeatedly acquire their respective vertical accelerations (S51).

[0161] The behavior estimation unit 67 determines whether the amount of change in the acceleration of the activity sensor 9B exceeds the threshold a2 (S52). It may also determine whether the acceleration of the activity sensor 9A has changed. This determination is for detecting whether the user has begun to lean forward. If the determination in step S52 is No, the process returns to step S51.

[0162] If the determination in step S52 is Yes, the behavior estimation unit 67 determines whether the difference between the acceleration of the activity sensor 9A and the acceleration of the activity sensor 9B exceeds the threshold d within a certain period of time after the determination in step S52 is Yes (S53). Since the acceleration of the activity sensor 9A is greater than the acceleration of the activity sensor 9B, it is determined whether the "acceleration of the activity sensor 9A - the acceleration of the activity sensor 9B" exceeds the threshold d. If the determination in step S53 is No, the process returns to step S51.

[0163] If the determination in step S53 is Yes, the behavior estimation unit 67 determines whether the acceleration of the activity sensors 9A and 9B has not changed for a certain period of time or more (S54). This is a determination to detect that the forward leaning state has continued. If the determination in step S54 is No, the process returns to step S51.

[0164] If the determination in step S54 is Yes, the behavior estimation unit 67 estimates that the user has performed a forward leaning behavior (S55). The communication unit 68 notifies the information processing system 10 in the repeated communication process that a forward leaning posture has been detected.

[0165] In this way, the behavior estimation unit 67 can estimate that the user has assumed a forward leaning posture, based on the acceleration detected by the two activity sensors 9A and 9B.

[0166] The user's activity sensor 9A may output the fact that the user has taken a forward leaning posture by itself. When the activity sensor 9A indicates that the user has taken a forward leaning posture by vibration or sound, the user can understand that he or she is concentrating.

[0167] Although detection of posture changes in the front-back direction has been described in Fig. 24, as shown in Fig. 27, the behavior estimation unit 67 can also detect posture changes in the left-right direction. Fig. 27 shows a state in which the user is leaning to the left or right side. Fig. 27(a) shows an upright state, Fig. 27(b) shows a state in which the user is leaning to the left side, and Fig. 27(c) shows a state in which the user is leaning to the right side.

[0168] The left and right leaning postures are often relaxed or spaced out. Since the chest is tilted, the downward acceleration detected by the activity sensor 9A is smaller than the gravitational acceleration. Since the neck is upright, the downward acceleration detected by the activity sensor 9A is approximately the same as the gravitational acceleration. Therefore, the behavior estimation unit 67 estimates the left and right leaning posture when the "difference between the downward acceleration detected by the activity sensor 9B and the downward acceleration detected by the activity sensor 9B" exceeds the threshold value. Note that the left and right leaning postures can be distinguished from the backward leaning in FIG. 24 by the fact that the acceleration in the left and right direction increases in the left and right leaning postures.

[0169] <<Hunched back estimation>> The estimation of hunched posture will be described with reference to Fig. 28. Fig. 28(a) shows an upright state, and Fig. 28(b) shows a hunched posture. In the case of a hunched posture, the activity sensor 9A on the chest is suspended in mid-air, so the downward acceleration detected by the activity sensor 9A is approximately the same as the acceleration of gravity. On the other hand, the neck moves or tilts forward, so the downward acceleration detected by the activity sensor 9B is smaller than the acceleration of gravity. Therefore, when the difference between the downward acceleration detected by the activity sensor 9A and the downward acceleration detected by the activity sensor 9B exceeds the threshold value e, the behavior estimation unit 67 detects a hunched posture.

[0170] The difference between this posture and the leaning forward posture is that in the case of the leaning forward posture, the user is sitting, whereas in the case of the hunched posture, the user is standing or walking. Whether the user is sitting or not can be determined by whether the activity sensors 9A and 9B detect a downward acceleration of a certain level or more.

[0171] FIG. 29 is a flowchart illustrating the process performed by the behavior estimation unit 67 to estimate that the posture is hunched.

[0172] The acceleration acquisition units 62, 112 of the activity sensors 9A, 9B repeatedly acquire their respective vertical accelerations (S61).

[0173] The behavior estimation unit 67 determines whether the amount of change in the acceleration of the activity sensor 9B exceeds the threshold a2 (S62). It may also determine whether the acceleration of the activity sensor 9A has changed. This determination is for detecting whether the user has begun to hunch over, but because changes in the hunch over posture are gradual, the determination in step S62 may not be necessary. If the determination in step S62 is No, the process returns to step S61.

[0174] If the determination in step S62 is Yes, the behavior estimation unit 67 determines whether the difference between the acceleration of the activity sensor 9A and the acceleration of the activity sensor 9B exceeds the threshold e within a certain period of time after the determination in step S62 is Yes (S63). Since the acceleration of the activity sensor 9A is greater than the acceleration of the activity sensor 9B, it is determined whether the "acceleration of the activity sensor 9A - the acceleration of the activity sensor 9B" exceeds the threshold e. If the determination in step S63 is No, the process returns to step S61.

[0175] If the determination in step S63 is Yes, the behavior estimation unit 67 determines whether the acceleration of the activity sensors 9A and 9B has not changed for a certain period of time or more (S64). This is a determination to detect whether the hunched posture has continued. If the determination in step S64 is No, the process returns to step S61.

[0176] If the determination in step S64 is Yes, the behavior estimation unit 67 determines whether or not the user is standing (S65). This determination is made based on whether or not the downward acceleration of the activity sensors 9A and 9B indicates a decrease in the threshold value at which the user is considered to have sat down by the start of the processing in Fig. 29. If the determination in step S65 is No, the processing returns to step S61.

[0177] If the determination in step S65 is Yes, the behavior estimation unit 67 estimates that the user has a hunched posture (S66). The communication unit 68 notifies the information processing system 10 that a hunched posture has been detected in the repeated communication process.

[0178] In this way, it can be estimated that the user has adopted a hunched posture based on the acceleration detected by the two activity sensors 9A and 9B.

[0179] The user's activity sensor 9A may output the fact that the user has taken a hunched posture by itself. When the activity sensor 9A outputs the fact that the user has taken a hunched posture, for example, by vibration or sound, the user can know that he or she has taken a hunched posture and that he or she should correct his or her posture.

[0180] <Estimation of head shaking> The estimation of the head shaking action will be described with reference to Figs. 30 and 31. Fig. 30 shows a schematic diagram of the body movement when the user shakes his / her head. First, Fig. 30(a) shows the upright state before shaking the head. Fig. 30(b) shows the state after the user has shaken his / her head to the left. When changing from the state of Fig. 30(a) to Fig. 30(b), the chest remains facing forward, so no angular velocity is generated in activity sensor 9A. On the other hand, activity sensor 9B detects the angular velocity of rotation to the left around axis 3.

[0181] Fig. 30(c) shows a state where the user shakes his / her head to the right. When changing from Fig. 30(b) to Fig. 30(c), the chest remains facing forward, so no angular velocity is generated in activity sensor 9A. On the other hand, activity sensor 9B detects the angular velocity of rotation to the right around axis 3.

[0182] Therefore, behavior estimation unit 67 can detect the head shaking motion based on the difference between the angular velocity of activity sensor 9A and the angular velocity of activity sensor 9B.

[0183] FIG. 31 is a flowchart illustrating the process performed by the behavior estimating unit 67 to estimate that a head shaking motion has occurred.

[0184] The angular velocity acquisition units 63 and 114 of the activity sensors 9A and 9B repeatedly acquire the angular velocity of rotation around the respective axes 3 (S71). The angular velocity acquired by the angular velocity acquisition unit 63 of the activity sensor 9A is an example of a first angular velocity, and the angular velocity acquired by the angular velocity acquisition unit 114 of the activity sensor 9B is an example of a second angular velocity.

[0185] The behavior estimation unit 67 judges whether the difference between the angular velocity of the activity sensor 9A and the angular velocity of the activity sensor 9B exceeds either of the thresholds f and g (S72). Since the angular velocity of the activity sensor 9B can be positive or negative, the threshold f is a threshold for detecting a change in the angular velocity in the positive direction, and the threshold g is a threshold for detecting a change in the angular velocity in the negative direction. If the judgment in step S72 is No, the process returns to step S71.

[0186] If the determination in step S72 is Yes, the behavior estimation unit 67 determines whether the difference between the angular velocity of the activity sensor 9A and the angular velocity of the activity sensor 9B exceeds the thresholds g and f (S73). That is, if the head was swung to the left in step S72, it is determined in S73 whether the head was swung to the right, and if the head was swung to the right in step S72, it is determined in S73 whether the head was swung to the left. If the determination in step S73 is No, the process returns to step S71.

[0187] If the determination in step S73 is Yes, the behavior estimation unit 67 estimates that the user performed a head shaking behavior (S74). The communication unit 68 notifies the information processing system 10 in the repeated communication process that a head shaking behavior has been detected.

[0188] In this way, the behavior estimation unit 67 can estimate that the user has shook his / her head, based on the acceleration detected by the two activity sensors 9A and 9B.

[0189] As with nodding, the LED light 708 of the activity sensor 9 of the user's device lights up and the information processing system 10 notifies the activity sensors 9 of other users in the same user group of the fact. The difference between a nod and a head shake can be distinguished by the color of the light or the way it blinks of the LED light 708, the rhythm of the vibration, the message, etc. User A (the speaker) can understand that user B (the listener) has shown a negative reaction, and can therefore take action such as explaining the situation to user B individually.

[0190] <Head tilting presumption> The estimation of the head tilting action will be described with reference to Figs. 32 and 33. Fig. 32 shows a schematic diagram of the body movement when the user tilts his / her head. First, Fig. 32(a) shows the upright state before the user tilts his / her head. Fig. 32(b) shows the state after the user tilts his / her head. When changing from the state of Fig. 32(a) to the state of Fig. 32(b), the chest does not tilt, so the downward acceleration of activity sensor 9A is approximately the same as the acceleration of gravity. On the other hand, activity sensor 9B tilts to the left together with the neck, so it detects an acceleration smaller than the acceleration of gravity.

[0191] FIG. 33 is a flowchart illustrating the process performed by the behavior estimating unit 67 to estimate that a head tilting motion has occurred.

[0192] The acceleration acquisition units 62, 112 of the activity sensors 9A, 9B repeatedly acquire their respective vertical accelerations (S81).

[0193] The behavior estimation unit 67 determines whether the amount of change in the acceleration of the activity sensor 9B exceeds the threshold a2 (S82). It may also determine whether the acceleration of the activity sensor 9A has changed. This determination is for detecting whether the user has started to tilt his / her head. If the determination in step S82 is No, the process returns to step S81.

[0194] If the determination in step S82 is Yes, the behavior estimation unit 67 determines whether the difference between the acceleration of the activity sensor 9A and the acceleration of the activity sensor 9B exceeds the threshold h within a certain period of time after the determination in step S82 is Yes (S83). Since the acceleration of the activity sensor 9A is greater than the acceleration of the activity sensor 9B, it is determined whether the acceleration of the "acceleration of the activity sensor 9A - the activity sensor 9B" exceeds the threshold h. If the determination in step S83 is No, the process returns to step S81.

[0195] If the determination in step S83 is Yes, the behavior estimation unit 67 determines whether the acceleration of the activity sensors 9A and 9B has not changed for a certain period of time or more (S84). This is a determination to detect whether the tilted head posture has continued. If the determination in step S84 is No, the process returns to step S81.

[0196] If the determination in step S84 is Yes, the behavior estimation unit 67 estimates that the user is in a tilted head position (S85). The communication unit 68 notifies the information processing system 10 in the repeated communication process that the head tilting behavior has been detected.

[0197] In this way, the behavior estimation unit 67 can estimate that the user has tilted his / her head based on the acceleration detected by the two activity sensors 9A and 9B. It is possible to estimate the tilted head state, which is representative of a mental state of dissatisfaction.

[0198] As with nodding, the activity sensor 9 of the user's own device may emit light and the information processing system 10 may notify the activity sensors 9 of other users in the same user group of the head tilting. The difference between a head tilting and a head tilting can be distinguished by the color of the light of the LED light 708, the way it blinks, the rhythm of the vibration, the message, etc. User A (the speaker) can understand that user B (the listener) has shown a reaction that expresses a question, and can therefore respond by explaining the situation to user B individually.

[0199] Note that in step S83, it may be determined that the acceleration of activity sensor 9A does not change, and the angular velocity about axis 2 detected by activity sensor 9B has changed beyond a threshold value.

[0200] <Estimating the face direction of the speaker and listener> The information processing system 10 can estimate the quality of communication by estimating whether the listener is facing the speaker.

[0201] Fig. 34 is a diagram for explaining a method for estimating a face direction. User A is speaking, and users B and C are listening attentively. Users A to C are determined to be a user group. User C turns his body and neck toward user A, while user B turns only his neck toward user A. In such a case, listening level estimation unit 42 can determine that user C has a higher listening level.

[0202] Whether each user is turning only their head or their body and neck can be determined by the direction. That is, when the direction detected by the activity sensor 9A and the direction detected by the activity sensor 9B are the same, it can be determined that the degree of attentive listening is high.

[0203] FIG. 35 is a flowchart illustrating the process in which the listening level estimation unit 42 estimates the listening level based on the face direction.

[0204] The direction acquisition units 64 and 113 of the activity sensors 9A and 9B repeatedly acquire their respective directions (S91). The direction acquired by the direction acquisition unit 64 of the activity sensor 9A is an example of first direction information, and the direction acquired by the direction acquisition unit 113 of the activity sensor 9B is an example of second direction information.

[0205] The direction may be estimated from a signal from a geomagnetic sensor, or from an angular velocity signal from the acceleration and angular velocity sensors 705 and 712. For example, the direction acquisition units 64 and 113 may determine that the direction in which the user faces immediately after entering the conference room is with their back to the door, and thereafter, the direction can be estimated by integrating the angular velocity.

[0206] The listening level estimation unit 42 judges whether or not the users A to C are judged to belong to a user group (S92). The fact that they are currently judged to belong to a user group is notified by the information processing system 10. If the judgment in step S92 is No, the process returns to step S91.

[0207] If the determination in step S92 is Yes, the listening level estimation unit 42 determines whether the directions detected by the activity sensors 9A and 9B are the same for each user (S93). They do not have to be completely the same, and a difference of about 5 degrees, for example, may be considered to be the same.

[0208] If the determination in step S93 is Yes, the listening level estimation unit 42 determines that the listening level of the user is high (S94). When determining the listening level, the listening level estimation unit 42 may determine whether another user is speaking and whether the user is facing toward the other user. Both of these can be notified by the information processing system 10.

[0209] If the determination in step S94 is No, the listening level estimation unit 42 determines that the listening level of the user is low (S95). The communication unit 68 notifies the information processing system 10 that the listening level is high or low in the repeated communication process.

[0210] In this way, the listening attentiveness estimation unit 42 can estimate the listening attentiveness of the user based on the directions detected by the two activity sensors 9A and 9B, and can infer the quality of communication.

[0211] For example, when user A is speaking and user B is listening attentively, if user B's activity sensor 9A lights up the LED light 908 in a color or blinking pattern according to the degree of listening attentiveness of user B, user A can understand that he is being listened to carefully. Similar to nodding, the information processing system 10 notifies user A's activity sensor 9A of the degree of listening attentiveness of user B, and the activity sensor 9B vibrates with an intensity according to the degree of listening attentiveness to inform user A of the degree of listening attentiveness of user B.

[0212] <Behavior reinforcement using speech data> Behavior reinforcement using speech data will be described with reference to Figs. 36 and 37. Fig. 36 shows a user nodding and saying "yes." A microphone 702 is incorporated in the activity sensor 9A, and since it is close to the mouth, it is possible to acquire voice data with high sensitivity. For example, when a user nods and mutters at the same time, a soft voice can be acquired at the same time because the microphone 702 is close to the mouth. In addition, the degree of behavior reinforcement can be estimated based on the volume of the voice.

[0213] FIG. 37 is a flow chart for explaining behavior reinforcement using speech data.

[0214] First, the behavior estimation unit 67 judges whether or not it has been estimated that a nodding action has occurred (S101). The method of estimating that a nodding action has occurred may be any of the above. If the judgment in step S101 is No, the process ends.

[0215] If the determination in step S101 is Yes, the behavior estimation unit 67 determines whether or not predetermined speech data is detected almost simultaneously with the nod (S102). The predetermined speech data is speech data associated with the estimated behavior in the behavior speech information 52. This determination may be made by the information processing system 10. Also, the behavior estimation unit 67 may estimate the degree of reinforcement based on the volume of the speech.

[0216] If the determination in step S102 is Yes, the behavior estimation unit 67 detects that the nodding is reinforced (S103). If the determination in step S102 is No, the behavior estimation unit 67 detects that the nodding is not reinforced (S104). The communication unit 68 notifies the information processing system 10 of the presence or absence of reinforcement in repeated communication processing.

[0217] In this way, the behavior estimation unit 67 can reinforce the estimated behavior of the user by using the speech data. For example, when user A is speaking and user B is listening attentively, if user B nods strongly and the activity sensor 9A of user B lights up an LED light in a color or blinking pattern corresponding to the degree of reinforcement, user A can understand that consent has been obtained. As with nodding, the information processing system 10 notifies the activity sensor 9A of user A that user B has nodded strongly. User A can understand the degree of reinforcement of user B by the vibration motor 714 of the activity sensor 9A of user A vibrating with a vibration pattern corresponding to the degree of reinforcement or outputting a sound.

[0218] <Psychological safety during behavior using vital data> FIG. 38 is a diagram for explaining that the activity sensor 9B can detect vital data. As shown in FIG. 38, the activity sensor 9B can naturally come into contact with the neck where a large artery passes, so that it is possible to measure the heart rate by the photoplethysmography method and the amount of sweating by a hygrometer. By using the vital data together with the behavior estimated by the behavior estimation unit 67, it is possible to estimate psychological safety in communication with a specific person. For example, in a conversation between users A and B, if user B has a high heart rate, it can be said that the user feels a sense of urgency in communication with user A, and psychological safety is low.

[0219] Psychological safety is an environment with a calm atmosphere where you can be completely open and natural without hiding anything, without being afraid of other people's reactions or feeling embarrassed. Psychological safety is the degree of that. A calm atmosphere means that you can honestly express your thoughts and feelings, and it can be said to be a state in which creative output is expected.

[0220] Fig. 39 is a flow chart illustrating the process of estimating psychological safety based on vital data by the behavior estimation unit 67. Note that the explanation of Fig. 39 will mainly focus on the differences from Fig. 35.

[0221] The direction acquisition units 64 of the activity sensors 9A and 9B repeatedly acquire their respective directions, and the vital data acquisition units 66 and 115 repeatedly acquire vital data (S111). The subsequent steps S112 to S115 may be similar to steps S92 to S95 in FIG.

[0222] In step S116, the psychological safety estimation unit 41 determines whether the user's heart rate exceeds a threshold value i (S116).

[0223] If the determination in step S116 is Yes, the psychological safety estimation unit 41 determines that the user has low psychological safety (S117). In repeated communication processing, the communication unit 68 of the activity sensor 9A notifies the information processing system 10 of the user's identification information and that psychological safety is low. The information processing system 10 can output tones and images that improve psychological safety from an output device, and can encourage creative output. The information processing system 10 may notify the speaker's activity sensor 9A of a user with low psychological safety. The speaker's activity sensor 9A can notify the speaker by vibration or sound. The activity sensor 9A of a user with low psychological safety may cause its own activity sensor 9B to output tones that improve psychological safety from a speaker.

[0224] In this way, the user's psychological safety can be inferred based on the direction and vital data detected by the two activity sensors 9A, 9B, making it possible to perform control that improves psychological safety.

[0225] <Major Effects> The activity sensor 9 of this embodiment is easy for the user to wear as it only needs to be worn around the neck, and since it detects acceleration and angular velocity signals at different parts of the body, it can detect various movements such as nodding, shaking the head, and tilting the head, and various postures such as leaning back or leaning forward.

[0226] <Other application examples> The present invention is not limited to the above-mentioned specifically disclosed embodiment, and various modifications and changes are possible without departing from the scope of the claims. Note that the information processing system 10 described in the present embodiment is only one example, and it goes without saying that there are various system configuration examples according to the application and purpose.

[0227] For example, one user may carry multiple activity sensors 9. This allows the orientation and behavior of the user to be detected more accurately.

[0228] Furthermore, the two activity sensors 9A and 9B of this embodiment can detect behaviors other than those disclosed in the examples, as long as they involve different movements of the torso and neck.

[0229] In addition, the left-right acceleration and the front-back acceleration detected by the activity sensor 9A, and the left-right acceleration and the front-back acceleration detected by the activity sensor 9B may be used to determine the behavior. The three-axis angular velocities detected by the activity sensor 9A and the three-axis angular velocities detected by the activity sensor 9B may be used to determine the behavior.

[0230] In addition, when a model for determining behavior from signals detected by the activity sensors 9A and 9B is created by machine learning, the three-axis acceleration and angular velocity detected by the activity sensor 9A and the three-axis acceleration and angular velocity detected by the activity sensor 9B may all be input. For example, an arbitrary PC creates a model that learns the correspondence between the sensor signals and behavior by deep learning or the like. The activity sensor 9A inputs data points sampled from the sensor signals at regular intervals into a model built by deep learning, and outputs the behavior. The model may be created by a support vector machine, a gradient boosting decision tree, or the like.

[0231] Furthermore, the information processing system 10 may determine the user's behavior by combining the direction and behavior information from the activity sensor 9 with the user's movement detected by the camera 18.

[0232] The configuration example in Fig. 10 and the like is divided according to main functions in order to facilitate understanding of the processing by the information processing system 10. The present invention is not limited by the manner in which the processing units are divided or the names of the processing units. The processing of the information processing system 10 can be divided into more processing units depending on the processing content. Also, it can be divided so that one processing unit includes more processes.

[0233] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0234] The devices described in the example are merely illustrative of one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, the information processing system 10 includes a plurality of computing devices, such as a server cluster, configured to communicate with each other via any type of communication link, including a network, shared memory, etc., to perform the processes disclosed herein.

[0235] Furthermore, the information processing system 10 can combine the disclosed processing steps in various ways. Each element of the information processing system 10 may be integrated into one device, or may be divided into multiple devices. Each process performed by the information processing system 10 may be performed by the information processing terminal 22.

[0236] <Additional Notes> [Appendix 1] A sensor device having a first inertial sensor in contact with a user's torso and a second inertial sensor in contact with a user's neck, a behavior estimation unit that estimates a behavior of the user based on a first signal detected by the first inertial sensor and a second signal detected by the second inertial sensor; A sensor device having the following: [Appendix 2] the first signal is a first acceleration acting in a vertical direction of the user, and the second signal is a second acceleration acting in a forward / backward direction of the user; The sensor device according to claim 1, wherein the behavior estimation unit estimates a nodding behavior of the user based on the first acceleration and the second acceleration. [Appendix 3] the first signal is an acceleration acting on the user in a vertical direction, the second signal is an angular velocity of rotation about an axis in a left-right direction of the user, The sensor device according to claim 1, wherein the behavior estimation unit estimates a nodding behavior of the user based on the acceleration and the angular velocity. [Appendix 4] The sensor device according to claim 1, wherein the behavior estimation unit calculates a frequency spectrum from each of the first signal and the second signal, and estimates the user's nodding behavior based on the intensity of each of the frequency spectra. [Appendix 5] the first signal is a first acceleration acting on the user in a vertical direction, the second signal is a second acceleration acting on the user in a vertical direction, The sensor device according to claim 1, wherein the behavior estimation unit estimates a behavior of the user leaning backward based on the first acceleration and the second acceleration. [Appendix 6] the first signal is a first acceleration acting on the user in a vertical direction, the second signal is a second acceleration acting on the user in a vertical direction, The sensor device according to claim 1, wherein the behavior estimation unit estimates a behavior of the user leaning forward based on the first acceleration and the second acceleration. [Appendix 7] the first signal is a first acceleration acting on the user in a vertical direction, the second signal is a second acceleration acting on the user in a vertical direction, The sensor device according to claim 1, wherein the behavior estimation unit estimates a behavior of the user leaning to the right or left based on the first acceleration and the second acceleration. [Appendix 8] the first signal is a first acceleration acting on the user in a vertical direction, the second signal is a second acceleration acting on the user in a vertical direction, The sensor device according to claim 1, wherein the behavior estimation unit estimates a behavior of the user being standing and hunched based on the first acceleration and the second acceleration. [Appendix 9] the first signal being a first angular velocity of rotation about a vertical axis of the user, and the second signal being a second angular velocity of the user about a vertical axis; The sensor device according to claim 1, wherein the behavior estimation unit estimates a behavior of the user shaking his / her head based on the first angular velocity and the second angular velocity. [Appendix 10] the first signal is a first acceleration acting on the user in a vertical direction, the second signal is a second acceleration acting on the user in a vertical direction, The sensor device according to claim 1, wherein the behavior estimation unit estimates a behavior of the user tilting his / her head based on the first acceleration and the second acceleration. [Appendix 11] The first inertial sensor detects first direction information of the user, and the second inertial sensor detects second direction information of the user; The sensor device communicates with an information processing system via a network; The information processing system includes: a user group estimation unit that groups a plurality of users according to the user position information and the first direction information received from the sensor device; The sensor device according to claim 1, further comprising a listening level estimation unit configured to estimate a listening level for the users included in a group based on the first direction information and the second direction information. [Appendix 12] the first inertial sensor having a microphone; a voice data acquisition unit that acquires speech data of the user acquired by the microphone, When the behavior estimated by the behavior estimation unit and the utterance data acquired by the voice data acquisition unit are registered in behavior utterance information in which a type of behavior is associated with utterance data reinforcing the behavior, The sensor device according to any one of claims 2 to 6, wherein the behavior estimation unit determines that the behavior is estimated to be stronger than when the utterance data associated with the behavior is not acquired. [Appendix 13] The second inertial sensor detects vital data of the user; The sensor device according to claim 10, wherein the behavior estimation unit estimates psychological safety for the users included in a group based on the vital data. [Appendix 14] 13. The sensor device according to claim 1, wherein, when the behavior estimation unit estimates the behavior, the behavior estimation unit outputs information indicating that the behavior has been estimated by using light, sound, or vibration. [Appendix 15] The first inertial sensor detects first direction information of the user, and the second inertial sensor detects second direction information of the user; The sensor device communicates with an information processing system via a network; The information processing system includes: a user group estimation unit that groups a plurality of users according to the user position information and the first direction information received from the sensor device; transmitting behavior information regarding the behavior estimated by the behavior estimation unit to the information processing system; the information processing system transmits the behavior information to a second sensor device in the same group as the sensor device that transmitted the behavior information, the second sensor device being different from the sensor device that transmitted the behavior information; The sensor device of claim 1, wherein the second sensor device outputs light, sound, or vibration in response to the behavior information. [Appendix 16] 16. The sensor device according to any one of claims 1 to 15, having a necklace shape in which the first inertial sensor and the second inertial sensor are communicatively connected via a serial cable. [Appendix 17] 17. The sensor device of claim 16, wherein a dummy having the same shape as the second inertial sensor but no sensor is arranged symmetrically to the second inertial sensor around the neck, and the first inertial sensor, the second inertial sensor and the dummy have a necklace shape. [Explanation of symbols]

[0237] 9 Activity Sensor 10. Information Processing Systems 100 Communication Systems [Prior art documents] [Patent documents]

[0238] [Patent Document 1] JP 2019-017673 A

Claims

1. A sensor device having a first inertial sensor in contact with a user's torso and a second inertial sensor in contact with a user's neck, a behavior estimation unit that estimates a behavior of the user based on a first signal detected by the first inertial sensor and a second signal detected by the second inertial sensor; A sensor device having the following:

2. the first signal is a first acceleration acting in a vertical direction of the user, and the second signal is a second acceleration acting in a forward / backward direction of the user; The sensor device according to claim 1 , wherein the behavior estimator estimates a nodding behavior of the user based on the first acceleration and the second acceleration.

3. the first signal is an acceleration acting on the user in a vertical direction, the second signal is an angular velocity of rotation about an axis in a left-right direction of the user, The sensor device according to claim 1 , wherein the behavior estimator estimates a nodding behavior of the user based on the acceleration and the angular velocity.

4. The sensor device according to claim 1 , wherein the behavior estimator calculates a frequency spectrum from each of the first signal and the second signal, and estimates the user's nodding behavior based on an intensity of each of the frequency spectra.

5. the first signal is a first acceleration acting on the user in a vertical direction, the second signal is a second acceleration acting on the user in a vertical direction, The sensor device according to claim 1 , wherein the behavior estimator estimates that the user is leaning backward based on the first acceleration and the second acceleration.

6. the first signal is a first acceleration acting on the user in a vertical direction, the second signal is a second acceleration acting on the user in a vertical direction, The sensor device according to claim 1 , wherein the behavior estimator estimates a behavior of the user leaning forward based on the first acceleration and the second acceleration.

7. the first signal is a first acceleration acting on the user in a vertical direction, the second signal is a second acceleration acting on the user in a vertical direction, The sensor device according to claim 1 , wherein the behavior estimator estimates the behavior of the user leaning to the right or left based on the first acceleration and the second acceleration.

8. the first signal is a first acceleration acting on the user in a vertical direction, the second signal is a second acceleration acting on the user in a vertical direction, The sensor device according to claim 1 , wherein the behavior estimator estimates that the user is standing and hunched, based on the first acceleration and the second acceleration.

9. the first signal being a first angular velocity of rotation about a vertical axis of the user, and the second signal being a second angular velocity of the user about a vertical axis; The sensor device according to claim 1 , wherein the behavior estimator estimates a head-shaking behavior of the user based on the first angular velocity and the second angular velocity.

10. the first signal is a first acceleration acting on the user in a vertical direction, the second signal is a second acceleration acting on the user in a vertical direction, The sensor device according to claim 1 , wherein the behavior estimator estimates a behavior of the user tilting his / her head based on the first acceleration and the second acceleration.

11. The first inertial sensor detects first direction information of the user, and the second inertial sensor detects second direction information of the user; The sensor device communicates with an information processing system via a network; The information processing system includes: a user group estimation unit that groups a plurality of users according to the user position information and the first direction information received from the sensor device; The sensor device according to claim 1 , further comprising a listening level estimating unit configured to estimate a listening level of the users included in a group based on the first direction information and the second direction information.

12. the first inertial sensor has a microphone; a voice data acquisition unit that acquires speech data of the user acquired by the microphone; When the behavior estimated by the behavior estimation unit and the utterance data acquired by the voice data acquisition unit are registered in behavior utterance information in which a type of behavior is associated with utterance data reinforcing the behavior, The sensor device according to any one of claims 2 to 6, wherein the behavior estimation unit determines that the behavior is estimated to be stronger than when the speech data associated with the behavior is not acquired.

13. The second inertial sensor detects vital data of the user; The sensor device according to claim 10 , wherein the behavior estimation unit estimates psychological safety for the users included in a group based on the vital data.

14. The sensor device according to claim 1 , wherein when the behavior estimation unit estimates the behavior, the behavior estimation unit outputs information indicating that the behavior has been estimated by using light, sound, or vibration.

15. The first inertial sensor detects first direction information of the user, and the second inertial sensor detects second direction information of the user; The sensor device communicates with an information processing system via a network; The information processing system includes: a user group estimation unit that groups a plurality of users according to the user position information and the first direction information received from the sensor device; transmitting behavior information regarding the behavior estimated by the behavior estimation unit to the information processing system; the information processing system transmits the behavior information to a second sensor device in the same group as the sensor device that transmitted the behavior information, the second sensor device being different from the sensor device that transmitted the behavior information; The sensor device according to claim 1 , wherein the second sensor device outputs light, sound, or vibration in response to the behavior information.

16. The sensor device according to claim 1 , wherein the first inertial sensor and the second inertial sensor are communicatively connected to each other via a serial cable in the form of a necklace.

17. The sensor device of claim 16, wherein a dummy having the same shape as the second inertial sensor but no sensor is arranged symmetrically to the second inertial sensor around the neck, and the first inertial sensor, the second inertial sensor and the dummy have the shape of a necklace.

18. A sensor device capable of communicating with an information processing system via a network, a first inertial sensor in contact with the user's torso and a second inertial sensor in contact with the user's neck; a communication unit that transmits a first signal detected by the first inertial sensor and a second signal detected by the second inertial sensor to the information processing system, The communication unit is a sensor device that receives, from the information processing system, behavior information regarding the behavior of the user that is estimated by the information processing system based on the first signal and the second signal.

19. A sensor device having a first inertial sensor in contact with a user's torso and a second inertial sensor in contact with a user's neck, a behavior estimation unit that estimates a behavior of the user based on a first signal detected by the first inertial sensor and a second signal detected by the second inertial sensor; A program to function as a

20. An estimation method performed by a sensor device having a first inertial sensor in contact with a user's torso and a second inertial sensor in contact with a user's neck, the method comprising: An estimation method for estimating a behavior of the user based on a first signal detected by the first inertial sensor and a second signal detected by the second inertial sensor.

21. A communication system capable of communicating with an information processing system via a network, the communication system including a sensor device having a first inertial sensor in contact with a user's torso and a second inertial sensor in contact with a user's neck, the communication system comprising: a behavior estimation unit that estimates a behavior of the user based on a first signal detected by the first inertial sensor and a second signal detected by the second inertial sensor; A communication system having the above configuration.

Images