Psychological State Display System

Abstract

A psychological state display system (1) comprises: a near-infrared light source (100) that outputs output light to irradiate a plurality of persons, the output light including first near-infrared rays having a wavelength of at least 780 nm but below 2500 nm; an imaging device (200) that receives the first near-infrared rays included in the outputted output light and that captures an infrared image of the plurality of persons; a generation device (300) that generates psychological data representing a psychological state by using a trained machine-learning model (321) in which the captured infrared image is input and the psychological state of at least one of the plurality of persons shown in the infrared image is output; and a display device (400) that displays the changes over time of the generated psychological data.

Description

[Technical Field]

[0001] The present invention relates to a psychological state display system. [Background technology]

[0002] 2. Description of the Related Art In recent years, techniques have become known for estimating a person's psychological state (emotions) from an image of the person.

[0003] For example, Patent Document 1 discloses a system (facial expression estimation device) that estimates a facial expression that matches the psychological state and emotions of a person based on the facial expression and actions of the person shown in an image of the person. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-206903 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the system disclosed in Patent Document 1 only reveals the temporary psychological state of the person included in the image. As time passes, the psychological state of a person often changes, and there is a need to visualize these changes.

[0006] Therefore, an object of the present invention is to provide a psychological state display system that can visualize changes in a person's psychological state while suppressing the influence of illumination light. [Means for solving the problem]

[0007] A psychological state display system according to one embodiment of the present invention includes a near-infrared light source that outputs output light having first near-infrared rays with wavelengths of 780 nm or more and less than 2500 nm to a plurality of people; an imaging device that receives the first near-infrared rays contained in the output light and captures infrared images of the plurality of people; a generation device that generates psychological data indicating the psychological states using a trained machine learning model that takes the captured infrared images as input and outputs the psychological state of at least one of the plurality of people shown in the infrared images; and a display device that displays changes over time in the generated psychological data. [Effects of the Invention]

[0008] The psychological state display system of the present invention can visualize changes in a person's psychological state while suppressing the influence of illumination light. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing the configuration of a psychological state display system according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing a characteristic functional configuration of the psychological state display system according to the embodiment. [Figure 3] FIG. 3 is a schematic diagram of a near-infrared light source according to the embodiment and a person viewed in the vertical direction. [Figure 4] FIG. 4 is a schematic side view of FIG. [Figure 5] FIG. 5 is a flowchart showing a processing procedure for the psychological state display system according to the embodiment to train a machine learning model. [Figure 6] FIG. 6 is a flowchart showing a processing procedure for visualizing a person's mental state by the mental state display system according to the embodiment. [Figure 7] FIG. 7 is a diagram showing an infrared image captured by the imaging device according to the embodiment. [Figure 8] FIG. 8 is an example of psychological data based on the infrared image shown in FIG. [Figure 9]FIG. 9 shows another example of psychological data based on the infrared image shown in FIG. [Figure 10] FIG. 10 shows another example of psychological data according to the embodiment. [Figure 11] FIG. 11 is a block diagram showing a characteristic functional configuration of a psychological state display system according to a modified example of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection forms, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present invention. Furthermore, among the components in the following embodiments, components not recited in independent claims will be described as optional components.

[0011] It should be noted that the drawings are schematic diagrams and are not necessarily strict illustrations. In addition, in the drawings, substantially the same components are denoted by the same reference numerals, and overlapping descriptions may be omitted or simplified.

[0012] (Embodiment) [Configuration of the psychological state display system] The configuration of the psychological state display system 1 according to this embodiment will be described.

[0013] Fig. 1 is a diagram showing the configuration of a psychological state display system 1 according to this embodiment. Fig. 2 is a block diagram showing the characteristic functional configuration of the psychological state display system 1 according to this embodiment.

[0014] The psychological state display system 1 outputs output light L having a first near-infrared ray to a plurality of people (for example, several to several tens of people) and captures infrared images of the plurality of people. The psychological state display system 1 generates psychological data indicating the psychological state of at least one of the plurality of people based on the captured infrared image and displays changes in this psychological data over time. For example, a manager of the psychological state display system 1 can understand changes in the psychological states of the plurality of people by looking at the changes in the displayed psychological data over time. In other words, the psychological state display system 1 can visualize changes in the psychological states of people.

[0015] Furthermore, the psychological state display system 1 determines stimuli based on the psychological data and provides the determined stimuli to multiple people. Therefore, the psychological state display system 1 can provide stimuli to people to change their psychological states according to their psychological states, thereby controlling the psychological states of the people.

[0016] The psychological state in this embodiment refers to, for example, happy, sad, energetic, fearful, angry, depressed, awake, absent-minded, interested, tense, concentrated, etc., but is not limited to these.

[0017] As shown in FIGS. 1 and 2, the psychological state display system 1 includes a near-infrared light source 100, an image capturing device 200, a generating device 300, a display device 400, a control device 500, and a stimulation device 600.

[0018] The near-infrared light source 100 outputs output light L having first near-infrared rays that include a wavelength of 780 nm or more and less than 2500 nm to a plurality of people. Using such a near-infrared light source 100, a wide space is irradiated with the first near-infrared rays.

[0019] The imaging device 200 captures infrared images of the plurality of people by receiving the first near-infrared rays contained in the output light L. The imaging device 200 captures infrared images of the plurality of people by receiving the first near-infrared rays reflected by the plurality of people.

[0020] The generating device 300 generates psychological data indicating psychological states using a trained machine learning model 321 that receives a captured infrared image as input and outputs the psychological state of at least one person among multiple people shown in the infrared image. The generating device 300 also determines stimuli to be given to the multiple people based on the generated psychological data. Note that the generating device 300 may utilize, for example, the technology described in Japanese Patent No. 6467965.

[0021] The display device 400 displays the time-varying changes in the generated psychological data.

[0022] The control device 500 controls the stimulation device 600 so that the determined stimulation is given to a plurality of people.

[0023] The stimulation device 600 provides stimulation to multiple individuals.

[0024] The psychological state display system 1 according to this embodiment is a system used in a space such as an entertainment space, a public space, or a learning space, but is not limited to such spaces. At least the near-infrared light source 100, the imaging device 200, and the stimulation device 600 are installed in the space. The generation device 300, the display device 400, and the control device 500 may be installed in the space, or may be installed in a remote location away from the space.

[0025] 1, in a movie theater, which is an example of an entertainment space, output light L is output to a plurality of people watching a movie, infrared images of the plurality of people are taken, and changes in the psychological states of the plurality of people over time are visualized. Furthermore, stimuli are given to the plurality of people.

[0026] Here, the components of the psychological state display system 1 will be described.

[0027] The near-infrared light source 100 is a light source that outputs output light L having a first near-infrared ray to multiple people. The first near-infrared ray is light having a wavelength of 780 nm or more and less than 2500 nm. The first near-infrared ray may include the entire wavelength range of 780 nm or more and less than 2500 nm, but it is sufficient that the first near-infrared ray includes at least a part of the wavelength range. Furthermore, the first near-infrared ray includes a peak wavelength in the wavelength range of 780 nm or more and less than 2500 nm.

[0028] The first near-infrared ray is preferably (i) light having a wavelength of 780 nm or more and less than 1700 nm, and more preferably (ii) light having a wavelength of 780 nm or more and less than 1500 nm. The first near-infrared ray is more preferably (iii) light having a wavelength of 780 nm or more and less than 900 nm, and even more preferably (iv) light having a wavelength of 800 nm or more and less than 900 nm. The first near-infrared ray preferably has a peak wavelength within the wavelength ranges (i), (ii), (iii), and (iv) above.

[0029] In this embodiment, the near-infrared light source 100 outputs output light L having the first near-infrared light and visible light. Visible light is light having a wavelength of 380 nm or more and less than 780 nm. The visible light may be monochromatic light such as red light, green light, or blue light, but in this embodiment, it is white light.

[0030] The near-infrared light source 100 controls the first near-infrared light and the visible light independently, and controls at least one of turning on / off and dimming the first near-infrared light and the visible light. More specifically, the control unit 110 included in the near-infrared light source 100 controls the first near-infrared light and the visible light independently, and controls at least one of turning on / off and dimming the first near-infrared light and the visible light. For example, the near-infrared light source 100 can control at least one of turning on / off and dimming only the first near-infrared light while keeping the emission mode of visible light constant. Similarly, it can control at least one of turning on / off and dimming only the visible light while keeping the emission mode of the first near-infrared light constant.

[0031] The near-infrared light source 100 according to this embodiment includes a near-infrared light source unit 120 that irradiates first near-infrared light and a white light source unit 130 that irradiates visible light (white light). The control unit 110 controls the near-infrared light source unit 120 to irradiate the first near-infrared light and the white light source unit 130 to irradiate visible light (white light).

[0032] The near-infrared light source unit 120 has, as an example, a first solid-state light-emitting element. The first solid-state light-emitting element may be, for example, a laser diode element, but in this case, it is a near-infrared LED element. The near-infrared LED element converts given power into light (first near-infrared light) and irradiates it.

[0033] As another example, the near-infrared light source unit 120 has a second solid-state light-emitting element and a phosphor member. The second solid-state light-emitting element is an element that outputs excitation light, and may be, for example, a laser diode element, but in this case, is a blue LED element. In other words, the second solid-state light-emitting element outputs blue light as excitation light. The phosphor member is a member that outputs wavelength-converted light as the first near-infrared light based on the output excitation light (blue light). The phosphor member is, for example, made of Cr, which is easy to manufacture. 3+ Activated phosphor or rare earth activated phosphor (especially Tm 3+ , Er 3+ , Nd 3+ , and Yb 3+ The phosphor is activated with at least one selected from the group consisting of:

[0034] Next, a case where a near-infrared light source unit 120 having a second solid-state light-emitting element and a phosphor member is used will be described. As described above, the first near-infrared light is light having a wavelength of 780 nm or more and less than 2500 nm. However, when the light has wavelengths (i), (ii), (iii), and (iv), a commercially available near-infrared light source unit 120 can be used. As a commercially available near-infrared light source unit 120, one having a fluorescence peak at any wavelength selected from 780 nm, 850 nm, 940 nm, 980 nm, 1050 nm, 1200 nm, 1300 nm, 1400 nm, 1450 nm, 1500 nm, 1550 nm, 1600 nm, and 1650 nm is used. In addition, the above-mentioned Cr3+ The activated phosphors used have a fluorescent peak at any wavelength selected from the group consisting of 790 nm, 800 nm, 825 nm, 850 nm, 880 nm, 925 nm, 950 nm, 970 nm, and 1035 nm. The rare earth activated phosphors are as follows: For example, Tm 3+ The activated phosphor generally has the property of having a line-shaped fluorescent peak wavelength at wavelengths of 785 nm, 800 nm, and 820 nm. For example, Er 3+ The activated phosphor generally has the property of having a line-shaped fluorescent peak wavelength at wavelengths of 970 nm, 1005 nm, 1470 nm, 1530 nm, 1570 nm, 1615 nm, and 1645 nm. For example, Nd 3+ The activated phosphor has the property of having line-shaped fluorescent peak wavelengths at wavelengths of 880 nm, 935 nm, 1060 nm, 1105 nm, 1335 nm, and 1420 nm. For example, Yb 3+ The activated phosphor has the property of having line-shaped fluorescent peak wavelengths at wavelengths of 970 nm, 1000 nm, and 1025 nm.

[0035] Next, the white light source unit 130 that emits visible light (white light) has, for example, a yellow phosphor member and a blue LED element. A portion of the excitation light, which is blue light output from the blue LED element, is wavelength-converted by the yellow phosphor member to become yellow light. This yellow light is combined with the blue light that has not been wavelength-converted to output white light from the white light source unit 130. Note that the white light source unit 130 is not limited to this configuration as long as it can output visible light such as white light.

[0036] In this way, the near-infrared light source 100 outputs to a plurality of people output light L having the first near-infrared light emitted by the near-infrared light source unit 120 and visible light (white light) emitted by the white light source unit 130. In this embodiment, the output light L continues to be lit, that is, is output continuously, while the psychological state display system 1 generates psychological data and displays the changes in this psychological data over time.

[0037] With this configuration, the imaging device 200 can continuously capture images of multiple people, and the generation device 300 can continuously generate psychological data. Therefore, the display device 400 can continuously display the generated psychological data, making it possible to display changes in the psychological data over time.

[0038] Furthermore, the output of the first near-infrared rays emitted by the near-infrared light source 100 (more specifically, the near-infrared light source unit 120) is 10 W or more and 3 kW or less. For example, the output of the first near-infrared rays may be one of 30 W, 100 W, 300 W, 1 kW, and 3 kW. Note that, in order to output first near-infrared rays with a higher output from the near-infrared light source 100, the near-infrared light source 100 may have a plurality of near-infrared light source units 120. Furthermore, in the present embodiment, the psychological state display system 1 includes one near-infrared light source 100, but is not limited thereto and may include a plurality of near-infrared light sources 100.

[0039] This allows the near-infrared light source 100 to irradiate the first near-infrared light to reach a plurality of people located in an area distant from the near-infrared light source 100. In other words, a psychological state display system 1 is realized that can visualize changes in the psychological states of a plurality of people located in an area distant from the near-infrared light source 100.

[0040] The imaging device 200 is a device, such as an infrared camera, that receives the first near-infrared rays contained in the output light L output by the near-infrared light source 100 and captures infrared images of multiple people. The imaging device 200 may be configured to capture infrared images at all times while the near-infrared light source 100 is outputting the first near-infrared rays.

[0041] When output light L (first near-infrared light) is output to a plurality of people, the first near-infrared light is reflected by the plurality of people. The imaging device 200 captures an infrared image of the plurality of people by receiving the first near-infrared light reflected by the plurality of people. This infrared image is a still image, and the plurality of people are captured. In this embodiment, the infrared image captures at least one of the face and body of each of the plurality of people.

[0042] Furthermore, the imaging device 200 captures infrared images at predetermined time intervals. That is, the imaging device 200 captures one infrared image, and after the predetermined time has elapsed, it captures another infrared image, and this process is repeated. The predetermined time is, for example, from several seconds to several minutes, but is not limited to this. As an example, the imaging device 200 captures an infrared image (still image) every five seconds. Note that the imaging device 200 may capture an infrared video image instead of an infrared image (still image), and in this case, each frame image included in the infrared video image may be used as the infrared image.

[0043] The imaging device 200 outputs the captured infrared image to the generation device 300. As described above, the imaging device 200 repeatedly captures infrared images. For example, after capturing one infrared image and outputting it to the generation device 300, the imaging device 200 captures the next infrared image and outputs it to the generation device 300. In other words, the imaging device 200 repeats the process of outputting one infrared image to the generation device 300 every time it captures one infrared image. Furthermore, for example, the imaging device 200 may capture multiple infrared images and then output the multiple infrared images collectively to the generation device 300.

[0044] In addition, in this embodiment, a communication unit included in the imaging device 200 outputs the captured infrared image to the generating device 300. The communication unit is a communication circuit that enables the imaging device 200 to communicate with the generating device 300. In this embodiment, the communication unit is a circuit for performing wireless communication, and specifically, performs wireless communication in accordance with a communication standard such as BLE (Bluetooth (registered trademark) Low Energy) or Wi-Fi (registered trademark). Note that the communication unit may also be a circuit for performing wired communication.

[0045] Here, the intensity of the light-receiving signal of the imaging device 200 will be described.

[0046] As described above, when output light L having the first near-infrared rays is output, the image capture device 200 receives the first near-infrared rays reflected by a plurality of people. The intensity of the received light signal obtained by this light reception is referred to as the first received light signal intensity. The first received light signal intensity is the intensity of the received light signal corresponding to the charge generated by the image capture device 200 (more specifically, the image capture element included in the image capture device 200) receiving the first near-infrared rays.

[0047] In addition, there is a case where second near-infrared rays different from the first near-infrared rays are irradiated onto multiple people. The second near-infrared rays are near-infrared rays output from a light source different from the near-infrared light source 100. The second near-infrared rays are, for example, near-infrared rays contained in sunlight, but are not limited to this and may be near-infrared rays output from a lighting device. Here, a case where the second near-infrared rays are irradiated onto multiple people when the output light L having the first near-infrared rays is not output will be described. The imaging device 200 receives the second near-infrared rays reflected by the multiple people. The intensity of the received light signal obtained by this light reception is defined as the second received light signal intensity. The second received light signal intensity is the intensity of the received light signal corresponding to the charge generated by the imaging device 200 (more specifically, the imaging element included in the imaging device 200) receiving the second near-infrared rays.

[0048] In this embodiment, the minimum value of the first received light signal strength is greater than the maximum value of the second received light signal strength. For example, the minimum value of the first received light signal strength is preferably at least twice, more preferably at least five times, and even more preferably at least ten times the maximum value of the second received light signal strength.

[0049] Furthermore, by setting the output of the first near-infrared ray to be 10 W or more and 3 kW or less, in other words, by sufficiently increasing the output of the first near-infrared ray, the minimum value of the first received signal intensity becomes greater than the maximum value of the second received signal intensity.

[0050] This sufficiently reduces the intensity of the second received light signal, which becomes noise in the signal used to recognize multiple people. Therefore, because such noise is sufficiently reduced, the imaging device 200 can obtain an infrared image in which multiple people are captured with high accuracy.

[0051] The imaging device 200 may include a magnification control mechanism (not shown) that controls the magnification of the captured infrared image. In this case, it is advantageous to obtain infrared images focused on each of multiple people using lenses or the like provided in the magnification control mechanism.

[0052] The generation device 300 is a device that generates psychological data indicating psychological states using captured infrared images and a trained machine learning model 321. The generation device 300 also determines stimuli to be given to multiple people based on the generated psychological data. The generation device 300 is, for example, a personal computer, but may also be a server device with high computing power connected to a network.

[0053] Here, the generating device 300 includes a communication unit 310, a generating unit 320, and a learning unit 330.

[0054] The communication unit 310 is a communication circuit that enables the generating device 300 to communicate with the imaging device 200, the display device 400, and the control device 500. In this embodiment, the communication unit 310 is a circuit for performing wireless communication, and specifically, performs wireless communication in accordance with a communication standard such as BLE or Wi-Fi (registered trademark). Note that the communication unit 310 may also be a circuit for performing wired communication.

[0055] The communication unit 310 includes an acquisition unit 311 and an output unit 312. The acquisition unit 311 acquires infrared images captured and output by the imaging device 200. As described above, the imaging device 200 repeats the process of capturing and outputting infrared images, so the acquisition unit 311 sequentially acquires the output infrared images. In other words, the acquisition unit 311 acquires a plurality of output infrared images. Furthermore, the output unit 312 outputs the generated psychological data to the display device 400 and stimulus data indicating the determined stimulus to the control device 500.

[0056] The generation unit 320 generates psychological data based on the infrared image acquired by the acquisition unit 311. More specifically, the generation unit 320 has a machine learning model 321, and generates psychological data indicating the psychological state of at least one person among a plurality of people using this machine learning model 321. Furthermore, each time the generation unit 320 acquires an infrared image, it generates one piece of psychological data corresponding to that infrared image.

[0057] As mentioned above, the psychological state refers to, by way of example only, states such as happy, sad, energetic, fearful, angry, depressed, alert, absent-minded, interested, and tense, but is not limited to these.

[0058] The machine learning model 321 is a trained model that receives an infrared image acquired by the acquisition unit 311 as an input and outputs the psychological state of at least one person among a plurality of people shown in the infrared image. In this embodiment, the training unit 330 trains the machine learning model 321.

[0059] The psychological data is, for example, data indicating the psychological state of at least one person among a plurality of people, and is data to be displayed by the display device 400, more specifically, digital data. Examples of the psychological data include display data such as number display data, color display data, pattern display data, graphic symbol display data, pictogram display data, and illustration display data. In the present embodiment, the psychological data is image data in which at least one of these display data is superimposed on a captured infrared image. When the psychological data is image data, the plurality of people captured in the infrared image are linked to the psychological states of the people and displayed on the display device 400.

[0060] Furthermore, the generation unit 320 determines stimuli to be given to the plurality of people based on the generated psychological data. For example, the generation unit 320 determines stimuli to be given to the plurality of people based on the generated psychological data and correspondence data stored in a storage unit (not shown) of the generation device 300. The correspondence data is data indicating a correspondence between a psychological state indicated by the psychological data and a stimulus corresponding to the psychological state. The generation unit 320 refers to the correspondence data and determines stimuli corresponding to the generated psychological data as stimuli to be given to the plurality of people. Note that giving a stimulus to a person is expected to change the psychological state of the person. The stimulus data indicates the stimuli determined in this manner. This makes it possible to determine a stimulus to change the psychological state of the person according to the psychological state of the person.

[0061] Specifically, the generation unit 320 is realized by a processor that executes a program, a microcomputer, or a dedicated circuit.

[0062] The learning unit 330 uses the training data to train the machine learning model 321. Specifically, the learning unit 330 is realized by a processor that executes a program, a microcomputer, or a dedicated circuit.

[0063] The learning unit 330 learns and constructs the machine learning model 321. The learning unit 330 provides the constructed machine learning model 321 to the generation unit 320. Note that the learning unit 330 is not an essential component, and may not be included in the psychological state display system 1.

[0064] This machine learning model 321 is a model for generating psychological data.

[0065] In this embodiment, the machine learning model 321 is a model constructed by machine learning using one or more datasets as training data. One dataset is composed of a combination of an infrared image captured by the imaging device 200 and the psychological state of a person represented by the infrared image. The infrared image captures at least one of the person's face and body. The infrared image may capture the entire face and the entire body of the person, but is not limited to this. For example, the infrared image may capture only a part of the person's face or body.

[0066] In other words, the machine learning model 321 is a recognition model constructed by machine learning using one or more data sets as training data, each of which is a data set of infrared images and the psychological states of people shown in the infrared images. More specifically, the machine learning model 321 is a recognition model constructed using infrared images belonging to each of the one or more data sets that are the training data as input data, and psychological states of people shown in the infrared images belonging to the data sets as output data.

[0067] As an example, the learning unit 330 learns the model using machine learning as described above. Therefore, in this embodiment, the model is the machine learning model 321.

[0068] Furthermore, the learning unit 330 may use, for example, a neural network, more specifically, a convolutional neural network (CNN), to train the machine learning model 321. When the machine learning model 321 is a convolutional neural network model, the learning unit 330 determines the coefficients (weights) of the filters of the convolutional layer by machine learning based on training data.

[0069] Furthermore, the learning unit 330 may train the machine learning model 321 using an algorithm other than a neural network.

[0070] The display device 400 displays the change over time of the psychological data output from the generating device 300. More specifically, since the generating unit 320 generates and outputs one piece of psychological data corresponding to one infrared image every time the generating unit 320 acquires the infrared image, the display device 400 sequentially acquires and displays the output psychological data. Therefore, the display device 400 displays the change over time of the psychological data.

[0071] The display device 400 is a light-emitting or non-light-emitting monitor device. The display device 400 is configured by an organic EL (Electro Luminescence) panel, which is a light-emitting monitor device, or a liquid crystal panel, which is a non-light-emitting monitor device. The display device 400 may be a stationary or portable type. An example of the stationary display device 400 is a display for a personal computer monitor, and an example of the portable display device is a smartphone or a tablet terminal.

[0072] The display device 400 is a device that is visually recognized by, for example, an administrator of the psychological state display system 1. The administrator can understand changes in the psychological states of multiple people by viewing the changes over time in the displayed psychological data. In other words, the psychological state display system 1 can visualize changes in people's psychological states.

[0073] By visualizing changes in a person's psychological state in this way, for example, a manager looking at the display device 400 can learn the following:

[0074] When the mental state display system 1 is used in a space where a plurality of people are present, the administrator can quickly discover among the people a suspicious person who is nervous or a person who is depressed because he is unable to speak due to the large number of people present.

[0075] Furthermore, when the psychological state display system 1 is used in a seminar hall (an example of a learning space) where a lecture is being held, it can also reveal the psychological states of the lecturers, such as their level of interest in the lecture. In this case, it is preferable for the administrator to inform the lecturer of the timing of breaks, etc., according to the level of interest.

[0076] The control device 500 is a device that controls the stimulation device 600, and is, for example, a personal computer, but may also be a server device with high computing power connected to a network. The control device 500 may also be any other device that can control the stimulation device 600, for example, a device dedicated to controlling the stimulation device 600.

[0077] The control device 500 acquires the stimulus data output from the generation device 300. The control device 500 controls the stimulus device 600 so that the stimulus indicated by the acquired stimulus data is given to a plurality of people.

[0078] This allows a stimulus to be given to a person to change that psychological state in accordance with the person's psychological state, thereby realizing a psychological state display system 1 that can control the psychological state of the person.

[0079] The stimulation device 600 is a device controlled by the control device 500 and provides stimulation to multiple people. The stimulation acts on at least one of the multiple people's senses of sight, hearing, smell, and touch. The stimulation is, for example, at least one stimulation selected from light, sound, smell, vibration, wind, temperature, and humidity. When the stimulation is light, it also means an image. Examples of the stimulation device 600 include, but are not limited to, a light source device, a projector device, a speaker, a vibration device, a blower device, a temperature control device, and a humidity control device.

[0080] The control device 500 may control the stimulation device 600 so that the stimulation given to the plurality of people is blocked or suppressed. For example, when the stimulation device 600 is a light source, the control device 500 may block (turn off) or suppress (reduce the amount of light) the light after providing light as a stimulation to the plurality of people.

[0081] Here, the relationship between the near-infrared light source 100 and a person will be described with reference to FIGS.

[0082] Fig. 3 is a schematic diagram of the near-infrared light source 100 according to this embodiment and a person viewed vertically. More specifically, Fig. 3 is a diagram of the near-infrared light source 100 and one person out of multiple people viewed vertically downward, in other words, a diagram of the near-infrared light source 100 and one person viewed from above the person's head. Fig. 4 is a schematic diagram of Fig. 3 viewed from the side.

[0083] Here, the front direction D1 shown in FIGS. 3 and 4 will be described. The front direction D1 is the direction in which the head of at least one of the multiple people is facing. In this embodiment, the near-infrared light source 100 outputs output light L to one person from a direction other than the front direction D1 in which the head of the person is facing. The first near-infrared light is irradiated to the one person from a direction that does not create a shadow directly behind the face of the one person facing forward. In other words, the first near-infrared light contained in the output light L is not irradiated from the front direction D1 of the head (face) of the one person; in other words, it is irradiated from above or below, or from the side (left or right) or obliquely to the side of the face of the one person.

[0084] Here, the optical axis La is defined as the direction of the first near-infrared light traveling toward the angle at which the radiation intensity is highest. When the first near-infrared light contained in the output light L is irradiated from the side or obliquely from the side of the face of the person, the optical axis La and the front direction D1 are not parallel. When the first near-infrared light contained in the output light L is irradiated from above or below the face of the person, the optical axis La and the front direction D1 may be parallel when viewed vertically as shown in FIG. 3.

[0085] In this way, the psychological state display system 1 according to the present embodiment does not need to irradiate the first near-infrared light from the front direction D1 of the person's face.

[0086] The area of ​​the area irradiated by the first near-infrared light is 10 m 2More than 10000m 2 The area is less than 25m 2 More than 100m is better. 2 It is even better if it is more than this.

[0087] Here, the upper limit of the area of ​​the irradiated region illuminated by the first near-infrared rays is not particularly limited. The reason for this is that imaging with the first near-infrared rays depends on the amount of light of the first near-infrared rays. If an upper limit is to be set in consideration of the actual state of lighting technology and imaging technology, the above 10,000 m 2 Less than this is good.

[0088] This allows the first near-infrared light to irradiate multiple people in a large space, thereby realizing a psychological state display system 1 that can visualize changes in the psychological states of multiple people in a large space.

[0089] Next, the distance R1 shown in FIG. 4 will be described.

[0090] Distance R1 is the distance between the near-infrared light source 100 and the person who is closest to the near-infrared light source 100 among the multiple people. More specifically, distance R1 is the distance between the person who is closest to the near-infrared light source 100 among the multiple people and a light output surface of the near-infrared light source 100. The output surface is the surface from which the output light L is output.

[0091] The distance R1 is preferably 50 cm or more and less than 30 m. Alternatively, the distance R1 is 50 cm or more and less than 3 m (first lighting distance), 3 m or more and less than 5 m (second lighting distance), or 5 m or more and less than 20 m (third lighting distance).

[0092] When the distance R1 is the first lighting distance, it is convenient for visualizing the psychological state of a person in an office, seminar venue, classroom, store, public space (such as a government office or library), public transportation space (inside a train or bus), etc. When the distance R1 is the second lighting distance, it is convenient for visualizing the psychological state of a person in an indoor event venue, movie theater, gymnasium, small outdoor event venue, relatively small area in a city, etc. When the distance R1 is the third lighting distance, it is convenient for visualizing the psychological state of a person in a large outdoor event venue, sports venue, relatively large area in a city, etc.

[0093] Furthermore, it is preferable that the distance between each of the plurality of people and the near-infrared light source 100 is within the above range.

[0094] The light distribution pattern of the first near-infrared light can be selected from a narrow-angle light distribution with a light distribution angle of less than 15°, a medium-angle light distribution with a light distribution angle of 15° or more and less than 30°, a wide-angle light distribution with a light distribution angle of 30° or more and less than 90°, and a Lambertian light distribution.

[0095] The closer the light distribution pattern is to a narrow-angle light distribution, the more the first near-infrared rays become beam-shaped, which realizes a light distribution pattern that is advantageous for irradiating a region far from the near-infrared light source 100 with the first near-infrared rays.

[0096] On the other hand, the closer the light distribution pattern is to a Lambertian light distribution, the softer and more uniform the first near-infrared rays will be irradiated over a wider area, thereby achieving a light distribution pattern that is advantageous for irradiating a wider area with the first near-infrared rays.

[0097] The near-infrared light source 100 may have a light distribution control mechanism (not shown) that controls the light distribution of the first near-infrared light to be output. In this case, the first near-infrared light may be shaped into a beam or may have a light distribution similar to a Lambertian light distribution through a lens and / or mirror provided in the light distribution control mechanism. Therefore, it is possible to irradiate the first near-infrared light suitable for generating psychological data, for example.

[0098] Next, the horizontal divergence angle θ of the first near-infrared ray of the output light L when the near-infrared light source 100 is viewed vertically as shown in FIG. 3 will be described.

[0099] The spread angle θ refers to the range of angles within which the first near-infrared rays are output, at which the radiation intensity at a given distance is half of the maximum intensity. In this embodiment, the spread angle θ is preferably 30° to 180°, and more preferably 45° to 120°. When the spread angle θ is within this range, the first near-infrared rays can be irradiated over a wide area far from the near-infrared light source 100. This allows the psychological state display system 1 to be space-saving.

[0100] [Processing procedure for mental state visualization method] Next, a specific processing procedure of the psychological state visualization method executed by the psychological state display system 1 will be described.

[0101] FIG. 5 is a flowchart showing the processing steps performed by the psychological state display system 1 according to this embodiment to train the machine learning model 321.

[0102] First, the learning unit 330 acquires training data for the machine learning model 321 (S101). The training data may be generated by the generating unit 320, but is not limited to this and may be generated by another processing unit or another device.

[0103] The teacher data will be explained below.

[0104] As mentioned above, the training data is one or more datasets.

[0105] A data set is composed of a combination of an infrared image captured by the image capture device 200 and a person's psychological state represented by the infrared image. For example, the infrared image captures a person's face and body. The infrared image is also combined with one of the person's psychological states, such as happy, sad, energetic, fearful, angry, depressed, alert, absent-minded, interested, and nervous.

[0106] The training data may also be visual data collected using a visible light sensor under visible light.

[0107] After step S101 is performed, the learning unit 330 uses the training data acquired in step S101 to train the machine learning model 321 (S102). More specifically, the learning unit 330 trains the machine learning model 321 through machine learning. Furthermore, the learning unit 330 outputs the trained machine learning model 321 to the generation unit 320.

[0108] After the processing of the flowchart shown in FIG. 5 is performed, the psychological state display system 1 performs processing to visualize the psychological state of a person.

[0109] FIG. 6 is a flowchart showing the processing procedure for visualizing a person's mental state by the mental state display system 1 according to this embodiment.

[0110] The near-infrared light source 100 outputs output light L having a first near-infrared ray to a plurality of people (S201).

[0111] The imaging device 200 receives the first near-infrared ray contained in the output light L output by the near-infrared light source 100 and captures infrared images of a plurality of people (S202). In this operation example, the imaging device 200 captures infrared images at predetermined time intervals, and after capturing one infrared image and outputting it to the generation device 300, the imaging device 200 captures the next infrared image and outputs it to the generation device 300. In other words, the imaging device 200 sequentially outputs the captured infrared images to the generation device 300, and the generation device 300 sequentially acquires the infrared images sequentially output by the imaging device 200.

[0112] The generating device 300 uses a machine learning model 321 that receives an infrared image as an input to generate psychological data that indicates the psychological state of at least one person among a plurality of people shown in the infrared image (S203).

[0113] In the machine learning model 321, when an infrared image acquired by the generating device 300 is input, the psychological state of at least one person among the multiple people shown in the infrared image is output. The generating device 300 generates psychological data indicating this psychological state.

[0114] Note that, each time the generation device 300 acquires an infrared image, it generates one piece of psychological data corresponding to the infrared image. As described above, the generation device 300 acquires infrared images sequentially, and therefore generates psychological data sequentially. Each time the generation device 300 generates psychological data, it outputs the generated psychological data to the display device 400. In other words, the display device 400 acquires the generated psychological data sequentially.

[0115] The display device 400 displays the acquired psychological data (S204). As described above, the display device 400 acquires the generated psychological data sequentially, and therefore displays the psychological data sequentially. As described above, the imaging device 200 captures infrared images at predetermined time intervals. Therefore, after displaying one piece of psychological data based on one infrared image, the display device 400 displays the next piece of psychological data based on the next infrared image captured a predetermined time after the previous infrared image. In other words, the display device 400 displays the changes in the psychological data over time. In other words, the display device 400 shows how the person's psychological state changes from moment to moment.

[0116] Here, the psychological state display system 1 according to this embodiment will be compared with the technology disclosed in Patent Document 1.

[0117] Patent Document 1 discloses that a person's temporary psychological state is displayed. However, the psychological state of the person before or after the displayed psychological state is unknown. Patent Document 1 also discloses that a plurality of images (still images) are extracted from a moving image, an average value of each of the plurality of images is calculated, and one psychological state is displayed based on the calculated average value. Even in this case, since only one psychological state is displayed, it is unknown whether there has been a change in the person's psychological state in the plurality of images.

[0118] It is natural that the psychological state of a person changes from moment to moment. In Patent Document 1, the temporary psychological state of a person is displayed, but how the psychological state of the person changes is not displayed.

[0119] In this embodiment, for example, the psychological state of a person that changes from moment to moment is displayed as shown in step S204 of Fig. 6. In other words, the psychological state display system 1 according to this embodiment can visualize the temporal change of the psychological state of a person.

[0120] Furthermore, when the image used in Patent Document 1 is a visible light image, the image is affected by the illuminating light (visible light) in the space where the image is captured. For example, the image quality, etc. of the image is affected by the illumination state of the illuminating light in the space, which in turn affects the displayed psychological state. In other words, in Patent Document 1, the displayed psychological state is affected by the illuminating light (visible light) in the space where the image is captured.

[0121] On the other hand, in the mental state display system 1 according to the present embodiment, first near-infrared light (more specifically, infrared images) is used to visualize temporal changes in a person's mental state. Therefore, when visualizing changes in a person's mental state, the mental state display system 1 is less susceptible to the influence of illumination light (visible light) in the space in which the mental state display system 1 is used. In other words, a mental state display system 1 is realized that can visualize changes in a person's mental state with the influence of illumination light suppressed.

[0122] Here, the psychological data will be described in more detail with reference to FIGS.

[0123] FIG. 7 is a diagram showing an infrared image P1 captured by imaging device 200 according to this embodiment.

[0124] FIG. 8 is an example of psychological data based on the infrared image P1 shown in FIG.

[0125] In the infrared image P1 shown in FIG. 7, a plurality of people are captured, and more specifically, four people, Mr. A, Mr. B, Mr. C, and Mr. D, are captured.

[0126] As described above, when infrared image P1 is input to machine learning model 321, it is sufficient to output the psychological state of at least one of the four people, but here, the psychological states of all four people (i.e., all people) are output. In other words, the psychological data shown in Fig. 8 indicates the psychological states of each of the four people.

[0127] Here, the interest level, which indicates the degree of interest, is used as the psychological state. In Fig. 8, the interest levels of four people are shown as psychological data.

[0128] The displayed psychological state level should be between 2 and 11 levels, and preferably 6 levels or less. For example, if the interest level is 6 levels, the levels are displayed in descending order of interest level as 5 points, 4 points, 3 points, 2 points, 1 point, and 0 points. In addition to the interest level, the levels of other psychological states (for example, the joy level) should also be between 2 and 11 levels, and preferably 6 levels or less.

[0129] When the displayed levels are within the above range, the psychological states of multiple people can be roughly grasped and the amount of data handled can be reduced, which is advantageous for downsizing the devices used in the psychological state display system 1 and for improving data processing speed.

[0130] As described above, the psychological data may be, for example, display data such as numeric display data, color display data, pattern display data, symbol display data, pictogram display data, and illustration display data. In the present embodiment, the psychological data is image data in which at least one of these display data is superimposed on a captured infrared image. When the psychological data is image data, multiple people captured in the infrared image are linked to the psychological states of the people and displayed on the display device 400.

[0131] The use of numerical display data is advantageous for displaying psychological states in detail or for expressing them numerically. The use of color display data is advantageous for expressing psychological states in color. The use of pattern display data, graphic symbol display data, and pictogram display data is advantageous for expressing psychological states metaphorically using figures and patterned symbols. The use of illustration display data is advantageous for expressing psychological states in illustrations. Furthermore, the use of image data is advantageous for linking a person to their psychological state and expressing it.

[0132] Here, an example in which color display data is used will be described with reference to FIG.

[0133] FIG. 8 shows image data P2 in which color display data is superimposed on a captured infrared image P1.

[0134] In the psychological data shown in Figure 8, colors corresponding to the psychological states of each of the four people are displayed, with darker colors indicating higher interest levels and lighter colors indicating lower interest levels. In Figure 8, dark colors are superimposed on Mr. A, Mr. B, and Mr. C, indicating a high level of interest, and light colors are superimposed on Mr. D, indicating a low level of interest. Note that although the colors corresponding to psychological states are expressed by intensity in Figure 8, they may also be expressed by color.

[0135] In this way, it is particularly convenient to display a person's psychological state using color (tone). In particular, when displayed in color, an excited state can be displayed in red, a slightly excited state in yellow, a slightly calm state in green, a calm state in blue, and a depressed state in gray or black. In this way, the psychological state display system 1 according to the present embodiment can visualize the time change of a person's psychological state using color, so that, for example, a user looking at the display device 400 can intuitively understand the time change of the person's psychological state.

[0136] Furthermore, as mentioned above, it is preferable that the number of levels of psychological state displayed be six or less. However, if there are six levels, for example, if the person's psychological state is displayed in rainbow colors, the user will be able to more easily understand the changes in the person's psychological state over time.

[0137] Another example of psychological data will be described with reference to FIG.

[0138] Fig. 9 shows another example of psychological data based on the infrared image P1 shown in Fig. 7. More specifically, Fig. 9 shows image data P3 in which graphic symbol display data is superimposed on the captured infrared image P1.

[0139] In the psychological data shown in FIG. 9, graphic symbols corresponding to the psychological states of each of the four people are displayed. The graphic symbols are display bodies, and are dashed framed as shown in FIG. 9. The larger the frame, the higher the level of interest, and the smaller the frame, the lower the level of interest. In other words, the level of interest is indicated by the size of the display body. In FIG. 9, large frames are superimposed on Mr. A, Mr. B, and Mr. C, indicating a high level of interest, and a small frame is superimposed on Mr. D, indicating a low level of interest. Note that the level of interest may be indicated by other graphic symbols.

[0140] When the graphic symbol display data is used, the psychological state of the person is displayed in an emphasized manner, as shown in Fig. 9. In this way, the psychological state display system 1 according to the present embodiment can visualize the time-varying change in the psychological state of the person by using graphic symbols, so that, for example, a user looking at the display device 400 can intuitively understand the time-varying change in the psychological state of the person.

[0141] Another example of psychological data will be described with reference to FIG.

[0142] FIG. 10 shows another example of psychological data according to the present embodiment.

[0143] More specifically, Fig. 10 shows image data in which numeric display data is superimposed on a captured infrared image. Fig. 10(a) shows image data P4, which is psychological data based on infrared image P1, in which numeric display data is superimposed on the captured infrared image P1. Fig. 10(b) shows image data P5 relating to psychological data based on an infrared image captured a predetermined time after the infrared image P1 was captured. Fig. 10(c) shows image data P6 relating to psychological data based on an infrared image captured a predetermined time after the infrared image of Fig. 10(b) was captured. That is, Fig. 10 shows, for example, the time change in psychological data displayed on the display device 400 in step S204 of Fig. 6.

[0144] In the psychological data shown in Figure 10, numbers corresponding to the psychological states of each of the four people are displayed. Note that here, the level of interest, which is a psychological state, is shown on six levels, and in descending order of interest level, it is displayed as 5 points, 4 points, 3 points, 2 points, 1 point, and 0 points. In (a) and (c) of Figure 10, "5 points of interest" is superimposed on Mr. A, Mr. B, and Mr. C, indicating a high level of interest, and "1 point of interest" is superimposed on Mr. D, indicating a low level of interest. In (b) of Figure 10, "5 points of interest" is superimposed on Mr. A and Mr. C, indicating a high level of interest, "3 points of interest" is superimposed on Mr. B, indicating a medium level of interest, and "1 point of interest" is superimposed on Mr. D, indicating a low level of interest.

[0145] 10, when numeric display data is used, it becomes easy to display the psychological state in detail and to express it numerically. In this way, the psychological state display system 1 according to the present embodiment can visualize the change over time of a person's psychological state using numbers, so that, for example, a user looking at the display device 400 can intuitively understand the change over time of the person's psychological state.

[0146] Furthermore, in FIGS. 8 to 10, image data in which at least one of the display data items is superimposed on a captured infrared image is used as psychological data, but the present invention is not limited to this.

[0147] For example, if only numerical display data is used as psychological data, the psychological data based on the infrared image P1 may be text data containing numbers, such as "Mr. A, Mr. B, and Mr. C have an interest level of 5, and Mr. D has an interest level of 1."

[0148] In addition, in this embodiment, psychological data showing the psychological state of each of the plurality of people is displayed, but this is not limited to this. For example, psychological data showing the average value of the psychological states of the plurality of people may be displayed. In this case, data such as "The interest levels of the plurality of people are 4 points" may be used as the psychological data corresponding to (a) of FIG. 10.

[0149] In this way, a psychological state display system 1 is realized that can visualize the temporal change in the average psychological state of a plurality of people.

[0150] Furthermore, when numerical display data is used as psychological data, the scores indicating the interest levels may be integrated and displayed. For example, when the scores indicating the interest levels shown in (a) to (c) of Fig. 10 are integrated, Mr. A and Mr. C get 15 points, Mr. B gets 13 points, and Mr. D gets 3 points, and these numerical values ​​may be displayed on the display device 400.

[0151] In this way, a psychological state display system 1 that can visualize a person's psychological state over a certain period of time is realized.

[0152] Furthermore, when numerical display data is used as psychological data, the score indicating the interest level may be differentiated and displayed.

[0153] In this way, a psychological state display system 1 is realized that can visualize temporal changes in a person's psychological state over a short period of time.

[0154] Returning now to FIG. 6, further processing will be described.

[0155] The generation device 300 determines stimuli to be given to the plurality of people based on the generated psychological data (S205). Note that the generation device 300 may determine stimuli to be given to all of the plurality of people, or may determine stimuli to be given to some of the plurality of people.

[0156] Furthermore, if the psychological data generated by the generation device 300 indicates that the psychological state of one or more persons is a predetermined psychological state, the generation device 300 may determine a stimulus to be given to one or more persons based on the predetermined psychological state.

[0157] This limits the number of people to whom a stimulus is applied to change their psychological state, eliminating the need to apply a stimulus to people who are not the target of the stimulus. This allows for the realization of a psychological state display system 1 that can efficiently control the psychological state of a target person.

[0158] The predetermined psychological state and stimulus will now be described.

[0159] For example, when a predetermined psychological state is "interest level of 1 point or less," the stimulus to be given to one or more persons is light. As shown in Fig. 10, the psychological state of one person, Mr. D, is "interest level of 1 point or less," and in this case, the generation device 300 determines that light is the stimulus to be given to Mr. D. Note that in this case, the stimulus device 600 is a light source device that emits the light (more specifically, visible light).

[0160] The communication unit 310 of the generation device 300 outputs stimulus data indicating the determined stimulus to the control device 500. The control device 500 acquires the output stimulus data.

[0161] The control device 500 controls the stimulation device 600 based on the output stimulation data (S206). For example, the control device 500 controls the stimulation device, which is a light source device, to irradiate light toward Mr. D. This is expected to change Mr. D's psychological state, for example, increasing his level of interest. In this case, there is no need to provide stimulation to Mr. A, Mr. B, and Mr. C. In other words, stimulation can be provided efficiently only to Mr. D.

[0162] In addition, the above example is an example in which the generation device 300 determines to give a stimulus to some of the people (i.e., Mr. D) and the control device 500 controls the stimulation device 600 so that the determined stimulus is given to at least some of the people, but this is not limiting. The generation device 300 may determine to give a stimulus to all of the people (e.g., Mr. A, Mr. B, Mr. C, and Mr. D), and the control device 500 may control the stimulation device 600 so that the stimulus is given to all of the people.

[0163] The generation device 300 may determine stimuli to be given to some or all of the multiple people based on changes over time in the psychological states indicated by the generated psychological data. In Fig. 10, there was no change over time in the psychological states of people A, C, and D, and their interest levels were always constant, but there was a change over time in the psychological state of person B, and his interest level fluctuated. In this case, for example, the generation device 300 may determine to give stimuli to people A, C, and D, whose psychological states did not change over time, or to give a stimuli to person B, whose psychological state changed over time.

[0164] In the above description, light is used as a stimulus when the psychological state is "interest level of 1 point or less," but the present invention is not limited to this. For example, other examples will be described below.

[0165] As another example, if there is a suspicious person whose psychological state is "tense" as described above, it would be advisable to determine stimuli that will deter criminal activity (for example, a bright light that illuminates the suspicious person, or a sound (voice) such as "Police officers are on patrol").

[0166] As another example, when the psychological state display system 1 is used in an entertainment space, stimuli such as sound and light can be controlled according to the level of interest in the psychological state, which can liven up an event or evoke strong emotions in people in the space. Conversely, it is good to provide stimuli to calm down people who are in a state where it is difficult to control their emotions.

[0167] Furthermore, display device 400 according to this embodiment may display vital information of a person. For example, psychological data and the person's vital information may be displayed in step S204 shown in FIG. 6. In this case, the vital information may be obtained, for example, as follows: imaging device 200 captures an infrared image including vascular information related to the contraction of blood vessels of each of a plurality of people. generation device 300 generates vital information indicating the vital signs of at least one of the plurality of people based on the vascular information included in the captured infrared image.

[0168] Blood vessel information is, for example, information on the light reflectance of a person's skin captured in an infrared image. Furthermore, vital signs are, for example, heart rate. Because blood has the property of absorbing light, the light reflectance changes in response to blood vessel contraction caused by pulse waves. Therefore, based on information on the light reflectance of a person's skin (blood vessel information), the pulse wave is estimated and the heart rate is calculated. Vital sign information is information indicating the person's heart rate (vital signs).

[0169] Furthermore, the display device 400 displays the generated vital information. As an example, when displaying the image data P2 shown in Fig. 8, the display device 400 may display the vital information superimposed on the image data P2.

[0170] As a result, the psychological state display system 1 can visualize not only the temporal change in a person's psychological state, but also the vital information of the person and the temporal change in the vital information.

[0171] Here, the space in which the psychological state display system 1 is used and the effects obtained in that space will be described in more detail.

[0172] As described above, the psychological state display system 1 is a system used in spaces such as entertainment spaces, public spaces, learning spaces, etc. Entertainment spaces, public spaces, and learning spaces are often relatively large spaces.

[0173] Entertainment spaces include spaces such as (1) or (2) below: (1) Staged lighting spaces where the lighting conditions using visible light change rapidly (such as live music venues or affordance lighting spaces); (2) Dark spaces and shadowed spaces where people experience images or sound (such as movie theaters, snack bars, and karaoke houses). Affordance lighting spaces are outdoor staged lighting spaces that affect people's psychological state and behavior (such as "wandering" and "staying") by adding movement, brightness, and color changes to the lighting to create a sense of liveliness in the space.

[0174] Furthermore, public spaces include spaces of public transport (trains, buses, taxis, airplanes, etc.), and include, for example, the following (3) spaces: (3) Spaces where external light such as sunlight is repeatedly incident or blocked. noon and spaces where people move alternately between dark places (such as tunnels) and light places (such as the inside of a bus traveling on a highway).

[0175] Furthermore, the learning space includes a space where a person is studying, such as at home, at school, at a cram school, or at a seminar venue, and the space may contain people studying and may also contain people (such as instructors) who are guiding the students. The psychological state display system 1 is particularly suitable for use in the following learning space (4): (4) At least one lighting space illuminated with white lighting light of different light colors (such as chromaticity).

[0176] In entertainment spaces and public spaces (particularly the spaces indicated by (1), (2), and (3) above), the presence, intensity, and color tone of visible light in the space are prone to change significantly, and the psychological state of people in the space is easily affected. In this embodiment, first near-infrared light (more specifically, infrared images) is used to visualize changes in people's psychological states. In other words, even if the presence, intensity, and color tone of visible light in the space change significantly, the first near-infrared light, which has a different wavelength range from visible light, is used, so the psychological state display system 1 can easily visualize changes in the psychological state of people whose psychological states are easily affected.

[0177] Furthermore, particularly in a learning space such as (4), if a person is illuminated with a plurality of white illumination lights each having a different chromaticity, and the person's psychological state is estimated using visible light and a visible light image based on the visible light, the accuracy of the estimation may decrease depending on the chromaticity. Even in such a learning space, the psychological state display system 1 according to this embodiment uses the first near-infrared light, so that changes in the person's psychological state can be easily visualized. Note that the same effect is expected not only in a learning space but also in spaces such as (4).

[0178] In this way, the psychological state display system 1 is realized, which can easily visualize changes in a person's psychological state even in a space where the presence, intensity, and color tone of visible light are likely to change significantly, or in a space where a person is illuminated with multiple white illumination lights with different chromaticities. In particular, in such environments, it becomes possible to detect suspicious individuals early, and use of the stimulation device 600 can help prevent crime. Therefore, such a psychological state display system 1 is a system that can be expected to have an effect of resolving social problems, such as revitalizing the economy, eliminating inequality, and creating a safe and secure society.

[0179] Furthermore, a comparison will be made between the prior art and the psychological state display system 1 according to the present embodiment.

[0180] For example, Patent Document 2 (JP 2009-87303 A) discloses a technology for estimating facial expressions such as anxiety or surprise of a driver in a vehicle. Also, Patent Document 3 (JP 2020-48149 A) ​​discloses a technology for displaying the emotions of participants in an online conference in a conference room.

[0181] The technology disclosed in Patent Document 2 is applicable to narrow spaces such as inside a vehicle, and it is difficult to apply it to large spaces such as entertainment spaces and public spaces (particularly the spaces (1), (2) and (3) above).

[0182] Furthermore, since conference rooms and other such spaces in which the technology disclosed in Patent Document 3 is used are often general white lighting spaces, it is difficult to apply the technology disclosed in Patent Document 3 to spaces such as those described in (4) above.

[0183] However, as described above, the psychological state display system 1 according to this embodiment can easily visualize changes in a person's psychological state even in entertainment spaces, public spaces, or learning spaces (particularly spaces (1), (2), (3), or (4) above).

[0184] [Variations] Here, a modification of the embodiment will be described. Modification 1 of the embodiment differs from the embodiment in that the stimulation device is a near-infrared light source 100a. The following description will focus on the differences from the embodiment, and the description of the commonalities will be omitted or simplified.

[0185] An example of the configuration of a psychological state display system 1a according to the first modification of the embodiment will be described with reference to FIG.

[0186] FIG. 11 is a block diagram showing the functional configuration of a psychological state display system 1a according to the first modification of the present embodiment.

[0187] The psychological state display system 1a includes a near-infrared light source 100a, an imaging device 200, a generating device 300, a display device 400, and a control device 500.

[0188] In this modification, the stimulus device is a near-infrared light source 100a that outputs output light L having visible light. Furthermore, the generation device 300 determines a stimulus to be given to the plurality of people based on the generated psychological data, and the control device 500 controls the stimulus device (near-infrared light source 100a) so that the determined stimulus is given to the plurality of people. Here, visible light output by the near-infrared light source 100a is determined as the stimulus, and the visible light is output toward the plurality of people.

[0189] This reduces the number of parts in the psychological state display system 1a according to this modification, and simplifies the configuration of the psychological state display system 1a.

[0190] [Effects, etc.] The psychological state display system 1 according to this embodiment includes a near-infrared light source 100, an imaging device 200, a generation device 300, and a display device 400. The near-infrared light source 100 outputs output light L having first near-infrared rays with wavelengths equal to or greater than 780 nm and less than 2500 nm to a plurality of people. The imaging device 200 receives the first near-infrared rays contained in the output light L and captures infrared images of the plurality of people. The generation device 300 generates psychological data indicating the psychological states using a trained machine learning model 321 that receives the captured infrared images as input and outputs the psychological state of at least one person among the plurality of people represented by the infrared images. The display device 400 displays changes over time in the generated psychological data.

[0191] As a result, for example, as shown in step S204 of FIG. 6, after one piece of psychological data based on one infrared image is displayed on the display device 400, the next piece of psychological data based on the next infrared image captured a predetermined time after the first infrared image is displayed. In other words, the display device 400 shows how the psychological state of a person changes from moment to moment. By viewing the changes in the displayed psychological data over time, a manager of the psychological state display system 1 or the like can understand changes in the psychological states of multiple people, etc. In other words, the psychological state display system 1 can visualize changes in a person's psychological state.

[0192] Furthermore, in the psychological state display system 1 according to this embodiment, first near-infrared light (more specifically, infrared images) is used to visualize temporal changes in a person's psychological state. Therefore, when visualizing changes in a person's psychological state, the psychological state display system 1 is less susceptible to the influence of illumination light (visible light) in the space in which the psychological state display system 1 is used. In other words, a psychological state display system 1 is realized that can visualize changes in a person's psychological state with reduced influence from illumination light.

[0193] Furthermore, for example, the first near-infrared light has a peak wavelength in the wavelength range of 780 nm or more and less than 2500 nm.

[0194] This irradiates the subject with first near-infrared light, which contains a large amount of near-infrared light components that are invisible to the human eye. This realizes a psychological state display system 1 that can visualize changes in a person's psychological state without causing discomfort to the person.

[0195] Furthermore, for example, the near-infrared light source 100 includes a first solid-state light-emitting element that outputs a first near-infrared ray.

[0196] This allows electrical energy to be converted directly into the first near-infrared rays (light energy), making it easy to obtain strong first near-infrared rays. Therefore, the imaging device 200 can obtain infrared images of multiple people with high accuracy. This realizes a psychological state display system 1 that can visualize changes in a person's psychological state with high accuracy.

[0197] Furthermore, for example, the near-infrared light source 100 includes a second solid-state light-emitting element that outputs excitation light, and a phosphor member that outputs wavelength-converted light as first near-infrared light based on the output excitation light.

[0198] This reduces the unevenness in the spectral distribution and intensity of the first near-infrared light. Therefore, the imaging device 200 can obtain infrared images of multiple people with higher accuracy. This realizes the psychological state display system 1 that can visualize changes in people's psychological states with higher accuracy.

[0199] Furthermore, for example, the output light L includes visible light.

[0200] This irradiates output light L having visible light that can be seen by the human (person) eye. This realizes a psychological state display system 1 that is advantageous for recognizing the presence of the near-infrared light source 100 that is emitting the first near-infrared light.

[0201] Furthermore, for example, the near-infrared light source 100 controls the first near-infrared light and the visible light independently, and controls at least one of turning on and off and dimming of each of the first near-infrared light and the visible light.

[0202] By controlling the first near-infrared light and visible light in this manner by the near-infrared light source 100, a psychological state display system 1 is realized that is advantageous in terms of freely controlling the visualization of psychological states based on the first near-infrared light and the light production effects based on visible light.

[0203] Further, for example, when output light L having a first near-infrared ray is output, the intensity of a received light signal obtained by the imaging device 200 receiving the first near-infrared ray reflected by a plurality of people is defined as the first received light signal intensity. When output light L having the first near-infrared ray is not output, second near-infrared ray different from the first near-infrared ray is irradiated onto a plurality of people, and the second near-infrared ray reflected by the plurality of people is received by the imaging device 200. The minimum value of the first received light signal intensity is greater than the maximum value of the second received light signal intensity.

[0204] This sufficiently reduces the intensity of the second received light signal, which becomes noise in the signal that recognizes multiple people. Therefore, because such noise is sufficiently reduced, the imaging device 200 can obtain infrared images in which multiple people are captured with high accuracy. This realizes a psychological state display system 1 that can visualize changes in a person's psychological state with high accuracy.

[0205] Furthermore, for example, the near-infrared light source 100 outputs the output light L to one person from a direction other than the front direction toward which the head of at least one person among the plurality of people is facing.

[0206] In this way, even if the first near-infrared light is not emitted from the front direction D1 of the person's face, the mental state display system 1 can visualize changes in the person's mental state. For example, if the mental state display system 1 is used in a large space and multiple people in the large space move in completely different ways, the first near-infrared light is not emitted from the front direction D1 of the person's face. Even in such a case, the mental state display system 1 can be realized that can visualize changes in the person's mental state.

[0207] As described above, the infrared image used by the learning unit 330 when training the machine learning model 321 preferably captures the entire face and the entire body of a person, but is not limited to this and may capture, for example, a part of the person's face or a part of the body. By performing learning using an infrared image capturing a part of the person's face or a part of the body, the psychological state display system 1 can visualize changes in the person's psychological state even if the first near-infrared light is not irradiated from the front direction D1 of the person's face.

[0208] For example, the area of ​​the irradiated region irradiated by the first near-infrared rays is 10 m 2 More than 10000m 2 is less than.

[0209] This allows the first near-infrared light to irradiate multiple people in a large space, thereby realizing a psychological state display system 1 that can visualize changes in the psychological states of multiple people in a large space.

[0210] Furthermore, for example, the distance between the near-infrared light source 100 and the person who is closest to the near-infrared light source 100 among the multiple people is 50 cm or more and less than 30 m.

[0211] Furthermore, the distance R1 is 50 cm or more and less than 3 m (first lighting distance), 3 m or more and less than 5 m (second lighting distance), or 5 m or more and less than 20 m (third lighting distance).

[0212] When the distance R1 is the first lighting distance, it is convenient for visualizing the psychological state of a person in an office, seminar venue, classroom, store, public space (such as a government office or library), public transportation space (inside a train or bus), etc. When the distance R1 is the second lighting distance, it is convenient for visualizing the psychological state of a person in an indoor event venue, movie theater, gymnasium, small outdoor event venue, relatively small area in a city, etc. When the distance R1 is the third lighting distance, it is convenient for visualizing the psychological state of a person in a large outdoor event venue, sports venue, relatively large area in a city, etc.

[0213] Furthermore, for example, when the near-infrared light source 100 is viewed vertically, the horizontal spread angle of the first near-infrared light is not less than 30° and not more than 180°.

[0214] When the divergence angle θ is within this range, the first near-infrared light can be irradiated onto a range area that is far from the near-infrared light source 100. This allows the psychological state display system 1 to be made more space-saving.

[0215] Furthermore, for example, the generation device 300 determines stimuli to be given to multiple people based on the generated psychological data.

[0216] This makes it possible to determine a stimulus for changing a person's psychological state in accordance with that person's psychological state.

[0217] Furthermore, for example, the psychological state display system 1 includes a control device 500 that controls a stimulation device 600 that applies stimulation to a plurality of people. The control device 500 controls the stimulation device 600 so that the determined stimulation is applied to the plurality of people.

[0218] This allows a stimulus to be given to a person to change that psychological state in accordance with the person's psychological state, thereby realizing a psychological state display system 1 that can control the psychological state of the person.

[0219] Furthermore, for example, the psychological state display system 1 includes a stimulation device 600. The control device 500 controls the stimulation device 600 so that the determined stimulation is given to at least some of the plurality of people.

[0220] This makes it possible to provide a stimulus to at least some of the people, thereby realizing a psychological state display system 1 that can control the psychological states of the people.

[0221] Also, for example, the determined stimulus is a stimulus that affects at least one of the senses of sight, hearing, smell, and touch of the multiple people.

[0222] This realizes a psychological state display system 1 that can change the psychological states of a plurality of people by stimulating their five senses except for the sense of taste.

[0223] Also, for example, the control device 500 controls the stimulation device 600 so that stimulation given to multiple people is blocked or suppressed.

[0224] This realizes a psychological state display system 1 that can change the psychological states of a plurality of people by alleviating the stimuli given to the people.

[0225] Also, for example, if the generated psychological data indicates that the psychological state of one or more persons is a predetermined psychological state, the generation device 300 determines a stimulus to be given to one or more persons based on the predetermined psychological state.

[0226] This limits the number of people to whom a stimulus is applied to change their psychological state, eliminating the need to apply a stimulus to people who are not the target of the stimulus. This allows for the realization of a psychological state display system 1 that can efficiently control the psychological state of a target person.

[0227] Also, for example, in a modified example, the stimulator is a near-infrared light source 100a that outputs output light L having visible light.

[0228] This reduces the number of parts in the psychological state display system 1a according to the modified example, and simplifies the configuration of the psychological state display system 1a.

[0229] For example, the imaging device 200 captures an infrared image including vascular information related to the contraction of blood vessels of each of a plurality of people. The generating device 300 generates vital information indicating the vital signs of at least one of the plurality of people based on the vascular information included in the captured infrared image. The display device 400 displays the generated vital information.

[0230] As a result, the psychological state display system 1 can visualize not only the temporal change in a person's psychological state, but also the vital information of the person and the temporal change in the vital information.

[0231] For example, the near-infrared light source 100 and the imaging device 200 are installed in an entertainment space, a public space, or a learning space.

[0232] This realizes a psychological state display system 1 that can easily visualize changes in a person's psychological state even in a space where the presence, intensity, and color tone of visible light are prone to change significantly, or in a space where a person is illuminated with multiple white illumination lights each with different chromaticities.

[0233] (Other embodiments) Although the embodiments have been described above, the present invention is not limited to the above-described embodiments.

[0234] In the above-described embodiment and modified examples, the output light L includes the first near-infrared light and visible light, but is not limited to this. The output light L may include only the first near-infrared light. Furthermore, the output light L may further include electromagnetic waves other than the first near-infrared light and visible light.

[0235] As such electromagnetic waves, for example, at least one of ultraviolet rays, mid-infrared rays, and far-infrared rays can be used.

[0236] For example, ultraviolet light has the property of being converted into visible light by acting on a phosphor. Phosphors are contained in bleaching agents contained in clothing, etc. Therefore, when output light L having ultraviolet light is irradiated onto a person wearing the clothing, visible light is output from the clothing. Therefore, for example, multiple people or an administrator of the psychological state display system can easily recognize that output light L having ultraviolet light and first near-infrared light is being output.

[0237] Furthermore, for example, mid-infrared and far-infrared rays have a relatively high heating effect. Therefore, if the output light L includes mid-infrared and far-infrared rays, a psychological state display system that has a slight heating effect can be realized in cold regions.

[0238] The near-infrared light source 100 controls the first near-infrared light, visible light, and electromagnetic waves independently, and controls at least one of turning on and off and dimming of each of the first near-infrared light, visible light, and electromagnetic waves.

[0239] By having the near-infrared light source 100 control the first near-infrared light, visible light, and electromagnetic waves in this manner, a psychological state display system 1 is realized that is advantageous in visualizing psychological states based on the first near-infrared light and freely controlling light effects based on visible light and electromagnetic waves.

[0240] The communication method between the devices described in the above embodiment is merely an example, and is not particularly limited to this.

[0241] In the above-described embodiment, the processing performed by a specific processing unit may be performed by another processing unit. The order of multiple processing operations may be changed, or multiple processing operations may be performed in parallel.

[0242] In the above-described embodiments, components such as the generation unit may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0243] Furthermore, components such as the generation unit may be realized by hardware. For example, components such as the generation unit may be circuits (or integrated circuits). These circuits may form a single circuit as a whole, or may be separate circuits. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0244] Furthermore, the general or specific aspects of the present invention may be realized as a system, device, method, integrated circuit, computer program, or computer-readable recording medium such as a CD-ROM. Also, the present invention may be realized as any combination of a system, device, method, integrated circuit, computer program, and recording medium. For example, the present invention may be realized as the psychological state display system of the above-described embodiment, or as a psychological state display method executed by the psychological state display system. The present invention may be realized as a program for causing a computer to execute such a psychological state display method, or as a non-transitory recording medium on which such a program is recorded. Such programs include an application program for causing a computer such as a general-purpose information terminal to function as the generation device of the above-described embodiment.

[0245] In addition, the present invention also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the spirit of the present invention. [Explanation of symbols]

[0246] 1, 1a Psychological state display system 100, 100a near infrared light source 200 Imaging device 300 generator 321 Machine Learning Models 400 display device 500 control device 600 Stimulator L output light

Claims

1. a near-infrared light source that outputs output light having a first near-infrared ray having a peak wavelength in a wavelength range of 780 nm or more and less than 925 nm and an output of 10 W or more and 3 kW or less to a plurality of people; an imaging device that receives the first near-infrared ray included in the output light and captures infrared images of the plurality of people; a generating device that generates psychological data indicating the psychological state using a trained machine learning model that receives the captured infrared image as an input and outputs the psychological state of at least one person among the plurality of people shown in the infrared image; a display device that displays the generated psychological data over time. Psychological state display system.

2. When the output light having the first near-infrared rays is output, the intensity of the received light signal obtained by the imaging device receiving the first near-infrared rays reflected by the plurality of persons is defined as a first received light signal intensity, when the output light having the first near-infrared rays is not being output, second near-infrared rays different from the first near-infrared rays are irradiated onto the plurality of people; When the intensity of a received light signal obtained by the imaging device receiving the second near-infrared rays reflected by the plurality of people is defined as a second received light signal intensity, The minimum value of the first received light signal strength is greater than the maximum value of the second received light signal strength. The psychological state display system according to claim 1 .

3. The near-infrared light source includes a first solid-state light-emitting element that outputs the first near-infrared light. The psychological state display system according to claim 2 .

4. The near-infrared light source is a second solid-state light-emitting element that outputs excitation light; a phosphor member that outputs wavelength-converted light as the first near-infrared light based on the output excitation light. The psychological state display system according to claim 2 .

5. The output light comprises visible light. The psychological state display system according to claim 2 .

6. The near-infrared light source is The first near-infrared light and the visible light are independently controlled; At least one of turning on and off the first near-infrared light and dimming the visible light is controlled. The psychological state display system according to claim 5.

7. The near-infrared light source outputs the output light to at least one person among the plurality of people from a direction other than a front direction toward which the head of the at least one person is facing. The psychological state display system according to any one of claims 1 to 6.

8. The area of ​​the irradiated region irradiated with the first near-infrared rays is 10 m ２ More than 10000m ２ is less than The psychological state display system according to claim 7.

9. The distance between the person closest to the near-infrared light source and the near-infrared light source is 50 cm or more and less than 30 m. The psychological state display system according to claim 7.

10. When the near-infrared light source is viewed vertically, The horizontal spread angle of the first near-infrared ray is 30° or more and 180° or less. The psychological state display system according to claim 7.

11. The generating device determines stimuli to be given to the plurality of people based on the generated psychological data. The psychological state display system according to claim 2 .

12. a control device for controlling a stimulation device that stimulates the plurality of persons; The control device controls the stimulation device so that the determined stimulation is given to the plurality of persons. The psychological state display system according to claim 11.

13. The stimulation device is provided, The control device controls the stimulation device so that the determined stimulation is applied to at least some of the plurality of people. The psychological state display system according to claim 12.

14. The determined stimulus is a stimulus that affects at least one of the senses of sight, hearing, smell, and touch of the plurality of persons. The psychological state display system according to claim 12.

15. The control device controls the stimulation device so that the stimulation applied to the plurality of persons is blocked or suppressed. The psychological state display system according to any one of claims 12 to 14.

16. When the generated psychological data indicates that the psychological state of one or more of the persons is a predetermined psychological state, The generating device determines a stimulus to be given to the one or more people based on the predetermined psychological state. The psychological state display system according to claim 11.

17. The stimulation device is a near-infrared light source that outputs the output light having visible light. The psychological state display system according to claim 13.

18. the imaging device captures the infrared image including vascular information related to vascular contraction of each of the plurality of persons; the generating device generates vital information indicating a vital sign of at least one person among the plurality of people based on the blood vessel information included in the captured infrared image; The display device displays the generated vital information. The psychological state display system according to claim 2 .

19. The near-infrared light source and the imaging device are installed in an entertainment space, a public space, or a learning space. The psychological state display system according to claim 2 .

Images