Psychological state display system

Abstract

A psychological state display system (1) is provided with: a near-infrared light source (100) that outputs, to a plurality of persons, output light having first near-infrared light having a wavelength of 780 nm or more and less than 2500 nm; an imaging device (200) that receives the first near-infrared rays of the output light that has been output, and captures infrared images of a plurality of persons; a generation device (300) that generates psychological data indicating a psychological state using a learned machine learning model (321) that uses the captured infrared image as an input and the psychological state of at least one of the plurality of persons indicated by the infrared image as an output; and a display device (400) that displays a temporal change in the generated psychological data.

Description

Technical Field

[0001] The invention relates to a psychological state display system. Background Art

[0002] In recent years, there has been known a technique for estimating a person's psychological state (emotions) and the like based on an image of the person.

[0003] For example, Patent Document 1 discloses a system (expression estimation device) that estimates an expression suitable for the psychological state emotion of a person based on the expression and action of the person shown in an image of the person.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2014-206903 Summary of the invention

[0007] Problem that the invention aims to solve

[0008] In the system disclosed in Patent Document 1, only the temporary mental state of a person included in an image is made clear. The mental state of a person (character) often changes over time, and visualization of these changes is sought.

[0009] Therefore, an object of the present invention is to provide a psychological state display system capable of visualizing changes in a person's psychological state while suppressing the influence of illumination light.

[0010] Means used to solve problems

[0011] According to one aspect of the present invention, a psychological state display system comprises: a near-infrared light source, which outputs a first near-infrared output light having a wavelength of greater than 780nm and less than 2500nm to a plurality of persons; a shooting device, which receives the first near-infrared output light and shoots infrared images of the plurality of persons; a generating device, which generates psychological data representing a psychological state using a learned machine learning model, the machine learning model taking the captured infrared image as input and taking the psychological state of at least one of the plurality of persons shown in the infrared image as output; and a display device, which displays the time variation of the generated psychological data.

[0012] Effects of the Invention

[0013] The psychological state display system of the present invention can visualize changes in a person's psychological state while suppressing the influence of illumination light. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a diagram showing the structure of a mental state display system according to an embodiment.

[0015] Figure 2 This is a block diagram showing a characteristic functional structure of the mental state display system according to the embodiment.

[0016] Figure 3 This is a schematic diagram of a near-infrared light source and a person according to an embodiment as viewed from a vertical direction.

[0017] Figure 4 Observed from the side Figure 3 Schematic diagram of .

[0018] Figure 5 It is a flowchart showing the processing sequence for enabling the mental state display system of an implementation method to learn a machine learning model.

[0019] Figure 6 This is a flowchart showing a processing procedure for visualizing a person's psychological state by the psychological state display system according to the embodiment.

[0020] Figure 7 The diagram shows an infrared image captured by the imaging device according to the embodiment.

[0021] Figure 8 is based on Figure 7 An example of psychological data of the infrared image shown.

[0022] Fig. 9 is based on Figure 7 Another example of psychological data of an infrared image is shown.

[0023] Fig.10 This is another example of psychological data in the implementation mode.

[0024] Fig.11 This is a block diagram showing a characteristic functional structure of a mental state display system according to a modified example of the embodiment. DETAILED DESCRIPTION

[0025] Hereinafter, the embodiments are described in detail with reference to the accompanying drawings. In addition, the embodiments described below are all general or specific examples. The numerical values, shapes, materials, constituent elements, configuration positions and connection methods of constituent elements, steps, and the order of steps shown in the following embodiments are only examples, and the main purpose is not to limit the present invention. In addition, among the constituent elements of the following embodiments, constituent elements not recorded in the independent technical solutions are described as arbitrary constituent elements.

[0026] In addition, each figure is a schematic diagram and does not necessarily illustrate strictly. In addition, in each figure, the same reference numerals are attached to substantially the same structure, and the repeated description is sometimes omitted or simplified.

[0027] (Implementation Method)

[0028] [Structure of the mental state display system]

[0029] The configuration of the mental state display system 1 according to the present embodiment will be described.

[0030] Figure 1 It is a diagram showing the configuration of a mental state display system 1 according to the present embodiment. Figure 2 1 is a block diagram showing a characteristic functional structure of the mental state display system 1 according to the present embodiment.

[0031] The psychological state display system 1 outputs the output light L having the first near infrared ray to a plurality of persons (e.g., more than several persons and less than dozens of persons), and captures infrared images of the plurality of persons. The psychological state display system 1 generates psychological data representing the psychological state of at least one person among the plurality of persons based on the captured infrared images, and displays the temporal changes of the psychological data. For example, an administrator of the psychological state display system 1 can understand the changes in the psychological states of the plurality of persons, etc., by viewing the temporal changes of the displayed psychological data. That is, the psychological state display system 1 can visualize the changes in the psychological states of the persons.

[0032] Furthermore, the psychological state display system 1 determines stimulation based on the psychological data and applies the determined stimulation to a plurality of characters. Therefore, since stimulation for changing the psychological state of the characters can be applied according to the psychological state of the characters, the psychological state display system 1 can control the psychological state of the characters.

[0033] The mental state of this embodiment refers to states such as happiness, sadness, energy, fear, anger, frustration, wakefulness, absent-mindedness, interest, tension, and concentration, but is not limited thereto.

[0034] like Figure 1 and Figure 2 As shown, the mental state display system 1 includes a near-infrared light source 100 , a photographing device 200 , a generating device 300 , a display device 400 , a control device 500 , and a stimulation device 600 .

[0035] The near-infrared light source 100 outputs the first near-infrared light L including the wavelength of 780 nm or more and less than 2500 nm toward the plurality of people. By using such a near-infrared light source 100, a wide space is irradiated with the first near-infrared light.

[0036] The photographing device 200 receives the first near infrared rays included in the output light L and photographs infrared images of a plurality of persons. The photographing device 200 photographs infrared images of a plurality of persons by receiving the first near infrared rays reflected by the plurality of persons.

[0037] The generation device 300 generates psychological data representing a psychological state using the learned machine learning model 321, wherein the machine learning model 321 takes the captured infrared image as input and outputs the psychological state of at least one person among the multiple persons shown in the infrared image. In addition, the generation device 300 determines the stimulation to be applied to the multiple persons based on the generated psychological data. In addition, the generation device 300 may use, for example, the technology described in Japanese Patent No. 6467965.

[0038] The display device 400 displays the time change of the generated psychological data.

[0039] The control device 500 controls the stimulation device 600 to apply the determined stimulation to the plurality of characters.

[0040] The stimulation device 600 applies stimulation to a plurality of characters.

[0041] As an example, the mental state display system 1 of this embodiment is a system used in a space such as an entertainment space, a public space, or a learning space, but is not limited to such a space. In addition, at least the near-infrared light source 100, the camera 200, and the stimulation device 600 are set in the space. The generation device 300, the display device 400, and the control device 500 can be set in the space or in a remote location away from the space.

[0042] exist Figure 1 In the present invention, in a movie theater as an example of an entertainment space, output light L is output to a plurality of people watching a movie, infrared images of the plurality of people are captured, and temporal changes in the psychological states of the plurality of people are visualized. In addition, stimulation is applied to the plurality of people.

[0043] Here, the components included in the mental state display system 1 will be described.

[0044] The near-infrared light source 100 is a light source that outputs an output light L having a first near-infrared ray to a plurality of people. The first near-infrared ray is a light including a wavelength of 780 nm or more and less than 2500 nm. In addition, although the first near-infrared ray may include a wavelength in all wavelength ranges of 780 nm or more and less than 2500 nm, it may also include a wavelength in at least a portion of the range. In addition, the first near-infrared ray includes a peak wavelength in a wavelength range of 780 nm or more and less than 2500 nm.

[0045] In addition, the first near infrared ray preferably (i) includes light with a wavelength of 780 nm or more and less than 1700 nm, and more preferably (ii) includes light with a wavelength of 780 nm or more and less than 1500 nm. The first near infrared ray further preferably (iii) includes light with a wavelength of 780 nm or more and less than 900 nm, and further preferably (iv) includes light with a wavelength of 800 nm or more and less than 900 nm. In addition, the first near infrared ray preferably includes a peak wavelength within the range of wavelengths of (i), (ii), (iii), and (iv) above.

[0046] In this embodiment, the near infrared light source 100 outputs the output light L including the first near infrared ray and visible light. Visible light is light having a wavelength of 380 nm or more and less than 780 nm. Visible light can be monochromatic light such as red light, green light, blue light, etc., but here it is white light.

[0047] The near-infrared light source 100 controls the first near-infrared ray and the visible light independently, and controls at least one of the on / off and dimming of each of the first near-infrared ray and the visible light. More specifically, the control unit 110 of the near-infrared light source 100 controls the first near-infrared ray and the visible light independently, and controls at least one of the on / off and dimming of each of the first near-infrared ray and the visible light. For example, the near-infrared light source 100 can control at least one of the on / off and dimming of only the first near-infrared ray while keeping the luminous mode of the visible light unchanged. Similarly, it can control at least one of the on / off and dimming of only the visible light while keeping the luminous mode of the first near-infrared ray unchanged.

[0048] The near infrared light source 100 of this embodiment includes a near infrared light source unit 120 for emitting first near infrared rays and a white light source unit 130 for emitting visible light (white light). The control unit 110 controls the near infrared light source unit 120 to emit first near infrared rays and the white light source unit 130 to emit visible light (white light).

[0049] As an example, the near-infrared light source unit 120 includes a first solid-state light emitting element. The first solid-state light emitting element may be, for example, a laser diode element, but is a near-infrared LED element in this case. The near-infrared LED element converts applied power into light (first near-infrared rays) and irradiates.

[0050] In addition, as another example, the near-infrared light source unit 120 includes a second solid-state light-emitting element and a fluorescent component. The second solid-state light-emitting element is an element that outputs excitation light, and may be, for example, a laser diode element, but is a blue LED element in this case. That is, the second solid-state light-emitting element outputs blue light as excitation light. The fluorescent component is a component that outputs wavelength-converted light as the first near-infrared ray based on the output excitation light (blue light). The fluorescent component includes, for example, Cr, which is easy to manufacture. 3+Activated phosphor or rare earth activated phosphor (especially from Tm 3+ , Er 3+ 、Nd 3+ and Yb 3+ At least one activated phosphor selected from the group consisting of

[0051] In addition, the case of using a near-infrared light source unit 120 having a second solid-state light-emitting element and a fluorescent member is described. As described above, the first near-infrared ray is light having a wavelength of 780 nm or more and less than 2500 nm. In the case of light having wavelengths (i), (ii), (iii), and (iv), a commercially available near-infrared light source unit 120 can be used. As a commercially available near-infrared light source unit 120, a near-infrared light source unit having a fluorescence peak at a wavelength selected from 780 nm, 850 nm, 940 nm, 980 nm, 1050 nm, 1200 nm, 1300 nm, 1400 nm, 1450 nm, 1500 nm, 1550 nm, 1600 nm, and 1650 nm is used. In addition, as the above-mentioned Cr 3+ The phosphor is activated, for example, by using Cr having a fluorescence peak at a wavelength selected from 790 nm, 800 nm, 825 nm, 850 nm, 880 nm, 925 nm, 950 nm, 970 nm, and 1035 nm. 3+ Activated phosphor. In addition, the rare earth activated phosphor is as follows. For example, Tm 3+ The activated phosphor has the property of having bright line-shaped fluorescence peak wavelengths at wavelengths of approximately 785 nm, 800 nm, and 820 nm. 3+ The activated phosphor has the property of having bright line-shaped fluorescence peak wavelengths at wavelengths of approximately 970 nm, 1005 nm, 1470 nm, 1530 nm, 1570 nm, 1615 nm, and 1645 nm. 3+ The activated phosphor has the property of having bright line-shaped fluorescence peak wavelengths at wavelengths of 880 nm, 935 nm, 1060 nm, 105 nm, 1335 nm, and 1420 nm. 3+ The activated phosphor has a property of having bright line-shaped fluorescence peak wavelengths at wavelengths of 970 nm, 1000 nm, and 1025 nm.

[0052] Next, the white light source unit 130 that radiates visible light (white light) has, for example, a yellow phosphor component and a blue LED element. A portion of the excitation light, which is the blue light output from the blue LED element, undergoes wavelength conversion through the yellow phosphor component and becomes yellow light. The yellow light and the blue light whose wavelength has not been converted are combined to output white light from the white light source unit 130. In addition, the white light source unit 130 is not limited to such a structure as long as it can emit visible light such as white light.

[0053] In this way, the near-infrared light source 100 outputs the output light L having the first near-infrared ray irradiated by the near-infrared light source unit 120 and the visible light (white light) irradiated by the white light source unit 130 to the plurality of characters. In the present embodiment, while the psychological state display system 1 generates psychological data and displays the temporal changes of the psychological data, the output light L is continuously lit, that is, continuously output.

[0054] With such a configuration, the image capturing device 200 can continuously capture multiple people, and the generating device 300 can continuously generate psychological data. Therefore, since the display device 400 can continuously display the generated psychological data, the temporal changes of the psychological data can be displayed.

[0055] In addition, the output of the first near infrared ray irradiated by the near infrared light source 100 (more specifically, the near infrared light source unit 120) is an output of 10W or more and 3kW or less. For example, the output of the first near infrared ray may be one of 30W, 100W, 300W, 1kW, and 3kW. In addition, in order to output a higher output of the first near infrared ray from the near infrared light source 100, the near infrared light source 100 may have a plurality of near infrared light source units 120. In addition, in the present embodiment, the mental state display system 1 includes one near infrared light source 100, but is not limited thereto, and may also include a plurality of near infrared light sources 100.

[0056] Thus, the near infrared light source 100 can make the first near infrared rays reach and irradiate multiple people in an area far from the near infrared light source 100. That is, the mental state display system 1 can visualize changes in the mental states of multiple people in an area far from the near infrared light source 100.

[0057] The imaging device 200 is a device that receives the first near infrared rays contained in the output light L outputted by the near infrared light source 100 and captures infrared images of a plurality of people, and is an infrared camera as an example. In addition, the imaging device 200 can capture infrared images all the time while the near infrared light source 100 is outputting the first near infrared rays.

[0058] When the output light L (first near infrared ray) is output to a plurality of people, the first near infrared ray is reflected by the plurality of people. The photographing device 200 photographs infrared images of the plurality of people by receiving the first near infrared ray reflected by the plurality of people. The infrared image is a still image and photographs the plurality of people. In this embodiment, the infrared image photographs at least one of the face and body of each of the plurality of people.

[0059] In addition, the camera 200 captures an infrared image at every specified time. That is, the camera 200 captures an infrared image, and when the specified time has passed, captures another infrared image, and repeats the process. The specified time is, for example, more than a few seconds and less than a few minutes, but is not limited to this. As an example, the camera 200 captures an infrared image (still image) every 5 seconds. In addition, the camera 200 can capture an infrared motion image instead of an infrared image (still image). In this case, each frame image contained in the infrared motion image can be used as an infrared image.

[0060] The photographing device 200 outputs the photographed infrared image to the generating device 300. As described above, the photographing device 200 repeatedly photographs infrared images. For example, when photographing one infrared image and outputting it to the generating device 300, the photographing device 200 further photographs the next infrared image and outputs it to the generating device 300. That is, the photographing device 200 repeatedly performs the process of outputting one infrared image to the generating device 300 each time it photographs one infrared image. In addition, for example, after photographing a plurality of infrared images, the photographing device 200 may aggregate the plurality of infrared images and output them to the generating device 300.

[0061] In addition, in the present embodiment, the communication unit of the photographing device 200 outputs the photographed infrared image to the generating device 300. The communication unit is a communication circuit for enabling the photographing device 200 to communicate with the generating device 300. In the present embodiment, the communication unit is a circuit for wireless communication, specifically, wireless communication is performed in accordance with communication standards such as BLE (Bluetooth (registered trademark) Low Energy) or Wi-Fi (registered trademark). In addition, the communication unit can also be a circuit for wired communication.

[0062] In addition, the intensity of the light-receiving signal of the imaging device 200 will be described here.

[0063] As described above, when outputting the output light L having the first near infrared ray, the camera 200 receives the first near infrared ray reflected by the plurality of people. The intensity of the light-receiving signal obtained by the light reception is set as the first light-receiving signal intensity. The first light-receiving signal intensity is the intensity of the light-receiving signal corresponding to the charge generated by the camera 200 (more specifically, the camera element of the camera 200) by receiving the first near infrared ray.

[0064] In addition, there is also a case where a plurality of people are irradiated with a second near infrared ray different from the first near infrared ray. The second near infrared ray is a near infrared ray output from a light source different from the near infrared light source 100. The second near infrared ray is, for example, a near infrared ray contained in sunlight, but is not limited thereto, and may also be a near infrared ray output from a lighting device. Here, a case is described where a plurality of people are irradiated with the second near infrared ray when the output light L having the first near infrared ray is not output. The shooting device 200 receives the second near infrared ray reflected by a plurality of people. The light receiving signal intensity obtained by the light reception is set to the second light receiving signal intensity. The second light receiving signal intensity is the intensity of the light receiving signal corresponding to the charge generated by the shooting device 200 (more specifically, the shooting element possessed by the shooting device 200) by receiving the second near infrared ray.

[0065] In this embodiment, the minimum value of the first light receiving signal intensity is greater than the maximum value of the second light receiving signal intensity. For example, the minimum value of the first light receiving signal intensity is preferably more than 2 times the maximum value of the second light receiving signal intensity, more preferably more than 5 times, and further preferably more than 10 times.

[0066] Furthermore, by setting the output of the first near infrared ray to 10 W or more and 3 kW or less, that is, by sufficiently increasing the output of the first near infrared ray, the minimum value of the first light receiving signal intensity becomes greater than the maximum value of the second light receiving signal intensity.

[0067] As a result, the intensity of the second light receiving signal that becomes noise in the signal for identifying the plurality of persons becomes sufficiently small. Therefore, since such noise is sufficiently reduced, the imaging device 200 can obtain an infrared image of the plurality of persons with higher accuracy.

[0068] The photographing device 200 may include a magnification control mechanism (not shown) for controlling the magnification of the photographed infrared image. In this case, the lens or the like provided by the magnification control mechanism is advantageous in obtaining an infrared image focused on each of the plurality of persons.

[0069] The generating device 300 is a device that generates psychological data representing a psychological state using the captured infrared image and the learned machine learning model 321. In addition, the generating device 300 determines the stimulation to be applied to a plurality of characters based on the generated psychological data. The generating device 300 is, for example, a personal computer, but may also be a server device connected to a network and having high computing power.

[0070] Here, the generating device 300 includes a communication unit 310 , a generating unit 320 , and a learning unit 330 .

[0071] The communication unit 310 is a communication circuit for the generation device 300 to communicate with the camera 200, the display device 400, and the control device 500. In this embodiment, the communication unit 310 is a circuit for wireless communication, specifically, wireless communication according to a communication standard such as BLE or Wi-Fi (registered trademark). In addition, the communication unit 310 may be a circuit for wired communication.

[0072] The communication unit 310 includes an acquisition unit 311 and an output unit 312. The acquisition unit 311 acquires the infrared image captured and output by the camera 200. As described above, the camera 200 repeatedly performs the process of capturing the infrared image and the process of outputting the infrared image, so the acquisition unit 311 sequentially acquires the output infrared images. In other words, the acquisition unit 311 acquires a plurality of output infrared images. In addition, the output unit 312 outputs the generated psychological data to the display device 400, and outputs the stimulation data representing the determined stimulation to the control device 500.

[0073] The generating unit 320 generates psychological data based on the infrared image acquired by the acquiring unit 311. More specifically, the generating unit 320 stores a machine learning model 321 and generates psychological data representing the psychological state of at least one person among the plurality of persons using the machine learning model 321. In addition, each time the generating unit 320 acquires an infrared image, it generates psychological data corresponding to the infrared image.

[0074] As described above, the mental state refers to states such as happiness, sadness, energy, fear, anger, frustration, wakefulness, absent-mindedness, interest, and tension as an example, but is not limited thereto.

[0075] The machine learning model 321 is a learned model that takes as input the infrared image acquired by the acquisition unit 311 and outputs as output the psychological state of at least one person among the plurality of persons shown in the infrared image. In this embodiment, the learning unit 330 causes the machine learning model 321 to learn.

[0076] The psychological data is, for example, data indicating the psychological state of at least one person among the plurality of persons, and is data for display on the display device 400, more specifically, digital data. Examples of the psychological data include display data such as digital display data, color display data, pattern display data, graphic symbol display data, emoticon display data, and illustration display data. In addition, in the present embodiment, the psychological data is image data obtained by superimposing at least one of these display data on the captured infrared image. In the case where the psychological data is image data, the plurality of persons captured in the infrared image are associated with the psychological states of the persons and displayed on the display device 400.

[0077] In addition, the generation unit 320 determines the stimulation to be applied to the plurality of characters based on the generated psychological data. For example, the generation unit 320 determines the stimulation to be applied to the plurality of characters based on the generated psychological data and the corresponding data stored in the storage unit (not shown) possessed by the generation device 300. The corresponding data is data representing the psychological state indicated by the psychological data and the stimulation corresponding to the psychological state in correspondence. The generation unit 320 refers to the corresponding data and determines the stimulation corresponding to the generated psychological data as the stimulation to be applied to the plurality of characters. In addition, by applying stimulation to the character, it is expected that the psychological state of the character will change. The stimulation data shows the stimulation determined in this way. Thus, the stimulation used to change the psychological state can be determined according to the psychological state of the character.

[0078] Specifically, such a generation unit 320 is realized by a processor, a microcomputer, or a dedicated circuit that executes a program.

[0079] The learning unit 330 uses the training data to make the machine learning model 321 learn. Specifically, the learning unit 330 is implemented by a processor, a microcomputer, or a dedicated circuit that executes a program.

[0080] The learning unit 330 constructs the machine learning model 321 by learning. The learning unit 330 supplies the constructed machine learning model 321 to the generating unit 320. The learning unit 330 is not an essential component and may not be provided in the mental state display system 1.

[0081] The machine learning model 321 is a model for generating psychological data.

[0082] In this embodiment, the machine learning model 321 is a model constructed by machine learning using one or more data sets as training data. A data set refers to a combination of an infrared image captured by the camera 200 and the psychological state of a person shown in the infrared image. In addition, at least one of the face and body of the person is captured in the infrared image. The entire face and body of the person are preferably captured in the infrared image, but are not limited to this. For example, a portion of the face or a portion of the body of the person may also be captured in the infrared image.

[0083] In other words, the machine learning model 321 is a recognition model constructed by machine learning using one or more data sets as training data, wherein the one or more data sets are respectively data sets of infrared images and psychological states of persons shown in the infrared images. More specifically, the machine learning model 321 is a recognition model constructed using infrared images belonging to each of the one or more data sets as the training data as input data, and using the psychological states of persons shown in the infrared images belonging to the data sets as output data.

[0084] As an example, as described above, the learning unit 330 uses machine learning to train the model. Therefore, in this embodiment, the model is the machine learning model 321.

[0085] In addition, the learning unit 330 can use, for example, a neural network, more specifically, a convolutional neural network (CNN) to learn the machine learning model 321. When the machine learning model 321 is a convolutional neural network model, the learning unit 330 determines the coefficients (weights) of the filters of the convolutional layer through machine learning based on the training data.

[0086] In addition, the learning unit 330 may use an algorithm other than a neural network to cause the machine learning model 321 to learn.

[0087] The display device 400 displays the temporal changes of the psychological data output from the generating device 300. More specifically, since the generating unit 320 generates and outputs a piece of psychological data corresponding to each infrared image obtained, the display device 400 sequentially obtains and displays the outputted psychological data. Therefore, the display device 400 displays the temporal changes of the psychological data.

[0088] The display device 400 is a light-emitting or non-light-emitting monitoring device. The display device 400 is composed of an organic EL (Electro Luminescence) panel as a light-emitting monitoring device or a liquid crystal panel as a non-light-emitting monitoring device. The display device 400 can be fixed or portable. Examples of fixed display devices 400 include monitors for personal computers, and examples of portable display devices include smartphones and tablet terminals.

[0089] The display device 400 is, for example, a device that can be visually recognized by an administrator of the psychological state display system 1. The administrator can understand changes in the psychological states of a plurality of characters, etc., by viewing changes in displayed psychological data over time. That is, the psychological state display system 1 can visualize changes in the psychological states of characters.

[0090] By visualizing changes in the psychological state of a character in this manner, for example, an administrator viewing the display device 400 can know the following.

[0091] When the mental state display system 1 is used in a space with a plurality of characters, the administrator can early detect a suspicious person who is nervous among the plurality of characters, or a frustrated person who cannot speak because of the large number of characters.

[0092] Furthermore, when the mental state display system 1 is used in a seminar venue (an example of a learning space) where lectures are held, the mental state of the lecturers such as the interest level can be made clear. In this case, the administrator can inform the lecturer of the timing of a break according to the interest level.

[0093] The control device 500 is a device for controlling the stimulation device 600, such as a personal computer, but may also be a server device connected to a network and having high computing power. In addition, the control device 500 may also be any other device if it can control the stimulation device 600, such as a dedicated device for controlling the stimulation device 600.

[0094] The control device 500 acquires the stimulation data output from the generation device 300. The control device 500 controls the stimulation device 600 so as to apply the stimulation indicated by the acquired stimulation data to a plurality of characters.

[0095] Thereby, a psychological state display system 1 is realized which can control the psychological state of a person by applying a stimulus for changing the psychological state to the person according to the psychological state of the person.

[0096] The stimulation device 600 is a device controlled by the control device 500 and applies stimulation to multiple characters. The stimulation is a stimulation that acts on at least one of the vision, hearing, smell and touch of the multiple characters. The stimulation is, for example, at least one stimulus selected from light, sound, smell, vibration, wind, temperature and humidity. In the case where the stimulus is light, it also includes the meaning of the image. As an example, the stimulation device 600 includes but is not limited to a light source device, a projector device, a speaker, a vibration device, an air supply device, a temperature control device and a humidity control device.

[0097] In addition, the control device 500 preferably controls the stimulation device in a manner that cuts off or suppresses the stimulation applied to the multiple characters. For example, when the stimulation device 600 is a light source, the control device 500 can cut off (extinguish) or suppress (reduce the light amount) the light after applying light as a stimulus to the multiple characters.

[0098] Here, use Figure 3 and Figure 4 The relationship between the near-infrared light source 100 and the person will be described below.

[0099] Figure 3 is a schematic diagram of the near infrared light source 100 and a person according to the present embodiment viewed in the vertical direction. More specifically, Figure 3 This is a diagram in which the near-infrared light source 100 and one of the plurality of persons are viewed vertically downward, in other words, this is a diagram in which the near-infrared light source 100 and one of the persons are viewed from the top of the person's head. Figure 4 Observed from the side Figure 3 Schematic diagram of .

[0100] Here, it is explained Figure 3 and Figure 4 The front direction D1 shown. The front direction D1 is the direction that the head of at least one of the multiple characters faces. In the present embodiment, the near-infrared light source 100 outputs the output light L to a character from a direction other than the front direction D1 that the head of the character faces. The first near-infrared ray is irradiated to the character from a direction that will not create a shadow directly behind the face of the character facing the front. That is, the first near-infrared ray possessed by the output light L is not irradiated from the front direction D1 of the head (face) of the character, in other words, it is irradiated from above or below, or from the side (left or right) or obliquely to the face of the character.

[0101] Here, the optical axis La is set to be the direction of the light that advances toward the angle with the highest radiation intensity in the first near infrared ray. When the first near infrared ray contained in the output light L is irradiated from the side or oblique side of the face of the one person, the optical axis La is not parallel to the front direction D1. In addition, when the first near infrared ray contained in the output light L is irradiated from above or below the face of the one person, as Figure 3As shown, when viewed in the vertical direction, the optical axis La may be parallel to the front direction D1.

[0102] In this way, the mental state display system 1 of the present embodiment does not need to irradiate the first near infrared ray from the front direction D1 of the face of the person.

[0103] In addition, the area of ​​the irradiated area by the first near infrared ray is 10m 2 Above and less than 10000m 2 In addition, the area is preferably 25m 2 More than 100m 2 above.

[0104] Here, the upper limit of the area of ​​the irradiated region of the first near-infrared irradiation is not particularly limited. The reason is that the first near-infrared imaging depends on the light quantity of the first near-infrared ray. In addition, when setting the upper limit in consideration of the actual conditions of the lighting technology and the imaging technology, it is preferably less than the above 10000m 2 degree.

[0105] Thus, the first near infrared rays can irradiate a plurality of people in a wide space. That is, the psychological state display system 1 can visualize changes in the psychological states of a plurality of people in a wide space.

[0106] In addition, Figure 4 The distance R1 shown is used for illustration.

[0107] The distance R1 is the distance between the person located closest to the infrared light source 100 among the multiple persons and the near-infrared light source 100. More specifically, the distance R1 is the distance between the person located closest to the infrared light source 100 among the multiple persons and the light emitting surface of the near-infrared light source 100. The output surface is a surface from which the output light L is output.

[0108] The distance R1 may be 50 cm or more and less than 30 m. In addition, the distance R1 is 50 cm or more and less than 3 m (first lighting distance), 3 m or more and less than 5 m (second lighting distance), or 5 m or more and less than 20 m (third lighting distance).

[0109] When the distance R1 is the first lighting distance, it is suitable for visualizing the psychological state of people in offices, seminar venues, classrooms, stores, public spaces (government offices and libraries, etc.), and public mobile spaces (inside trains and buses), etc. When the distance R1 is the second lighting distance, it is suitable for visualizing the psychological state of people in indoor event venues, movie theaters, gymnasiums, small outdoor event venues, and relatively narrow areas in cities, etc. When the distance R1 is the third lighting distance, it is suitable for visualizing the psychological state of people in large outdoor event venues, sports venues, and relatively wide areas in cities, etc.

[0110] Furthermore, it is preferable that the distance between each of all the persons among the plurality of persons and the near-infrared light source 100 is within the above range.

[0111] The first near infrared ray illumination pattern can be selected from narrow-angle illumination with an illumination angle of less than 15°, medium-angle illumination with an illumination angle of 15° or more and less than 30°, wide-angle illumination with an illumination angle of 30° or more and less than 90°, and Lambert illumination.

[0112] The closer the illumination pattern is to narrow-angle illumination, the more the first near infrared rays become beam-shaped. Therefore, an illumination pattern is achieved that is advantageous for irradiating a region far from the near infrared light source 100 with the first near infrared rays.

[0113] On the other hand, as the illumination pattern approaches Lambert illumination, uniform first near infrared rays are softly irradiated over a wide range. Thus, an illumination pattern is achieved that is advantageous for irradiating a wider area with the first near infrared rays.

[0114] In addition, the near-infrared light source 100 may have an illumination control mechanism (not shown) for controlling the illumination of the output first near-infrared light. In this case, the first near-infrared light may be beam-shaped or close to Lambert illumination by the lens and / or reflector provided by the illumination control mechanism. Therefore, for example, the first near-infrared light suitable for generating psychological data may be irradiated.

[0115] In addition, the description is as follows Figure 3 When the near-infrared light source 100 is viewed in the vertical direction shown, a horizontal divergence angle θ of the first near-infrared ray of the output light L is obtained.

[0116] The divergence angle θ refers to the range of the angle of the first near infrared ray outputted so that the radiation intensity at the same distance is half of the maximum intensity. In the present embodiment, the divergence angle θ is preferably greater than 30° and less than 180°, and more preferably greater than 45° and less than 120°. When the divergence angle θ is within such a range, a wider area away from the near infrared light source 100 can be irradiated with the first near infrared ray. Therefore, the space of the mental state display system 1 can be saved.

[0117] [Processing order of mental state visualization method]

[0118] Next, a specific processing sequence of the mental state visualization method executed by the mental state display system 1 will be described.

[0119] Figure 5 This is a flowchart showing the processing procedure for the mental state display system 1 of this embodiment to make the machine learning model 321 learn.

[0120] First, the learning unit 330 obtains training data for learning the machine learning model 321 (S101). The training data may be generated by the generating unit 320, but the present invention is not limited thereto and may be generated by other processing units or other devices.

[0121] The following describes the training data.

[0122] As mentioned above, the training data is more than one dataset.

[0123] One data set is composed of a combination of an infrared image captured by the camera 200 and a psychological state of a person shown in the infrared image. For example, the face and body of the person are captured in the infrared image. In addition, the infrared image is combined with one of happiness, sadness, energy, fear, anger, frustration, wakefulness, absent-mindedness, interest, and tension as the psychological state of the person.

[0124] In addition, visible data collected by a visible light sensor under visible light can also be used as training data.

[0125] After step S101, the learning unit 330 uses the training data obtained in step S101 to learn the machine learning model 321 (S102). More specifically, the learning unit 330 learns the machine learning model 321 through machine learning. In addition, the learning unit 330 outputs the learned machine learning model 321 to the generation unit 320.

[0126] In progress Figure 5 After the process of the flowchart shown, the mental state display system 1 performs a process of visualizing the mental state of the person.

[0127] Figure 6 This is a flowchart showing a processing procedure for visualizing a person's psychological state by the psychological state display system 1 according to the present embodiment.

[0128] The near-infrared light source 100 outputs output light L having first near-infrared rays to a plurality of persons ( S201 ).

[0129] The photographing device 200 receives the first near infrared rays contained in the output light L output by the near infrared light source 100, and photographs infrared images of a plurality of persons (S202). In this operation example, the photographing device 200 photographs infrared images at each predetermined time. When an infrared image is photographed and output to the generating device 300, the photographing device 200 further photographs the next infrared image and outputs it to the generating device 300. That is, the photographing device 200 sequentially outputs the photographed infrared images to the generating device 300, and the generating device 300 sequentially obtains the infrared images sequentially output by the photographing device 200.

[0130] The generating device 300 uses the machine learning model 321 that receives the infrared image as input to generate psychological data representing the psychological state of at least one person among the multiple persons shown in the infrared image (S203).

[0131] When the machine learning model 321 receives an infrared image acquired by the generation device 300, it outputs the psychological state of at least one person among the plurality of people shown in the infrared image. The generation device 300 generates psychological data representing the psychological state.

[0132] In addition, each time the generating device 300 acquires an infrared image, it generates a psychological data corresponding to the infrared image. As described above, since the generating device 300 acquires infrared images sequentially, the generating device 300 generates psychological data sequentially. Each time the generating device 300 generates psychological data, it outputs the generated psychological data to the display device 400. That is, the display device 400 acquires the generated psychological data sequentially.

[0133] The display device 400 displays the acquired psychological data (S204). In addition, as described above, the display device 400 acquires the generated psychological data in sequence, so it displays the psychological data in sequence. As described above, the shooting device 200 shoots infrared images at predetermined intervals. Therefore, on the display device 400, after displaying one psychological data based on one infrared image, the next psychological data based on the next infrared image shot after a predetermined time of the one infrared image is displayed. That is, the display device 400 displays the time change of the psychological data. In other words, the display device 400 displays the situation in which the psychological state of the person changes from moment to moment.

[0134] Here, the mental state display system 1 of the present embodiment is compared with the technology disclosed in Patent Document 1.

[0135] Patent document 1 discloses displaying a temporary psychological state of a person. However, the psychological state of the person before or after the displayed psychological state is unclear. In addition, Patent document 1 discloses extracting a plurality of images (still images) from a moving image, calculating the average value of each of the plurality of images, and displaying one psychological state based on the calculated average value. In this case, since only one psychological state is displayed, it is unclear whether the psychological state of the person has changed in the plurality of images.

[0136] It is natural that the psychological state of a person (character) changes from moment to moment. In Patent Document 1, the temporary psychological state of a person is shown, but how the psychological state of the person changes is not shown.

[0137] In this embodiment, for example, Figure 6 As shown in step S204, the psychological state of the person that changes moment by moment is displayed. That is, the psychological state display system 1 of this embodiment can visualize the time change of the psychological state of the person.

[0138] Furthermore, when the image used in Patent Document 1 is a visible light image, the image is affected by the illumination light (visible light) of the space where the image is taken. For example, the image quality of the image is affected by the way the illumination light of the space emits light, and as a result, the displayed psychological state is also affected. That is, in Patent Document 1, the displayed psychological state is affected by the illumination light (visible light) of the space where the image is taken.

[0139] On the other hand, in the psychological state display system 1 of the present embodiment, the temporal change of the psychological state of the person is visualized using the first near infrared rays (more specifically, infrared images). Therefore, when visualizing the change of the psychological state of the person, the psychological state display system 1 is not easily affected by the illumination light (visible light) of the space where the psychological state display system 1 is used. That is, the psychological state display system 1 is realized that can visualize the change of the psychological state of the person while suppressing the influence of the illumination light.

[0140] Here, use Figure 7 and Figure 8 Explain the psychological data in more detail.

[0141] Figure 7 3 is a diagram showing an infrared image P1 captured by the imaging device 200 according to the present embodiment.

[0142] Figure 8 is based on Figure 7 An example of psychological data of the infrared image P1 shown.

[0143] Figure 7The infrared image P1 shown captures a plurality of persons, more specifically, four persons, A, B, C, and D. FIG.

[0144] As described above, when the infrared image P1 is used as input in the machine learning model 321, the psychological state of at least one of the four characters is output. Here, the psychological states of the four characters (i.e., all the characters) are output. In other words, Figure 8 The psychological data shown represent the psychological states of each of the four characters.

[0145] Here, the degree of interest indicating the degree of interest is used as the psychological state. Figure 8 In FIG. 1 , the interest levels of each of the four characters are shown as psychological data.

[0146] In addition, the displayed psychological state level is preferably 2 or more and 11 or less, and more preferably 6 or less. For example, when the interest level is 6, the interest level is displayed from high to low as 5 points, 4 points, 3 points, 2 points, 1 point, and 0 points. In addition, not limited to the interest level, the level of other psychological states (such as the happiness level) is also preferably 2 or more and 11 or less, and preferably 6 or less.

[0147] In this way, when the displayed level is within the above range, the psychological state of each of the plurality of characters can be roughly grasped, and the amount of data to be processed can be reduced. Therefore, it is advantageous to miniaturize the device used in the psychological state display system 1 and to improve the data processing speed.

[0148] In addition, as described above, as psychological data, for example, display data such as digital display data, color display data, pattern display data, graphic symbol display data, emoticon display data, and illustration display data can be cited. In addition, in the present embodiment, psychological data is image data obtained by superimposing at least one of these display data on the captured infrared image. In the case where the psychological data is image data, a plurality of people captured in the infrared image are associated with the psychological states of the people and displayed on the display device 400.

[0149] When using numbers to display data, it is advantageous to display psychological states in detail and to express them numerically. When using colors to display data, it is advantageous to express psychological states with colors. When using patterns to display data, graphic symbols to display data, and emoticons to display data, it is advantageous to express psychological states by using pictures or text pattern symbols as metaphors. The method of displaying data with illustrations is advantageous in expressing psychological states with illustrations. In addition, when using image data, it is advantageous to express a person and his or her psychological state in association.

[0150] Here, use Figure 8 To illustrate an example of using color to display data.

[0151] Figure 8 The image data P2 is shown in which the color display data is superimposed on the captured infrared image P1.

[0152] exist Figure 8 The psychological data shown shows the colors corresponding to the psychological states of the four characters. The darker the color, the higher the interest level, and the lighter the color, the lower the interest level. Figure 8 In the example, A, B, and C are overlaid with dark colors, indicating a high level of interest, and D is overlaid with light colors, indicating a low level of interest. Figure 8 In the text, the colors corresponding to the psychological state are expressed by shades, but they can also be expressed by colors.

[0153] In this way, it is particularly convenient to display the psychological state of a person using colors (hues). In particular, when displaying using colors, an excited state can be displayed in red, a slightly excited state can be displayed in yellow, a calm state can be displayed in green, a calm state can be displayed in blue, and a depressed state can be displayed in gray or black for distinction. In this way, the psychological state display system 1 of this embodiment can visualize the temporal changes of the psychological state of a person by using colors, so that, for example, a user viewing the display device 400 can intuitively understand the temporal changes of the psychological state of the person.

[0154] Furthermore, as described above, the displayed psychological state level is preferably not more than level 6. For example, when the psychological state of a person is level 6, if the psychological state of the person is displayed in rainbow colors, it is easier to understand the temporal changes in the psychological state of the person.

[0155] In addition, use Fig. 9 Other examples of psychological data will be described.

[0156] Fig. 9 is based on Figure 7 Another example of psychological data of the infrared image P1 shown. More specifically, Fig. 9 The image data P3 in which the graphic symbol display data is superimposed on the captured infrared image P1 is shown.

[0157] exist Fig. 9 The psychological data shown in FIG. 1 show graphic symbols corresponding to the psychological states of the four characters. The graphic symbols are displayed as Fig. 9 The dashed box shown in FIG. The larger the box, the higher the interest level, and the smaller the box, the lower the interest level. In other words, the interest level is shown by the size of the display area. Fig. 9In FIG. 1 , large frames are superimposed on A, B, and C, indicating a high degree of interest, and a small frame is superimposed on D, indicating a low degree of interest. In addition, the degree of interest may be indicated by other graphic symbols.

[0158] like Fig. 9 As shown, when the data is displayed using graphic symbols, the psychological state of the person is emphasized. In this way, the psychological state display system 1 of this embodiment can visualize the time change of the psychological state of the person through the graphic symbols, so that, for example, a user who views the display device 400 can intuitively understand the time change of the psychological state of the person.

[0159] In addition, use Fig.10 Other examples of psychological data will be described.

[0160] Fig.10 This is another example of psychological data in this embodiment.

[0161] More specifically, Fig.10 The image data in which the digital display data is superimposed on the captured infrared image is shown. Fig.10 (a) is a diagram showing image data P4. The image data P4 is psychological data based on the infrared image P1, and is image data in which digital display data is superimposed on the captured infrared image P1. Fig.10 (b) is a diagram showing image data P5, which is image data based on psychological data of an infrared image captured a predetermined time after the infrared image P1 is captured. Fig.10 (c) is a diagram showing image data P6. Image data P6 is based on the Fig.10 (b) is the image data of the psychological data of the infrared image taken after a prescribed time of the infrared image. That is, Fig.10 It means, for example, Figure 6 In step S204, a graph of temporal changes in psychological data is displayed on the display device 400.

[0162] exist Fig.10 The psychological data shown shows numbers corresponding to the psychological states of the four characters. In addition, here, the interest level as a psychological state is expressed in 6 levels, and is displayed from high to low as interest level 5 points, interest level 4 points, interest level 3 points, interest level 2 points, interest level 1 point, and interest level 0 points. Fig.10 In (a) and (c), "5 points of interest" are added to A, B, and C, indicating a high level of interest, and "1 point of interest" is added to D, indicating a low level of interest. Fig.10In (b), "interest level 5 points" is superimposed on A and C, indicating a high interest level, "interest level 3 points" is superimposed on B, indicating a medium interest level, and "interest level 1 point" is superimposed on D, indicating a low interest level.

[0163] like Fig.10 As shown, when digital display data is used, it is easy to display the psychological state in detail and to express it digitally. In this way, the psychological state display system 1 of this embodiment can visualize the time change of the psychological state of a person by digital means, so that, for example, a user who views the display device 400 can intuitively understand the time change of the psychological state of the person.

[0164] In addition, Figures 8 to 10 In the embodiment, image data in which at least one of the display data is superimposed on the captured infrared image is used as psychological data, but it is not limited thereto.

[0165] For example, when only numerical display data is used as psychological data, the psychological data based on the infrared image P1 may be text data including numbers such as "The interest levels of A, B, and C are 5 points, and the interest level of D is 1 point."

[0166] In addition, in this embodiment, the psychological data representing the psychological states of the plurality of characters are displayed, but the present invention is not limited thereto. For example, the psychological data representing the average psychological state of the plurality of characters may be displayed. In this case, data such as "the interest level of the plurality of characters is 4 points" may be used as the equivalent of Fig.10 (a) Psychological data.

[0167] In this way, the mental state display system 1 capable of visualizing the temporal changes of the average mental states of a plurality of persons is realized.

[0168] Furthermore, in the case of using numerical display data as psychological data, the scores indicating the degree of interest may be integrated and displayed. Fig.10 When the scores indicating the degree of interest shown in (a) to (c) are integrated, A and C are 15 points, B is 13 points, and D is 3 points, and these values ​​can be displayed on the display device 400.

[0169] In this way, the psychological state display system 1 capable of visualizing the psychological state of a person during a certain period of time is realized.

[0170] Furthermore, in the case of using numerical display data as psychological data, a score indicating the degree of interest may be differentiated and displayed.

[0171] In this way, the mental state display system 1 capable of visualizing the temporal changes of a person's mental state in a short period of time is realized.

[0172] Here, return to Figure 6 , indicating further processing.

[0173] The generating device 300 determines the stimulus to be applied to the plurality of characters based on the generated psychological data (S205). In addition, the generating device 300 may determine to apply the stimulus to all of the plurality of characters, or may determine to apply the stimulus to a portion of the plurality of characters.

[0174] Furthermore, when the psychological data generated by the generating device 300 indicates that the psychological state of one or more characters is a predetermined psychological state, the generating device 300 may determine the stimulus to be applied to the one or more characters based on the predetermined psychological state.

[0175] This limits the characters to which stimulation is applied to change the psychological state, and eliminates the need to apply stimulation to characters that are not the target among the multiple characters. Therefore, a psychological state display system 1 that can efficiently control the psychological state of the target character is realized.

[0176] Here, the prescribed mental state and stimulus are explained.

[0177] For example, when the predetermined psychological state is "interest level is 1 point or less", the stimulus applied to the one or more persons is light. Fig.10 As shown, the psychological state of D as a character is "interest level is less than one point", in which case the generation device 300 determines light as the stimulus applied to D. In this case, the stimulus device 600 is a light source device that emits the light (more specifically, visible light).

[0178] The communication unit 310 of the generation device 300 outputs stimulation data indicating the determined stimulation to the control device 500. The control device 500 acquires the output stimulation data.

[0179] The control device 500 controls the stimulation device 600 based on the output stimulation data (S206). For example, the control device 500 controls so that the stimulation device as a light source device irradiates light toward D. As a result, it can be expected that the psychological state of D will change, for example, the degree of interest will increase. In this case, it is not necessary to apply stimulation to A, B, and C. In other words, only D can be subjected to efficient stimulation.

[0180] In addition, the above example is an example in which the generation device 300 determines to apply a stimulus to a part of the multiple characters (i.e., D), and the control device 500 controls the stimulation device 600 to apply the determined stimulus to at least a part of the multiple characters, but the present invention is not limited thereto. The generation device 300 may determine to apply a stimulus to all of the multiple characters (e.g., A, B, C, and D), and the control device 500 controls the stimulation device 600 to apply the stimulus to all of the multiple characters.

[0181] In addition, the generating device 300 may determine the stimulus to be applied to some or all of the multiple characters based on the temporal changes in the psychological state indicated by the generated psychological data. Fig.10 In the example, the psychological states of A, C, and D do not change over time, and their interest levels remain constant, while the psychological state of B changes over time, and the interest level changes. In this case, for example, the generating device 300 may determine to apply stimulation to A, C, and D whose psychological states do not change over time, or may determine to apply stimulation to B whose psychological state changes over time.

[0182] In the above, light is used as a stimulus when the psychological state is "the interest level is 1 point or less", but the present invention is not limited to this. For example, other examples are described below.

[0183] As another example, as described above, when there is a suspicious person whose psychological state is in a "tense state", a stimulus to suppress criminal behavior (for example, shining a strong light on the suspicious person, a sound (voice) such as "the police are patrolling") may be determined.

[0184] As another example, when the psychological state display system 1 is used in an entertainment space, by controlling stimuli such as sound and light according to the interest level of the psychological state, the activity can be enlivened. Or, it is possible to evoke strong emotions in the people in the space. In addition, on the contrary, it is possible to apply stimuli to calm people who are in a state where they cannot suppress their emotions.

[0185] In addition, the display device 400 of this embodiment can display the vital signs information of a person. Figure 6 In step S204 shown, psychological data and vital sign information of the person may be displayed. In this case, the vital sign information may be obtained, for example, in the following manner. The photographing device 200 photographs an infrared image including blood vessel information related to the contraction of blood vessels of each of the plurality of persons. The generating device 300 generates vital sign information representing the vital sign of at least one person among the plurality of persons based on the blood vessel information included in the photographed infrared image.

[0186] Blood vessel information is, for example, information on the light reflectance of the skin of a person captured in an infrared image. In addition, vital signs refer to, for example, heart rate. Since blood has the property of absorbing light, its light reflectance changes according to the contraction of blood vessels caused by the pulse. Therefore, based on the information on the light reflectance of the person's skin (blood vessel information), the pulse is estimated and the heart rate is calculated. Vital sign information is information showing the heart rate (vital sign) of the person.

[0187] In addition, the display device 400 displays the generated vital sign information. As an example, when the display Figure 8 When the image data P2 is shown, the display device 400 can superimpose the vital sign information on the image data P2 and display it.

[0188] Thus, the mental state display system 1 can visualize not only the temporal changes in the mental state of a person, but also the vital sign information of the person and the temporal changes in the vital sign information.

[0189] Here, the space in which the mental state display system 1 is used and the effects obtained in the space are described in more detail.

[0190] As described above, the mental state display system 1 is, as an example, a system used in a space such as an entertainment space, a public space, or a learning space. The entertainment space, the public space, or the learning space is a relatively wide space.

[0191] In addition, entertainment spaces include spaces such as (1) or (2) below. (1) Performance lighting spaces where the lighting state formed by visible light changes rapidly (live performance venues, lighting spaces for preset purposes, etc.). (2) Dark and dim spaces where people experience images or sounds (movie theaters, snack bars, and karaoke halls, etc.). In addition, lighting spaces for preset purposes refer to outdoor performance lighting spaces that create a sense of liveliness in the space by applying movement, brightness, and color changes to the lighting light, while also influencing people's psychological state and actions (such as "touring" and "staying").

[0192] In addition, public space means a space including public moving bodies (trains, buses, taxis, airplanes, etc.), for example, including the following (3) space. (3) Moving space at night where external light such as sunlight is repeatedly incident or blocked, and moving space where people move alternately between darkness (tunnel space, etc.) and light (the interior of a bus traveling on a highway, etc.).

[0193] In addition, the learning space includes a space where a person is learning at home, school, a class, or a seminar venue, and the space may include a person who is learning and a person who guides the learning (a lecturer, etc.). The mental state display system 1 can be used in a learning space, especially in the following space (4). (4) At least one illuminated space illuminated by white lighting with different light colors (chromaticity, etc.).

[0194] In entertainment spaces and public spaces (particularly the spaces shown in (1), (2), and (3) above), the presence, intensity, and hue of visible light in the space are prone to significant changes, and the psychological state of the person in the space is prone to fluctuate. In this embodiment, the first near-infrared ray (more specifically, an infrared image) is used to visualize the changes in the psychological state of the person. That is, even if the presence, intensity, and hue of visible light in the space change significantly, the psychological state display system 1 can easily visualize the changes in the psychological state of a person whose psychological state is prone to fluctuate because the first near-infrared ray having a wavelength region different from that of visible light is used.

[0195] In addition, in particular, in a learning space such as (4), for example, when estimating the psychological state of a person using visible light and a visible light image based on the visible light while illuminating the person with a plurality of white illumination lights each having a different chromaticity, the accuracy of the estimation sometimes decreases depending on the chromaticity. Even in such a learning space, since the psychological state display system 1 of the present embodiment uses the first near infrared ray, it is possible to easily visualize changes in the psychological state of the person. In addition, not limited to the learning space, the same effect is expected in a space such as (4).

[0196] Thus, even in a space where the presence, intensity, and color tone of visible light are prone to change greatly, or in a space where a person is illuminated by a plurality of white lighting lights of different chromaticities, a psychological state display system 1 that can easily visualize changes in a person's psychological state can be realized. In particular, in these environments, suspicious persons can be found early, and crime can be suppressed by using the stimulation device 600. Therefore, such a psychological state display system 1 is a system that can be expected to have the effect of eliminating social problems, such as revitalizing the economy, eliminating the gap between the rich and the poor, and creating a safe and reliable society.

[0197] Furthermore, a comparison is made between the conventional art and the mental state display system 1 of the present embodiment.

[0198] For example, Patent Document 2 (Japanese Patent Publication No. 2009-87303) discloses a technology for estimating the expression of anxiety or surprise of a driver in a vehicle. In addition, Patent Document 3 (Japanese Patent Publication No. 2020-48149) discloses a technology for displaying the emotions of participants in an online meeting in a conference room.

[0199] The technology disclosed in Patent Document 2 is suitable for narrow spaces such as inside a vehicle, but is difficult to apply to wide spaces such as entertainment spaces and public spaces (particularly the spaces of (1), (2), and (3) above).

[0200] Furthermore, since conference rooms and the like in which the technology disclosed in Patent Document 3 is used are generally white-lit spaces, the technology disclosed in Patent Document 3 is difficult to apply to spaces such as the above-mentioned (4).

[0201] However, as described above, the psychological state display system 1 of this embodiment can also easily visualize the changes in a person's psychological state in an entertainment space, a public space, or a learning space (especially the spaces (1), (2), (3) or (4) mentioned above).

[0202] [Modifications]

[0203] Here, a modification of the embodiment is described. In modification 1 of the embodiment, the difference from the embodiment is that the stimulation device is a near-infrared light source 100a. Hereinafter, the description will be centered on the differences from the embodiment, and the description of common features will be omitted or simplified.

[0204] use Fig.11 A configuration example of a mental state display system 1 a according to a first modification of the embodiment will be described.

[0205] Fig.11 This is a block diagram showing the functional configuration of a mental state display system 1 a according to Modification 1 of the present embodiment.

[0206] The mental state display system 1 a includes a near-infrared light source 100 a , an imaging device 200 , a generating device 300 , a display device 400 , and a control device 500 .

[0207] In this modification, the stimulation device is a near-infrared light source 100a that outputs output light L having visible light. In addition, the generation device 300 determines the stimulation to be applied to the plurality of characters based on the generated psychological data, and the control device 500 controls the stimulation device (near-infrared light source 100a) to apply the determined stimulation to the plurality of characters. Here, the visible light output from the near-infrared light source 100a is determined as the stimulation, and the visible light is output toward the plurality of characters.

[0208] As a result, the number of components of the mental state display system 1 a of the modified example is reduced, and the structure of the mental state display system 1 a is simplified.

[0209] [Effects, etc.]

[0210] The psychological state display system 1 of this embodiment includes a near-infrared light source 100, a photographing device 200, a generating device 300, and a display device 400. The near-infrared light source 100 outputs an output light L having a first near-infrared ray having a wavelength of 780 nm or more and less than 2500 nm to a plurality of people. The photographing device 200 receives the first near-infrared ray contained in the output light L and photographs infrared images of a plurality of people. The generating device 300 generates psychological data representing a psychological state using a machine learning model 321, which is a learned model that takes a photographed infrared image as input and takes the psychological state of at least one person among a plurality of people shown in the infrared image as output. The display device 400 displays the temporal changes of the generated psychological data.

[0211] Thus, for example, Figure 6 As shown in step S204, after displaying a psychological data based on an infrared image, the display device 400 displays the next psychological data based on the next infrared image taken after a specified time of the infrared image. That is, the display device 400 displays the situation in which the psychological state of the person changes moment by moment. The administrator of the psychological state display system 1 can understand the psychological state changes of multiple persons by observing the time changes of the displayed psychological data. That is, the psychological state display system 1 can visualize the psychological state changes of the person.

[0212] Furthermore, in the psychological state display system 1 of the present embodiment, the temporal change of the psychological state of the person is visualized using the first near infrared rays (more specifically, infrared images). Therefore, when visualizing the change of the psychological state of the person, the psychological state display system 1 is not easily affected by the illumination light (visible light) of the space where the psychological state display system 1 is used. That is, the psychological state display system 1 is realized that can visualize the change of the psychological state of the person while suppressing the influence of the illumination light.

[0213] In addition, for example, the first near infrared ray includes a peak wavelength within a wavelength range of 780 nm or more and less than 2500 nm.

[0214] Thus, the first near infrared ray having a large amount of near infrared light components invisible to human eyes is irradiated. Therefore, the mental state display system 1 is realized that can visualize changes in the mental state of a person without causing discomfort to the person.

[0215] In addition, for example, the near-infrared light source 100 includes a first solid-state light-emitting element that outputs a first near-infrared ray.

[0216] Thus, since the electrical energy can be directly converted into the first near infrared rays (light energy), it is easy to obtain stronger first near infrared rays. Therefore, the infrared images of multiple people can be obtained with higher accuracy through the shooting device 200. Therefore, a psychological state display system 1 that can visualize the changes in the psychological state of a person with higher accuracy is realized.

[0217] For example, the near-infrared light source 100 includes: a second solid-state light emitting element that outputs excitation light; and a fluorescent member that outputs wavelength-converted light as the first near-infrared ray based on the output excitation light.

[0218] As a result, the spectral distribution and intensity unevenness of the first near infrared ray are reduced. Therefore, the infrared image of a plurality of people can be obtained with higher accuracy through the camera 200. Therefore, the psychological state display system 1 is realized that can visualize the changes in the psychological state of a person with higher accuracy.

[0219] In addition, the output light L includes visible light, for example.

[0220] Thus, the output light L having visible light visible to human eyes is radiated. Therefore, the mental state display system 1 is realized which facilitates recognition of the presence of the near-infrared light source 100 that outputs the first near-infrared ray.

[0221] In addition, for example, the near-infrared light source 100 controls the first near-infrared ray and the visible light independently, and controls at least one of the on / off and dimming of each of the first near-infrared ray and the visible light.

[0222] By controlling the first near infrared rays and visible light in this manner by the near infrared light source 100, a mental state display system 1 is realized that facilitates free control of visualization of mental states based on the first near infrared rays and light presentation effects based on visible light.

[0223] In addition, for example, when the output light L having the first near infrared ray is output, the intensity of the light-receiving signal obtained by receiving the first near infrared ray reflected by the plurality of people through the camera 200 is used as the first light-receiving signal intensity. When the output light L having the first near infrared ray is not output, the plurality of people are irradiated with the second near infrared ray different from the first near infrared ray, and the intensity of the light-receiving signal obtained by receiving the second near infrared ray reflected by the plurality of people through the camera 200 is used as the second light-receiving signal intensity. The minimum value of the first light-receiving signal intensity is greater than the maximum value of the second light-receiving signal intensity.

[0224] As a result, the intensity of the second light receiving signal that becomes noise in the signal for identifying multiple people becomes sufficiently small. Therefore, since such noise is sufficiently reduced, an infrared image of multiple people can be obtained with higher accuracy through the camera 200. Therefore, a psychological state display system 1 that can visualize changes in the psychological state of a person with higher accuracy is realized.

[0225] In addition, for example, the near-infrared light source 100 outputs the output light L toward one person from a direction other than the front direction toward which the head of at least one person among the plurality of persons faces.

[0226] In this way, even if the first near infrared ray is not irradiated from the front direction D1 of the face of the person, the psychological state display system 1 can visualize the change of the psychological state of the person. For example, when the psychological state display system 1 is used in a wide space and a plurality of people in the wide space perform completely different activities, the first near infrared ray is not irradiated from the front direction D1 of the face of the person. The psychological state display system 1 that can visualize the change of the psychological state of the person even in such a situation is realized.

[0227] In addition, as described above, in the infrared image used when the learning unit 330 causes the machine learning model 321 to learn, the entire face and the entire body of the person may be captured, but the present invention is not limited thereto. For example, a portion of the face or a portion of the body of the person may be captured. By using the infrared image of a portion of the face or a portion of the body of the person for learning, the mental state display system 1 can visualize changes in the mental state of the person even if the first near infrared ray is not irradiated from the front direction D1 of the face of the person.

[0228] For example, the area of ​​the irradiated region by the first near infrared ray is 10 m 2 Above and less than 10000m 2 .

[0229] Thus, the first near infrared rays can irradiate a plurality of people in a wide space. That is, the psychological state display system 1 can visualize changes in the psychological states of a plurality of people in a wide space.

[0230] In addition, for example, the distance between the person located closest to the infrared light source 100 among the plurality of people and the near-infrared light source 100 is greater than or equal to 50 cm and less than 30 m.

[0231] In addition, the distance R1 is 50 cm or more and less than 3 m (first lighting distance), 3 m or more and less than 5 m (second lighting distance), or 5 m or more and less than 20 m (third lighting distance).

[0232] When the distance R1 is the first lighting distance, it is suitable for visualizing the psychological state of people in offices, seminar venues, classrooms, stores, public spaces (government offices and libraries, etc.), and public mobile spaces (inside trains and buses), etc. When the distance R1 is the second lighting distance, it is suitable for visualizing the psychological state of people in indoor event venues, movie theaters, gymnasiums, small outdoor event venues, and relatively narrow areas in cities, etc. When the distance R1 is the third lighting distance, it is suitable for visualizing the psychological state of people in large outdoor event venues, sports venues, and relatively wide areas in cities, etc.

[0233] For example, when the near-infrared light source 100 is observed from the vertical direction, the divergence angle of the first near-infrared ray in the horizontal direction is greater than or equal to 30° and less than or equal to 180°.

[0234] When the divergence angle θ is within such a range, the first near infrared rays can be irradiated to a range region that is a region away from the near infrared light source 100. Therefore, the space in the mental state display system 1 can be saved.

[0235] In addition, for example, the generating device 300 determines stimulation to be applied to a plurality of characters based on the generated psychological data.

[0236] In this way, it is possible to determine a stimulus for changing the psychological state according to the psychological state of the character.

[0237] In addition, for example, the mental state display system 1 includes a control device 500 that controls a stimulation device 600 that applies stimulation to a plurality of characters. The control device 500 controls the stimulation device 600 so as to apply a determined stimulation to a plurality of characters.

[0238] Thereby, a psychological state display system 1 is realized which can control the psychological state of a person by applying a stimulus for changing the psychological state to the person according to the psychological state of the person.

[0239] In addition, for example, the mental state display system 1 includes a stimulation device 600. The control device 500 controls the stimulation device 600 so that the determined stimulation is applied to at least a part of the plurality of characters.

[0240] This makes it possible to apply stimulation to at least a portion of the plurality of characters, thereby realizing the psychological state display system 1 that can control the psychological state of the portion of characters.

[0241] In addition, for example, the determined stimulus is a stimulus that acts on at least one of the senses of vision, hearing, smell, and touch of the plurality of characters.

[0242] Thus, the psychological state display system 1 is realized which can change the psychological states of a plurality of characters by stimulating their five senses other than taste.

[0243] In addition, for example, the control device 500 controls the stimulation device 600 so that the stimulation applied to the plurality of characters is cut off or suppressed.

[0244] Thereby, the psychological state display system 1 is realized which can change the psychological states of a plurality of characters by alleviating the stimulation applied to the plurality of characters.

[0245] Furthermore, for example, when the generated psychological data indicates that the psychological state of one or more characters is a predetermined psychological state, the generating device 300 determines the stimulus to be applied to the one or more characters based on the predetermined psychological state.

[0246] This limits the characters to which stimulation is applied to change the psychological state, and eliminates the need to apply stimulation to characters that are not the target characters. Therefore, a psychological state display system 1 that can efficiently control the psychological state of the target characters is realized.

[0247] In addition, for example, in a modified example, the stimulation device is a near-infrared light source 100a that outputs output light L including visible light.

[0248] As a result, the number of components of the mental state display system 1 a of the modified example is reduced, and the structure of the mental state display system 1 a is simplified.

[0249] In addition, for example, the camera 200 captures an infrared image containing blood vessel information related to the contraction of blood vessels of each of the plurality of persons. The generator 300 generates vital sign information indicating the vital sign of at least one of the plurality of persons based on the blood vessel information contained in the captured infrared image. The display device 400 displays the generated vital sign information.

[0250] Thus, the mental state display system 1 can visualize not only the temporal changes in the mental state of a person, but also the vital sign information of the person and the temporal changes in the vital sign information.

[0251] In addition, for example, the near-infrared light source 100 and the photographing device 200 are provided in an entertainment space, a public space, or a learning space.

[0252] Thus, a psychological state display system 1 is realized that can easily visualize changes in a person's psychological state even in a space where the presence, intensity and color tone of visible light are prone to significant changes, or in a space where a person is illuminated by multiple white lighting lights of different chromaticities.

[0253] (Other embodiments)

[0254] As mentioned above, although embodiment was described, this invention is not limited to the said embodiment.

[0255] In the above-mentioned embodiments and modified examples, the output light L includes the first near infrared ray and the visible light, but the present invention is not limited thereto. The output light L may include only the first near infrared ray. Furthermore, the output light L may include electromagnetic waves other than the first near infrared ray and the visible light.

[0256] In addition, as such electromagnetic waves, for example, at least one of ultraviolet rays, mid-infrared rays, and far-infrared rays can be used.

[0257] For example, ultraviolet rays have the property of acting on a fluorescent body and being converted into visible light. A fluorescent body is contained in a bleaching agent or the like contained in clothes. Therefore, when a person wearing the above clothes is irradiated with the output light L having ultraviolet rays, visible light is output from the above clothes. Therefore, for example, an administrator of a plurality of characters or a psychological state display system can easily recognize that the output light L having ultraviolet rays and the first near infrared ray is being output.

[0258] In addition, for example, mid-infrared rays and far-infrared rays have relatively high heating effects. Therefore, when the output light L has mid-infrared rays and far-infrared rays, a mental state display system having a slight heating effect in cold areas or the like is realized.

[0259] The near-infrared light source 100 controls the first near-infrared ray, the visible light, and the electromagnetic wave independently, and controls at least one of the on / off and dimming of the first near-infrared ray, the visible light, and the electromagnetic wave.

[0260] By controlling the first near infrared ray, visible light and electromagnetic wave in this way by the near infrared light source 100, a mental state display system 1 is realized that is conducive to freely controlling the visualization of the mental state based on the first near infrared ray and the light presentation effect based on visible light and electromagnetic wave.

[0261] Furthermore, the communication method between devices described in the above embodiment is an example, and the communication method between devices is not particularly limited.

[0262] Furthermore, in the above-described embodiment, a process executed by a specific processing unit may be executed by another processing unit. Furthermore, the order of a plurality of processes may be changed, or a plurality of processes may be executed in parallel.

[0263] In the above embodiments, the components such as the generator can be implemented by executing a software program suitable for each component. Each component can be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0264] In addition, the components such as the generating unit can be implemented by hardware. For example, the components such as the generating unit can be circuits (or integrated circuits). These circuits can constitute a circuit as a whole, or they can be separate circuits. In addition, these circuits can be general circuits or dedicated circuits.

[0265] In addition, the general or specific aspects of the present invention may be implemented by a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM. In addition, the present invention may be implemented by any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. For example, the present invention may be implemented as a psychological state display system of the above-mentioned embodiment, or as a psychological state display method executed by a psychological state display system. The present invention may be implemented as a program for causing a computer to execute such a psychological state display method, or as a non-temporary recording medium having such a program recorded thereon. These programs include application programs for causing a computer such as a general-purpose information terminal to function as a generating device of the above-mentioned embodiment.

[0266] In addition, the present invention also includes various modifications that can be conceived by those skilled in the art to the embodiments, or embodiments achieved by arbitrarily combining components and functions in the embodiments within the scope not departing from the gist of the present invention.

[0267] Description of the attached figure

[0268] 1.1a Psychological state display system

[0269] 100, 100a near infrared light source

[0270] 200 Camera

[0271] 300 Generator

[0272] 321 Machine Learning Model

[0273] 400 display devices

[0274] 500 control device

[0275] 600 Stimulation Device

[0276] L Output light

Claims

1. A psychological state display system, wherein: have: A near-infrared light source that outputs a first near-infrared light having a wavelength of 780 nm or more and less than 2500 nm toward the plurality of people; a photographing device, receiving the first near infrared rays contained in the output light, and photographing infrared images of the plurality of persons; a generating device for generating psychological data representing a psychological state using a learned machine learning model, wherein the machine learning model takes the captured infrared image as an input and takes the psychological state of at least one person among the plurality of persons shown in the infrared image as an output; as well as A display device displays the time change of the generated psychological data.

2. The mental state display system according to claim 1, wherein: The first near infrared ray includes a peak wavelength within a wavelength range of 780 nm to less than 2500 nm.

3. The mental state display system according to claim 1, wherein: The near-infrared light source includes a first solid-state light-emitting element that outputs the first near-infrared ray.

4. The mental state display system according to claim 1, wherein: The near-infrared light source has: A second solid-state light emitting element outputs excitation light; and The fluorescent member outputs wavelength converted light as the first near infrared light based on the output excitation light.

5. The mental state display system according to claim 1, wherein: The output light comprises visible light.

6. The mental state display system according to claim 5, wherein: The near-infrared light source, independently controlling the first near infrared ray and the visible light, At least one of turning on and off and dimming of each of the first near infrared ray and the visible light is controlled.

7. The mental state display system according to claim 1, wherein: When the output light having the first near infrared ray is output, the intensity of the light-receiving signal obtained by receiving the first near infrared ray reflected by the plurality of people through the photographing device is used as the first light-receiving signal intensity, When the output light having the first near infrared ray is not output, second near infrared ray different from the first near infrared ray is irradiated to the plurality of characters, The intensity of the light-receiving signal obtained by receiving the second near-infrared rays reflected by the plurality of people through the photographing device is used as the second light-receiving signal intensity, The minimum value of the first light-receiving signal strength is greater than the maximum value of the second light-receiving signal strength.

8. The mental state display system according to any one of claims 1 to 7, wherein: The near-infrared light source outputs the output light toward the one person from a direction other than a front direction that a head of at least one person among the plurality of persons faces.

9. The mental state display system according to claim 8, wherein: The area of ​​the irradiated area by the first near infrared ray is 10m 2 Above and less than 10000m 2 .

10. The mental state display system according to claim 8, wherein: A distance between a person among the plurality of people that is located closest to the near-infrared light source and the near-infrared light source is greater than or equal to 50 cm and less than 30 m.

11. The mental state display system according to claim 8, wherein: When observing the near-infrared light source from a vertical direction, The first near infrared rays have a horizontal divergence angle of not less than 30° and not more than 180°.

12. The mental state display system according to claim 1, wherein: The generating device determines stimulation to be applied to the plurality of characters based on the generated psychological data.

13. The mental state display system according to claim 12, wherein: A control device is provided, wherein the control device controls a stimulation device for applying stimulation to the plurality of characters, The control device controls the stimulation device to apply the determined stimulation to the plurality of characters.

14. The mental state display system according to claim 13, wherein: having the stimulation device, The control device controls the stimulation device to apply the determined stimulation to at least a part of the plurality of characters.

15. The mental state display system according to claim 13, wherein: The determined stimulus is a stimulus that acts on at least one of the senses of vision, hearing, smell, and touch of the plurality of characters.

16. The mental state display system according to any one of claims 13 to 15, wherein: The control device controls the stimulation device to cut off or inhibit the stimulation applied to the multiple characters.

17. The mental state display system according to claim 12, wherein: When the generated psychological data indicates that the psychological state of one or more of the characters is a predetermined psychological state, The generating device determines the stimulus to be applied to the one or more characters based on the predetermined psychological state.

18. The mental state display system according to claim 14, wherein: The stimulation device is the near-infrared light source that outputs the output light having visible light.

19. The mental state display system according to claim 1, wherein: The imaging device captures the infrared image including blood vessel information related to the contraction of blood vessels of each of the plurality of persons, The generating device generates vital sign information representing a vital sign of at least one of the plurality of persons based on the blood vessel information included in the captured infrared image, The display device displays the generated vital sign information.

20. The mental state display system according to claim 1, wherein: The near-infrared light source and the shooting device are arranged in an entertainment space, a public space or a learning space.

Images