Non-contact state detection device and non-contact state detection program

Abstract

To provide a non-contact type state detection device capable of grasping a health state and a psychological state of an object person in a remote place over a screen by acquiring biological information on the object person from moving image information captured in a non-contact manner, and estimating a state of the object person, and capable of managing the health state and the psychological state easily.SOLUTION: A non-contact type state detection device includes: a measurement area extraction unit 32 for specifying a measurement area for measuring biological information from moving image information in which an object person 11 is imaged; a detection unit 34 for detecting a pulse wave of the object person 11 on the basis of a change in a luminance value between the frames of the moving image information in a skin area of the measurement area; and a state estimation unit 35 for estimating the state of the object person 11 on the basis of a result of the detection by the detection unit 34. As needed, respiration of the object person 11 is detected on the basis of a change in a vertical position of a shoulder area detected between the frames of the moving image information, and a change in a facial expression of the object person 11 is detected on the basis of a change in a predetermined feature point detected between the frames of the moving image information in a face area.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a non-contact type state detection device for detecting biological information without contact, and particularly to a non-contact type state detection device for estimating the state of a target person from the acquired biological information.

Background Art

[0002] Techniques for estimating the state of a target person, such as stress and emotion, based on biological information and the like acquired from the target person are known. The technique shown in Patent Document 1 is to photograph a person being measured with a camera before applying stress and in a state where stress has been applied when the person being measured is in a healthy state, extract feature points from the photographed image data of the body of the person being measured, store the coordinates of these feature points in a standard database, represent the difference between the coordinates of the feature points in the healthy state of the person being measured stored in the standard database and the coordinates of the feature points at the time of measurement as a vector, and evaluate the stress received by the person being measured from this vector.

[0003] Further, the technique shown in Patent Document 2 is such that an input interface receives a first biological signal corresponding to the pulse of a target person and a second biological signal corresponding to the brain wave of the target person, a processor converts a first data set including a first value corresponding to the first biological signal and a second value corresponding to the second biological signal into a second data set including a plurality of state values corresponding to a plurality of states arranged on a Russell's annulus model, a storage stores colors associated with each of the plurality of states, a display device displays a coordinate plane formed by a first coordinate axis corresponding to the first biological signal and a second coordinate axis corresponding to the second biological signal, the processor displays a label at a position on the coordinate plane determined by the first data set, and causes the display device to display a color associated with one having the maximum value among the plurality of state values.

[0004] On the other hand, there are known technologies that use video data to augment and visualize minute changes that are imperceptible to the human eye. These technologies make it possible to visualize pulse rate by augmenting slight changes in skin color caused by changes in blood flow, or to visualize the vibration state of a crane by augmenting its minute vibrations (see Non-Patent Literature 1). [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2017-176762 [Patent Document 2] Japanese Patent Publication No. 2020-185138 [Non-patent literature]

[0006] [Non-Patent Document 1] Massachusetts Institute of Technology, “Video Magnification,” [online], last updated June 2015, [Retrieved February 12, 2021], Internet<URL: http: / / people.csail.mit.edu / mrub / vidmag / > [Overview of the project] [Problems that the invention aims to solve]

[0007] The technology described in Patent Document 1 extracts predetermined feature points before and after stress is applied and evaluates the stress from their coordinate vectors. However, it has the problem that it requires the prior collection of a large amount of data before and after stress application, which is time-consuming.

[0008] Furthermore, while it is described that heart rate can be obtained from the fingers or the chest near the heart, this would require a contact-type sensor to acquire the heart rate. For example, some individuals with mental disabilities may strongly dislike having sensors come into contact with their bodies, and in situations where it is desirable to acquire biometric information without using contact-type sensors, it becomes difficult to use the technology described in Patent Document 1.

[0009] The technology described in Patent Document 2 is a technology that estimates a person's state from their pulse and brain waves, but it has the problem that it may be difficult to apply in some cases for the same reasons as above, as it requires sensors to be attached to the body to acquire the pulse and brain waves.

[0010] The technology described in Non-Patent Document 1 enables visualization of pulse rate and other functions by taking camera images in a non-contact manner, but it is not capable of determining the emotional state or stress level of the subject.

[0011] The present invention provides a non-contact state detection device that acquires a subject's biometric information from video information captured non-contactually and estimates the subject's state, thereby enabling easy management of the health and psychological state of a subject located remotely, via a screen. [Means for solving the problem]

[0012] The non-contact state detection device according to the present invention comprises: a measurement area identification means for identifying a measurement area for measuring biological information from video information captured by a subject; a detection means for detecting the pulse wave of the subject in the skin area of ​​the measurement area based on changes in brightness values ​​between frames of the video information; and a state estimation means for estimating the state of the subject based on the detection result of the detection means.

[0013] Thus, in the non-contact state detection device according to the present invention, a measurement area for measuring biological information is identified from video information captured of a subject, the subject's pulse wave is detected in the skin area of ​​the measurement area based on changes in brightness values ​​between frames of the video information, and the subject's state is estimated based on the detection result. Therefore, it is possible to easily estimate the subject's state simply by acquiring video information captured through a camera in a non-contact manner, and this has the effect of allowing the subject's health and psychological state to be understood.

[0014] The non-contact state detection device according to the present invention detects the breathing of the subject based on changes in the vertical position of the shoulder region detected between frames of the video information in the shoulder region of the measurement area.

[0015] Thus, in the non-contact state detection device according to the present invention, the breathing of the subject is detected in the shoulder region of the measurement area based on the change in the vertical position of the shoulder region detected between frames of video information. This has the effect of enabling a more accurate estimation of the subject's state from pulse wave and breathing information.

[0016] Furthermore, since there is a correlation between changes in pulse waves in response to changes in state and changes in respiration in response to changes in state, even if a situation arises where pulse wave information cannot be obtained, it can be compensated for by respiration, thus providing an effect.

[0017] The non-contact state detection device according to the present invention detects changes in the facial expression of the subject based on changes in predetermined feature points detected between frames of the video information in the face region of the measurement area.

[0018] Thus, in the non-contact state detection device according to the present invention, changes in the subject's facial expression are detected in the face region of the measurement area based on changes in predetermined feature points detected between frames of the video information. This has the effect of enabling more accurate estimation of the subject's state from information on pulse waves and changes in facial expression.

[0019] In the non-contact state detection device according to the present invention, the detection means expands minute changes between frames by vector amplification to detect the changes.

[0020] Thus, in the non-contact state detection device according to the present invention, since the detection means expands minute changes between frames by vector amplification to detect the changes, minute changes in the skin area due to blood flow, minute vertical vibrations in the shoulder area due to breathing, minute changes in expressions, etc. can be expanded and captured as clear changes, which has the effect of enabling such detection.

[0021] The non-contact state detection device according to the present invention includes operation information storage means for storing information on appropriate operations according to the state, operation information extraction means for extracting appropriate operation information according to the state estimated by the state estimation means from the operation information storage means, and display control means for displaying the extracted operation information on a display.

[0022] Thus, in the non-contact state detection device according to the present invention, since it includes operation information storage means for storing information on appropriate operations according to the state, operation information extraction means for extracting appropriate operation information according to the state estimated by the state estimation means from the operation information storage means, and display control means for displaying the extracted operation information on a display, biofeedback according to the estimated state of the subject becomes possible, and the mental state of the subject can be stabilized, which has the effect of enabling such stabilization.

[0023] In the non-contact state detection device according to the present invention, the operation information stored in the operation information storage means is classified into parasympathetic nerve dominant operation information for making the parasympathetic nerve of the subject dominant and sympathetic nerve dominant operation information for making the sympathetic nerve of the subject dominant.

[0024] Thus, in the non-contact state detection device according to the present invention, since the operation information stored in the operation information storage means is classified into parasympathetic-dominant operation information for predominating the parasympathetic nerve of the subject and sympathetic-dominant operation information for predominating the sympathetic nerve of the subject, for example, when it is desired to shift the mood to an excited state (sympathetic-dominant state) a little from the current state or when it is desired to calm down the mood (parasympathetic-dominant state), by presenting operation information suitable for each case, it is possible to control the mood of the subject so as to adapt to various situations, and thus the effect is achieved.

[0025] In the non-contact state detection device according to the present invention, the operation information stored in the operation information storage means is classified into long-term operation information that is an appropriate operation for a long-term state and short-term operation information that is an appropriate operation for a short-term state.

[0026] Thus, in the non-contact state detection device according to the present embodiment, since the operation information stored in the operation information storage means is classified into long-term operation information that is an appropriate operation for a long-term state and short-term operation information that is an appropriate operation for a short-term state, for example, by distinguishing and corresponding to an appropriate operation for a state that requires long-term treatment such as psychosomatic disorder and a coping operation suitable for excessive stress received in the short term, the effect that more appropriate coping becomes possible is achieved.

[0027] The non-contact state detection device according to the present invention includes temperature information acquisition means for acquiring temperature information of the subject imaged by a thermal sensor, and face extraction means for extracting the face region of the subject based on the temperature information, and the measurement region specifying means specifies the measurement region by superimposing the face region extracted by the face extraction means on the video information.

[0028] Thus, the non-contact state detection device according to this embodiment includes a temperature information acquisition means for acquiring temperature information of the subject captured by a thermal sensor, and a face extraction means for extracting the face region of the subject based on the temperature information. The measurement region identification means identifies the measurement region by superimposing the face region extracted by the face extraction means onto the video information. This has the effect of easily extracting the face region of the subject from the temperature distribution without performing complex processing such as image recognition (for example, using AI), thereby reducing processing load. [Brief explanation of the drawing]

[0029] [Figure 1] This is a system configuration diagram of a non-contact type state detection system using a non-contact type state detection device according to the first embodiment. [Figure 2] This is a functional block diagram showing the configuration of the state detection device according to the first embodiment. [Figure 3] This is an illustrative diagram showing the process when detecting a pulse wave in the state detection device according to the first embodiment. [Figure 4] This is an illustrative diagram showing the process when detecting respiratory information in the state detection device according to the first embodiment. [Figure 5] This is the first figure showing an example of a moving image when vector expansion is performed in the state detection device according to the first embodiment. [Figure 6] The second figure shows an example of a moving image when vector expansion is performed in the state detection device according to the first embodiment. [Figure 7] The third figure shows an example of a moving image when vector expansion is performed in the state detection device according to the first embodiment. [Figure 8] This figure shows an example of a Lorentz plot in the estimation process of the state detection device according to the first embodiment. [Figure 9] This is a flowchart showing the processing of the state detection device according to the first embodiment. [Figure 10] This is a functional block diagram showing the configuration of the state detection device according to the second embodiment. [Figure 11] This is a flowchart showing the processing of the state detection device according to the second embodiment. [Figure 12] This figure shows the configuration of the sensor unit in the state detection device according to the third embodiment. [Figure 13] This figure shows the process of identifying a measurement area using a thermal sensor in a state detection device according to the third embodiment. [Figure 14] This figure shows an overview of the processing when the state detection device according to the fourth embodiment is used in the workplace. [Figure 15] This figure shows an overview of the processing when the state detection device according to the fourth embodiment is used in an educational setting. [Figure 16] This figure shows an example of the screen displayed on the administrator terminal when the process described in Figure 14 or Figure 15 is performed. [Modes for carrying out the invention]

[0030] (First embodiment of the present invention) A non-contact state detection device according to this embodiment will be described with reference to Figures 1 to 9. The non-contact state detection device according to this embodiment detects the biological information of a subject without contact and estimates and manages the subject's state from the detected biological information.

[0031] Figure 1 is a system configuration diagram of a non-contact state detection system using a non-contact state detection device according to this embodiment. The non-contact state detection system 1 includes a subject terminal 10 which has at least a camera 12 for capturing images of a subject 11 and a display 13 that displays the image information of the person with whom the subject 11 is communicating via a telecommunication line such as the Internet, or the image information of the subject 11 itself captured by the camera 12.

[0032] Furthermore, the non-contact state detection system 1 includes an administrator terminal 20 which has at least a camera 22 for capturing images of the administrator 21 who manages the entire system, and a display 23 for displaying image information of the subject 11 with whom the administrator is interacting via a telecommunications line such as the internet.

[0033] Furthermore, the non-contact state detection system 1 includes a non-contact state detection device 30 (hereinafter referred to as the state detection device 30) that receives imaging information of the subject 11 captured by the camera 12 of the subject terminal 10, performs state estimation processing of the subject 11 using the received imaging information, and transmits various information including the processing results to each subject terminal 10 and the administrator terminal 20.

[0034] Note that the administrator terminal 20 in Figure 1 is not a mandatory configuration, and the subject 11 does not necessarily need to interact with the administrator 21 through the display 13. However, as an example, this explanation assumes that the subject 11's biometric information is acquired and their state is estimated while interacting with the administrator 21.

[0035] While the subject 11 and the administrator 21 engage in a one-to-one conversation, the camera 12 on the subject terminal 10 captures video images of the subject 11, and the information from the captured video images is transmitted to the state detection device 30. The state detection device 30 detects the subject 11's biometric information from the video image information transmitted from the subject terminal 10. The process of detecting biometric information from video image information will be described in detail later.

[0036] The state detection device 30 estimates the state of the subject 11 from the detected biological information. The state of the subject 11 includes mental elements such as a state of excessive stress, a relaxed state, a state of anxiety, or a state of excitement. These states are managed daily as the health status of the subject 11 and used for health management.

[0037] If the administrator 21 and the subject 11 are able to communicate, the status detection device 30 transmits biometric information and status information of the subject 11, who is the conversation partner, to the administrator terminal 20, and this information is made available on the display 23. The administrator 21 can communicate with the subject 11 while referring to the biometric information and mental state of the subject 11 displayed on the display 23.

[0038] The relationship between the administrator 21 and the subject 11 can be, for example, that of a superior and subordinate, teacher and student, employer and employee, organizer and participant, etc., and it is possible to manage the health of subordinates, students, employees, participants, etc., on a daily basis through face-to-face interviews. In particular, with the expansion of remote communication using telecommunication lines such as teleworking, the opportunities to photograph the subject 11 with a camera will increase, which will increase the frequency of use of the non-contact state detection device according to this embodiment, making it possible to easily manage the health of the subject 11.

[0039] Furthermore, this method can also be applied to other situations, such as job interviews or arranged marriage meetings conducted via telecommunication lines, allowing for responses that take into account the other person's feelings and emotions.

[0040] Figure 2 is a functional block diagram showing the configuration of the state detection device 30 according to this embodiment. The state detection device 30 includes an input unit 31 that receives video information of a subject 11 captured by the camera 12 of the subject terminal 10, a measurement area extraction unit 32 that extracts a measurement area necessary for measuring biological information from the input video information of the subject 11 based on information in a basic information storage unit 33 which stores basic information such as information about the area necessary for measuring the biological information of the subject 11, information necessary for detecting the biological information, and / or parameters necessary for estimation calculations, a detection unit 34 that detects the biological information of the subject 11 from the changes between frames of the video information of the extracted measurement area, a state estimation unit 35 that estimates the state of the subject 11 based on the detected biological information, a state information storage unit 36 ​​that stores the estimated state and biological information of the subject 11, and an output control unit 37 that outputs the state and biological information of the subject 11 to the subject terminal 10 and the administrator terminal 20.

[0041] In this embodiment, non-contact detectable biological information includes, for example, pulse wave, respiratory rate, facial expression, and body temperature, and at least one of these pieces of information is detected from the video information. Measurement areas to be extracted from the video information are set according to the biological information to be detected, and this information is stored in the basic information storage unit 33. The measurement area extraction unit 32 extracts the measurement areas to be extracted based on the information in the basic information storage unit 33, according to the type of biological information to be detected.

[0042] Specifically, for example, when detecting pulse wave information, it is necessary to capture minute changes in brightness values ​​on the skin surface caused by blood flow, so it is necessary to extract at least the skin area as the measurement area. Generally, when imaging is performed with camera 12, the face area of ​​the subject 11, where skin is likely to be exposed, is extracted. Among the face areas, the cheeks and forehead are particularly easy to extract skin from, but in the case of the cheeks, it may be difficult to extract the skin area if a mask is worn, and in the case of the forehead, it may be difficult to extract the skin area due to bangs, etc. Therefore, it is desirable to extract both areas and then extract the skin area from one or both of these areas.

[0043] Furthermore, when detecting respiratory rate, it is necessary to capture slight vertical vibrations of the shoulders, so the shoulder area of ​​subject 11 is extracted as the measurement area. When detecting facial expressions, the face area is extracted as the measurement area. When detecting body temperature, the skin area is extracted as the measurement area. Note that for body temperature, a thermal sensor capable of measuring the temperature of the body surface must be used instead of a regular camera.

[0044] Based on the above, it is sufficient to extract the area from the shoulders up as the measurement area for detecting pulse waves, respiratory rate, facial expression, body temperature, etc., and there is no need to image the whole body or back of the subject 11. The above biometric information can be detected simply by having the subject sit in front of the camera 12, i.e., the subject terminal 10.

[0045] The detection unit 34 extracts the above-mentioned biological information from the changes between frames of the video image in the measurement area extracted by the measurement area extraction unit 32. Figure 3 is an image diagram showing the process when detecting a pulse wave. Here, the process is shown from extracting the measurement area (in this case, the cheek area in the skin region) as shown in Figure 3(B) from each frame of the video information as shown in Figure 3(A) to obtain the brightness value, and detecting the pulse wave waveform from the change in brightness value as shown in Figure 3(C). As shown in Figure 3, by capturing the change in brightness value, it is possible not only to detect the pulse rate but also to detect pulse wave information (peak interval, pulse strength, etc.) as a waveform as shown in Figure 3(C). Such pulse wave information is used in the next stage state estimation unit 35 and is also stored in the state information storage unit 36 ​​and used as information for health management.

[0046] Furthermore, the detection unit 34 detects the respiratory rate as needed. Figure 4 is an illustrative diagram showing the process when respiratory information is detected. As mentioned above, the respiratory rate is detected by capturing changes in the movement of the shoulder region, which vibrates slightly up and down in accordance with the breathing of the subject 11. Figure 4(A) shows the video information, Figure 4(B) shows the process of detecting changes in the up and down movement of the shoulder region, which is the measurement area, and Figure 4(C) shows the process of detecting the respiratory waveform from the changes in the up and down movement of the shoulder region. Here, not only the respiratory rate but also the interval of breathing (normal breathing / deep breathing / rapid breathing, etc.) and the intensity of breathing can be detected as waveforms, as shown in Figure 4. Such respiratory information can be used in the next stage state estimation unit 35, and may also be stored in the state information storage unit 36 ​​and used as information for health management.

[0047] Furthermore, the detection unit 34 detects facial expressions as needed. For facial expression detection, for example, it is possible to detect changes in the corners of the eyes or changes in the mouth from side to side, and generally known facial expression detection techniques can be used. Body temperature can also be detected by imaging the subject 11 with a generally known infrared thermal sensor. This information on facial expressions and body temperature can be used in the next stage state estimation unit 35, and may also be stored in the state information storage unit 36 ​​and used as information for health management.

[0048] Furthermore, the detection unit 34 may be configured to more reliably detect biological information using vector augmentation technology. Figures 5 to 7 show examples of moving images when vector augmentation is applied. Figure 5 shows an example of augmentation of changes in brightness values ​​in the skin region, Figure 6 shows an example of augmentation of changes in vertical movement of the shoulder region, and Figure 7 shows an example of augmentation of changes in facial expression. In each figure, (A) shows the moving image without vector amplification, and (B) shows the moving image with vector amplification.

[0049] In Figure 5, by amplifying the change in brightness value between frames in the video information to increase the degree of color change, subtle color changes on the skin surface caused by pulse, which would be unrecognizable without vector amplification, can be displayed in a recognizable state. In Figure 6, by amplifying the change in the vertical movement of the shoulder region between frames in the video information, vertical vibrations of the shoulder region caused by breathing, which would be almost unrecognizable without vector amplification, can be displayed in a recognizable state. In Figure 7, by amplifying the change in facial expression between frames in the video information, even slight changes in facial expression can be exaggerated. Note that Figures 5 to 7 show an example of how the actual changes in subject 11 are exaggerated by comparing the case with and without vector amplification, and the degree of exaggeration by vector amplification can be set as appropriate.

[0050] Regarding the changes in facial expressions shown in Figure 7, the exaggeration of the changes makes it easier to estimate emotions, especially for subjects 11, such as those with intellectual disabilities, whose facial expressions hardly change in response to emotions. Furthermore, the exaggeration of changes in the moving images makes it possible to easily confirm changes in biological information by visual inspection, as shown in Figures 5 to 7.

[0051] The state estimation unit 35 estimates the state of the subject 11 from the biological information detected by the detection unit 34. For example, it analyzes the autonomic nervous system using a Lorentz plot based on the detected pulse wave information. More specifically, as shown in Figure 8, it calculates the L (long axis length) / T (short axis length) value from the Lorentz plot and determines the emotion corresponding to this L / T. The table showing the relationship between L / T and the state of the subject 11, as shown in Figure 8(B), can be arbitrarily set by the administrator 21, etc., and is stored in the basic information storage unit 33. Here, it is set to determine irritation, tension, reassurance, and relaxation. In addition to these, it can also determine, for example, sympathetic nervous system dominance / parasympathetic nervous system dominance, concentration / distraction, etc. The estimated information regarding the state of the subject 11 is stored in the state information storage unit 36 ​​and can be used as information for health management.

[0052] If respiratory information is detected, a Lorentz plot using the respiratory information may be created to estimate the state of subject 11, or a Lorentz plot may be created using respiratory information when pulse waves cannot be detected. If both pulse wave and respiratory information are detected, only one of the pieces of information may be used according to a predetermined priority, or a table like the one shown in Figure 8(B) may be created for L / T (pulse wave) + L / T (respiration), and the relationship between L / T (pulse wave) + L / T (respiration) and the state may be determined.

[0053] Furthermore, if information regarding facial expressions is detected, the state may be estimated by quantifying those expressions. For example, the intensity of a smile or anger can be quantified based on the degree to which the corners of the eyes are raised or lowered, or the degree to which the mouth is widened (for example, quantifying the range from anger to joy on a scale of -10 to 10), and then normalized and reflected in the above L / T ratio, allowing for a more accurate estimation of the subject's state. Similarly, body temperature can be quantified as a state of tension if it is high and a state of relaxation if it is low, and finally normalized and reflected in the above L / T ratio, allowing for a more accurate estimation of the subject's state.

[0054] In the above explanation, we described a case where information such as pulse wave, respiration, facial expression, and body temperature are normalized and treated as a single parameter. However, it is also possible to estimate the state based on different indicators for each piece of biological information. For example, one could estimate the state from "tension" to "relaxation" from pulse wave and respiration, "joy, anger, sadness, and pleasure" from facial expression, and "excitement" to "normal state" from body temperature, thereby estimating the state using multiple indicators.

[0055] The output control unit 37 transmits the biometric information detected by the detection unit 34 and the state of the subject 11 estimated by the state estimation unit 35 to the subject terminal 10 and the administrator terminal 20. The biometric information and state information transmitted to the subject terminal 10 and the administrator terminal 20 are displayed on the respective terminal's display for reference by the subject 11 and the administrator 21.

[0056] When the administrator 21 remotely interacts with the subject 11 via a display, they can monitor the subject's biometric information and condition in real time, enabling them to respond appropriately to the subject 11's state at that time. For example, if the administrator determines that the subject 11 is in an agitated state, they can speak slowly to help the subject transition to a relaxed state. Furthermore, by checking their own biometric information and condition, the subject 11 can understand their own state and take actions that help stabilize their mental state.

[0057] Furthermore, the administrator 20 can manage the health of the subject 11 by periodically checking the subject's biometric information and status information stored in the status information storage unit 36 ​​(for example, every six months or every year).

[0058] Next, the operation of the state detection device will be described. Figure 9 is a flowchart showing the operation of the state detection device according to this embodiment. First, the input unit 31 receives and acquires video information of the subject 11 transmitted from the subject terminal 10 (S1). The measurement area extraction unit 32 extracts the face area (especially the skin area) and shoulder area, which will be the measurement area, from the acquired video information (S2). The detection unit 34 detects biological information such as pulse waves, respiration, facial expressions and / or body temperature from the changes between frames of the video information in the extracted measurement area and from the thermal sensor (S3). The state estimation unit 35 estimates the state of the subject 11 from the detected biological information (S4). The detected biological information and the state information of the subject 11 are output (transmitted) to the administrator terminal 20 and the subject terminal 10 (S5) to end the process.

[0059] Thus, in the state detection device according to this embodiment, a measurement area for measuring biological information is identified from the video information captured of the subject 11, and the pulse wave of the subject 11 is detected in the skin area of ​​the measurement area based on the change in brightness value between frames of the video information. Based on the detection result, the state of the subject is estimated. Therefore, it is possible to easily estimate the state of the subject 11 simply by acquiring video information captured through a camera without contact, and the health and psychological state of the subject 11 can be understood.

[0060] Furthermore, in the shoulder region of the measurement area, the subject's respiration is detected based on the change in the vertical position of the shoulder region detected between frames of the video information. This makes it possible to more accurately estimate the subject's condition from pulse wave and respiration information. Moreover, since there is a certain degree of correlation between changes in pulse wave and changes in respiration in response to changes in the subject's condition, even if pulse wave information cannot be obtained, it can be compensated for by respiration.

[0061] Furthermore, in the facial region of the measurement area, changes in the subject's facial expression are detected based on changes in predetermined feature points detected between frames of the video information. This makes it possible to more accurately estimate the state of the subject 11 from the pulse wave and facial expression change information.

[0062] Furthermore, since the detection unit 34 amplifies minute changes between frames by vector amplification to detect these changes, it can exaggerate minute changes in the skin area due to blood flow, minute vertical vibrations in the shoulder area due to breathing, minute changes in facial expressions, etc., and capture them as clear changes.

[0063] In addition to pulse waves, respiration, facial expressions, and body temperature, it is also possible to detect the state of the autonomic nervous system by, for example, the degree of pupil dilation. In this case, by extracting an image of the pupil and detecting the pupil size, it is possible to estimate the state of sympathetic nervous system dominance or parasympathetic nervous system dominance. Then, similar to the above, the pupil size (change from the normal state in daily life) may be quantified, normalized, and reflected in the above L / T.

[0064] (Second embodiment of the present invention) A non-contact state detection device according to this embodiment will be described with reference to Figures 10 and 11. The non-contact state detection device according to this embodiment extracts an appropriate course of action based on the state of the subject 11 estimated from biological information and presents it to the subject 11. In this embodiment, explanations that overlap with the first embodiment will be omitted.

[0065] The subjects 11 can include a variety of individuals, such as healthy individuals, people with disabilities, children, the elderly, people with symptoms of illness, healthy individuals, people with mental illnesses, motivated individuals, unmotivated individuals, people who are easily anxious, and calm individuals. Among these, those whose condition is relatively stable do not require immediate intervention, but for those whose condition is unstable, it is desirable to shift their emotions and autonomic nervous system to a state that is as stable as possible. Furthermore, even if there are no particular mental or physical symptoms at present, maintaining a stable mental state and autonomic nervous system on a daily basis can help maintain long-term health. In this embodiment, the aim is to stabilize the mental state and autonomic nervous system by specifically presenting the subjects 11 with coping methods for stabilizing their mental state and autonomic nervous system.

[0066] Figure 10 is a functional block diagram showing the configuration of the state detection device 30 according to this embodiment. The difference from the case shown in Figure 2 in the first embodiment is that it includes a countermeasure information storage unit 81 that stores information on countermeasures according to the state of the subject 11, and a countermeasure information extraction unit 82 that extracts an appropriate countermeasure from the countermeasure information storage unit 81 according to the state estimated by the state estimation unit 35.

[0067] The response information stored in the response information storage unit 81 includes sympathetic nervous system dominance action information, which instructs the subject 11 to perform actions to shift to sympathetic nervous system dominance according to the subject 11's state, and parasympathetic nervous system dominance action information, which instructs the subject 11 to perform actions to shift to parasympathetic nervous system dominance. Appropriate response information is extracted according to the subject 11's state. For example, if the state estimation unit 35 determines that the subject 11 is in an extremely excited state and needs to be relaxed (for example, if they are overworked and stressed, or if they are experiencing stress in their relationships), parasympathetic nervous system dominance action information is extracted. Conversely, if the subject 11 is in an extremely relaxed state and needs to be urgently increased in tension (for example, if they are withdrawn in their room and too relaxed), sympathetic nervous system dominance action information is extracted.

[0068] Actions that shift the body towards sympathetic nervous system dominance include actions that inevitably increase breathing rate or increase sweating. Specifically, these include tensing the hands, arms, legs, and torso (making a fist), doing gymnastics, and watching exciting videos. On the other hand, actions that shift the body towards parasympathetic nervous system dominance include actions that slow breathing rate or induce a relaxed state. Specifically, these include relaxing the entire body, opening the fingers (making a flat hand), getting a massage, reading, listening to classical music, deep breathing / diaphragmatic breathing, and chewing gum.

[0069] Generally, it is considered desirable to shift to a parasympathetic nervous system dominant state, which is associated with relaxation. However, since both the sympathetic and parasympathetic nervous systems can cause health problems if either becomes too dominant, it is desirable to maintain a good balance. For example, if a state of emergency is declared and people are forced to stay home all day, they may feel relaxed mentally but still experience physical discomfort. In other words, by distinguishing between information that promotes sympathetic nervous system dominance and information that promotes parasympathetic nervous system dominance, and then providing appropriate coping information for each, it becomes possible to maintain the balance of the autonomic nervous system.

[0070] Furthermore, the response information stored in the response information storage unit 81 includes long-term action information that instructs the subject 11 to perform actions to address the subject 11 in the long term, and short-term action information that instructs the subject 11 to perform actions to address the subject 11 in the short term, and appropriate response information is extracted according to the subject 11's condition. For example, if the condition estimation unit 35 estimates that the subject 11 is in a condition that requires long-term action (for example, a state in which psychosomatic disorders (depression, headache, atopic dermatitis, irritable bowel syndrome, etc.) have developed due to the accumulation of stress), long-term action information is extracted. Conversely, if the unit estimates that the subject 11 is in a condition that requires short-term action (for example, a state in which the subject is angry after being reprimanded by a superior, or a state of extreme tension due to handling a customer complaint), short-term action information is extracted.

[0071] Long-term actions include, for example, actions that are habitually performed in daily life and are particularly helpful in maintaining good physical condition. Specifically, these include living in accordance with your internal clock (fixing wake-up and bedtime), drinking water regularly, maintaining good posture, smiling, eating dinner three hours before bedtime, taking a long bath at a comfortable temperature, and tidying up your room. On the other hand, short-term actions include actions that can be performed immediately and do not require much space or time. In addition to the actions mentioned above that shift to sympathetic and parasympathetic nervous system dominance, these include solving difficult problems in a short amount of time, solving easy problems without stress, looking up at the sky, and looking at colors that evoke excitement or relaxation.

[0072] Furthermore, long-term and short-term activities may be further subdivided into long-term activities that activate the sympathetic nervous system, long-term activities that activate the parasympathetic nervous system, short-term activities that activate the sympathetic nervous system, and short-term activities that activate the parasympathetic nervous system. By making these classifications, it becomes possible to present treatment information that is more appropriate to the condition of the subject 11.

[0073] The response information storage unit 81 stores information for instructing the subject 11 to perform actions that promote sympathetic nervous system dominance, actions that promote parasympathetic nervous system dominance, actions for long-term coping, and actions for short-term coping. For example, if the subject is to perform actions such as tensing or relaxing muscles, the unit will announce this in text or voice and provide a recommended duration for the action. The unit also stores videos / music that promote excitement / relaxation, videos and audio of making fists / open hands, videos and audio of making smiles, videos and audio of counters corresponding to appropriate breathing rates, and other videos and audio associated with each of the aforementioned actions. Based on these, the unit extracts the appropriate method of execution for the above actions (imitate, watch, move / move as instructed, etc.).

[0074] The information extracted by the information extraction unit 82 is transmitted to the user terminal 10 by the output control unit 37 and displayed on the display 13. The user 11 then performs the action while referring to the information displayed on the display 13. For example, if the information extracted is "Take deep breaths at a predetermined rhythm," then a display or sound corresponding to the predetermined rhythm will be output to the display 13 (or speaker in the case of sound) of the user terminal 10. By taking deep breaths in accordance with the sound or display, the user 11 can activate the parasympathetic nervous system and transition to a relaxed state.

[0075] Furthermore, if necessary, the system may be equipped with evaluation tools to assess the coping actions performed by the subject 11 (especially short-term actions that can be observed on the spot). For example, if the coping action is to make a smile, the AI ​​may be used to judge the smile actually made by the subject 11 and evaluate (assign a score) whether it is a sufficient smile. It is also possible to evaluate whether breathing occurred at the instructed timing (confirming shoulder movements using vector amplification) and whether the video was properly viewed (whether the gaze was directed towards the screen). Based on these evaluation results, the subject 11 will be able to perform more appropriate coping actions.

[0076] Next, the operation of the state detection device according to this embodiment will be described. Figure 11 is a flowchart showing the operation of the state detection device according to this embodiment. The processes from S1 to S4 are the same as those shown in Figure 9 of the first embodiment, so their explanation will be omitted. When the state of the subject 11 is estimated in S4, the response information extraction unit 82 extracts appropriate response information for the state of the subject 11 from the response information storage unit 81 (S5). The output control unit 37 outputs (transmits) the extracted response information together with the state information to the subject terminal 10 (S6), and the process ends.

[0077] Thus, in the state detection device according to this embodiment, biofeedback is possible according to the estimated state of the subject 11, and the mental state of the subject 11 can be stabilized. In other words, by performing actions according to the presented instructions, while controlling biological information that can be intentionally controlled, such as respiratory rate and facial expressions, it becomes possible to stabilize biological information that is difficult to intentionally control, such as heart rate, body temperature, and autonomic nervous system function. Furthermore, it is possible to make the unconscious subject 11 aware of the degree and effects of stress, thereby preventing various psychosomatic disorders that may occur in the future as much as possible.

[0078] Furthermore, by presenting appropriate action information for situations such as shifting from a state of heightened excitement (sympathetic nervous system dominance) or calming down (parasympathetic nervous system dominance), it becomes possible to control the subject's emotions to adapt to various situations.

[0079] Furthermore, by differentiating between appropriate actions for conditions requiring long-term management, such as psychosomatic disorders, and appropriate actions for dealing with excessive stress experienced in the short term, more appropriate responses become possible.

[0080] Furthermore, when presenting coping information on the display 13 of the target terminal 10, the vector amplification technique described in the first embodiment may be utilized. Specifically, for example, if making a smile to relax one's facial expression is extracted as coping information, the presented content can be emphasized by displaying a vector-amplified reference video of a smile on the display 13. In addition, by displaying a video of the target 11 themselves as coping information on the display 13 along with the reference video of a smile, it becomes possible to vector-amplified and exaggerated the user's own facial expression, allowing them to perform coping information while being aware of their own exaggerated facial changes.

[0081] (Third embodiment of the present invention) The state detection device according to this embodiment will be described with reference to Figures 12 and 13. In the state detection device according to this embodiment, the measurement area extraction unit 32 extracts the measurement area from the temperature information of the subject, thereby reducing the processing load. In this embodiment, explanations that overlap with the previous embodiments will be omitted.

[0082] Figure 12 shows the configuration of the sensor unit of the target terminal 10. In the first embodiment, a configuration in which a camera 12 is installed in the target terminal 10 was described, but in this embodiment, the target terminal 10 (with or without the camera 12) and the sensor unit shown in Figure 12 constitute the target terminal 10 as a single unit. In Figure 12, the sensor unit 40 includes a camera 41 (having at least the same function as the camera 12) that images the target in visible light, a thermal sensor 42 for detecting temperature (far-infrared), a near-infrared LED 43 that emits light for near-infrared imaging, a control unit 44 that controls the operation of each sensor, an input / output interface 45 for input / output with external devices, and a communication interface 46 for wired or wireless communication with external devices.

[0083] Furthermore, the target terminal 10 may be configured with only the sensor unit 40. That is, the target terminal 10 may be configured as a single device with only the sensor unit 40, without input / output devices such as a display or keyboard. In this case, input / output and communication with other external devices may be performed directly via the input / output interface 45 or the communication interface 46.

[0084] Camera 41 is an image sensor that captures images of the subject and the surrounding area. It performs imaging using visible light sources such as sunlight and fluorescent lights, and near-infrared imaging using a near-infrared LED 43 as the light source. When imaging using visible light as the light source, a filter that cuts out near-infrared light is used. When performing night vision imaging, such as at night, the near-infrared LED 43 is used as the light source and the filter that cuts out near-infrared light is removed.

[0085] The thermal sensor 42 detects far-infrared radiation and determines the temperature according to its intensity. Generally, the stronger the far-infrared radiation, the higher the temperature and the red it is displayed, and the weaker the far-infrared radiation, the lower the temperature and the blue it is displayed. In the first embodiment, a configuration is described in which the thermal sensor 42 is provided for measuring the body temperature of a subject. In this embodiment, in addition to collecting body temperature information, the measurement area is identified using the measured temperature information.

[0086] Figure 13 shows the process of identifying a measurement area using a thermal sensor in the state detection device according to this embodiment. As shown in Figure 13(A), by imaging a subject with the thermal sensor 42, it is possible to detect a range of body temperature (approximately 35°C to 37°C, including the temperature reduction due to clothing as needed, from 25°C) within a somewhat fixed range on the subject, that is, the subject's body temperature range. On the other hand, as shown in Figure 13(B), the subject is also imaged by the camera 41. By superimposing the body temperature range in Figure 13(A) imaged by the thermal sensor 42 and Figure 13(B) imaged by the camera 41, it becomes possible to identify the subject's measurement area in the image captured by the camera 41, as shown in Figure 13(C). Without this method, it would be necessary to recognize the subject (human face) using, for example, AI, and then extract the measurement area. However, in this embodiment, it is only necessary to detect a predetermined temperature range with the thermal sensor 42, so the processing can be made significantly lighter.

[0087] Furthermore, in this embodiment, in addition to pulse wave, respiratory rate, facial expression, body temperature, and autonomic nervous system state (sympathetic / parasympathetic nerves) described in the first and second embodiments, it is possible to detect or obtain multiple other biometric information.

[0088] Specifically, for example, by illuminating a subject with a near-infrared LED 43 and imaging the subject with camera 41 in night vision mode (a mode in which the filter that cuts out near-infrared light is removed), it is possible to measure the subject's blood glucose level. That is, when the absorbance of the subject's skin is measured from the near-infrared image captured by camera 41, areas with a high concentration of sugar (glucose) absorb more near-infrared light, resulting in a stronger black color. Therefore, it is possible to determine the blood glucose level from the difference in density between areas where sugar is detected and areas where sugar is not detected. Conventionally, measuring blood glucose levels requires actually collecting blood, but by determining the estimated value non-destructively as in this embodiment, the physical burden on the subject can be significantly reduced. In addition, since it is desirable to image blood vessels as close to the body surface as possible when measuring blood glucose levels, the subject may be instructed by voice or text to face camera 41 towards an area where blood vessels are easily visible, such as the inner arm.

[0089] Furthermore, it is possible to calculate a subject's blood pressure from acquired pulse wave information. Several methods are known for calculating blood pressure from pulse waves, but for example, it can be calculated as follows: The vascular elasticity E is

[0090]

number

[0091] Here, E0 is the vascular elastic modulus at no pressure, a is the correction factor, and P is the blood pressure. From this equation, the blood pressure P is:

[0092]

number

[0093] It can be calculated as follows. For the vascular elasticity, statistically known values ​​may be used, or it may be determined by the velocity pulse wave obtained by the first derivative of the pulse wave, or the acceleration pulse wave obtained by the second derivative.

[0094] Furthermore, it is possible to determine, for example, the blood oxygen concentration of a subject. Near-infrared light emitted by the near-infrared LED 43 has the property of penetrating biological tissues while being absorbed by hemoglobin in the blood. In other words, it is possible to determine the blood oxygen concentration by measuring the concentration of hemoglobin in the blood. In order to measure the relative concentrations of oxygenated hemoglobin and deoxygenated hemoglobin, near-infrared light of different wavelength bands may be emitted. In this case, multiple near-infrared LEDs 43 with different wavelengths may be used, or near-infrared LEDs 43 with variable wavelengths may be used to alternately emit near-infrared light of different wavelength bands.

[0095] Furthermore, it is also possible to estimate, for example, the level of concentration of a subject. A correlation between RRI (R-wave interval) and concentration is known; when RRI increases, concentration decreases, and when RRI remains stable, concentration is maintained. In other words, it is possible to estimate the presence or absence of concentration by monitoring pulse waves.

[0096] The collection and transmission of the above-mentioned biological information in the sensor unit 40 are controlled by the control unit 44. For example, when recognizing a subject or acquiring body temperature, the thermal sensor 42 is activated to perform sensing. When measuring blood glucose levels or blood oxygen saturation, or when imaging during nighttime hours, the near-infrared LED 43 is activated as a light source while the camera 41 performs imaging in night vision mode.

[0097] Information acquired by the sensor unit 40 is transmitted to the state detection device 30 via the communication interface 46, and the state of the subject is estimated. In the first embodiment, a method was described in which the state estimation unit 35 estimates the state of the subject using a Lorentz plot with the acquired biological information, but in this embodiment, the subject's information is mainly estimated by artificial intelligence (AI). In this embodiment, it is possible to acquire or calculate multiple types of biological information as described above, in addition to the biological information in each embodiment. Therefore, the state of the subject is estimated by artificial intelligence (AI) that takes this large amount of biological information as input. Alternatively, a unique Lorentz plot may be created for each piece of biological information, and the state of the subject may be estimated by comprehensively judging them.

[0098] For example, staff members working in childcare facilities or facilities for people with disabilities can sometimes judge a person's mood or emotions just by looking at them. This is thought to be based on the experience of veteran staff members, who instantly make a comprehensive judgment of the atmosphere the person is giving off (e.g., appearance, voice, facial expression, movements, clothing, etc.), but it is very difficult to explain or quantify logically. Therefore, by using an AI that has learned the state of the person based on various biometric information as described above, it becomes possible to make estimations similar to those of veteran staff members. Note that the above AI may make estimations for all service recipients, or it may make estimations individually for each service recipient.

[0099] In addition to the emotional state described in the first embodiment, the AI ​​function of the state estimation unit 35 can estimate the subject's state, including, for example, health status (physical health + mental health), signs of impending tantrums, manic-depressive states, degree of fatigue, mood, etc., based on the above-mentioned biometric information.

[0100] (Fourth embodiment of the present invention) Here, we will describe some specific examples of how the non-contact type state detection device according to the present invention can be used. Note that in this embodiment, explanations that overlap with the previous embodiments will be omitted.

[0101] First, as an example, we will explain the case where the non-contact state detection system 1 is used in the workplace. By adopting the non-contact state detection system 1 in the workplace, it is possible to improve the workplace environment and increase productivity as described later. Figure 14 is a diagram showing the processing overview when the non-contact state detection system 1 is used in the workplace. In Figure 14, the work content and work location are set in advance for workers working in a factory or office (Figure 14(A)). The camera 12 of the target terminal 10 captures images of the worker's work (Figure 14(B)). The worker is recognized and identified by facial recognition and monitored (Figure 14(C)). The worker's state is monitored and information is accumulated by processing as described in each embodiment above. In addition, an alert is output when the worker's concentration decreases (Figure 14(D)).

[0102] In this way, by issuing an alert when it is estimated that a worker's concentration is decreasing, it becomes possible to take accident prevention and safety measures as a risk management measure, especially when dangerous work is being performed. Furthermore, by analyzing the accumulated information, it is possible to understand what kind of work the target worker is doing when their state is stable and their concentration is maintained, which can be used to appropriately allocate work content and workload. In addition, it is possible to understand what kind of work environment allows workers to maintain a stable state and concentration, which can be used to improve the workplace environment.

[0103] As another example, we will describe the use of the non-contact state detection system 1 in an educational setting. By adopting the non-contact state detection system 1 in an educational setting, it can be used to improve the classroom environment and enhance learning ability, as described later. Figure 15 is a diagram showing an overview of the processing when the non-contact state detection system 1 is used in an educational setting. Although Figure 15 describes the use in classrooms such as schools, it can also be applied to online individual lessons and e-learning. In Figure 15, for students taking lessons in a school classroom, the learning content and classroom location are set in advance (Figure 15(A)). The camera 12 of the target terminal 10 captures images of the students' lessons (Figure 15(B)). Individual students are recognized and identified using facial recognition and monitored (Figure 15(C)). Information is accumulated while monitoring the students' status using the processing described in each embodiment above. In addition, if a student's concentration decreases, an alert is output to the administrator terminal 20 used by the teacher (Figure 15(D)).

[0104] In this way, when it is estimated that students' concentration levels are declining, the teacher, as the administrator, can understand how much concentration each student is receiving during class, allowing them to flexibly adjust the teaching method and achieve effective lessons. Furthermore, by analyzing the accumulated data, it is possible to understand in detail how biological information and autonomic nervous systems change when a student is receiving a lesson from a particular teacher, enabling detailed support tailored to their learning style, lesson content, strengths and weaknesses in different subjects.

[0105] Furthermore, schools can closely monitor changes in students' conditions and respond quickly to changes in their physical and mental health. Understanding students' conditions allows for the implementation of strategies to improve their academic performance. Moreover, it becomes possible to draw out latent interests and passions that students themselves may not be aware of, allowing for the discovery of new possibilities for students and the suggestion of appropriate classes and career paths. In addition, by understanding the classroom environment and time periods in which each student is most stable and able to maintain concentration, it becomes possible to improve the classroom environment.

[0106] Furthermore, students can receive detailed support that helps them discover potential they weren't aware of and improve their academic performance.

[0107] Figure 16 shows an example of the screen displayed on the administrator terminal 20 when processing is performed as shown in Figures 14 and 15. Figure 16(A) shows a heat map of status changes for each monitored person in the entire monitored area, such as a workplace or classroom, and Figure 16(B) shows a heat map of status changes over time for each monitored person.

[0108] The heatmap in Figure 16(A) allows us to see at a glance which areas within a monitored area (such as a workplace or classroom) indicate that monitored individuals (such as workers or students) are in a stable state, and which areas indicate that they are in an unstable state. In other words, if changes in state are observed due to placement (for example, being in an area with a lot of foot traffic, near a window, close to an air conditioner, in a front seat, in a back seat, being near a specific person, being near a supervisor or teacher, being near a close friend, etc.), then sequentially changing the placement can help stabilize the monitored individuals' state as much as possible.

[0109] The heatmap in Figure 16(B) allows for real-time monitoring of each monitored individual's state over time, enabling appropriate responses to be taken for each individual. For example, if concentration levels are low, encourage a break; if accumulated information indicates that an unstable state is caused by performing a task the individual dislikes, switch to a task they excel at; or, in the case of a school lesson, engage in relaxing conversation to stabilize the monitored individual's state.

[0110] Thus, by using the non-contact state detection device according to this embodiment, it can be used to improve the environment in workplaces and schools, and to enable detailed monitoring that ensures the safety of workers and students while enabling efficient work and lessons.

[0111] (Other embodiments) In each of the above embodiments, parallel processing enables communication while viewing the other party's biometric information on the screen in near real-time. For example, by dividing the frames of the video information acquired from the target terminal 10 and performing color extraction in parallel processing for each CPU core, it is possible to detect pulse waves in near real-time. In fact, in the system developed by the inventors, it was possible to detect pulse waves within a time lag range of 0.5 to 1.5 seconds.

[0112] In this way, by appropriately dividing the frames of the video information and processing the extraction of color and motion information in parallel for each CPU core, it becomes possible to detect biological information in real time and view it on the screen. [Explanation of symbols]

[0113] 1. Non-contact state detection system 10 Target user's device 11 Target Persons 12 cameras 13 displays 20 Administrator terminals 21 Administrator 22 cameras 23 displays 30. Non-contact type state detection device (state detection device) 31 Input section 32 Measurement area extraction part 33 Basic information storage section 34 Detection unit 35 State Estimation Unit 36 State Information Storage Unit 37 Output Control Unit 40 Sensor section 41 Cameras 42 Thermal Sensors 43 Near-infrared LED 44 Control Unit 45 Input / Output Interfaces 46 Communication Interfaces 81 Handling Information Storage Unit 82. Response Information Extraction Unit

Claims

1. A measurement area identification means for identifying a measurement area for measuring biological information from video information captured of a subject, Detection means for detecting the subject's pulse wave in the skin region of the measurement area based on changes in brightness values ​​between frames of the video information, the subject's respiration in the shoulder region of the measurement area based on changes in the vertical position of the shoulder region detected between frames of the video information, and changes in the subject's facial expression in the face region of the measurement area based on changes in predetermined feature points detected between frames of the video information. A state estimation means for estimating the state of the subject based on the detection result of the detection means, An operation information storage means that stores information on appropriate actions according to the state, An operation information extraction means extracts appropriate operation information from the operation information storage means according to the state estimated by the state estimation means, The system includes a display control means for displaying the extracted operation information on a display, The aforementioned operational information includes operational information that utilizes biometric information that the subject can intentionally control. A non-contact type state detection device characterized by the following:

2. In the non-contact type state detection device according to claim 1, The detection means, A non-contact state detection device that detects minute, unrecognizable changes between frames by amplifying them to recognizable changes through vector amplification.

3. In the non-contact type state detection device according to claim 1 or 2, A non-contact state detection device in which the information of the operation stored in the operation information storage means is divided into parasympathetic dominance operation information for making the parasympathetic nervous system dominant of the subject, and sympathetic dominance operation information for making the sympathetic nervous system dominant of the subject.

4. In the non-contact type state detection device according to claim 1 or 2, A non-contact type state detection device in which the operation information stored in the operation information storage means is divided into long-term operation information, which is appropriate for long-term states, and short-term operation information, which is appropriate for short-term states.

5. In the non-contact type state detection device according to claim 1 or 2, A temperature information acquisition means for acquiring temperature information of the subject captured by a thermal sensor, The system includes a body temperature region extraction means for extracting the body temperature region of the subject based on the aforementioned temperature information, A non-contact type state detection device characterized in that the measurement area identification means identifies the measurement area by superimposing the body temperature area extracted by the body temperature area extraction means onto the video information.

6. Measurement area identification means for identifying a measurement area for measuring biological information from video information captured of a subject, Detection means for detecting the subject's pulse wave in the skin region of the measurement area based on changes in brightness values ​​between frames of the video information, the subject's respiration in the shoulder region of the measurement area based on changes in the vertical position of the shoulder region detected between frames of the video information, and changes in the subject's facial expression in the face region of the measurement area based on changes in predetermined feature points detected between frames of the video information. A state estimation means that estimates the state of the subject based on the detection result of the detection means, Operation information storage means for storing information on appropriate actions according to the state, An operation information extraction means that extracts appropriate operation information from the operation information storage means according to the state estimated by the state estimation means. A computer is used as a display control means to display the extracted operation information on a display. The aforementioned operational information includes operational information that utilizes biometric information that the subject can intentionally control. A non-contact type state detection program characterized by the following:

Images