Stress coping style determination system and method, learning device and method, program and learning completion model
By obtaining biological information and facial images without contact, combining hemodynamic parameters and machine learning, we can determine the pressure response method of the subject, solve the problem that cannot be determined in the existing technology, and realize the detailed analysis of the pressure of the subject and management.
Patent Information
- Application Number
- CN202080068209.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-08-30
- Filing Date
- 2020-08-28
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-08-28
AI Technical Summary
The prior art cannot accurately determine the pressure response method of the examinee in a non-contact state, and cannot distinguish the types and properties of the pressure, which limits the detailed analysis and management of the pressure of the examinee.
Through non-contact acquisition of the subject's biological information, using hemodynamic parameters and facial images, combined with machine learning technology, the characteristic quantity is extracted and stored, and the subject's stress response method is determined to be active, passive or non-coping mode.
It realizes that the pressure response method is accurately determined without imposing physical constraints on the subject, and can analyze and manage the pressure status of the subject in detail, and applies it to productivity improvement and personnel management.
Smart Images

Figure CN114466618B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a technology for determining a subject's stress coping style in a non-contact state. Background Art
[0002] The technology for understanding the stress state of the subject is a well-known technology. For example, Patent Document 1 describes a technology for estimating the psychological changes of the subject by measuring the thermal radiation heat of the face.
[0003] In addition, Patent Document 2 describes a technique for measuring the level of a subject's psychological state based on facial image information. According to these techniques, it is possible to detect a certain psychological change in a subject in a non-contact state, or to quantitatively analyze a subject's psychological state in a non-contact state.
[0004] However, in Patent Documents 1 and 2, it is not possible to understand the type of stress felt by the subject, and therefore it is not possible to perform a qualitative analysis of the stress of the subject.
[0005] Prior art literature
[0006] Patent Literature
[0007] Patent Document 1: Japanese Patent Application Laid-Open No. 6-54836
[0008] Patent Document 2: Japanese Patent Application Publication No. 2007-68620 Summary of the invention
[0009] Technical problem to be solved by the invention
[0010] It is known that humans show characteristic response patterns in order to achieve the goal of the cardiovascular system meeting the metabolic requirements of various body tissues when facing stress stimulation. That is, there are active response patterns, passive response patterns, and patterns that do not take specific responses to stress. These patterns are called "stress response methods". When an active response pattern appears, it can be inferred that the subject is in a good stress state. On the contrary, when a passive response pattern appears, it can be inferred that the subject is in a poor stress state.
[0011] That is, by determining the subject's coping method for stress, the type of stress felt by the subject can be understood.
[0012] The present invention provides a stress response mode determination system, a stress response mode determination method, a learning device, a learning method, a program for implementing these using a computer, and a learning completion model, which are capable of determining a subject's stress response mode in a non-contact state.
[0013] Solutions for solving the above technical problems
[0014] In order to solve the above-mentioned technical problems, the pressure response mode determination system of Scheme 1 is characterized in that it comprises: a biological information acquisition unit, which acquires the biological information of the subject in a non-contact state; and a determination unit, which determines the pressure response mode of the subject based on the biological information and a predetermined response mode, wherein the response mode is determined by hemodynamic parameters.
[0015] The stress coping mode determination system acquires biological information of a subject in a non-contact state, and determines the stress coping mode of the subject in a non-contact state based on the biological information and a response pattern determined by hemodynamic parameters.
[0016] In addition, the stress response method determination system of Scheme 2 is characterized in that, in the stress response method determination system described in Scheme 1, the hemodynamic parameters include multiple parameters of mean blood pressure, heart rate, cardiac output, stroke volume and total peripheral vascular resistance.
[0017] "Hemodynamics" here refers to a branch of circulatory physiology that focuses on blood circulation. It is a research field that applies the theories of mechanics, elastomer dynamics, and fluid dynamics to biological systems.
[0018] Specifically, the heart's internal pressure, pulsation, work done, stroke volume, elasticity of blood vessels and myocardium, pulse, blood flow velocity, blood viscosity, etc. are studied. Therefore, the "hemodynamic parameters" in the present invention refer to the numerical parameters of the heart's internal pressure, pulsation, work done, stroke volume, elasticity of blood vessels and myocardium, pulse, blood flow velocity, blood viscosity, etc.
[0019] The stress coping mode determination system acquires biological information of the subject in a non-contact state, and determines the stress coping mode of the subject by combining the biological information with a response pattern determined by a plurality of parameters including average blood pressure, heart rate, cardiac output, stroke volume, and total peripheral vascular resistance. In addition, hemodynamic parameters can usually be identified by a continuous sphygmomanometer.
[0020] Furthermore, the stress coping method determination system of Scheme 3 is characterized in that, in the stress coping method determination system described in Scheme 2, the biological information is a facial image.
[0021] The stress coping style determination system acquires a facial image of a subject in a non-contact state, and determines the stress coping style of the subject based on the facial image and the response pattern.
[0022] Furthermore, the stress coping style determination system of Scheme 4 is characterized in that, in the stress coping style determination system described in Scheme 3, the facial image is a facial thermal image or a facial visible image.
[0023] The stress coping style determination system acquires a facial thermal image or a facial visible image of a subject in a non-contact state, and determines the stress coping style of the subject based on the facial thermal image or the facial visible image and the above-mentioned response pattern.
[0024] In this case, the "visible face image" is an image obtained by photographing the face of the subject using a generally widely used camera, i.e., a device having an optical system for imaging and used to photograph images. In this case, a color image is preferred. In addition, the "facial thermal image" is an image obtained by analyzing infrared rays radiated from the face of the subject and displaying the heat distribution as a graph, and is an image obtained by photographing with an infrared thermal imager.
[0025] In addition, the stress response style determination system of Scheme 5 is characterized in that, in the stress response style determination system described in Scheme 3 or 4, the determination unit determines the stress response style of the subject by observing the pressure response of a specific part of the face contained in the facial image.
[0026] The stress coping style determination system determines the stress coping style of the subject based on the pressure response of a specific part of the subject's face and the response pattern.
[0027] Furthermore, the stress coping method determination system of claim 6 is characterized in that the response mode includes three modes consisting of "active coping", "passive coping" and "no coping".
[0028] The stress response method determination system obtains biological information of the subject in a non-contact state, and based on the biological information, determines whether the stress response method of the subject shows a response mode of "active response", "passive response" or "no response".
[0029] In addition, the stress response method determination system of Scheme 7 is characterized in that, in the stress response method determination system described in Scheme 6, the determination unit has a determination feature quantity storage unit that stores spatial feature quantities corresponding to "active response", spatial feature quantities corresponding to "passive response" and spatial feature quantities corresponding to "no response", and based on the biological information and the spatial feature quantities stored in the determination feature quantity storage unit, it is determined whether the stress response method is a response mode of "active response", "passive response" and "no response".
[0030] The stress response mode determination system stores spatial feature quantities corresponding to "active response", spatial feature quantities corresponding to "passive response" and spatial feature quantities corresponding to "no response", and determines, based on the biological information of the subject and each spatial feature quantity, whether the stress response mode of the subject is "active response", "passive response" or "no treatment".
[0031] In addition, the stress coping method determination system of Scheme 8 is characterized in that, in the stress coping method determination system described in Scheme 7, the spatial feature value stored in the determination feature value storage unit is a spatial feature value extracted by a machine learning unit, and the machine learning unit has: a learning data storage unit, which stores a plurality of learning facial images labeled with "active response", "passive response" and "no response", respectively; a feature value extraction unit, which extracts the spatial feature value of the facial image from the learning facial image using a learned model; and a feature value learning unit, which changes the network parameters of the learned model based on the relationship between the extraction result obtained by the feature value extraction unit and the label annotated on the learning facial image as its extraction object so as to increase the extraction accuracy of the spatial feature value obtained by the feature value extraction unit.
[0032] In the stress coping method determination system, a spatial feature quantity corresponding to "active coping", a spatial feature quantity corresponding to "passive coping", and a spatial feature quantity corresponding to "no coping" are extracted by a machine learning unit.
[0033] The machine learning unit stores a plurality of learning facial images labeled with "active response", "passive response" and "no response" respectively, and uses a learned model to extract spatial feature quantities of the facial image of the subject from the learning facial images. Based on the relationship between the extraction results and the labels annotated on the learning facial images as extraction objects, the network parameters of the learned model are changed to increase the extraction accuracy of the spatial feature quantities of the facial image of the subject.
[0034] Furthermore, the stress coping style determining system of claim 9 is characterized in that, in the stress coping style determining system of claim 7 or 8, the spatial feature quantity is a fractal dimension calculated based on a facial image of the subject.
[0035] In addition, the program of Scheme 10 is a program for causing a computer to function as a mechanism for determining a subject's stress response method, and is characterized in that it comprises: a feature quantity storage step for determination, which stores spatial feature quantities corresponding to "active response", spatial feature quantities corresponding to "passive response", and spatial feature quantities corresponding to "no response"; and a determination step, which determines, based on the subject's facial image and each spatial feature quantity stored by the feature quantity storage step for determination, whether the subject's stress response method is a method showing a response mode of "active response", "passive response", and "no response", wherein the response mode is determined by hemodynamic parameters.
[0036] The program is installed in one or multiple computers working in cooperation with each other and executed, so that the system composed of the one or multiple computers working in cooperation with each other functions as the following mechanism: based on the spatial feature value corresponding to "active response", the spatial feature value corresponding to "passive response" and the spatial feature value corresponding to "no response" and the facial image of the subject, it is determined whether the stress response method of the subject shows a response mode of "active response", "passive response" and "no response".
[0037] In addition, the program of Scheme 11 is characterized in that, in the program described in Scheme 10, it has: a learning data storage step, storing a plurality of learning facial images labeled with "active response", "passive response" and "no response" respectively; a feature value extraction step, using the learned model to extract the spatial feature values of the learning facial images; a learning step, based on the relationship between the extraction result obtained by the feature value extraction step and the label annotated to the learning facial image as its extraction object, changing the network parameters of the learned model so that the extraction accuracy of the feature value obtained by the feature value extraction step becomes higher, and the judgment feature value storage step is a step of storing the spatial feature values extracted by the feature value extraction step.
[0038] The program is installed in one or multiple computers working in collaboration with each other and executed, so that the system composed of the one or more computers functions as the following system: storing multiple learning facial images labeled with "active response", "passive response" and "no response", respectively, using the learned model to extract spatial feature quantities of the learning facial images, based on the relationship between the extraction results and the labels annotated for the learning facial images that are the extraction objects, changing the network parameters of the learned model to increase the extraction accuracy of the spatial feature quantities, and storing the extracted spatial feature quantities.
[0039] Furthermore, the program of aspect 12 is characterized in that, in the program of aspect 10 or 11, the spatial feature amount is a fractal dimension calculated based on a facial image of the subject.
[0040] In addition, the stress response mode determination method of Scheme 13 is characterized in that it comprises: a biological information acquisition step for acquiring the biological information of the subject in a non-contact state; and a determination step for determining the stress response mode of the subject based on the biological information and a predetermined response mode, wherein the response mode is determined by hemodynamic parameters.
[0041] The stress coping style determination method acquires biological information of a subject in a non-contact state, and determines the stress coping style of the subject based on a response pattern determined by the biological information and hemodynamic parameters.
[0042] In addition, the learning device of Scheme 14 is characterized in that it comprises: a learning data storage unit, which stores a plurality of learning facial images labeled corresponding to response patterns determined by hemodynamic parameters; a feature extraction unit, which extracts spatial feature values of the facial images of the subject from the learning facial images using a learned model; and a feature learning unit, which changes the network parameters of the learned model based on the relationship between the extraction result obtained by the feature extraction unit and the label annotated to the learning facial image as the extraction object, so as to increase the extraction accuracy of the spatial feature values obtained by the feature extraction unit.
[0043] The learning device stores a plurality of learning facial images labeled corresponding to response patterns determined by hemodynamic parameters, extracts spatial feature quantities of the facial image of the subject from the learning facial images using a learned model, and based on the relationship between the extraction results and the labels annotated on the learning facial images that are the extraction objects, changes the network parameters of the learned model to increase the extraction accuracy of the spatial feature quantities of the facial image of the subject.
[0044] Furthermore, the learning device of claim 15 is characterized in that, in the learning device of claim 14, the spatial feature quantity is a fractal dimension calculated based on a facial image of the subject.
[0045] In addition, the learning method of Scheme 16 is characterized in that it has: a learning data storage step for storing a plurality of learning facial images labeled corresponding to response patterns determined by hemodynamic parameters; a feature extraction step for extracting spatial feature values of the facial images of the subject from the learning facial images using a learned model; and a feature learning step for changing the network parameters of the learned model based on the relationship between the extraction result obtained by the feature extraction unit and the label annotated to the learning facial image as the extraction object so as to increase the extraction accuracy of the spatial feature values obtained by the feature extraction unit.
[0046] The learning method stores a plurality of learning facial images labeled corresponding to response patterns determined by hemodynamic parameters, uses a learned model to extract spatial feature quantities of the facial image of the subject from these learning facial images, and based on the relationship between the extraction results and the labels annotated on the learning facial images that are the extraction objects, changes the network parameters of the learned model to increase the extraction accuracy of the spatial feature quantities of the facial image of the subject.
[0047] Furthermore, the learning method of item 17 is characterized in that, in the learning method of item 16, the spatial feature quantity is a fractal dimension calculated based on a facial image of the subject.
[0048] In addition, the program of Scheme 18 is used to enable the computer to function as a mechanism for learning spatial feature quantities of facial images, and is characterized in that it has: a learning data storage step for storing a plurality of learning facial images labeled corresponding to response patterns determined by hemodynamic parameters; a feature quantity extraction step for extracting spatial feature quantities of the facial images of the subject from the learning facial images using a learned model; and a feature quantity learning step for changing the network parameters of the learned model based on the relationship between the extraction result obtained by the feature quantity extraction step and the label annotated to the learning facial image as the extraction object so as to increase the extraction accuracy of the spatial feature quantities obtained by the feature quantity extraction step.
[0049] The program is installed in one or multiple computers working in cooperation with each other and executed, so that the system composed of the one or more computers functions as the following learning device: multiple learning facial images with labels corresponding to response patterns determined by hemodynamic parameters are stored, spatial feature quantities of the facial image of the subject are extracted from these learning facial images using a learned model, and based on the relationship between the extraction results and the labels annotated on the learning facial images that are the extraction objects, the network parameters of the learned model are changed to increase the extraction accuracy of the spatial feature quantities of the facial image of the subject.
[0050] Furthermore, the program of claim 19 is characterized in that, in the program of claim 18, the spatial feature quantity is a fractal dimension calculated based on a facial image of the subject.
[0051] In addition, the learned model of Scheme 20 is characterized in that a plurality of learning facial images labeled corresponding to response patterns determined by hemodynamic parameters are used as training data, and the learned model is generated by performing machine learning on spatial feature quantities of the facial images of the subject.
[0052] The learned model takes the subject's facial image as input and outputs the spatial feature value of the subject's facial image.
[0053] Furthermore, the learned model of claim 21 is characterized in that, in the program of claim 18, the spatial feature quantity is a fractal dimension calculated based on a facial image of the subject.
[0054] Effects of the Invention
[0055] The pressure response mode determination system according to Scheme 1 can obtain the biological information of the subject in a non-contact state, and determine the pressure response mode of the subject based on the biological information and the response pattern determined by the hemodynamic parameters in a non-contact state. Therefore, the type of pressure felt by the subject can be understood without imposing any restrictions on the subject's movements.
[0056] Usually, the word "stress" is widely used, but in psychology, "stress" refers to "a general term for non-specific reactions to external stimuli (stressors) acting on an organism" (psychologist Dr. Han Selye).
[0057] It is known that there are various stressors based on human life. In recent years, stressors in social environment have increased in particular, which have various effects on living bodies. It has been medically confirmed that stressors may cause diseases in some cases.
[0058] However, it is known that stress is not always bad for human beings. Even if a person is under stress, the organism will be activated when trying to cope with the stress, which will bring about beneficial effects. From this point of view, Dr. Sells clearly expressed the idea that stress can be good stress (enstress) or bad stress (distress) depending on the difference in the biological conditions of the recipient or the degree of stress.
[0059] Therefore, from such a psychological point of view, when considering stress, we cannot regard all stress as bad, but need to classify the types of stress for organisms as described above. In addition, based on such a point of view, we can actively consider how to manage stress in various modern social activities from the perspective of sociality and productivity, such as whether to improve production efficiency and work efficiency.
[0060] However, as described above, the technology and patent inventions for understanding the stress state of the subject are known, but according to the existing methods, only the stress felt by the subject can be roughly grasped, and it is impossible to perform a qualitative analysis based on the type of stress felt by the subject.
[0061] According to the invention of this application, by classifying and understanding the types of stress according to the subject, the relationship between the stressor and the stress response of the subject can be understood in more detail. As a result, the relationship between the social environment and the person that becomes the stressor can be analyzed and studied more accurately, and can be applied to various social environment fields to achieve problem solving, and can be applied to various industrial fields to improve production efficiency and promote production activities.
[0062] The stress response determination system according to Scheme 2 can determine the stress response of the subject in detail and accurately based on the subject's biological information and a response pattern determined by any of the mean blood pressure, heart rate, cardiac output, stroke volume and total peripheral vascular resistance.
[0063] The stress coping style determination system according to Scheme 3 is capable of acquiring a facial image of the subject in a non-contact state, and based on the facial image and a response pattern determined by hemodynamic parameters instead of using a continuous sphygmomanometer as in the past, the stress coping style of the subject can be quickly determined without restraining the subject and imposing a physical burden.
[0064] The stress coping style determination system according to Scheme 4 can obtain a thermal image or a visible image of the face of the subject in a non-contact state, and determine the stress coping style of the subject according to psychophysiology based on the thermal image or the visible image of the face and a response pattern determined by hemodynamic parameters.
[0065] The stress coping style determination system according to Scheme 5 can easily and accurately determine the stress coping style of the subject based on the response pattern determined by the state of a specific part of the subject's face and the hemodynamic parameters.
[0066] According to the pressure response method determination system of Scheme 6, it is possible to obtain the biological information of the subject in a non-contact state, and determine based on the biological information whether the pressure response method of the subject is "active response", "passive response" or "no response".
[0067] Therefore, since stress response methods can be classified into the three response modes of "active response", "passive response" and "no response" mentioned above, analysis can be performed based on the response mode, and the analysis results can be applied to human resources management business, quality management business, etc. in various production fields, which will help improve the quality of various businesses.
[0068] The stress response mode determination system according to Scheme 7 is capable of storing spatial feature quantities corresponding to "active response", spatial feature quantities corresponding to "passive response" and spatial feature quantities corresponding to "no response", and determining, based on the biological information of the subject and each spatial feature quantity, whether the stress response mode of the subject is "active response", "passive response" or "no treatment".
[0069] According to the stress coping style determination system of scheme 8, it is possible to change the network parameters of the learned model so as to increase the accuracy of extracting the spatial feature value of the facial image of the subject.
[0070] According to the stress coping style determination system of claim 9, the spatial feature quantity can be digitized with higher accuracy using the fractal dimension, and thus the stress coping style of the subject can be determined with high accuracy in a non-contact state.
[0071] According to the program of Scheme 10, the following system can be implemented using one or multiple computers working in collaboration with each other: based on the spatial feature quantity corresponding to "active response", the spatial feature quantity corresponding to "passive response", the spatial feature quantity corresponding to "no response" and the facial image of the subject, it is determined whether the subject's stress response method shows one of the response modes of "active response", "passive response" and "no response".
[0072] According to the program of Scheme 11, the following system can be implemented using one or multiple computers working in collaboration with each other: storing multiple learning facial images labeled with "active response", "passive response" and "no response", respectively, using the learned model to extract spatial feature quantities of the learning facial images, based on the relationship between the extraction results and the labels annotated for the learning facial images that are the extraction objects, changing the network parameters of the learned model to increase the extraction accuracy of the spatial feature quantities, and storing the extracted spatial feature quantities.
[0073] According to the program of scheme 12, the spatial feature quantity can be digitized with higher precision using fractal dimension, so that a system that can determine the stress coping style of the subject with high precision in a non-contact state can be realized using one or multiple computers working in cooperation with each other.
[0074] According to the pressure response mode determination method of Scheme 13, the biological information of the subject can be obtained in a non-contact state, and the pressure response mode of the subject can be determined in a non-contact state based on the biological information and the response pattern determined by the hemodynamic parameters. Therefore, the type of pressure felt by the subject can be understood without imposing any restrictions on the subject's movements.
[0075] According to the learning device of scheme 14, it is possible to change the network parameters of the learned model so as to increase the accuracy of extracting the spatial feature value of the facial image of the subject.
[0076] According to the learning device of claim 15, since the spatial feature quantity can be digitized with higher accuracy using the fractal dimension, the network parameters of the learned model can be changed so that the extraction accuracy of the spatial feature quantity of the subject's facial image becomes higher.
[0077] According to the learning method of Scheme 16, the network parameters of the learned model can be changed so as to increase the accuracy of extracting the spatial feature value of the facial image of the subject.
[0078] According to the learning method of Scheme 17, since the spatial feature quantity can be digitized with higher accuracy using the fractal dimension, the network parameters of the learned model can be changed to further increase the accuracy of extracting the spatial feature quantity of the subject's facial image.
[0079] According to the program of scheme 18, by installing the program on one or multiple computers working in cooperation with each other and executing the program, a learning device can be realized that can change the network parameters of the learned model to increase the extraction accuracy of the spatial feature value of the subject's facial image.
[0080] According to the program of Scheme 19, since the fractal dimension can be used to quantify the spatial feature quantity with higher accuracy, by installing and executing the program on one or multiple computers working in collaboration, the network parameters of the learned model can be changed to make the extraction accuracy of the spatial feature quantity of the subject's facial image higher.
[0081] According to the learned model of claim 20, by inputting a facial image of a subject into the learned model, it is possible to extract the spatial feature value of the facial image of the subject.
[0082] According to the learned model of Scheme 21, since the spatial feature value can be digitized with higher accuracy using the fractal dimension, the spatial feature value of the facial image of the subject can be extracted with higher accuracy by inputting the facial image of the subject into the learned model. BRIEF DESCRIPTION OF THE DRAWINGS
[0083] Figure 1 This is a block diagram of one embodiment of the stress coping method determination system of the present invention.
[0084] Figure 2 (A) is an explanatory diagram conceptually illustrating a learning face image group labeled "active response". (B) is an explanatory diagram conceptually illustrating a learning face image group labeled "passive response". (C) is an explanatory diagram conceptually illustrating a learning face image group labeled "no response".
[0085] Figure 3 It shows the composition Figure 1 Flowchart of processing contents of a determination device of a stress coping method determination system.
[0086] Figure 4 It shows the composition Figure 1 Flowchart of processing contents of a learning device of a stress coping method determination system.
[0087] Figure 5 TABLE 1 is a table showing the hemodynamic model responses of the prior art study of Experimental Example 1.
[0088] Figure 6 is a conceptual diagram showing a measurement system.
[0089] Figure 7 This is a conceptual diagram showing the topic of mirror drawing.
[0090] Figure 8 It is a graph showing the time series change of MBP (mean blood pressure).
[0091] Fig. 9 is a schematic diagram showing the structure of a CNN (Convolutional Neural Network).
[0092] Fig.10 It is a table showing the filter size, stride, and number of filters of the convolutional layer.
[0093] Fig.11 is a table showing the filter size and stride of the pooling layer.
[0094] Fig.12 The diagram shows facial features of each subject in each case of active response, passive response, and no response, and compares the thermal image and the facial feature map.
[0095] Fig.13 3 is a conceptual diagram showing the experimental protocol of Experimental Example 2.
[0096] Fig.14 This is a graph showing the relationship between the degree of preference and the degree of concentration.
[0097] Fig.15 This is a diagram showing the arrangement state of electroencephalogram electrodes on a subject and a measurement system.
[0098] Fig.16 This is a table showing changes in the multifaceted emotion scale for “positive” content.
[0099] Fig.17 This is a table showing changes in the multifaceted emotion scale for "negative" content.
[0100] Fig.18 This is a table showing changes in the multifaceted emotion scale for the "horror (focus)" content.
[0101] Fig.19 This is a table showing changes in the multifaceted emotion scale for the "horror (non-focus)" content.
[0102] Fig. 20 This is a table showing changes in subjective psychological indicators for “positive” content.
[0103] Fig.21 This is a graph showing changes in subjective psychological indicators for “negative” content.
[0104] Fig. 22 This is a table showing changes in subjective psychological indicators for "horror (concentration)" content.
[0105] Fig.23This is a graph showing changes in subjective psychological indicators for "horror (non-focus)" content.
[0106] Fig.24 This is a table showing time-series changes in physiological indicators for viewing "positive" and "negative" contents.
[0107] Fig.25 This is a table showing the evaluation of physiological indicators for each content.
[0108] Fig.26 It is a graph showing the time series changes of the physiological indexes for watching "horror (concentration)" and "horror (non-concentration)" contents.
[0109] Fig. 27 This is a graph showing the relationship between preferences for TV content and viewing patterns.
[0110] Fig.28 3 is a conceptual diagram showing the experimental protocol of Experimental Example 3.
[0111] Fig.29 is a conceptual diagram showing the experimental protocol.
[0112] Fig.30 This is a table showing the structure of a neural network for estimating “excitement-calmness”, “stress coping style”, and “preference”.
[0113] Fig.31 is a graph showing a method of extracting a feature vector.
[0114] Fig.32 This is a table showing the evaluation of the subjective psychological index for each content.
[0115] Fig.33 This is a table showing the evaluation of physiological indicators for each content.
[0116] Fig.34 This is a table showing the positive discrimination rate of preferences for TV contents and viewing methods.
[0117] Fig.35 This is a graph showing the positive discrimination rate of preferences for TV contents and viewing patterns.
[0118] Fig.36 It is a graph showing the temporal changes of the measured values and the estimated values of the “excited-sedated” state.
[0119] Fig.37 Graph showing the estimation error of the "excited-sedated" state.
[0120] Fig.38 It is a graph showing the relationship between the measured value and the estimated value of the "excited-sedated" state.
[0121] Fig.39 This is a flowchart illustrating a method of obtaining a fractal dimension as a spatial feature quantity.
[0122] Fig.40 It is shown Fig.39 An explanatory diagram of an embodiment of clustering processing in FIG.
[0123] Fig.41 It is shown Fig.39 An illustration of an embodiment of image extraction processing and edge extraction processing in FIG.
[0124] Fig.42 It is shown Fig.39 An explanatory diagram of an embodiment of fractal analysis processing in FIG. DETAILED DESCRIPTION
[0125] Hereinafter, one embodiment of the present invention will be described with reference to the drawings.
[0126] [constitute]
[0127] Figure 1 The stress coping method determination system 100 according to the illustrated embodiment includes a biological information acquisition device (biological information acquisition unit) 110 , a determination device (determination unit) 120 , and a learning device (machine learning unit) 130 .
[0128] The biological information acquisition device 110 is a device for acquiring biological information of a subject P in a non-contact state.
[0129] As the most suitable example of biometric information, a face image IF can be cited. In the following description, a case where the face image IF is used as the biometric information will be described.
[0130] The facial image IF may be a facial thermal image or a facial visible image. When the facial image IF is a facial thermal image, an infrared thermal imager is used as the biological information acquisition device 110. When the facial image IF is a facial visible image, a so-called camera as a visible image shooting device is used as the biological information acquisition device 110.
[0131] As described above, the "visible face image" is a color image obtained by photographing the face of the subject using a generally widely used camera, i.e., a device having an optical system for imaging and used to photograph images. In addition, the "facial thermal image" is an image obtained by analyzing infrared rays radiated from the face of the subject and expressing the heat distribution as a graph, and is an image obtained by photographing with an infrared thermal imager.
[0132] In this case, the image taken by the camera is formed by visible light (wavelength of 380nm to 800nm), and on the other hand, the thermal distribution image obtained by the infrared thermal imager is formed by infrared light (wavelength above 800nm). Therefore, in either case, there is only a difference in wavelength. Therefore, both the infrared thermal imager and the general camera can be used as the biological information acquisition device 110.
[0133] The determination device 120 can be realized by installing the program of the present invention in a general-purpose computer and executing it.
[0134] The determination device 120 is a device having the following function: determining the stress coping method of the subject based on the facial image IF acquired by the biological information acquisition device 110 and a predetermined response mode. The response mode includes three modes consisting of "active response" (mode I), "passive response" (mode II) and "no response" (mode III). The response mode is determined by hemodynamic parameters. The hemodynamic parameters include a plurality of parameters including mean blood pressure (MBP), heart rate (HR), cardiac output (CO), stroke volume (SV) and total peripheral vascular resistance (TPR).
[0135] The determination device 120 includes a determination feature quantity storage unit 121 , a specific site reaction detection unit 122 , and a response pattern determination unit 123 .
[0136] The determination feature storage unit 121 is a functional module that stores a spatial feature corresponding to "active response", a spatial feature corresponding to "passive response", and a spatial feature corresponding to "no response". The spatial feature stored in the determination feature storage unit 121 is a spatial feature extracted by the learning device 130. When the facial image IF is a facial thermal image, facial skin temperature distribution can be cited as an example of the spatial feature.
[0137] The specific part reaction detection unit 122 is a functional module that detects the pressure response of the anatomical specific part of the face of the subject P included in the facial image IF. The anatomical specific part is one or more parts determined from an anatomical point of view as a part that depends less on individual differences. As an example of the anatomical specific part, the top of the nose can be cited.
[0138] The response mode determination unit 123 is a functional module that determines whether the pressure response mode of the subject P is "active response" (mode I), "passive response" (mode II) and "no response" (mode III) based on the pressure response detected by the specific part reaction detection unit 122 and the various spatial feature quantities stored in the feature quantity storage unit 121 for determination.
[0139] The learning device 130 includes a learning data storage unit 131, a feature extraction unit 132, and a feature learning unit 133. The learning device 130 is implemented by installing the program of the present invention in a general-purpose computer and executing it.
[0140] The learning data storage unit 131 is a functional module that stores a plurality of learning face images LG labeled with labels corresponding to "active response", "passive response" and "no response", respectively. Figure 2 A learning face image LG is conceptually illustrated in FIG. Figure 2 LGA1, LGA2, LGA3, ... shown in (A) are a group of learning face images labeled with "active response". Figure 2 LGB1, LGB2, LGB3, ... shown in (B) are learning face image groups labeled with "passive response". Figure 2 LGC1, LGC2, LGC3, ... shown in (C) are a group of learning face images labeled "no response".
[0141] The feature extraction unit 132 is a functional module that extracts spatial feature values of facial images from the learning facial images LG using the learned model 134. A plurality of learning facial images LG labeled corresponding to response patterns determined by hemodynamic parameters are used as training data, and the learned model 134 is generated by machine learning the spatial feature values of the facial images of the subject P included in the learning facial images LG.
[0142] The feature learning unit 133 is a functional module that changes the network parameters of the learned model 134 based on the relationship between the extraction result of the feature extraction unit 132 and the label attached to the learning face image LG as the extraction target so as to improve the extraction accuracy of the spatial feature of the feature extraction unit 132.
[0143] [action]
[0144] Next, follow Figure 3 and Figure 4 The flowchart of will be used to illustrate the processing flow in the determination device 120 and the learning device 130 of the stress coping method determination system 100 constructed as described above.
[0145] like Figure 2 As shown, the determination device 120 executes determination feature quantity storage processing S1, specific site reaction detection processing S2, and response pattern determination processing S3.
[0146] The judgment feature quantity storage processing S1 is a processing of storing the spatial feature quantities extracted by the learning device 130, namely, the spatial feature quantities corresponding to "active response", the spatial feature quantities corresponding to "passive processing" and the spatial feature quantities corresponding to "no response" in the judgment feature quantity storage unit 121.
[0147] The specific site reaction detection process S2 is a process of detecting a pressure response of an anatomical specific site on the face of the subject P included in the face image IF captured by the biological information acquisition device 110 .
[0148] The response mode determination processing S3 is a processing for determining whether the pressure response mode of the subject P is "active response", "passive response" or "no response" based on the pressure response detected by the specific part reaction detection processing S2 and the spatial feature quantities stored in the determination feature quantity storage unit 121.
[0149] like Figure 4 As shown, the learning device 130 executes a learning data storage process S11 , a feature extraction process S12 , and a feature learning process S13 .
[0150] The learning data storage process S11 is a process of storing a plurality of learning face images LG labeled with “active response”, “passive response”, and “no response”, respectively, in the learning data storage unit 131 .
[0151] The feature extraction process S12 is a process of extracting the spatial feature of the facial image of the subject included in the learning facial image LG from the learning facial image LG using the learned model 134 .
[0152] The feature learning process S13 is a process for changing the network parameters of the learned model 134 based on the relationship between the extraction result of the feature extraction process S12 and the label attached to the learning face image LG as the extraction target so as to improve the extraction accuracy of the feature in the feature extraction process S12.
[0153] [Spatial feature quantity]
[0154] In this embodiment, the fractal dimension calculated based on the face image IF can be used as the spatial feature value. By using the fractal dimension, the spatial feature value can be easily and accurately quantified.
[0155] The method of obtaining the fractal dimension as a spatial feature quantity is arbitrary. Fig.39 In the illustrated processing flow, the fractal dimension is obtained by executing a series of processing including the clustering processing S21 , the image extraction processing S22 , the edge extraction processing S23 , and the fractal analysis processing S24 .
[0156] The clustering process S21 is to cluster the temperature distribution of the facial image IF. The clustering method is arbitrary, and as a method suitable for this embodiment, the fuzzy C-means method can be cited. The fuzzy C-means method is not a method that considers the binary problem of whether a certain data group belongs to a cluster. In addition to the situation where the data completely belongs to a unique cluster (K-means method), it is a method that assumes that there are data that belong to multiple clusters to a certain extent, and fuzzily shows the degree of belonging (degree of belonging) of the data to the cluster. Fig.40 An example of clustering the input image (face image IF) is shown in FIG. Fig.40 In the example, cluster 1 belongs to the lowest temperature distribution and cluster 12 belongs to the highest temperature distribution.
[0157] The image extraction process S22 is a process of extracting a clustered image of a temperature distribution in a temperature range above a predetermined temperature from the plurality of clustered images obtained by the clustering process S21. Fig.41 In the example, Fig.40 Of the images of clusters 1 to 12 illustrated as an example, the attribution degree images including the temperature of the face region, that is, the images of clusters 8 to 12 are extracted.
[0158] The edge extraction process S23 is a process for detecting the edge portion of the image extracted by the image extraction process S22, and generating an edge graphic composed of line segments represented by the edge portion. The edge detection method is arbitrary, and the Canny method can be cited as a method suitable for this embodiment. The Canny method is a method for detecting edges through noise removal processing using a Gaussian filter, brightness gradient (edge) extraction processing using a Sobel filter, non-maximum suppression processing to remove portions other than portions where the edge intensity reaches a maximum, and hysteresis threshold processing to determine whether it is a correct edge using a hysteresis threshold. Fig.41 In the example of , the edge portions of the images of clusters 8 to 12 are detected by the Canny method, and an edge graphic consisting of line segments representing the edge portions is generated.
[0159] The fractal analysis process S24 is a process for obtaining the fractal dimension of the edge graphics generated by the edge extraction process S23. The fractal dimension is an index that quantitatively represents the self-similarity and complexity of graphics and phenomena, and is generally a non-integer value. The method of fractal analysis is arbitrary, and as a method suitable for the present embodiment, the box counting method can be cited. The box counting method is the following method: when the graphics of the analysis object are divided into square boxes (grid-shaped), the fractal dimension is obtained based on the absolute value of the slope of the straight line when the relationship between the size of the box and the total number of boxes containing the graphics is approximated by a straight line on a logarithmic hyperbola.
[0160] The calculation formula of fractal dimension (D) is shown below. r is the size of the box, and N(r) is the number of boxes.
[0161] [Number 1]
[0162]
[0163] exist Fig.42 In Fig.41 An example of calculating the fractal dimension of the edge graph of cluster 12 in FIG. In this example, r is varied in the range of 2 to 128 for the edge graph of cluster 12, and r and N(r) are plotted on a logarithmic hyperbola, yielding a value of 1.349.
[0164] [Function and Effect]
[0165] In the stress coping mode determination system 100 configured as described above, the facial image IF of the subject P is captured by the biological information acquisition device 110. The captured facial image IF is input to the determination device 120. The determination device 120 determines which mode of the stress coping mode of the subject P is the mode of mode I, II, and III based on the input facial image IF and the response mode (mode I, II, and III) determined by the hemodynamic parameters, i.e., the mean blood pressure, heart rate, cardiac output, stroke volume, and total peripheral vascular resistance.
[0166] Therefore, according to the stress response method determination system 100, the stress response method of the subject P can be determined in a non-contact state based on the facial image IF of the subject P, so that the type of stress felt by the subject P can be understood without imposing any restrictions on the subject's behavior.
[0167] Furthermore, the stress coping style determination system 100 determines the stress coping style of the subject P based on the reaction and response pattern (patterns I, II, and III) of the anatomically specific facial parts included in the facial image IF of the subject P. The anatomically specific parts are one or more parts determined from an anatomical viewpoint as parts that are less dependent on individual differences.
[0168] Therefore, according to the stress coping method determination system 100, a general-purpose system for determining a stress coping method can be constructed.
[0169] In addition, in the stress coping method determination system 100, the feature quantity corresponding to "active coping", the feature quantity corresponding to "passive coping", and the feature quantity corresponding to "no coping" are extracted by the learning device 130. The learning device 130 stores a plurality of learning facial images LG labeled with labels corresponding to "active coping", "passive coping", and "no coping", respectively, and uses the learned model 134 to extract the spatial feature quantity of the facial image from these learning facial images LG. Based on the relationship between the extraction result and the label labeled for the learning facial image LG as the extraction object, the network parameters of the learned model LG are changed to increase the extraction accuracy of the spatial feature quantity of the facial image. As the learning of the learned model LG progresses, the extraction accuracy of the spatial feature quantity of the facial image is improved, and the accuracy of the spatial feature quantity stored in the determination feature quantity storage unit 121 is also improved.
[0170] Therefore, according to the stress coping method determination system 100, as the learning of the learned model LG in the learning device 130 progresses, the accuracy of determining the stress coping method can be improved.
[0171] Furthermore, according to the stress coping method determination system, by quantifying the spatial feature quantity using the fractal dimension, it is possible to further improve the determination accuracy of the stress coping method.
[0172] In addition, the present invention is not limited to the above-mentioned embodiment. For example, in the above-mentioned embodiment, the stress coping method determination system 100 includes the learning device 130, but the learning device 130 may be omitted. In the case where the learning device 130 is omitted, the spatial feature quantity extracted or generated by a mechanism other than the learning device 130 is stored in the determination feature quantity storage unit 121 of the determination device 120.
[0173] <Experimental Example 1>
[0174] An experimental example for collecting data of feature quantities stored in the determination feature quantity storage unit 121 of the present embodiment is described below.
[0175] 1. Experimental Methods
[0176] The experiment was conducted in a shielded room at 25.0±1.5℃, and the subjects were 8 healthy male adults aged 18 to 21 years old. Figure 6The subject sat down and wore a measuring cuff of a continuous blood pressure and hemodynamics measuring device (Finometer model 2, manufactured by Finapres Medical Systems BV) on the second joint of the left middle finger.
[0177] As hemodynamic parameters, mean blood pressure (MBP), heart rate (HR), cardiac output (CO), and total peripheral vascular resistance (TPR) were measured. In order to measure the facial thermal image, an infrared thermal imager (TVS-200EX, AVIONICS) was set at a position 1.2m in front of the face with a viewing angle that could measure the entire face, and the shooting interval was set to 1fps. When measuring the facial skin temperature, the patient sat in a chair with a backrest, and a projector was used to project the computer screen onto the wall at a distance of 1.55m.
[0178] The experiment consisted of 60 seconds of quiet eyes closing before the task (Rest1), 60 seconds of active task (Task1), 60 seconds of no response task (Task2), 60 seconds of passive task (Task3), and 60 seconds of quiet eyes closing after the task (Rest2). Mental arithmetic tasks were performed in the active task, mirror writing tasks were performed in the passive task, and quiet eyes closing was performed in the stress-free response task. The mental arithmetic task consisted of a 2-digit addition operation every 4 seconds on a computer screen. The correctness of the mental arithmetic was not announced each time. In addition, in the mirror writing task, the subjects were instructed to use the mouse with their right hand to move the star-shaped figure displayed on the screen (refer to Figure 7 ) and reproduced on the computer. The movement of the mouse and the movement of the cursor on the screen are reversed up and down and left and right.
[0179] The cursor and the cursor trajectory are displayed on the star graph. If the cursor is out of bounds, it returns to the original position and all the trajectories are deleted. The starting position of the circle is the top of the star. As a non-response subject, the patient is allowed to close his eyes quietly. The average value of Rest1 in the hemodynamic parameters is used as the baseline for normalization.
[0180] 2. Analysis method
[0181] 2-1 Determination of hemodynamic parameters
[0182] Figure 8The time series changes of the normalized MBP of subject A are shown in FIG. For each hemodynamic parameter, the range of values above the baseline + 2σ (σ: standard deviation of each hemodynamic parameter in Rest1) is defined as “+”, and the range of values below the baseline - 2σ is defined as “-”. According to the pattern response in Table 1, each hemodynamic parameter is distinguished as “Mode I” (active response) or “Mode II” (passive response), and the situation that does not meet any of the conditions is distinguished as “no response”. In addition, based on the determination of the hemodynamic parameters, the thermal images taken during Tasks 1 to 3 are labeled.
[0183] 2-2 Creation of input data
[0184] In order to use the labeled thermal images in machine learning, the facial part was cropped at 151×171 pixels and grayscaled to create a facial thermal image. In addition, in order to make the number of labeled facial thermal images (input data) consistent with each stress response, random cropping, white Gaussian noise addition, and contrast adjustment were performed to expand the data.
[0185] 2-3 Machine Learning Using CNN
[0186] In this experiment, a convolutional neural network (CNN) was used to construct a personal discrimination model of stress coping style based on facial skin temperature distribution, and feature quantities related to stress coping style were extracted.
[0187] The structure of CNN is composed of 3 convolutional layers for feature extraction, 3 pooling layers, and 1 fully connected layer for discrimination. The structure of CNN is shown in Fig. 9 , the parameters of the filters of the convolutional layer and the pooling layer are shown in Fig.10 , 11 Based on the feature map which is the weight of the convolutional layer of CNN, feature analysis related to stress coping style is performed.
[0188] 3. Results and Research
[0189] The feature map of the second convolutional layer when the facial thermal image of each stress coping method is input to CNN for each subject is shown in FIG. Fig.12 middle.
[0190] The results of observing the characteristic mapping of each coping style within the subjects showed that, in particular, in subject B, the characteristic locations expressed were different between the stress coping styles, such as the left cheek in active coping and no-stress coping, and the right cheek in passive coping. The results of observing the characteristic mapping between subjects showed that in most of the subjects, the characteristics were expressed in the nose, but it was confirmed that there were individual differences in the characteristic expression locations of each subject.
[0191] The main reason for the individual differences in the characteristic expression sites of each subject is believed to be differences in the structure of blood vessels or fat. However, it is considered that by identifying the characteristic sites with anatomical meaning, it is possible to lay the foundation for building a general model for stress response discrimination.
[0192] 4. Summary
[0193] In this experiment, we used CNN to try to identify stress coping styles based on facial skin temperature distribution and extract features related to stress coping styles. The results showed that the feature distribution of stress coping styles in facial skin temperature distribution changed, and we confirmed that there were individual differences in the feature distribution related to stress coping styles between individuals.
[0194] References
[0195] [1] Sawada Yukihiro: New Physiological Psychology (edited by Miyata Hiroshi), Chapter 10, Kitaoji Shobo, pp. 172-195, (1998)
[0196] [2] Nagano Yuichiro: “Cardiac and vascular responses in competitive mirror-writing tasks”, Physiological Psychology and Psychophysiology, Vol. 22, No. 3, pp. 237-246, (2004)
[0197] [3] Bando Shizuka, Asano Hirotoshi, and Nozawa Akio: "Analysis of physiological effects related to reading display media", Journal of the Institute of Electrical Engineering C (Electronics, Information, and Systems), Vol. 135, No. 5, pp. 520-525, (2015)
[0198] [4] Watanuki Takuya and Nozawa Akio: “Analysis of TV viewing behavior based on stress coping styles”, Journal of the Institute of Electrical Engineering C (Electronics, Information, and Systems), Vol. 134, No. 2, pp. 205-211, (2014)
[0199] [5] Hiroki, Akio Nozawa, Tsutomu Mizuno, and Eito Ide: “Physiological and psychological evaluation of mental workload under time pressure”, Journal of the Institute of Electrical Engineering C (Electronics, Information, and Systems), Vol. 127, No. 7, pp. 1000-1006, (2007)
[0200] [6] Hiroki Ito, Shizuka Bando, Kosuke Oiwa, Akio Nozawa: Evaluation of Variations in Autonomic Nervous System's Activity During the Day Based on Facial Thermal Images Using Independent Component Analysis, IEEJ Transactions on Electronics, Information and Systems, Vol. 138, No. 7, pp. 1-10, (2018)
[0201] [7] Matsumura Kenta, Sawada Yukihiro: “Cardiovascular responses to two mental arithmetic tasks”, Psychological Research, Vol. 79, No. 6, pp. 473-480, (2008)
[0202] Furthermore, two experimental research examples conducted by the inventors of the present application as basic research on the stress coping method determination system of the present invention are shown below as an application example (embodiment) of the present invention.
[0203] These experimental examples are not experimental examples of "obtaining biological information of the subject in a non-contact state" as in the present invention, but are conducted by using a continuous blood pressure monitor and having the subject wear a finger cuff. However, similarly to the present invention, the physiological and psychological state of the viewer when watching TV video content is analyzed, classified, and evaluated based on hemodynamic parameters. Therefore, the present invention can be applied to the determination of the stress mode of the subject in such a case.
[0204] <Experimental Example 2>
[0205] Purpose of the experiment
[0206] Nowadays, people are surrounded by a large number of information devices. Among them, television has been recognized as a core news media or entertainment media installed in the living room of a family since its invention. However, with the popularization of miniaturized and high-performance information and communication devices brought about by the development of IT technology in recent years and the expansion of the information network environment, the existence of information media has changed significantly, and the way of watching television has also changed significantly.
[0207] Nishigaki pointed out in the article that people who always turn on the TV while doing chores or using their mobile phones are obviously far from being "focused viewers". (1)However, if we can understand this shift in viewing habits, we can give television new added value, such as adjusting emotions, moods, and actions in the desired direction through TV content design.
[0208] Fujiwara and Saito conducted a public opinion survey, and Ohno conducted a questionnaire survey. The reasons for watching TV included "killing time", "to enjoy interesting programs", "to improve cultural literacy and gain knowledge", "out of habit", "to escape from the annoying reality", "to change the mood", "as background music", and "to reunite with the family". (2)(3) .
[0209] In addition, Takahashi et al. classified the time for watching TV into "breakfast", "commuting", "doing housework", "relaxing at home on weekdays", etc. (4) In addition, Tomoe et al. classified the way of watching TV into "dedicated watching" which means "both body and mind are focused on TV", "simultaneous and dedicated watching" which means "I want to watch this program but I watch it while doing other things", and "relaxation time watching" which means "there is nothing I want to watch but my body is focused on TV".
[0210] Thus, it is clear that viewing styles vary widely and change according to factors such as time, place, and mood. Preference for TV images is also one of the reasons for the transformation of viewing styles. However, there is no research that classifies viewing styles by preference. In addition, most of these studies that discriminate viewing styles use psychological responses such as opinion surveys or post-event introspective evaluations, and there is no research that classifies viewing styles using physiological responses.
[0211] Since the images and sounds of television are visual and auditory stimuli to organisms, they themselves can be considered stressors. Stressors cause organisms to cope with stress, which brings about physiological and psychological changes as a stress response. If various television contents are considered stressors, it can be expected that different responses to the content, that is, different viewing methods, will lead to different physiological stress responses. Nomura et al. classified the corresponding status of e-learning content using cardiovascular system indicators (6) .
[0212] Therefore, this study aimed to analyze the viewing patterns of TV content by using cardiovascular system indicators, and evaluated the central nervous system activity and autonomic nervous system activity by measuring the cardiovascular system using a continuous blood pressure monitor, measuring the facial skin temperature using an infrared thermal imaging device without contact, and measuring the electrocardiogram and brain waves using a digital bioamplifier. In particular, the viewing patterns when watching TV were analyzed by stress coping methods using hemodynamic parameters and preferences for TV images.
[0213] 2. Factor extraction experiment
[0214] Considering that viewing patterns are influenced by video content and personal preferences, a classification of factors was conducted through preliminary experiments.
[0215] <2.1> Experimental conditions
[0216] The subjects were 20 healthy university students (age: 19-22, average age: 21.1, 11 males, 9 females). The experimental content, purpose, and survey subjects were fully explained to the subjects verbally and in writing in advance, and their consent to assist in the experiment was confirmed by signing. The measurement was carried out in a non-convection shielded room with a room temperature of 26.0±1.6℃, without temperature input from the outside.
[0217] The subjects watched 10 kinds of video content (news, sports, documentaries, music, horror, comedy, TV series, cartoons, cooking, shopping) for 5 minutes on a LCD TV (55 inches, HX850, BRAVIA, SONY) set 2 meters in front of them. The video content was played by a DVD player (SCPH-3900, SONY). The video content was not made public until the subjects watched it. In addition, in order to eliminate the order effect, the order of the video content was set to random.
[0218] <2.2> Experimental procedures
[0219] like Fig.13 As shown in Figure 1, this experiment consisted of 5 minutes of video viewing and 1 minute of quiet eye-closing states R1 and R2 before and after. In addition, the preference and immersion of various video contents were evaluated using the Visual Analogue Scale (VAS) and immersion survey.
[0220] <2.3> Evaluation method
[0221] At the end of the experiment, the subjective feeling of "liking" was measured using VAS. In VAS, pairs of adjectives are placed at both ends of a line segment, and the subject checks any position on the line segment, thereby measuring the subject's psychophysical quantity. The sentences placed at both ends of the scale are set to "dislike very much" - "like very much". In addition, the immersion in the image is measured using a 5-point scale (1: completely unable to immerse - 5: quite immersed).
[0222] <2·4> Results and research of factor extraction experiment
[0223] Fig.14 The left side of the graph shows the average of all subjects' preferences and immersion in the 10 types of content. The error bars in the graph represent the standard errors of preferences and immersion. Fig.14 The right side of the graph shows the preferences of 20 subjects for news and horror among the 10 types of content, with particularly large individual differences. Fig.14 If we observe the preference and immersion of each content from the left, the average preference and immersion are in direct proportion. Fig.14 In, N=20.
[0224] Horror shows a unique result of low preference but high immersion. This is probably because people want to watch it out of curiosity even though they hate it, which is the so-called "the scarier the better". Fig.14 As can be seen from the right side of , since each subject has different preferences and immersion, there are large individual differences. Therefore, in the physiological measurement experiment, based on the above-mentioned factor classification experiment, each subject is divided into high-immersion positive preference (higher preference) image content and low-immersion negative preference (lower preference) image content, as well as horror images that show specific results such as negative preference but high immersion or low immersion, and prompts are given.
[0225] 3. Physiological Measurement Experiments
[0226] Based on the factors extracted by the factor extraction experiment, physiological measurements are performed again on the same subject using other images.
[0227] <3.1> Experimental conditions
[0228] The subjects were 14 healthy university students (age: 19-22, average age: 21.1, 7 males, 7 females). The experiment was conducted in the same shielded room as 2.1. In order to switch the video content and confirm the physiological measurement status, there was an experimenter in the same room. In order to achieve the adaptation of the body surface temperature to the room temperature, the experiment was carried out more than 20 minutes after the subjects entered the room. Based on the results of the factor extraction experiment, positive and negative video content preferences were set for each subject. The video content with the highest preference among the immersion of 4 or more and the preference of 0.6 or more was set as positive preference (hereinafter abbreviated as "positive"), and the video content with the lowest preference among the immersion of 2 or less and the preference of 0.4 or less was set as negative preference (hereinafter abbreviated as "negative"). Therefore, the video content watched was "positive", "negative", and "horror".
[0229] <3.2> Experimental procedures
[0230] As in the factor extraction experiment, Fig.13In addition, VAS, Multiple Mood Scale (MMS), and immersion survey were used to evaluate the emotional changes, preferences, and immersion of each video content.
[0231] <3.3> Measurement system
[0232] The measurement system and the configuration of the EEG measurement electrodes are shown in Fig.15 . The subject wore a continuous sphygmomanometer, EEG derivation electrodes, and ECG derivation electrodes while sitting. The subject wore a continuous sphygmomanometer (Finometer model 2, manufactured by Finapres Medical Systems BV) on the second joint of the left middle finger and recorded on the PC at a sampling frequency of 200 Hz. An infrared thermal imaging device (TVS-200X) was set at a position 0.7m in front of the subject at an angle capable of measuring the face. The skin emissivity was set to ε=0.98, the temperature resolution was set to less than 0.1°C, and recorded on the PC at a sampling frequency of 1Hz. EEG was measured by the reference electrode method with the left ear (A1) as a reference. The position of the EEG derivation electrode was set to 1 point (Pz) based on the international 10-20 method, and the electrode contact resistance was set to less than 10kΩ.
[0233] Usually, α waves are recorded as O1 and O2, but in this study, the α wave energy, which is a stable indicator of brain activity, is specifically calculated for the purpose of calculating its attenuation ratio. Therefore, compared with the left-right difference or local existence, more emphasis is placed on the ease of wearing the electrodes and reducing the measurement pressure on the subject, thereby measuring the nearby Pz. In order to minimize the mixing of myoelectric potentials, the electrodes for ECG measurement are worn at the upper edge of the sternum (+) and the apex of the heart (-) according to the NASA lead method. As a ground electrode, the EEG and ECG are shared and set to the top of the head (Cz). The EEG signal and the ECG signal are amplified by a digital bioamplifier (5102, manufactured by NFELECTRONIC INSTRUMENTS) and recorded on a PC via a processor box (5101, manufactured by NF ELECTRONIC INSTRUMENTS) at a sampling frequency of 200Hz.
[0234] <3.4> Evaluation method
[0235] In this study, the correlation between physiological and psychological indicators was analyzed. The physiological indicators were set as mean blood pressure (MP), heart rate (HR), stroke volume (SV), cardiac output (CO), total peripheral resistance (TPR), nasal skin temperature (NST), electrocardiograms (ECG) and electroencephalograms (EEG). The frequency component of EEG from 8Hz to 13Hz is called alpha wave, which is known to be obvious when quiet, eyes closed, and awake, and attenuated when any conditions are broken. (8) In this study, the 1024 sampling points of the EEG time series sampled at 200 Hz were Fourier transformed every 10 seconds to obtain the alpha wave power spectrum every 10 seconds. Fig.13 The average α wave power in each of the R1 and R2 intervals is taken, and the ratio of the average α wave power in the R2 interval to the average α wave power in the R1 interval is used as an indicator of the change in alertness before and after watching TV.
[0236] NST is known as an activity index of the sympathetic nervous system that controls peripheral blood flow. The hyperactivity and inhibition of the sympathetic nervous system are closely related to the temporal variation of NST, so in this study, the variation of NST every 10 seconds was used as a quantitative index of the kinetic energy of sympathetic nervous system activity related to image viewing. Positive values indicate inhibition of sympathetic nervous system activity, and negative values indicate hyperactivity of sympathetic nervous system activity. In each increment of the measured nasal thermal image time series, the spatial average temperature in 10×10 pixels of the nose was calculated and set as the NST time series. HF is a high-frequency component of 0.15Hz to 0.4Hz in the heart rate variability spectrum, and is known as a component of respiratory sinus arrhythmia. (7) .
[0237] HF is an indicator of the parasympathetic nervous system, and increases or decreases according to the hyperactivity or inhibition of the parasympathetic nerves. The time series of the R wave peak interval is obtained from the ECG sample by threshold processing, and after cubic spline interpolation, it is sampled at 20Hz. The sampled data is subjected to FFT processing once every 1 second to obtain the time series of the heart rate variation power spectrum. The number of data processed by FFT is set to 512 points. In the heart rate variation power spectrum of the time series, the integral value of the 0.15Hz to 0.4Hz area is obtained and set as the HF time series. It is known that hemodynamic parameters (MP, HR, SV, CO, TPR) show characteristic response patterns (mode I, mode II) according to external pressure. This is an important concept in understanding the physiological response of the cardiovascular system to pressure.
[0238] Specifically, pattern I is characterized by the hyperactivity of myocardial contractile activity and the increase in blood volume to skeletal muscle due to vasodilation, which can be said to be an energy-consuming response (active response). On the other hand, pattern II is characterized by the constriction of peripheral blood vessels and a substantial decrease in heart rate, which can be said to be an energy-saving response (passive response).
[0239] In addition, the psychological indexes are set as the Multifaceted Sense of Situation Scale (MMS) and VAS. MMS measures the temporary emotional and feeling states that change according to the conditions of the subject, and uses eight emotional state scales consisting of depression (hereinafter abbreviated as DA), hostility (hereinafter abbreviated as H), fatigue (hereinafter abbreviated as F), active pleasure (hereinafter abbreviated as A), inactive pleasure (hereinafter abbreviated as I), affinity (hereinafter abbreviated as AF), concentration (hereinafter abbreviated as D), and surprise (hereinafter abbreviated as S). (9) Before and after the experiment, VAS was used to measure four subjective feelings: "sense of wakefulness", "pleasure and unpleasantness", "fatigue", and "liking".
[0240] “Feelings of pleasure and displeasure” and “feeling of clarity” were selected as the items of psychological evaluation in this study as the basic components of Russell’s dualism of emotions. (10). In VAS, pairs of adjectives are placed at both ends of a line segment, and the subject ticks any position on the line segment, thereby measuring the subject's psychophysical quantity. For the sentences placed at both ends of the scale, the sense of wakefulness is set as "very sleepy" - "very awake", the sense of pleasure and unpleasantness is set as "very unpleasant" - "very pleasant", the mental level is set as "very tired" - "very energetic", and the preferences are set as "very disgusting" - "very fond of". Each VAS is prepared on a separate paper, and the subject is instructed to write it down in handwriting in sequence without recursive reference. Furthermore, after the experiment, the sense of immersion in the image is measured using a 5-point scale (1: completely unable to immerse - 5: quite immersed). In addition, as a statistical analysis method, a corresponding t-test is used in the test of the difference before and after watching the image of each psychological indicator, and the Wilcoxon signed order test is used for the overall change in the TEST interval in each physiological indicator.
[0241] <3.5> Results and Research
[0242] Statistical analysis was performed on psychological indicators to discuss psychological responses during and before and after watching TV images. In addition, all subjects' preferences for horror image stimuli were below 0.4, so only the sense of immersion was considered. In the 5-stage immersion evaluation after watching horror images, 4 or 5 was classified as horror (concentration) (hereinafter abbreviated as "horror (C)"), and 1 or 2 was classified as horror (non-concentration) (hereinafter abbreviated as "horror (N)"). Afterwards, the results were labeled by the classifications of "positive", "negative", "horror (C)", and "horror (N)". The number of people who responded to "horror (C)" was 10, and the number of people who responded to "horror (N)" was 4. The average of all subjects in each emotional scale of the MMS is shown in Figures 16 to 19 .
[0243] exist Fig.16 In N = 14, Fig.17 In N = 14, Fig.18 Where N = 10, Fig.19 N = 4. The average of all subjects in the immersion "pleasure and unpleasantness", "clearness", "vitality", "preference" and "immersion" is shown in Figures 20-23 .exist Fig. 20 In N = 14, Fig.21 N = 14, Fig. 22 Where N = 10, Fig.23 Where N=4.
[0244] according to Fig.16 , “positivity” increased significantly in A and AF, and decreased significantly in F. On the other hand, according to Fig.17 , “Negativity” increased significantly in H and F. This is consistent with the behavior of watching positive and negative video content respectively. “Horror (C)” increased significantly in Fig.18 A and I were significantly reduced, which can be inferred to generally evoke unpleasant emotions. Fig.19 , Fig.18 There are many scales with significant differences.
[0245] From this, it can be inferred that, for horror, the emotional changes when immersed are greater than when not immersed. Fig. 20 , "Positiveness" significantly increased in pleasantness and unpleasantness, clarity and vitality. From this, it can be considered that the positive video content brought positive comfort to the subjects accompanied by "refreshing" and "comfortable feeling".
[0246] On the contrary, according to Fig.21 Since the "negativity" is significantly reduced in pleasant and unpleasant feelings, sobriety and vitality, it can be considered that negatively liked video content brings negative unpleasant feelings accompanied by "depression" and "unpleasant feelings" to the subjects. In addition, according to Fig. 22 Since "fear (C)" significantly increases in wakefulness and significantly decreases in pleasure, unpleasantness, and vitality, it can be considered that it brings about positive unpleasantness accompanied by "awakeness" and "unpleasantness".
[0247] Next, we will discuss the physiological responses during and before and after watching TV videos. The average temporal changes of each physiological index among the subjects are shown in Fig.24 .exist Fig.24 N = 14. From the top are NST, α wave, HF, MP, HR, SV, CO, TPR. The start of the TEST interval is set to 0. The error bars in the figure represent the standard error every 10 seconds. In addition, the baselines of the R1 interval of NST, MP, HR, SV, CO, TPR and immersion are set to 0, and the baselines of the R1 interval of α wave and HF are set to 1 for normalization. The significance probability p of the Wilcoxon sign order test related to the displacement of the baseline from the entire TEST interval is shown in Fig.25 (+: p<0.1, *: p<0.05, **: p<0.01). P in the table indicates a positive response to each indicator, and N indicates a negative response. Fig.25 In this example, N=14.
[0248] according to Fig.24 If we observe the time series changes of each index, for NST, both "positive" and "negative" decrease with the start of the TEST interval, so it can be inferred that the sympathetic nerves are hyperactive. No significant changes were found in the α waves. Fig.25HF showed no significant change in the "positive" state, but decreased significantly in the "negative" state. HF is an indicator of the activity of the parasympathetic nervous system, and increases or decreases depending on the hyperactivity or inhibition of the parasympathetic nerves. Therefore, it can be considered that the "negative" state inhibits the parasympathetic nerves. This is consistent with the explanation in the NST mentioned above.
[0249] The average temporal changes of the physiological indicators of the horror images among the subjects are shown in Fig.26 .exist Fig.26 In, N = 14. Fig.26 , the NST of "fear (N)" decreases, so it can be inferred that the sympathetic nerves are hyperactive. No significant results were obtained for α waves. It can be seen that for HF, "fear (C)" is significantly reduced, and "fear (N)" is significantly increased. For HF, the explanation in "fear (N)" is inconsistent with that in the above NST. It is considered that this is due to the difference in mechanism. As shown below, "fear (N)" shows an active response characterized by an increase in TPR. As a result, it is considered that the blood flow in the peripheral vascular part is reduced, resulting in a decrease in NST.
[0250] Next, we classified the hemodynamic responses into "positive", "negative", "horror (C)" and "horror (N)" according to the modes of immersion and preference, and compared them. Fig. 27 In this way, it can be summarized on the axis of stress coping style and preference. Fig. 27 In this example, N=14.
[0251] Both "Positive" and "Horror (C)" had high immersion, but "Positive" was positively preferred and Horror (C) was negatively preferred. However, both groups saw increases in MP, HR, SV, and CO, and decreases in TPR. This is a typical mode I response (active response) that is mainly characterized by an increase in myocardial contractile activity.
[0252] In other words, it is clear that when actively coping, regardless of their preferences, they are immersed in the TV video content. In addition, if we extrapolate from the results of the factor extraction experiment, we can expect that the subjects who also have a high preference for horror videos in the physiological measurement experiment will also have a high sense of immersion, which is located in Fig. 27 , but no match exists. On the other hand, the immersion and preference of "Positive" and "Horror (N)" are both low. However, their physiological responses are different. That is, in "Negative", although TPR and HR decrease, no change in MP is seen. It can be considered that it is neither active nor passive, but no response. In addition, Fig. 27 In this example, N=14.
[0253] In contrast, in "fear (N)", although there is no significant change in HR, a significant increase in TPR and MP is characteristic. This is a typical mode II reaction (passive coping) mainly caused by an increase in MP due to an increase in peripheral vasoconstriction. From this, it is clear that when stress coping is not shown, the preference is usually low and the TV image content is not immersed, but a passive coping is shown for horror images. In other words, it is clear that the viewing state cannot be determined by the preference for TV images alone, and the viewing state can be classified according to the stress coping method.
[0254] 4. Summary
[0255] In this study, we conducted physiological and psychological measurements when watching video content with different preferences, and attempted to classify viewing styles according to stress coping styles with physiological evidence. We measured hemodynamic parameters (MP, HR, SV, CO, TPR) as cardiovascular system indicators, measured the alpha wave power spectrum of EEG as a central nervous system indicator, and measured nasal skin temperature and heart rate variability HF component as autonomic nervous system indicators. We also included simultaneous measurement of video preferences and psychological questionnaires, and conducted statistical analysis related to physiological and psychological states to quantitatively evaluate the physiological and psychological effects of watching video content.
[0256] As a result, the classification of psychological indicators related to preferences in this study has not been evaluated in existing studies. Therefore, by combining the stress coping and preferences of this study, a new classification of TV viewing can be obtained. As a conclusion, when actively coping, TV images are "positive" and "scary (C)", when passively coping, TV images are "scary (N)", and when no stress coping is performed, TV images are "negative". In other words, it is clear that the viewing style cannot be determined by the preference for TV images alone, but the viewing style can be determined based on the pressure response of hemodynamic parameters.
[0257] <References>
[0258] (1) Hirata Akihiro, Morofuji Emi, and Aramaki Hiroshi, “The Current State of Television Viewing and Media Use (2) - From the Survey on “Japanese People and Television 2010” -”, Journal of Broadcasting Research, pp. 2-27 (2010)
[0259] (2) Aramaki, Tomoko Masuda, and Sachiko Nakano: “How will television face people in their twenties?”, Broadcasting Research and Survey, pp. 2-21 (2008)
[0260] (3) Y. Siki, Y. Murakami and Y. Huzita: “Television viewing and simultaneous use of new media: an ethnographic study of young people in Japan” Institute for Media and Communication Research Keio University, Vol. 59, No. 1, pp. 131-140 (2009) (in Japanese) Shiki Yuko, Murayama Yo, Fujita Yuko: “Television viewing and simultaneous use of new media among young people: an ethnographic study of university students” Journal of the Institute for Media and Communication Research Keio University, Vol. 59, No. 1, pp. 131-140 (2009)
[0261] (4) S. Nomura, Y. Kurosawa, N. Ogawa, C. M. Althaff Irfan, K. Yajima, S. Handri, T. Yamagishi, K. T. Nakahira, and Y. Fukumura: “Psysiological Evaluation of student in E-learning Sessions by Hemodynamic Response”, IEEJ Trans. EIS, Vol. 131, No. 1, pp. 146-151 (2011) (in Japanese)
[0262] (5) T. Watanuki and A. Nozawa: “Visualization of a feeling of concentration for TV contents”, Bulletin of Institute of Electrical and Electronic Engineers of measurement, Vol. IM-12, No. 63, pp. 19-25 (2012) (in Japanese)
[0263] (6) K. Mori, C. Ono, Y. Motomura, H. Asoh, and A. Sakurai: “Empirical Analysis of User Preference Models for Movie Recommendation”, IEIC Technical Report. NC. neurocomputing, Vol. 104, No. 759, pp. 77-82 (2005) (in Japanese) Kurokawa Shigeki, Ono Tomohiro, Motomura Yoichi, Aso Hideki, Sakurai Akito: “Experimental Evaluation of User Preference Models for Movie Recommendation”, IEIC Technical Report. NC. neurocomputing, Vol. 104, No. 759, pp. 77-82 (2005)
[0264] (7) Sawada Yukihiro, "Hemodynamic Response: New Physiological Psychology Volume I (Edited by Fujisawa Kiyoshi, Kakigi Shoji, and Yamazaki Katsuo)", Kitaoji Shobo, Chapter 10, p. 187 (1998)
[0265] (8) J.A. Russell, “A circumplex model of affect”, J. Personality and Social Psychology, Vol. 39, pp. 1161-1178 (1980)
[0266] <Experimental Example 3>
[0267] 1. Purpose of the experiment
[0268] In recent years, the digitalization and multi-channelization of television broadcasting have been advancing. However, with the popularization of miniaturized and high-performance information and communication equipment and the expansion of the information network environment brought about by the development of IT technology, the phenomenon of de-televisioning of television as entertainment has accelerated, and the reasons for watching television have become more diverse. Hirata et al. cited "to understand what is happening and what is happening in society", "to relieve fatigue or relax", "to deepen or expand interpersonal relationships" as reasons for watching television. (1) . In addition, as for the reasons for watching TV and the way of watching TV, along with the de-televisionization, there is a shift in the view of weak participation, such as a non-caring attitude towards watching or a weakening of watching habits. Shiki et al. cited the following watching methods: "dedicated watching" where one focuses on watching TV, "simultaneous watching" where one watches TV while doing other life activities such as housework, eating, and studying, and "non-caring watching" where one watches many programs while changing channels. Thus, it is clear that there are many different ways of watching TV, and they change according to factors such as time, place, and mood.
[0269] However, most of these studies that classify viewing patterns have used psychological responses such as opinion surveys or post-event introspective evaluations, and there has been no research that has used physiological responses to classify viewing patterns. Since television images and sounds are visual and auditory stimuli to organisms, they themselves can be considered stressors. Stressors cause organisms to cope with stress, which leads to physiological and psychological changes as a stress response.
[0270] If we consider various TV contents as stressors, we can expect that different responses to the contents, that is, different viewing methods, will lead to different physiological stress responses. Nomura et al. classified the response status of e-learning content by cardiovascular system indicators (4) .
[0271] Therefore, attempts have been made to classify TV video content viewing patterns and preferences based on cardiovascular system indicators. (5) In addition, research is conducted not only on classification but also on the construction of inference models.
[0272] Kurokawa et al. constructed a user preference model for video content by using naive Bayes, decision trees, Bayesian networks, and neural networks, and extracted the prediction accuracy and important variables for content evaluation. (6) However, this study also only used a model of psychological responses, and did not conduct any research using physiological responses to construct an inference model of preferences and viewing patterns.
[0273] The results of previous studies suggest the possibility of classifying the physiological and psychological states of TV viewers, such as preference for content and immersion, by stress coping styles and heart rate based on hemodynamic parameters. Therefore, the purpose of this study is to conduct experimental research on the estimation of the physiological and psychological states of viewers related to TV viewing based on cardiovascular system indicators. Feature vectors are extracted from cardiovascular system indicators to create and evaluate estimation models for preference for TV video content, viewing style, and excitement-sedation.
[0274] 2. Experiment
[0275] Cardiovascular system measurements were performed using a continuous blood pressure monitor while watching TV video content. Pattern recognition using a hierarchical neural network was then performed based on the hemodynamic parameters to estimate preferences, viewing patterns, and excitement-sedation when watching TV video content.
[0276] <2.1> Experimental procedures
[0277] like Fig.28 As shown in the figure, this experiment consists of 5 minutes of video viewing and 1 minute of quiet eyes-closed state R1 and R2 before and after. The preference and immersion of various video contents were evaluated by Visual Analogue Scale (VAS) before and after video viewing and immersion survey before and after viewing. In addition, the changes in subjective excitement-calmness felt by the subjects were recorded in real time during video viewing.
[0278] <2.2> Experimental conditions
[0279] The subjects were 14 healthy university students in Japan (age: 19-22, average age: 21.4, 7 males, 7 females). The content, purpose, and survey subjects of the experiment were fully explained to the subjects verbally and in writing in advance, and their consent to cooperate in the experiment was confirmed by signing. The measurement was conducted in a non-convection shielded room with a room temperature of 26.0±1.6℃. In order to switch the image content and confirm the physiological measurement status, there was an experimenter in the same room. In order to achieve the adaptation of the body surface temperature to the room temperature, the experiment was carried out more than 20 minutes after the subject entered the room.
[0280] The subjects sat in their seats and watched three types of video content: positive preference and high immersion (hereinafter referred to as "positive"), negative preference and low immersion (hereinafter referred to as "negative"), and horror video content showing a unique result that immersion varies greatly among individuals despite low preference, using an LCD TV (55 inches, HX850, BRAVIA, manufactured by Sony) set 2 meters in front of them. The video content was played by a DVD player (SCPH-3900, manufactured by Sony).
[0281] <2.3> Measurement system
[0282] exist Fig.29 The measurement system is shown. The subject wears a continuous blood pressure monitor at a seat. The subject wears a continuous blood pressure monitor (Finometer model 2, manufactured by Finapres Medical Systems BV) at the second joint of the left middle finger and records the data on the PC at a sampling frequency of 200 Hz. In addition, a keyboard (K270, manufactured by Logicool) is set in front of the subject, and the software that inputs the changes in subjective excitement-sedation felt by the subject in real time through the up and down keys of the keyboard is used to record excitement-sedation successively and relatively.
[0283] <2.4> Evaluation method
[0284] The physiological indicators are set as mean pressure (MP), heart rate (HR), stroke volume (SV), cardiac output (CO), and total peripheral resistance (TPR). It is known that the hemodynamic parameters (MP, HR, SV, CO, TPR) show characteristic reaction patterns (mode I, mode II) according to the external pressure, and are important concepts for understanding the physiological response of the cardiovascular system to pressure. Specifically, mode I is characterized by the hyperactivity of the contractile activity of the myocardium and the increase in the amount of blood to the skeletal muscle caused by vasodilation, which can be said to be an energy-consuming reaction (active response). On the other hand, mode II is characterized by the contraction of peripheral blood vessels, and the heart rate is also generally reduced, which can be said to be an energy-saving reaction (passive response). (7) The baseline of the R1 interval was set to 0 to normalize MP, HR, SV, CO, and TPR.
[0285] In addition, as psychological indicators, VAS was measured using four subjective feeling quantities, namely "clearness", "pleasure and unpleasure", "vitality", and "liking", before and after the experiment. "Pleasure and unpleasure" and "clearness" were selected as the psychological evaluation items in this study as the basic components of Russell's emotion dualism. (8)In VAS, pairs of adjectives are placed at both ends of a line segment, and the subject ticks any position on the line segment, thereby measuring the subject's psychophysical quantity. For the sentences placed at both ends of the scale, the sense of wakefulness is set as "very sleepy" - "very awake", the sense of pleasure is set as "very unpleasant" - "very pleasant", the sense of vitality is set as "very tired" - "very energetic", and the sense of preference is set as "very dislike" - "very like".
[0286] Each VAS was prepared on a separate sheet of paper, and the subjects were instructed to write it down in person without recursive reference. Furthermore, after the experiment, the sense of immersion in the image was measured using a 5-point scale (1: completely unable to immerse - 5: quite immersed). In addition, a corresponding t-test was used in the test of the difference before and after watching the image for each psychological index, and the Wilcoxon signed order test was used for the overall change in the TEST interval in each physiological index. Excitement-sedation was normalized based on the maximum value of the addition.
[0287] Pattern recognition based on a hierarchical neural network was used to generate a preference and viewing pattern classification model and derive a discrimination rate, and to generate an excitement-sedation estimation model for each subject and derive an estimated value.
[0288] exist Fig.30 The structure of the hierarchical neural network is shown. The learning rule is set to the error back propagation algorithm, and the output function is set to the sigmoid function. The layers are divided into three layers: input layer, intermediate layer, and output layer. The cross-validation method is used to estimate the accuracy of the model. In addition, Fig.31 As shown in the figure, for the temporal changes of each index, it is assumed that there is a judgment time t d .t d is the 300s of video viewing. For each model, at t d In the time series of each feature quantity within the interval, we focus on each Δt i The slope of the network is set as the feature vector S at a series of slopes every 10 seconds. The number of elements of S is 30. In this regard, in the neural network of this article, the number of units in the input layer is set to 30, and the number of units in the middle layer is set to 16.
[0289] <2·4·1> Preference classification model
[0290] Using heart rate as feature quantity and Δt as iThe study was conducted for 10s, 20s, 30s, 40s, 50s, and 60s. The total learning data is 28 patterns of preferences (positive, negative) of 14 subjects, of which 27 patterns are used as learning data and the remaining 1 pattern is used as unknown data to evaluate the accuracy of the preference classification model. After learning the feature vector S when "positive" and "negative", the accuracy of the preference classification model is evaluated by inputting unknown data. The output layer is 2 units of "Positive" and "Negative".
[0291] <2·4·2> Viewing method classification model
[0292] As characteristic quantities, hemodynamic parameters, Δt i The study was conducted with the duration of 10s, 20s, 30s, 40s, 50s, and 60s. The total learning data is 42 patterns of various stress coping styles (active coping, passive coping, and stress-free coping) of 14 subjects, of which 41 patterns are used as learning data and the remaining 1 pattern is used as unknown data to evaluate the accuracy of the preference classification model. The output layer is composed of 3 units: active coping, passive coping, and stress-free coping.
[0293] <2·4·3> Excitation-sedation model
[0294] Hemodynamic parameters were used as feature quantities, and Δti was set to 30s. The total learning data consisted of 28 patterns of excitement-sedation values in each preference (positive, negative) of 14 subjects, of which 27 patterns were used as learning data and the remaining 1 pattern was used as unknown data to evaluate the accuracy of the preference classification model. The output layer was a unit of excitement-sedation values.
[0295] <2.5> Results and Research
[0296] In VAS, the average difference between the pre- and post-video viewing scores of all subjects in each emotional scale is calculated. Fig.32 (+: p<0.1, *: p<0.05, **: p<0.01). Fig.32 In, N = 14. Fig.32 All the subjects had a low preference for horror video stimulation. Therefore, in the 5-stage immersion evaluation after watching horror videos, 4 or 5 was classified as horror (concentration) (hereinafter abbreviated as "horror (C)"), and 1 or 2 was classified as horror (non-concentration) (hereinafter abbreviated as "horror (N)"). The subsequent results were classified as "positive", "negative", "horror (C)", and "horror (N)". In addition, the number of people who rated "horror (C)" was 10, and the number of people who rated "horror (N)" was 4.
[0297] The significance probability p of the Wilcoxon signed order test related to the shift of the baseline from the entire TEST interval of MP, HR, SV, CO, and TPR is shown in Fig.33 (+: p<0.1, *: p<0.05, **: p<0.01). P in the table indicates a positive response to each indicator, and N indicates a negative response. Fig.33 In this example, N=14.
[0298] according to Fig.33 , the viewing style of TV video viewing was classified. Both "Positive" and "Horror (C)" had a high sense of immersion, but "Positive" was a positive preference and "Horror (C)" was a negative preference.
[0299] However, both parties saw increases in MP, HR, SV, and CO, and decreases in TPR. This is a typical mode I response (active response) characterized by an increase in myocardial contractile activity. In other words, it is clear that during active response, people are immersed in the TV video content regardless of their preferences.
[0300] On the other hand, the immersion and preference of "Negativity" and "Fear (N)" are both low. However, their physiological responses are different. That is, although a decrease in TPR and HR is seen in "Negativity", no change in MP is seen. It is considered that it is not either an active response or a passive response, but a non-response. In contrast, although there is no significant change in HR in "Fear (N)", a significant increase in TPR and MP is characteristic. This is a typical mode II reaction (passive response) mainly caused by an increase in MP due to an increase in peripheral vasoconstriction.
[0301] Thus, it is clear that when stress coping is not shown, the preference is usually low and the TV image content is not immersed, but the horror image shows a passive response. In other words, it is clear that the viewing state cannot be determined by the preference for TV images alone, and the viewing state can be classified according to the stress coping method. Fig.33 In the data, HR increased significantly in the “positive” state and decreased significantly in the “negative” state.
[0302] Next, the results of the discrimination rate when the preference and viewing method are judged by the estimation model constructed using the neural network are shown in Fig.34 as well as Fig.35 In Fig.34 , Fig.35 In the i At 50s, the positive discrimination rate of preference is 83.3%, and the positive discrimination rate of viewing method is 75%, which is comparable to the other Δt i Compared with , it shows a higher positive discrimination rate.
[0303] In addition, Fig.36Measured excitement-sedation and predicted excitement-sedation when estimating excitement-sedation are shown in FIG.
[0304] Fig.36 The results for subject A, who had the smallest error between the measured excitement-sedation and the predicted excitement-sedation, are shown. Fig.36 In the "positivity" above, as the measured arousal-sedation increases, the predicted arousal-sedation also increases.
[0305] In addition, Fig.36 In the "negative" state under , as the measured excitement-sedation decreases, the predicted excitement-sedation also decreases. Next, the average error of each subject when the absolute value of the difference between the measured excitement-sedation and the predicted excitement-sedation is taken as the error is shown in Fig.37 . Fig.37 The average error of "positive" is 0.10 to 0.37, the average error of "negative" is 0.11 to 0.29, and the average of the average error among all subjects is 0.17. That is, the predicted excitement-sedation is estimated with an average error of about 17%. In addition, a graph showing the relationship between a series of measured excitement-sedation and predicted excitement-sedation obtained every 10 seconds for all subjects is shown in Fig.38 In. Fig.38 The correlation coefficient is 0.89, which shows a high correlation. Fig.38 It can be seen that the distribution of the measured excitement-sedation and the predicted excitement-sedation is approximately uniform regardless of the magnitude of the value.
[0306] 3. Summary
[0307] This study aimed to estimate viewing patterns, preferences, and excitement-calmness of television video content using a neural network. Feature vectors were extracted from cardiovascular system indicators, and estimation models for preferences, viewing patterns, and excitement-calmness when watching television video content were generated and evaluated.
[0308] As a result, "Positiveness" with positive preference and high immersion and "Horror (C)" with low preference and high immersion were classified as active coping, "Horror (N)" with negative preference and high immersion was classified as passive coping, and "Negativeness" with negative preference and low immersion was classified as stress-free coping. Furthermore, the heart rate was significantly different between "Positiveness" and "Negativeness", and the result was obtained that followed the general characteristics of heart rate response. In addition, the maximum positive discrimination rate when using the inference model to discriminate preferences and viewing methods was 83.3% for preferences and 75% for viewing methods.
[0309] The results of estimating excitement-sedation for each subject showed that the average error between the measured excitement-sedation and the predicted excitement-sedation was 10-37% in the "positive" and 11-29% in the "negative", with an average of 17% among all subjects. Furthermore, it can be seen that the correlation coefficient between the measured excitement-sedation and the predicted excitement-sedation is 0.89, which is a strong positive correlation. As mentioned above, this suggests the possibility of estimating preferences, viewing methods, and excitement-sedation when watching TV video content through hemodynamic parameters and heart rate. In the future, it is planned to further improve the accuracy through comparison and research with other physiological indicators and other discriminators.
[0310] <References>
[0311] (1) Nishigaki Michiru: “Will Television Resurrect Through the Internet?”, Monthly Public Broadcasting (2001)
[0312] (2) Fujiwara Kotatsu and Saito Yumiko, "Public Opinion Survey Report on Evaluation and Expectations of Television - From the Survey Report on "The Role of Television", NHK Broadcasting Research and Survey, June issue, pp. 2-15 (1989)
[0313] (3) Ohnoki Hiroaki: "Chapter 9: Female college students' attitudes towards television viewing and factors 2 and 3: A preliminary survey on the personality characteristics of viewers, their interest in program content, and their purpose of viewing television (A study of program analysis and learning behavior from viewing: with the goal of developing a classification system for educational programs)", Research Report, Vol. 18, pp. 153-172 (1990)
[0314] (4) Y.Takahashi and S.John: "Recommendation Models of TelevisionProgram Genre, Based on Survey and Analysis of Beha viour in WatchingTelevision: Toward Human Content Interface Design (2)", Bulletin of Japanese Society for Science of Design, Vol.46, No.133, pp.71-80 (1999) (in Japanese) Takahashi Yasushi, Shackleton John: "Recommendation Models of TelevisionProgram Genre, Based on Survey and Analysis of Beha viour in WatchingTelevision: Toward Human Content Interface Design (2)", Design Research, Vol.46, No.133, pp.71-80 (1999)
[0315] (5) Yumiko Tomoso and Yumiko Hara, “Television as a ‘relaxation time device’: the relationship between viewing attitudes and the overall variety orientation of programs”, Broadcasting Research and Survey, Vol. 51, pp. 2-17 (2001)
[0316] (6) S. Nomura, Y. Kurosawa, N. Ogawa, C. M. Althaff Irfan, K. Yajima, S. Handri, T. Yamagishi, K. T. Nakahira, and Y. Fukumura: “Psysiological Evaluation of student in E-learning Sessions by Hemodynamic Response”, IEEJ Trans. EIS, Vol. 131, No. 1, pp. 146-151 (2011)
[0317] (7) Sawada Yukihiro, "Hemodynamic Response: New Physiological Psychology Volume I (Edited by Fujisawa Kiyoshi, Kakigi Shoji, and Yamazaki Katsuo)", Kitaoji Shobo, Chapter 10, pp. 187 (1998)
[0318] (8) A. Michimori: "Relationship between the alpha attenuation test, subjective sleepiness and performance test", 10th Symposium on Human Interface, Vol. 10, No. 1413, pp. 233-236 (1992)
[0319] (9) M. Terasaki, Y. Kishimoto, and A. Koga: “Construction of a multiple moodscale”, The Japanese journal of psycholog y, Vol. 62, No. 6, pp. 350-356 (1992)
[0320] (10) J.A. Russell, “A circumplex model of affect”, J. Personality and Social Psychology, Vol. 39, pp. 1161-1178 (1980)
[0321] Industrial Applicability
[0322] The stress coping style determination system of the present invention can analyze the nature of the stress of the subject in a non-contact state. Therefore, it can be used in a wide range of technical fields, such as a mechanism for grasping the stress state of workers working in factories, a mechanism for grasping the stress state of drivers driving cars, and a means for grasping the stress state of students attending classes.
[0323] Description of Reference Numerals
[0324] 100 Stress Coping Method Determination System
[0325] 110 biological information acquisition device (biological information acquisition unit)
[0326] 120 determination device (determination unit)
[0327] 121 Determination feature storage unit
[0328] 122 Specific site reaction detection unit
[0329] 123 Response mode determination unit
[0330] 130 Learning Device (Machine Learning Department)
[0331] 131 Learning data storage unit
[0332] 132 Feature extraction unit
[0333] 133 Feature Learning Unit
[0334] 134 Learning Complete Model
[0335] P Subject
[0336] IF facial image
[0337] S1 Determination feature quantity storage process (determination feature quantity storage step)
[0338] S2 Specific site reaction detection processing (specific site reaction detection step)
[0339] S3 Response mode determination processing (response mode determination step)
[0340] S11 Learning data storage processing (learning data storage step)
[0341] S12 Feature extraction processing (feature extraction step)
[0342] S13 Feature quantity learning processing (feature quantity learning step)
[0343] S21 Clustering Processing (Clustering Step)
[0344] S22 Image extraction processing (image extraction step)
[0345] S23 Edge extraction processing (edge extraction step)
[0346] S24 Fractal analysis processing (fractal analysis step).
Claims
1. A stress coping method determination system, characterized in that: have: A biological information acquisition unit that acquires biological information of a subject in a non-contact state; a determination unit that determines the subject's stress coping style based on the biological information and a predetermined response pattern, The response mode is determined according to the hemodynamic parameters, The response modes include three modes: "active response", "passive response" and "no response". The determination unit includes a determination feature quantity storage unit for storing a spatial feature quantity corresponding to "active response", a spatial feature quantity corresponding to "passive response" and a spatial feature quantity corresponding to "no response", Based on the biological information and each spatial feature quantity stored in the feature quantity storage unit for determination, it is determined whether the stress response method is a mode of "active response", "passive response" and "no response", The feature quantity stored in the determination feature quantity storage unit is a feature quantity extracted by the machine learning unit. The machine learning department has: A learning data storage unit storing a plurality of learning face images labeled with "active response", "passive response" and "no response" respectively; a feature extraction unit that extracts a spatial feature of the facial image from the learning facial image using a learned model; The feature learning unit changes the network parameters of the learned model based on the relationship between the extraction result obtained by the feature extraction unit and the label annotated on the learning facial image as its extraction object so as to increase the extraction accuracy of the spatial feature obtained by the feature extraction unit.
2. The stress coping method determination system according to claim 1, characterized in that: The hemodynamic parameters include multiple parameters of mean blood pressure, heart rate, cardiac output, stroke volume and total peripheral vascular resistance.
3. The stress coping method determination system according to claim 1 or 2, characterized in that: The biometric information is a facial image.
4. The stress coping method determination system according to claim 3, characterized in that: The facial image is a facial thermal image or a facial visible image.
5. The stress coping method determination system according to claim 3, characterized in that: The determination unit determines the subject's stress coping style by observing a stress response of a specific part of the face included in the facial image.
6. The stress coping method determination system according to claim 1, characterized in that: The spatial feature quantity is a fractal dimension calculated based on the facial image of the subject.
