A method and device for detecting a mental state

Abstract

The present application discloses a method and device for detecting mental states. The method includes the following steps: collecting a video stream of a human head, and obtaining multiple head images from the video stream; processing the multiple head images to locate the contours of multiple organs related to the head, and obtaining the vibration frequency of each pixel point within the contours; calculating the temporal change data of the resonance frequency related to the head within a preset time according to the vibration frequency of each pixel point; and outputting multiple mental states and the levels of each mental state according to the temporal change data. According to the present application, the temporal change data of the resonance frequency within a preset time is calculated based on the vibration frequency of each pixel point within the contours of multiple organs related to the head, so that the mental state can be objectively and accurately detected, and thus has the advantages of wide application range, convenient use, and high accuracy.

Description

Technical Field

[0001] This application belongs to the technical field of artificial intelligence, and particularly relates to a method and device for detecting mental states. Background Art

[0002] Psychology refers to the process and result of the internal symbolic activity sorting of a person. Specifically, it refers to the subjective reflection of a living being on the objective material world. The manifestation form of psychology is called psychological phenomenon, including psychological processes and psychological characteristics. The psychological activities of people all have a process of occurrence, development, and disappearance. When humans are in different mental states (such as fatigue, anxiety, psychological stress, etc.), they will have different external manifestations. For example, when the psychological stress is moderate, the work efficiency is relatively high; when the psychological stress is too low or too high, the work efficiency will decrease.

[0003] Currently, the way to determine the mental state is to first perform single data detection, and then use the detected data to calculate the mental state, but this way is often not accurate.

[0004] Application Content

[0005] The purpose of the embodiments of this application is to provide a method and device for detecting mental states to solve the defect that the prior art cannot accurately detect mental states.

[0006] To solve the above technical problems, this application is implemented as follows:

[0007] In a first aspect, a method for detecting mental states is provided, including the following steps:

[0008] Collect a video stream of a human head, and obtain multiple head images from the video stream;

[0009] Process the multiple head images, locate the contours of multiple organs related to the head, and obtain the vibration frequency of each pixel point within the contours;

[0010] According to the vibration frequency of each pixel point, calculate the time-series change data of the resonance frequency related to the head within a preset time;

[0011] According to the time-series change data, output multiple mental states and the levels of each mental state.

[0012] In a second aspect, a device for detecting mental states is provided, including:

[0013] A collection module, configured to collect a video stream of a human head, and obtain multiple head images from the video stream;

[0014] A processing module for processing the multi-frame head images, locating the contours of multiple head-related organs, and obtaining the vibration frequency of each pixel point within the contours;

[0015] A calculation module for calculating the temporal variation data of the resonance frequency related to the head within a preset time according to the vibration frequency of each pixel point;

[0016] An output module for outputting multiple mental states and the levels of each mental state according to the temporal variation data.

[0017] In the embodiment of the present application, the temporal variation data of the resonance frequency within a preset time is calculated according to the vibration frequency of each pixel point within the contours of multiple head-related organs, so that the mental state can be objectively and accurately detected, and thus it has the advantages of wide application range, convenient use, and high accuracy. Description of the Drawings

[0018] Figure 1 is a flowchart of a mental state detection method provided by an embodiment of the present application;

[0019] Figure 2 is a specific implementation diagram of the mental state detection method provided by an embodiment of the present application;

[0020] Figure 3 is a specific implementation diagram of a non-contact mental state detection dedicated device provided by an embodiment of the present application;

[0021] Figure 4 is a schematic structural diagram of a mental state detection device provided by an embodiment of the present application. Detailed Embodiments

[0022] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0023] Among the currently widely used video-based non-contact psychological state detection technologies, micro-expression and vibration imaging are the main ones. Micro-expression detection mainly focuses on emotional expressions such as anger and fear. For vibration imaging, the mathematical formula derivation has not introduced deep learning to continuously optimize the detection accuracy. Any substance has wave-particle duality, that is, any substance in the universe, from living organisms to non-living things, is vibrating at all times. Moreover, each substance has its inherent natural vibration frequency, or resonance frequency. An organism can have multiple resonance frequencies. An organism itself is a delicate and weak natural electromagnetic wave vibration system. Each organ tissue, such as the brain and heart, has its specific vibration frequency. By detecting the weak resonance frequencies of multiple organs in the head and performing deep learning, a multi-resonance frequency psychological analysis model can be established to more accurately analyze the psychological state. However, the method of detecting the psychological state by detecting the resonance frequency of organs as a psychological signal was previously achieved using expensive and specialized medical electromyography equipment.

[0024] The embodiments of this application provide a non-contact psychological state detection method and device based on the multi-resonance frequencies of the head, which relates to the technical field of artificial intelligence psychological state detection. It uses a video-based method to detect the resonance frequencies of head organs and realizes the detection of psychological state in a non-contact manner. The detection principle is: each substance has its inherent natural vibration frequency, or resonance frequency. An organism itself is a delicate and weak natural electromagnetic wave vibration system. The detection process is to use a camera to capture the video of the human head, and then process the video signal frame by frame. In the frame image, the contours of the pupil, eyes, nose, and head are located. The pupil resonance frequency, eyeball resonance frequency, respiratory resonance frequency, and head resonance frequency caused by vestibular feedback are calculated through the displacement of the pixel points within these part contours in consecutive frame images. By associating and analyzing the time-series data of multiple resonance frequencies over a period of time using a deep learning engine, multiple human psychological states, such as fatigue, attention, stress, anxiety, depression, interpersonal relationships, and tension, can be obtained.

[0025] In addition, the psychological state detection device consists of an information collection unit, a resonance frequency calculation unit, and a psychological state detection unit. The information collection unit captures the video of the human head through a camera at 30 frames per second and obtains 30 head images per second with a resolution of no less than 400*400 pixels. The resonance frequency calculation unit identifies the defined contours of the pupil, eyes, nose, and head, and calculates the resonance frequency based on the displacement of the pixel points within the contours in consecutive frames. The psychological state detection unit realizes the detection of the psychological state through the deep learning results.

[0026] Specifically, the information acquisition unit includes a video acquisition module and a video processing module. The video acquisition module obtains the video stream in the camera, and the video processing module obtains the frame images. The resonance frequency calculation unit includes a contour location module for the pupil, eyes, nose, and head, a vibration frequency calculation module for the pixel points within the contour, a multi-resonance frequency calculation module, and a timing change data acquisition module. The mental state detection unit includes a mental state detection module and a deep learning engine module. The deep learning engine module is trained on a large amount of data under artificial adjustments such as mental state feedback and supervised learning to obtain a continuously optimized resonance frequency psychological analysis model. The mental state detection module uses the optimized resonance frequency psychological analysis model to accurately detect the mental state.

[0027] The following will, with reference to the accompanying drawings, explain in detail a mental state detection method provided by an embodiment of the present application through specific embodiments and their application scenarios.

[0028] As Figure 1 shown, it is a flowchart of a mental state detection method provided by an embodiment of the present application. The method includes the following steps:

[0029] Step 101: Collect the video stream of the human head and obtain multiple head images from the video stream.

[0030] Step 102: Process the multiple head images, locate the contours of multiple organs related to the head, and obtain the vibration frequency of each pixel point within the contour.

[0031] Specifically, based on human biological characteristics, the contours of the pupil, eyes, nose, and head, as well as each pixel point within the contour, can be located on each frame of the head image; according to the displacement of each pixel point in consecutive head images, the vibration frequency of each pixel point can be calculated.

[0032] Step 103: Calculate the timing change data of the resonance frequency related to the head within a preset time according to the vibration frequency of each pixel point.

[0033] Specifically, the timing change data of the pupil resonance frequency, eyeball resonance frequency, breathing resonance frequency, and head resonance frequency within a preset time can be obtained by respectively performing weighted average calculations on the vibration frequencies of each pixel point.

[0034] Step 104: Output multiple mental states and the levels of each mental state according to the timing change data.

[0035] Specifically, the timing change data can be input into a preset multi-resonance frequency psychological analysis model to output multiple mental states and the levels of each mental state.

[0036] In this embodiment, deep learning engine association and analysis can also be performed to optimize the multi-resonance frequency psychological analysis model.

[0037] In the embodiment of the present application, the time-series change data of the resonance frequency within a preset time is calculated based on the vibration frequency of each pixel point within the contours of multiple organs related to the head, and thus the psychological state can be objectively and accurately detected. Therefore, it has the advantages of wide application range, convenient use, and high accuracy.

[0038] In the embodiment of the present application, as Figure 2 shown, the specific implementation process includes the following steps:

[0039] (1) First, the camera captures the video stream of the human face. The camera needs to be aligned with the front of the human face, and the size of the human face in the camera should not be less than 400*400 pixels. The frame rate of the camera is required to reach 30fps, and 30 high-quality human face images can be obtained from the video stream per second.

[0040] (2) Process each frame of the image, locate the contours of the pupils, eyes, nose, and head on the frame image according to the human biological characteristics, and locate the pixel points within the contours row by row.

[0041] (3) According to the displacements of the pixel points within the contours of the pupils, eyes, nose, and head in consecutive frame images, calculate the vibration frequency of each pixel point. For a certain pixel point, there may be no displacement found in adjacent frame images, and it is necessary to accumulate multiple frames before calculating to find the change in displacement. Since the time interval between frames is fixed, the time interval of the displacement can be calculated, and thus the vibration frequency can be calculated.

[0042] (4) Calculate the as-accurate-as-possible pupil resonance frequency, eyeball resonance frequency, respiratory resonance frequency, head resonance frequency caused by vestibular feedback, etc. according to the weighted average of the vibration frequencies of the relevant pixel points.

[0043] (5) Like the heart rate, the resonance frequency is also a physiological parameter of the human body. Just as the heart rate variability can be represented by an electrocardiogram, the change of the resonance frequency over time also forms a resonance frequency change curve graph. Continuously collect and calculate the time-series change data of the above multi-resonance frequencies of the head for 60 seconds.

[0044] (6) Input the time-series change data into the preset multi-resonance frequency psychological analysis model to output the levels of multiple psychological states. The psychological states include fatigue, attention, stress, anxiety, depression, interpersonal relationship, tension, etc. Among them, fatigue refers to mental fatigue, and attention refers to attention disorder. Each psychological state is divided into four levels: normal, level 1, level 2, and level 3. The larger the level, the greater the possibility that there is a problem with this psychological state.

[0045] (7) Perform deep learning engine association and analysis to optimize the resonance frequency psychological analysis model. The optimized resonance frequency psychological analysis model replaces the originally preset resonance frequency psychological analysis model, making the accuracy of psychological state detection higher and higher.

[0046] Through the above technical solution, the system can hierarchically give the following psychological abnormalities: fatigue, attention, stress, anxiety, depression, interpersonal relationship, tension, etc., which are the most concerned psychological indicators among groups such as soldiers and students. The system uses video streams to detect psychological states, featuring non-contact, so as to detect the objective state of the subjects.

[0047] In the embodiments of the present application, the computer device can also be set as a dedicated device for non-contact psychological state detection, such as Figure 3 As shown, the device operates independently and is built-in with a camera with a frame rate of 30fps and a resolution of 720P. After the device is powered on, the psychological state detection application is automatically started. The psychological state detection application processes the head video stream collected by the built-in camera to complete psychological state detection. When the psychological state detection application is exited, the device automatically shuts down. The device has two displays, a main screen and a secondary screen, which display different interfaces respectively. The subjects and operators correspond to different interfaces, and the detected psychological states are unknown to the subjects.

[0048] In addition, to maintain the stability of detection, the system is a stand-alone version for offline use, with an overall design, and the camera is built into the system. The system adopts a double-sided screen design, and different contents are displayed on the front and back screens. Moreover, the screen faced by the subjects shall not display detection data and results to avoid negative impacts on the subjects. Compared with psychological scales, the system can perform multiple detections on the same subject. Compared with wearable devices such as electroencephalogram and electrocardiogram devices, the system is more convenient and flexible to use.

[0049] The embodiments of the present application adopt a non-contact and non-intrusive psychological state detection method, and through the method of calculating the inherent vibration frequency of human organs, which cannot be faked, realize non-contact psychological state detection based on video, making the non-contact psychological state detection based on video more objective and accurate, and thus having the advantages of wide application range, convenient use, and high precision.

[0050] Such as Figure 4 As shown, it is a psychological state detection device in the embodiments of the present application, including:

[0051] An acquisition module 410, configured to acquire a video stream of a human head and obtain multiple head images from the video stream.

[0052] A processing module 420, configured to process the multiple head images, locate the contours of multiple organs related to the head, and obtain the vibration frequency of each pixel point within the contours.

[0053] Specifically, the processing module 420 is specifically configured to locate the contours of the pupils, eyes, nose, and head, as well as each pixel point within the contours, on each frame of the head image based on human biological characteristics; and calculate the vibration frequency of each pixel point according to the displacement of each pixel point in consecutive head images.

[0054] The calculation module 430 is configured to calculate the temporal change data of the resonance frequency related to the head within a preset time according to the vibration frequency of each pixel point.

[0055] Specifically, the calculation module 430 is specifically configured to perform weighted average calculations on the vibration frequencies of each pixel point respectively to obtain the temporal change data of the pupil resonance frequency, eyeball resonance frequency, respiratory resonance frequency, and head resonance frequency within a preset time.

[0056] The output module 440 is configured to output multiple psychological states and the levels of each psychological state according to the temporal change data.

[0057] Specifically, the output module 440 is specifically configured to input the temporal change data into a preset multi-resonance frequency psychological analysis model and output multiple psychological states and the levels of each psychological state.

[0058] In addition, the above psychological state detection device further includes:

[0059] An optimization module for performing deep learning engine association and analysis to optimize the multi-resonance frequency psychological analysis model.

[0060] The embodiment of the present application calculates the temporal change data of the resonance frequency within a preset time according to the vibration frequency of each pixel point within the contours of multiple organs related to the head, and thus can objectively and accurately detect psychological states, and therefore has the advantages of wide application range, convenient use, and high accuracy.

[0061] It should be noted that in this article, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "including one..." does not exclude the existence of additional identical elements in the process, method, article or device including that element.

[0062] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described example methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in various embodiments of the present application.

[0063] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific implementation manners. The above specific implementation manners are merely illustrative and not restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and all of them belong to the protection scope of the present application.

Claims

1. A method for detecting a mental state, characterized in that, Including the following steps: Collect a video stream of a human head and obtain multiple head images from the video stream; Process the multiple head images to locate the contours of multiple organs related to the head and obtain the vibration frequency of each pixel point within the contours; Calculate the time-series change data of the resonance frequency related to the head within a preset time according to the vibration frequency of each pixel point; Output multiple psychological states and the levels of each psychological state according to the time-series change data; The processing of the multiple head images to locate the contours of multiple organs related to the head and obtain the vibration frequency of each pixel point within the contours specifically includes: Based on human biological characteristics, locate the contours of the pupils, eyes, nose, and head, as well as each pixel point within the contours, on each head image; Calculate the vibration frequency of each pixel point according to the displacement of each pixel point in consecutive head images; The calculation of the time-series change data of the resonance frequency related to the head within a preset time according to the vibration frequency of each pixel point specifically includes: Perform weighted average calculations on the vibration frequencies of each pixel point respectively to obtain the time-series change data of the pupil resonance frequency, eyeball resonance frequency, respiratory resonance frequency, and head resonance frequency within a preset time.

2. The method according to claim 1, wherein The output of multiple psychological states and the levels of each psychological state according to the time-series change data specifically includes: Input the time-series change data into a preset multi-resonance frequency psychological analysis model to output multiple psychological states and the levels of each psychological state.

3. The method according to claim 2, wherein It further includes: Perform deep learning engine association and analysis to optimize the multi-resonance frequency psychological analysis model.

4. A mental state detection device, characterized in that, It includes: A collection module for collecting a video stream of a human head and obtaining multiple head images from the video stream; A processing module for processing the multiple head images to locate the contours of multiple organs related to the head and obtain the vibration frequency of each pixel point within the contours; A calculation module for calculating the time-series change data of the resonance frequency related to the head within a preset time according to the vibration frequency of each pixel point; An output module for outputting multiple psychological states and the levels of each psychological state according to the time-series change data; The processing module is specifically configured to, based on human biological characteristics, locate the contours of the pupils, eyes, nose, and head, as well as each pixel point within the contours, on each head image; calculate the vibration frequency of each pixel point according to the displacement of each pixel point in consecutive head images; The calculation module is specifically configured to perform weighted average calculations on the vibration frequencies of each pixel point respectively to obtain the time-series change data of the pupil resonance frequency, eyeball resonance frequency, respiratory resonance frequency, and head resonance frequency within a preset time.

5. The device according to claim 4, wherein The output module is specifically configured to input the time-series change data into a preset multi-resonance frequency psychological analysis model to output multiple psychological states and the levels of each psychological state.

6. The device according to claim 5, characterized in that It further includes: An optimization module, which is used to perform deep learning engine association and analysis to optimize the multi-resonant frequency psychological analysis model.

Images