An emotion normalization monitoring system and method based on the domestic information technology innovation environment

Abstract

The present application discloses an emotion normalization monitoring system and method based on an information and communication technology (ICT) innovation environment. The system includes: a camera for collecting a video stream of a human face; an emotion computing gateway for generating a plurality of video files based on the video stream, obtaining multiple frames of human face images from each video file, processing the multiple frames of human face images, obtaining first time-series change data of the amplitude and vibration frequency of each pixel point within the contour of the facial muscle group within a preset time, and the vibration frequency of each pixel point within the contours of multiple organs related to the head; calculating second time-series change data of the resonance frequency related to the head within a preset time, and outputting emotion state data corresponding to the human face; and an emotion early warning server for analyzing based on the monitored emotion historical data and performing hierarchical early warning on personnel with a relatively high risk of psychological problems. Both the emotion computing gateway and the emotion early warning server support the ICT innovation environment. The present application can accurately, objectively, and safely achieve normalized emotion monitoring.

Description

Technical Field

[0001] This application belongs to the field of mental health monitoring, and specifically relates to an emotion normalization monitoring system and method based on an information technology application innovation environment. Background Art

[0002] In order to truly reflect the daily emotions of the target person, the best method is to conduct normalized monitoring of emotions. The currently adopted technologies and methods mainly include using wearable devices, such as wearing an electronic bracelet; and using micro-expression recognition technology for non-contact emotion normalization monitoring. The emotion indicators monitored by the electronic bracelet are relatively few, mainly several indicators such as stress based on heart rate variability detection, and it requires the wearer to actively cooperate. If the wearer does not actively cooperate, the monitoring effect will be relatively poor, which will have a greater impact on normalized monitoring and make it difficult for normalized monitoring to achieve the expected effect.

[0003] Application Content

[0004] The purpose of the embodiments of this application is to provide an emotion normalization monitoring system and method based on an information technology application innovation environment to solve the defect of poor monitoring effect in the prior art.

[0005] To solve the above technical problems, this application is implemented as follows:

[0006] In a first aspect, an emotion normalization monitoring system based on an information technology application innovation environment is provided, including:

[0007] A camera for collecting a video stream of a human face;

[0008] An emotion computing gateway for generating multiple video files based on the video stream, obtaining multiple frames of human face images from each video file, processing the multiple frames of human face images, locating multiple muscle groups on the face and the contours of multiple organs related to the head, and obtaining the first time-series change data of the amplitude and vibration frequency of each pixel point within the contour of the facial muscle group and the vibration frequency of each pixel point within the contour of multiple organs related to the head within a preset time; calculating the second time-series change data of the resonance frequency related to the head within a preset time according to the vibration frequency of each pixel point within the contour, and outputting the emotion state data corresponding to the human face according to the first time-series change data and the second time-series change data;

[0009] An emotion early warning server for analyzing based on the monitored emotion historical data and performing hierarchical early warning on personnel with a high risk of psychological problems;

[0010] Wherein, both the emotion computing gateway and the emotion early warning server support the information technology application innovation environment.

[0011] Second aspect, a method for normalizing emotion monitoring based on the Xinchuang environment is provided, including the following steps:

[0012] Collect a video stream of a human face;

[0013] Generate multiple video files according to the video stream, and obtain multiple frames of human face images from each video file;

[0014] Process the multiple frames of human face images, locate the contours of multiple muscle groups on the face and multiple organs related to the head, and obtain the first time-series change data of the amplitude and vibration frequency of each pixel point within the contour of the facial muscle group within a preset time, as well as the vibration frequency of each pixel point within the contour of multiple organs related to the head;

[0015] Calculate the second time-series change data of the resonance frequency related to the head within a preset time according to the vibration frequency of each pixel point within the contour, and output the emotion state data corresponding to the human face according to the first time-series change data and the second time-series change data;

[0016] Analyze based on the monitored emotion historical data, and conduct hierarchical early warning for personnel with a high risk of psychological problems.

[0017] By processing multiple frames of human face images in the video file, the embodiment of the present application outputs the emotion state data corresponding to the human face, which can accurately, objectively, and safely achieve normal emotion monitoring and truly reflect the daily emotions of the target personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic structural diagram of a system for normalizing emotion monitoring based on the Xinchuang environment provided by the embodiment of the present application;

[0019] Figure 2 is a schematic diagram of a specific implementation manner of a system for normalizing emotion monitoring based on the Xinchuang environment provided by the embodiment of the present application;

[0020] Figure 3 is a schematic hardware architecture diagram of a system for normalizing emotion monitoring based on the Xinchuang environment provided by the embodiment of the present application;

[0021] Figure 4 is a flowchart of a method for normalizing emotion monitoring based on the Xinchuang environment provided by the embodiment of the present application;

[0022] Figure 5 is a schematic diagram of a specific implementation manner of a method for normalizing emotion monitoring based on the Xinchuang environment provided by the embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0024] Monitoring the emotions of personnel involves information security, and the software and hardware of the emotion monitoring system should be autonomous and controllable. At present, there are very few emotion monitoring systems developed based on the domestic information technology innovation environment. Based on this, the embodiments of the present application provide an emotion normalization monitoring system based on the information technology innovation environment, as Figure 1 shown, including:

[0025] A camera 110 for collecting a video stream of a human face.

[0026] An emotion computing gateway 120 for generating a plurality of video files according to the video stream, obtaining multiple frames of human face images from each of the video files, processing the multiple frames of human face images, locating multiple muscle groups on the face and the contours of multiple organs related to the head, and obtaining the first time-series change data of the amplitude and vibration frequency of each pixel point within the contour of the facial muscle group and the vibration frequency of each pixel point within the contour of the multiple organs related to the head within a preset time; calculating the second time-series change data of the resonance frequency related to the head within a preset time according to the vibration frequency of each pixel point within the contour, and outputting the emotion state data corresponding to the human face according to the first time-series change data and the second time-series change data.

[0027] Among them, the emotion computing gateway 120 includes:

[0028] An emotion computing engine for locating each pixel point within the contour of the facial muscle group on each frame of human face image based on the muscle group contour definition model; calculating the amplitude and frequency of each pixel point according to the displacement of each pixel point in consecutive human face images, and obtaining the first time-series change data of the amplitude and the frequency within a preset time;

[0029] And, based on human biological characteristics, locating the contours of the pupils, eyes, nose, and head on each frame of human face image, and each pixel point within the contour; calculating the vibration frequency of each pixel point according to the displacement of each pixel point in consecutive human face images.

[0030] Specifically, the emotion computing engine is specifically used to perform weighted average calculations on the vibration frequencies of each pixel point respectively to obtain the second time-series change data of the pupil resonance frequency, eyeball resonance frequency, respiratory resonance frequency, and head resonance frequency within a preset time.

[0031] An emotion warning server 130 for analyzing based on monitored historical emotion data and performing hierarchical warnings on personnel with a high risk of psychological problems.

[0032] Specifically, the emotion warning server 130 is specifically configured to generate historical emotion data and historical emotion trends of the target personnel corresponding to the face according to the emotion state data corresponding to the face; and perform hierarchical warnings on the target personnel when the historical emotion trend triggers an emotion warning model.

[0033] Among them, both the emotion computing gateway 120 and the emotion warning server 130 support the Xinchuang environment and support running in the environment of domestic CPUs and domestic operating systems.

[0034] By processing multiple frames of face images in a video file, the embodiment of the present application outputs emotion state data corresponding to the face, which can accurately, objectively and safely achieve normalized emotion monitoring and truly reflect the daily emotions of the target personnel.

[0035] In a specific implementation manner of the embodiment of the present application, the normalized emotion monitoring system based on the Xinchuang environment includes: a camera at 30 frames per second, an emotion computing gateway, and an emotion warning server. As Figure 2 shown, both the emotion computing gateway and the emotion warning server support domestic CPUs and domestic operating systems. The camera is responsible for collecting the video of the human head and accessing it to the emotion computing gateway through the USB port to realize the transmission of the video signal to the emotion computing gateway. The emotion computing gateway can process the video signals input by four cameras at the same time. First, it transcribes the video signals into video files, and each video file only stores the video of the human head that meets the image quality requirements, and the videos without faces or with blurred faces are not transcribed; then it uses the self-developed emotion computing technology to perform distributed and fast processing on multiple video files to analyze the emotion status of the personnel in the video files, including the number of times, intensity and duration of various emotions, etc. The emotion computing gateway transmits the emotion state data to the emotion warning server through the network. The emotion warning server analyzes based on the monitored historical emotion data, makes a comprehensive judgment on the emotions of the personnel monitored by multiple cameras, and performs hierarchical warnings on the personnel with a high risk of psychological problems.

[0036] Specifically, the emotion computing gateway is composed of a video file generation program, an emotion computing engine and a concurrent computing scheduling program, and is used to realize video file generation and concurrent emotion computing. The emotion warning server is composed of an emotion warning generation program, an emotion warning application program and an external interface API, and is used to realize the generation and application of emotion warning messages. Among them, multiple emotion computing gateways can be connected to the same emotion warning server.

[0037] Among them, both the emotion computing gateway and the emotion early warning server support the Xinchuang environment and can run in software and hardware environments such as domestic CPUs and domestic operating systems.

[0038] The embodiments of this application can conduct normalized monitoring of people's daily emotions. Therefore, it can detect early those who are prone to psychological problems due to the accumulation of negative emotions, and can be applied to the non-contact and non-intrusive screening of people with psychological problems among groups such as special post personnel, window service personnel, students, and soldiers.

[0039] In this embodiment, the hardware of the system includes a small computer used as an emotion computing gateway and a general server used as an emotion early warning server. As Figure 3 shown, one emotion computing gateway can access 4 USB cameras and perform concurrent emotion computing processing on 4 video streams at the same time. The emotion computing gateway and the emotion early warning server are connected through a wired network, and there can be multiple emotion computing gateways in the system. Both the emotion computing gateway and the emotion early warning server run on domestic CPUs and domestic operating systems. At present, the adaptation is for Zhaoxin CPUs and Zhongke Fangde operating systems.

[0040] This system uses a USB camera to conduct one-on-one close-range monitoring of human faces, which can ensure the quality of video images. This system uses a non-real-time computing method to achieve normalized emotion monitoring. Through special video file processing and emotion computing engine scheduling, the maximum utilization of computing resources is achieved. Emotion computing can be completed with a device costing a thousand yuan, enabling the system to be deployed on a large scale, so as to achieve normalized emotion monitoring of all personnel within a team. By using a non-contact and non-intrusive method to conduct long-term emotion monitoring of personnel, it can truly reflect the daily emotions of personnel; the entire system runs based on the Xinchuang environment, supports domestic CPUs and domestic operating systems; using a USB camera for single-face monitoring can fully ensure the quality of face images, making emotion monitoring relatively accurate.

[0041] Next, in combination with the accompanying drawings, through specific embodiments and their application scenarios, a method for normalized emotion monitoring based on the Xinchuang environment provided by the embodiments of this application will be described in detail.

[0042] As Figure 4 shown, it is a flowchart of a method for normalized emotion monitoring based on the Xinchuang environment provided by the embodiments of this application. The method includes the following steps:

[0043] Step 401, collect a video stream of a human face.

[0044] Step 402, generate multiple video files according to the video stream, and obtain multiple frames of face images from each video file.

[0045] Step 403: Process the multiple frames of face images, locate multiple muscle groups on the face and the contours of multiple organs related to the head, and obtain the first time-series change data of the amplitude and vibration frequency of each pixel point within the contour of the facial muscle group within a preset time, as well as the vibration frequency of each pixel point within the contours of the multiple organs related to the head.

[0046] Specifically, a model can be defined based on the muscle group contour to locate each pixel point within the contour of the facial muscle group on each frame of the face image; based on the displacement of each pixel point in consecutive face images, calculate the amplitude and frequency of each pixel point, and obtain the first time-series change data of the amplitude and the frequency within a preset time; based on human biological characteristics, locate the contours of the pupils, eyes, nose, and head on each frame of the face image, as well as each pixel point within the contours; based on the displacement of each pixel point in consecutive face images, calculate the vibration frequency of each pixel point.

[0047] Step 404: Calculate the second time-series change data of the resonance frequency related to the head within a preset time according to the vibration frequency of each pixel point within the contour, and output the emotion state data corresponding to the face according to the first time-series change data and the second time-series change data.

[0048] Specifically, weighted average calculations can be performed on the vibration frequencies of each pixel point respectively to obtain the second time-series change data of the pupil resonance frequency, eyeball resonance frequency, respiratory resonance frequency, and head resonance frequency within a preset time.

[0049] Step 405: Analyze based on the monitored emotion history data and issue a graded warning for personnel with a relatively high risk of psychological problems.

[0050] In this embodiment, after outputting the emotion state data corresponding to the face, the multiple video files can be deleted, and the emotion history data and emotion history trend corresponding to the target person of the face can also be generated according to the emotion state data corresponding to the face; in the case where the emotion history trend triggers the emotion warning model, a graded warning is issued for the target person.

[0051] The embodiment of the present application processes multiple frames of face images in a video file and outputs the emotion state data corresponding to the face, which can accurately, objectively, and safely achieve normalized emotion monitoring and truly reflect the daily emotions of the target person.

[0052] In the embodiment of the present application, a specific implementation manner of the emotion normalization monitoring method based on the Xinchuang environment is as Figure 5 shown and includes the following steps:

[0053] (1) Continuously obtain video streams from a USB camera at 30 frames per second.

[0054] (2) Process the video stream frame by frame. When a face appears in the video, judge the pixel size occupied by the face, the angle of the face, etc. After meeting the acquisition requirements, transcribe it frame by frame into a video file. Stop transcription after the face disappears from the video. At this time, a video file of the face appearing and leaving the camera is generated. If a person appears in the camera multiple times during learning or working, a series of video files will be generated. When multiple cameras collect simultaneously, multiple video files will be generated concurrently.

[0055] (3) Process each video file. The video file is input into a self-developed emotion computing engine for emotion analysis and statistics. The self-developed emotion computing engine includes two algorithms, the emotion recognition algorithm based on facial muscle micro-tremors and the emotion recognition algorithm based on the multi-vibration frequencies of head organs. Since the video files are generated concurrently, in order to process them as soon as possible, the concurrent computing scheduler starts multiple emotion computing engines and automatically schedules the concurrent processing of video files. After a video file is processed, only one statistical information of the emotions of the person in the video is generated, and the video file is automatically deleted.

[0056] (4) By continuously monitoring the emotions of personnel, a large amount of emotion historical data of the personnel can be generated. For an individual, their emotion historical trend can be tracked, and for a group, the emotion differences between an individual and others in the group can be judged. When the individual emotion trend or the emotion difference from the group triggers the emotion warning model, a classification warning will be issued to this person.

[0057] (5) Since the warning messages are automatically generated during the daily and normalized monitoring of emotions, the warning messages need to be displayed for the management staff to view. By publishing the warning messages in the WEB server, the management staff can open a browser to view the warning messages and can track the normalized emotion monitoring data to understand why the warning occurred.

[0058] (6) The normalized emotion monitoring is often used to provide a basis for other management systems, such as whether a person is suitable for taking up a post, etc. Through the API, emotion warning messages and emotion monitoring data can be published for third-party applications to use.

[0059] For example, applying this system to subway train drivers during work, students in psychology or computer classes, soldiers on sentry duty, and staff at service windows can achieve the discovery of personnel with abnormal emotions without affecting learning and work, and can automatically and accurately warn personnel with higher psychological risks. Moreover, the cost of this system is relatively low and it is convenient for large-scale application.

[0060] The embodiments of this application realize the normal monitoring of emotions based on videos, featuring non-contact, non-intrusive, objective, and accurate characteristics. It can truly reflect the daily emotions of people, and can identify both facial emotions such as anger, sadness, joy, etc., and also deep emotions such as stress, tension, frustration, etc. Using self-developed emotion computing technology, it can analyze more than 20 kinds of emotions and is more likely to detect abnormal emotions; based on the analysis of the monitored emotion historical data, it automatically gives early warnings of emotional abnormalities.

[0061] It should be noted that in this article, the terms "including", "comprising", or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article, or device. Without further limitations, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article, or device including that element.

[0062] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to enable a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in various embodiments of this application.

[0063] The embodiments of this application have been described above in conjunction with the accompanying drawings. However, this application is not limited to the above specific implementation manners. The above specific implementation manners are merely illustrative and not restrictive. Under the inspiration of this application, those of ordinary skill in the art can also make many forms without departing from the purpose of this application and the scope protected by the claims, and all of them fall within the protection scope of this application.

Claims

1. An emotion normalization monitoring system based on the Xinchuang environment, characterized in that, Comprising: A camera for collecting a video stream of a human face; An emotion computing gateway for generating a plurality of video files according to the video stream, obtaining multiple frames of face images from each of the video files, processing the multiple frames of face images, locating multiple muscle groups on the face and the contours of multiple organs related to the head, and obtaining the first time-series change data of the amplitude and vibration frequency of each pixel point within the contour of the facial muscle group and the vibration frequency of each pixel point within the contour of the multiple organs related to the head within a preset time; calculating the second time-series change data of the resonance frequency related to the head within a preset time according to the vibration frequency of each pixel point within the contour, and outputting the emotion state data corresponding to the human face according to the first time-series change data and the second time-series change data; An emotion warning server for analyzing based on monitored emotion history data and performing hierarchical warnings on personnel with a relatively high risk of psychological problems; Wherein, both the emotion computing gateway and the emotion warning server support the Xinchuang environment; The emotion computing gateway includes: An emotion computing engine for locating each pixel point within the contour of the facial muscle group on each frame of face image based on a muscle group contour definition model; calculating the amplitude and frequency of each pixel point according to the displacement of each pixel point in consecutive face images, and obtaining the first time-series change data of the amplitude and the frequency within a preset time; And, based on human biological characteristics, locating the contours of the pupils, eyes, nose and head on each frame of face image, and each pixel point within the contour; calculating the vibration frequency of each pixel point according to the displacement of each pixel point in consecutive face images; The emotion computing engine is specifically configured to perform weighted average calculations on the vibration frequencies of each pixel point respectively to obtain the second time-series change data of the pupil resonance frequency, eyeball resonance frequency, respiratory resonance frequency and head resonance frequency within a preset time.

2. The system according to claim 1, wherein The emotion warning server is specifically configured to generate the emotion history data and emotion history trend of the target person corresponding to the human face according to the emotion state data corresponding to the human face; and perform a hierarchical warning on the target person when the emotion history trend triggers an emotion warning model.

3. The system according to claim 1, wherein Both the emotion computing gateway and the emotion warning server support running in an environment of domestic CPUs and domestic operating systems.

4. A method for emotion normalization monitoring based on the Xinchuang environment, characterized in that, Including the following steps: Collecting a video stream of a human face; Generating a plurality of video files according to the video stream and obtaining multiple frames of face images from each of the video files; Processing the multiple frames of face images, locating multiple muscle groups on the face and the contours of multiple organs related to the head, and obtaining the first time-series change data of the amplitude and vibration frequency of each pixel point within the contour of the facial muscle group and the vibration frequency of each pixel point within the contour of the multiple organs related to the head; Calculate the second time-series change data of the resonance frequency related to the head within a preset time according to the vibration frequency of each pixel point within the contour, and output the emotional state data corresponding to the face according to the first time-series change data and the second time-series change data; Based on the analysis of the monitored emotional history data, conduct hierarchical early warning for personnel with a relatively high risk of mental problems; The processing of the multiple face images includes locating the contours of multiple muscle groups on the face and multiple organs related to the head, and obtaining the first time-series change data of the amplitude and vibration frequency of each pixel point within the contour of the facial muscle group within a preset time, and the vibration frequency of each pixel point within the contours of multiple organs related to the head. Specifically, it includes: Based on the muscle group contour definition model, locate each pixel point within the contour of the facial muscle group on each face image; Calculate the amplitude and frequency of each pixel point according to the displacement of each pixel point in consecutive face images, and obtain the first time-series change data of the amplitude and the frequency within a preset time; Based on human biological characteristics, locate the contours of the pupil, eyes, nose, and head on each face image, and each pixel point within the contour; calculate the vibration frequency of each pixel point according to the displacement of each pixel point in consecutive face images; The calculation of the second time-series change data of the resonance frequency related to the head within a preset time according to the vibration frequency of each pixel point within the contour specifically includes: Perform weighted average calculations on the vibration frequencies of each pixel point respectively to obtain the second time-series change data of the pupil resonance frequency, eyeball resonance frequency, respiratory resonance frequency, and head resonance frequency within a preset time.

5. The method according to claim 4, characterized in that, After outputting the emotional state data corresponding to the face according to the first time-series change data and the second time-series change data, it further includes: Generate the emotional history data and emotional history trend of the target person corresponding to the face according to the emotional state data corresponding to the face; In the case where the emotional history trend triggers the emotional early warning model, conduct hierarchical early warning for the target person.

6. The method according to claim 4, wherein After outputting the emotional state data corresponding to the face according to the first time-series change data and the second time-series change data, it further includes: Delete the multiple video files.

Images