Non-contact psychological ability assessment method

Abstract

The invention provides a non-contact psychological ability assessment method, and relates to the technical field of psychological ability monitoring. Comprising the following steps: acquiring behavior clue information by using a multispectral camera and an RBG camera; obtaining non-behavior clues by using a millimeter wave radar and a pickup sensor; performing data fusion on the behavior clue information and the non-behavior clues to obtain fused data; and determining a psychological ability evaluation index according to the fused data. According to the method, the problems of low applicability and insufficient accuracy of a non-contact multi-mode evaluation technology in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of psychological ability monitoring, and in particular to a non-contact psychological ability assessment method. Background Art

[0002] In the current field of psychological or emotional monitoring and evaluation, the evaluation method using specific data and multimodal data fusion as data sources has a variety of technical directions. In terms of the acquisition of multimodal signals, contact, wearable and non-contact methods have also emerged. Its application scope is also expanding, such as in the fields of public health, health education, human resources, consumer medicine and even military training, and has great application value. Therefore, higher standards are also put forward for the objectivity, accuracy and practicality of psychological ability monitoring and evaluation technology.

[0003] At present, signal sensors based on non-contact multimodality mainly include visual sensors (cameras or cameras), millimeter-wave radars, and recording equipment. First, image signals and analysis technologies obtained based on visual sensors mainly include ippG technology (extracting pulse signal features through time-series images), facial expression recognition technology (extracting expression features through facial recognition technology), etc. Among them, ippG technology is limited by the large influence of light sources in the application scenarios and the displacement noise of the human head, making it impossible to carry out popular applications. For facial expression recognition technology, the accuracy of each expression has different probabilities, and the error in recognizing expressions in conversations is even greater, so it is also not suitable for popular applications. Secondly, based on millimeter-wave radar physiological signals and analysis technology, due to the entanglement of heart rate fluctuations and respiratory fluctuations, although the heart rate can be extracted in theory and ideal environments, the quality of the extracted heart rate signal cannot meet the analysis requirements of heart rate variability. The above two non-contact multimodal signal acquisition and evaluation technologies both face the bottleneck of being unable to be widely used in real usage scenarios. Summary of the invention

[0004] In order to overcome the deficiencies of the prior art, the object of the present invention is to provide a non-contact psychological ability assessment method.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A non-contact method of assessing psychological abilities, including:

[0007] Use multispectral cameras and RBG cameras to obtain behavioral clues;

[0008] Use millimeter-wave radar and sound pickup sensors to obtain non-behavioral clues;

[0009] fusing the behavioral clue information with the non-behavioral clue information to obtain fused data;

[0010] A psychological ability assessment index is determined based on the fused data.

[0011] Preferably, the method of obtaining behavioral clue information using a multispectral camera or an RBG camera includes:

[0012] Using the multispectral camera to acquire facial images and extract key points of eyes and mouth, wherein the facial images are two near-infrared images;

[0013] Using the RBG camera to obtain a panoramic image of the human body and extract activity features of the torso, hands and legs;

[0014] Performing a difference comparison on the two near-infrared images to obtain a pixel brightness difference;

[0015] determining a sweating feature based on the pixel brightness difference;

[0016] The behavior clue information is determined according to the activity characteristics of the eyes, mouth key points, trunk, hands and legs and the sweating characteristics.

[0017] Preferably, the calculation expression of the sweating characteristic is:

[0018]

[0019] Among them, ΔB is the sweating characteristic, B NIR1 Indicates the brightness value of the NIR1 spectrum, B NIR2 represents the brightness value of the NIR2 spectrum, k1 and k2 are the first fitting parameter and the second fitting parameter respectively, α and β are the first exponential parameter and the second exponential parameter respectively, and γ is the third exponential parameter.

[0020] Preferably, the method of obtaining non-behavioral clues by using millimeter wave radar and sound pickup sensor includes:

[0021] Use millimeter wave radar to obtain physiological signal data;

[0022] Obtaining a timing curve according to the physiological signal data;

[0023] A fusion signal curve of heart rate and respiration according to the timing curve;

[0024] Performing Fourier transformation on the fusion signal curve to obtain a neural dynamic spectrum diagram;

[0025] Calculating the frequency band data of the neural dynamic spectrum diagram to obtain first multi-frequency band data;

[0026] Determine a first frequency spectrum change feature according to the multi-band data;

[0027] The audio signal is acquired by using a sound pickup sensor and feature extraction is performed to obtain a sound rhythm spectrum;

[0028] Performing Fourier transformation on the sound rhythm spectrum to obtain a power spectrum density diagram of the sound rhythm spectrum;

[0029] Calculate and obtain second multi-band data according to the power spectrum density diagram;

[0030] fusing the first multi-band data and the second multi-band data to obtain a fusion feature;

[0031] The non-behavioral clue is determined according to the fused feature.

[0032] Preferably, the calculation expression of the neural dynamic spectrum graph is:

[0033]

[0034] Preferably, the multi-band data includes: first frequency band data, second frequency band data and third frequency band data, wherein the expressions of the first frequency band data, the second frequency band data and the third frequency band data are respectively:

[0035]

[0036] Among them, P VLF , P LF and P HF They are first frequency band data, second frequency band data and third frequency band data respectively.

[0037] Preferably, the calculation expression of the second multi-band data is:

[0038]

[0039] Among them, PSD sound (f) is the power spectrum density diagram, It is the second multi-band data.

[0040] Preferably, the calculation expression of the fusion feature is:

[0041]

[0042] Among them, F is the fusion feature, w1 and w2 are the first weight coefficient and the second weight coefficient respectively. For the fourth frequency band data, This is the fifth frequency band data.

[0043] The present invention discloses the following technical effects:

[0044] The present invention provides a non-contact psychological ability assessment method, including: using a multi-spectral camera and an RBG camera to obtain behavioral clue information; using a millimeter wave radar and a sound pickup sensor to obtain non-behavioral clues; fusing the behavioral clue information and the non-behavioral clues to obtain fused data; and determining a psychological ability assessment index according to the fused data. The present invention solves the problem of wide applicability by using multi-spectral imaging technology that is not affected by changes in scene light sources; (2) using a heart rate and breathing fusion spectrum analysis method to solve the problem of loss of change characteristics caused by separating heart rate signals in millimeter wave radar signal analysis, so that it has usability in actual work; (3) using more accurate and stable behavioral data, such as blinking, eye area, limb movements, sweating trends, etc. as a behavioral clue model data source, and fusing the heart rate & breathing rate fusion spectrum (HRV-R) with the sound rhythm spectrum as a non-behavioral clue model data source for multimodal fusion and analysis, solving the problem that traditional facial expressions and physiological signals cannot conduct in-depth and multi-dimensional evaluation of psychological abilities and phenomena; thereby realizing the true practicality and effectiveness of non-contact multimodal assessment technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0046] Figure 1 A flow chart of a non-contact psychological ability assessment method provided by an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of a non-contact psychological ability assessment method strategy provided by an embodiment of the present invention;

[0048] Figure 3 Schematic diagram of five spectra provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] like Figure 1-2 As shown, the present invention provides a non-contact psychological ability assessment method, comprising:

[0052] Step 100: Obtaining behavioral clue information using a multispectral camera and an RBG camera;

[0053] Step 200: using millimeter wave radar and sound pickup sensor to obtain non-behavioral clues;

[0054] Step 300: fusing the behavior clue information and the non-behavior clue information to obtain fused data;

[0055] Specifically, the behavioral clues (such as facial expressions and body movements) of the interviewees are obtained through multi-spectral cameras and RGB cameras, while the non-behavioral clues (such as heart rate, breathing rate, and voice features) are collected using millimeter-wave radar and sound pickup sensors. First, the multi-source data is synchronized and preprocessed to extract visual, physiological, and voice features respectively; then, early fusion, late fusion, or hybrid fusion strategies are used to perform multi-level data fusion of behavioral clues and non-behavioral clues to generate high-dimensional feature vectors; then, machine learning or deep learning models are used to analyze the fused data to infer the interviewee's emotional state, tension level, and other psychological characteristics; finally, the model output is used to provide support for interview decisions.

[0056] Step 400: Determine a psychological ability assessment index based on the fused data.

[0057] Specifically, the multispectral camera and RBG camera components in the contactless device are used to obtain the features of the eyes, mouth, body trunk, hands, legs and sweating, and a model including multiple conversation behavior parameters, namely, a behavior clue model, is constructed;

[0058] Among them, psychological ability assessment indicators include: personality adaptability, emotional stability, willpower, memory ability, language comprehension, personality tendencies and tension level.

[0059] Physiological signals and conversation audio signals are obtained through millimeter wave radar and sound pickup sensor components, and a spectral analysis model that includes the distribution of conversation neural activity and the distribution of conversation stress, i.e., non-behavioral clues, is constructed using a joint multi-task learning method.

[0060] Through the two data models of behavioral clues and non-behavioral clues obtained above, multimodal data fusion is performed, and deep learning models: convolutional neural network (CNN) and long short-term memory network (LSTM) are used to extract and learn the behavioral clues and non-behavioral clues related to psychological ability and extract them to obtain a psychological ability assessment model;

[0061] Through simulation technology, the target object in the actual conversation process is modeled and simulated, so as to achieve real-time monitoring, prediction and evaluation of the individual psychological ability of the target object during the conversation.

[0062] Furthermore, the use of a multispectral camera and an RBG camera to obtain behavioral clue information includes:

[0063] Using the multispectral camera to acquire facial images and extract key points of eyes and mouth, wherein the facial images are two near-infrared images;

[0064] Using the RBG camera to obtain a panoramic image of the human body and extract activity features of the torso, hands and legs;

[0065] Performing a difference comparison on the two near-infrared images to obtain a pixel brightness difference;

[0066] determining a sweating feature based on the pixel brightness difference;

[0067] The behavior clue information is determined according to the activity characteristics of the eyes, mouth key points, trunk, hands and legs and the sweating characteristics.

[0068] Specifically, a near-focus and far-focus synchronous image acquisition structure and device are used to obtain human body images of the same sequence, frame rate, pixel and space, thereby extracting the activity features and trends of the eyes, mouth, body trunk, hands and legs, and maintaining a sequence image that is completely consistent with the dynamics of human behavior in a real conversation scene; specifically including:

[0069] By configuring the multispectral camera with a close-focus lens, we can obtain clear facial images and extract the key points and change features of the eyes and mouth. By configuring the RBG camera with a long-focus lens, we can obtain a panoramic image of the human body and extract the activity features and trends of the torso, hands and legs.

[0070] Connect the multispectral camera and the RBG camera through a pulse generator to obtain the sequence images of the two cameras with the same sequence, frame rate and pixels;

[0071] Through structured design, the multispectral camera and RBG camera are positioned and combined to obtain sequence images of the same space of the two cameras;

[0072] A 3D digital human is drawn through three-dimensional modeling technology to obtain real-time behavior characterization, abnormal behavior indicators and behavior change trend data models that are consistent with the real individual behavior in real conversation scenarios.

[0073] More specifically, two facial images of the same size and pixel in near-infrared spectra are obtained through a multispectral camera, and the difference between the two near-infrared images is compared to obtain the pixel brightness difference caused by human sweat, and obtain the human sweating characteristics and trends, including:

[0074] like Figure 3 As shown in the figure, through a 5-channel multi-spectral camera and a 3MOS special lens, five spectral images are obtained, namely visible light images R\G\B and two near-infrared (NIR) images (spectra 700-800, 800-900); through the characteristics of dual-band infrared images and the brightness changes caused by human sweating, a model containing multiple parameters is constructed. This model will combine Planck's radiation law, the basic principles of infrared radiation and image processing technology to evaluate the characteristics and trends of human sweating during conversation.

[0075] Specifically, the calculation expression of the sweating characteristic is:

[0076]

[0077] Among them, ΔB is the sweating characteristic, B NIR1 Indicates the brightness value of the NIR1 spectrum (700-800nm), B NIR2 It represents the brightness value of NIR2 spectrum (800-900nm). k1 and k2 are the first fitting parameter and the second fitting parameter, respectively, which are used to adjust the weight of the brightness difference between the two bands. α and β are the first exponential parameter and the second exponential parameter, respectively, which are used to describe the nonlinear relationship between the brightness values ​​of different bands. γ is the third exponential parameter, which is used to describe the nonlinear relationship between the brightness difference ratios.

[0078] Furthermore, the use of millimeter wave radar and sound pickup sensor to obtain non-behavioral clues includes:

[0079] Use millimeter wave radar to obtain physiological signal data;

[0080] Obtaining a timing curve according to the physiological signal data;

[0081] Specifically, first, the millimeter-wave radar emits electromagnetic waves and receives reflected signals to collect the target object's tiny movement data; secondly, the original signal is denoised, and the frequency band components related to the physiological signal are extracted using Fourier transform or wavelet transform, and the breathing signal and heartbeat signal are separated therefrom; then, the extracted signal is sampled in time sequence to generate discrete data points, and converted into a continuous time series curve through smoothing and time-frequency analysis.

[0082] A fusion signal curve of heart rate and respiration according to the timing curve;

[0083] Performing Fourier transformation on the fusion signal curve to obtain a neural dynamic spectrum diagram;

[0084] Calculating the frequency band data of the neural dynamic spectrum diagram to obtain first multi-frequency band data;

[0085] Determine a first frequency spectrum change feature according to the multi-band data;

[0086] Specifically, the physiological signal of the millimeter wave radar spectrum, i.e., the heart rate and breathing fusion signal curve, is Fourier transformed to obtain a neural dynamic spectrum diagram as a physiological state feature, which specifically includes:

[0087] The original physiological signals of the millimeter-wave radar are preprocessed and feature extracted. Since filtering and separating the heart rate and breathing signals will reduce and minimize the changing characteristics of the body's own rhythm, for this specific situation, we plot the original signal containing both heart rate and breathing into a time series curve to obtain a fusion signal curve of heart rate and breathing; by performing Fourier transformation on the obtained fusion signal curve of heart rate and breathing, we obtain the HRV-R spectrum diagram and calculate the data of each frequency band, thereby observing the spectrum change characteristics caused by changes in neural activity and changes in heart rate-respiration rate resonance.

[0088] The audio signal is acquired by using a sound pickup sensor and feature extraction is performed to obtain a sound rhythm spectrum;

[0089] Performing Fourier transformation on the sound rhythm spectrum to obtain a power spectrum density diagram of the sound rhythm spectrum;

[0090] Calculate and obtain second multi-band data according to the power spectrum density diagram;

[0091] fusing the first multi-band data and the second multi-band data to obtain a fusion feature;

[0092] The non-behavioral clue is determined according to the fused feature.

[0093] Specifically, the heart rate and respiratory rate fusion spectrum (HRV-R) is fused with the sound rhythm spectrum through multimodal spectrum fusion, and is input into the machine learning model as a non-behavioral clue feature to analyze psychological ability through physiological state, i.e. non-behavioral clues, including:

[0094] Plot the power spectral density of the rhythmic spectrum of the sound through the audio signal.

[0095] By combining spectrum data from two different sources: heart rate & respiratory rate fusion spectrum (HRV-R), a fusion feature is obtained. The fusion feature F is input into the machine learning model to predict and analyze the changes in psychological ability characteristics caused by physiological state.

[0096] Specifically, the calculation expression of the neural dynamic spectrum graph is:

[0097]

[0098] Where, PSD(f) represents the power spectral density at frequency f.

[0099] Represents the normalization factor, which is used to normalize the power spectrum density. N is the length of the signal. ||RR(f)||^ 2 represents the power spectrum of the fused signal RR(f) at frequency f. RR(f) is the frequency domain representation obtained by Fourier transform. ||RR(f)|| represents its amplitude, and the power spectrum is obtained by squaring it.

[0100] Here, RR(f) represents the frequency domain representation of the signal that combines the heart rate (HR) and the respiratory rate (RR).

[0101] Specifically, the multi-band data includes: first frequency band data, second frequency band data and third frequency band data, wherein the expressions of the first frequency band data, the second frequency band data and the third frequency band data are respectively:

[0102]

[0103] Among them, P VLF , P LF and P HF They are first frequency band data, second frequency band data and third frequency band data respectively.

[0104] Specifically,

[0105] Among them, PSD sound (f) is the power spectrum density diagram, It is the second multi-band data.

[0106] Specifically, the calculation expression of the fusion feature is:

[0107]

[0108] Among them, F is the fusion feature, w1 and w2 are the first weight coefficient and the second weight coefficient respectively. For the fourth frequency band data, This is the fifth frequency band data.

[0109] Furthermore, the process for determining the psychological ability assessment indicators is as follows:

[0110] With the help of simulation platforms such as Unity3D, a virtual interactive system architecture is built to realize the simulation of behavioral clues and non-behavioral clues in virtual space, including: building a digital twin model, replicating the behavioral trajectory and non-behavioral clue characteristics of the target object during the conversation in the software system, and performing real-time simulation to realize the digital twin application of the entire conversation process from behavioral and non-behavioral aspects, that is, the interactive display model. A digital twin virtual simulation interactive system that can reflect individual behavior and physiological status in real time is built to achieve seamless connection and interaction between the physical world and the digital world.

[0111] The real-time evaluation calculation module designs digital labels for psychological abilities: personality adaptability, emotional stability, willpower, memory, and language comprehension, and generates shortcut command buttons. During the conversation, the corresponding buttons are triggered to obtain analysis indicators of various psychological abilities at the moment. Specifically, it includes:

[0112] Through model construction and verification, we obtained the evaluation indicators of psychological ability: personality adaptability, emotional stability, willpower, memory ability, language comprehension, personality tendency and tension level.

[0113] A one-to-one correspondence is established between shortcut commands and psychological ability assessment indicators. When the corresponding command is triggered, the system immediately calculates the corresponding assessment indicator and displays it on the visual interface of the digital twin to obtain the psychological ability assessment and interpretation. The corresponding command corresponds to the assessment indicator, and each corresponding command is implemented through different shortcut keys.

[0114] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0115] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A non-contact psychological ability assessment method, characterized in that: include: Use multispectral cameras and RBG cameras to obtain behavioral clues; Use millimeter-wave radar and sound pickup sensors to obtain non-behavioral clues; fusing the behavioral clue information with the non-behavioral clue information to obtain fused data; A psychological ability assessment index is determined based on the fused data.

2. A non-contact psychological ability assessment method according to claim 1, characterized in that: The method of using a multispectral camera and an RBG camera to obtain behavioral clue information includes: Using the multispectral camera to acquire facial images and extract key points of eyes and mouth, wherein the facial images are two near-infrared images; Using the RBG camera to obtain a panoramic image of the human body and extract activity features of the torso, hands and legs; Performing a difference comparison on the two near-infrared images to obtain a pixel brightness difference; determining a sweating feature based on the pixel brightness difference; The behavior clue information is determined according to the activity characteristics of the eyes, mouth key points, trunk, hands and legs and the sweating characteristics.

3. A non-contact psychological ability assessment method according to claim 2, characterized in that: The calculation expression of the sweating characteristic is: Among them, ΔB is the sweating characteristic, B NIR1 Indicates the brightness value of the NIR1 spectrum, B NIR2 represents the brightness value of the NIR2 spectrum, k1 and k2 are the first fitting parameter and the second fitting parameter respectively, α and β are the first exponential parameter and the second exponential parameter respectively, and γ is the third exponential parameter.

4. A non-contact psychological ability assessment method according to claim 1, characterized in that: The use of millimeter wave radar and sound pickup sensor to obtain non-behavioral clues includes: Use millimeter wave radar to obtain physiological signal data; Obtaining a timing curve according to the physiological signal data; A fusion signal curve of heart rate and respiration according to the timing curve; Performing Fourier transformation on the fusion signal curve to obtain a neural dynamic spectrum diagram; Calculating the frequency band data of the neural dynamic spectrum diagram to obtain first multi-frequency band data; Determine a first frequency spectrum change feature according to the multi-band data; The audio signal is acquired by using a sound pickup sensor and feature extraction is performed to obtain a sound rhythm spectrum; Performing Fourier transformation on the sound rhythm spectrum to obtain a power spectrum density diagram of the sound rhythm spectrum; Calculate and obtain second multi-band data according to the power spectrum density diagram; fusing the first multi-band data and the second multi-band data to obtain a fusion feature; The non-behavioral clue is determined according to the fused feature.

5. A non-contact psychological ability assessment method according to claim 4, characterized in that: The calculation expression of the neural dynamic spectrum graph is: Wherein, PSD(f) is the power spectral density at frequency f, N is the signal length, and RR(f) is the frequency domain representation of the signal that combines heart rate (HR) and respiratory rate (RR).

6. A non-contact psychological ability assessment method according to claim 4, characterized in that: The multi-band data includes: first-band data, second-band data and third-band data, wherein the expressions of the first-band data, the second-band data and the third-band data are respectively: Among them, P VLF , P LF and P HF They are first frequency band data, second frequency band data and third frequency band data respectively.

7. A non-contact psychological ability assessment method according to claim 6, characterized in that: The calculation expression of the second multi-band data is: Among them, PSD sound (f) is the power spectrum density diagram, It is the second multi-band data.

8. A non-contact psychological ability assessment method according to claim 7, characterized in that: The calculation expression of the fusion feature is: Among them, F is the fusion feature, w1 and w2 are the first weight coefficient and the second weight coefficient respectively. For the fourth frequency band data, This is the fifth frequency band data.

Images