A Method for Extracting Human Functional State Parameters Based on Thermal-Kinetic Image Data

Abstract

This invention discloses a method for extracting human functional state parameters based on thermal-kinetic image data and a target population screening device. The method includes: acquiring thermal-kinetic image data of a person; the thermal-kinetic image data includes: kinetic energy change data, thermal energy change data, and energy change cycle at various points on the head and neck of the person; extracting the at least one human functional state parameter from the thermal-kinetic image data according to a defined formula; wherein the at least one human functional state parameter includes one or more of the following: aggression index, stress index, anxiety index, balance index, confidence index, vitality index, and inhibition index. Using this invention allows for a more comprehensive and accurate understanding of human functional state.

Description

Technical Field

[0001] This invention belongs to the field of human life necessities, and specifically relates to a method for extracting human functional state parameters based on thermal-kinetic imaging data. Background Technology

[0002] Human functional state refers to the state of a person's psychological and physiological activities. Physiological activities influence psychological state; for example, eating sweets produces feelings of pleasure, while weightlessness evokes fear. Conversely, psychological activities also affect physiological state. For instance, prolonged depression can easily lead to dysfunction of the cerebral cortex, causing a series of physiological disorders. Studies show that approximately one-third of patients with depression exhibit low-level alpha waves on their electroencephalograms (EEGs) and show signs of low brain function.

[0003] This demonstrates that human psychological and physiological activities are closely related. By observing or testing a person's external physiological activities, we can learn not only about their physiological state but also their internal psychological state. The process of obtaining this information is the process of detecting or evaluating the functional state of the human body.

[0004] In existing technologies, most methods for assessing a person's functional state involve testing their electrocardiogram (ECG) or electroencephalogram (EEG). However, ECG and EEG only detect human functional state from a microscopic perspective of biological electrical activity, providing a narrow viewpoint. The resulting data is limited and cannot accurately correspond to different functional states of the body, making it difficult to reflect the body's functional state precisely. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a method for extracting human functional state parameters based on thermal-kinetic imaging data and a target population screening device. The technical problem to be solved by this invention is achieved through the following technical solution:

[0006] In a first aspect, the present invention provides a method for extracting human functional state parameters based on thermal-kinetic imaging data, including:

[0007] A method for extracting human functional state parameters based on thermal-kinetic image data includes:

[0008] Acquire thermal-kinetic image data of a person; the thermal-kinetic image data is used to generate thermal-kinetic images; wherein, the thermal-kinetic image data includes: kinetic energy change data, thermal energy change data, and energy change cycle of each point on the head and neck of the person;

[0009] According to the definition formula of at least one human functional state parameter to be extracted, the at least one human functional state parameter is extracted from the thermal-kinetic image data;

[0010] Wherein, the at least one human functional state parameter includes one or more of the following human functional state parameters:

[0011] Aggression index, stress index, anxiety index, balance index, confidence index, vitality index, and inhibition index.

[0012] In one embodiment, the aggressiveness index is defined by the following formula:

[0013]

[0014] Wherein, AGI represents the aggression index; f(t) is the probability density function of the energy change period at each point on the head and neck, t represents the energy change period, and t∈[T] min ,T max ]; T min T is the smallest period among the energy change periods at various points in the head and neck. max F(t) represents the maximum period among the energy change periods at each point in the head and neck; F(t) represents the number of points in the head and neck with an energy change period of t. Let F(t) be the expected value; F max (t) is the maximum value of F(t); A is a preset constant, 1≤A≤ 10.

[0015] In one embodiment, the kinetic energy change data includes multiple kinetic energy matrices, and the thermal energy change data includes multiple thermal energy matrices; the energy change period at each point of the head and neck is contained in a period matrix.

[0016] The formula for defining the pressure index is:

[0017]

[0018] Wherein, PRI represents the pressure index; PRI_1 is the kinetic energy sub-index of the pressure index, and PRI_2 is the thermal energy sub-index of the pressure index; the number of rows in the kinetic energy matrix, the thermal energy matrix, and the periodic matrix are all N;

[0019] δ E (n) is the kinetic energy difference between the left and right sides of the head and neck of the person being measured, calculated based on the elements of the nth row of a target kinetic energy matrix; the target kinetic energy matrix is ​​taken from the plurality of kinetic energy matrices;

[0020] |δ T(n)| is the energy change period difference value between the left and right sides of the head and neck of the person being described, calculated based on the element in the nth row of the period matrix;

[0021] The difference in thermal energy between the left and right sides of the head and neck of the person is calculated based on the element of the nth row of the m-th thermal energy matrix in [1,M].

[0022] In one embodiment, the anxiety index is defined by the following formula:

[0023]

[0024] Wherein, ANI represents the anxiety index, f(t) is the probability density function of the energy change cycle at each point in the head and neck, t represents the energy change cycle, and t∈[T] min ,B×T min ], T min B is the minimum period among the energy change periods at each point in the head and neck, and B is a preset constant, where 1 < B < 5.

[0025] In one embodiment, the balance index is defined by the following formula:

[0026]

[0027] Wherein, BAI represents the balance index; f(t) is the probability density function of the energy change period at each point in the head and neck, t represents the energy change period, and t∈[T] min ,T max ]; T min T is the minimum period among the energy change periods at various points on the head and neck of the person being described. max N(t) represents the maximum period among the energy change periods at various points on the head and neck of the person; N(t) is the standard normal probability density function of the energy change periods at various points on the head and neck.

[0028] In one embodiment, the kinetic energy change data includes multiple kinetic energy matrices; the thermal energy change data includes multiple thermal energy matrices.

[0029] The formula for defining the confidence index is:

[0030]

[0031] Wherein, SCI represents the confidence index, SCI_1 is the kinetic energy sub-index of the confidence index, and SCI_2 is the thermal energy sub-index of the confidence index; E l (n) represents the kinetic energy data of the left half of the nth row of any of the aforementioned kinetic energy matrices, E r(n) represents the kinetic energy data of the right half of the nth row of the kinetic energy matrix, H l (n) represents the thermal energy data of the left half of the nth row of any of the aforementioned thermal energy matrices, H r (n) represents the thermal energy data of the right half of the nth row of the thermal energy matrix, where n∈[1,N] and the number of rows of both the kinetic energy matrix and the thermal energy matrix is ​​N.

[0032] In one embodiment, the vitality index is defined by the following formula:

[0033]

[0034] Wherein, ENI represents the vitality index, f(t) is the probability density function of the energy change period at each point in the head and neck, t represents the energy change period, and t∈[T] min ,T max ]; T min T is the smallest period among the energy change periods at various points in the head and neck. max F is the maximum period among the energy change periods at all points in the head and neck; F(t) is the number of points in the head and neck with an energy change period of t. max (t) represents the maximum value of F(t); σ represents the standard deviation of the energy change period at each point in the head and neck.

[0035] In one embodiment, the inhibition index is defined by the following formula:

[0036]

[0037] Wherein, INI represents the inhibition index, I indicates that among the energy change cycles of various points on the head and neck of the subject, the energy change cycles of I points are less than a preset reference threshold, and t i This represents the energy change period of the i-th point among the I points.

[0038] Secondly, the present invention provides a target population screening device, comprising: a data processing module and a screening module;

[0039] The data processing module is used to extract human functional state parameters related to the screening purpose from the thermal-kinetic image data of the person to be screened; the thermal-kinetic image data is the data used to generate thermal-kinetic images; wherein, the thermal-kinetic image data includes: kinetic energy change data, thermal energy change data, and energy change cycle of each point of the head and neck of the person;

[0040] The screening module is used to calculate the human function status score of the person based on the difference between the various human function status parameters extracted by the data processing module and the corresponding screening standards, and to determine whether the person belongs to the target population based on the human function status score.

[0041] The human body functional state parameters include one or more of the following:

[0042] Aggression index, stress index, anxiety index, balance index, confidence index, vitality index, and inhibition index.

[0043] In one embodiment, the device further includes: a data acquisition module;

[0044] The data acquisition module is used to acquire video data and temperature data of the head and neck of the person being observed.

[0045] The data processing module is further configured to generate thermal-kinetic image data of the person based on the video data and the temperature data.

[0046] The beneficial effects of this invention are:

[0047] The method for extracting human functional state parameters based on thermal-kinetic imaging data provided by this invention acquires thermal-kinetic imaging data of a person. This data captures the movement of the head and neck and temperature changes from both thermal and kinetic perspectives, providing more comprehensive information. Furthermore, the two types of data complement each other and can be cross-referenced. Since people experience uncontrollable primary tension during periods of stress and anxiety, the vestibular system reflexes, leading to minute movements of the head and neck and changes in local body temperature. Therefore, this invention, through pre-collection and analysis of human thermal-kinetic imaging data, derives definition formulas for various indicators that accurately reflect the human functional state. Based on these formulas, human functional state parameters can be extracted from the thermal-kinetic imaging data of the person, thereby revealing the person's overall human functional state.

[0048] Furthermore, compared to existing technologies that use electrocardiograms or electroencephalograms to obtain information about human functional status, this invention provides a contactless detection solution. This reduces anxiety caused by physical contact with the detection instrument and minimizes errors introduced by the detection process itself, thereby obtaining more accurate information about the individual's functional status.

[0049] Furthermore, this invention can also visualize thermal-kinetic image data, thereby enabling observers to more intuitively understand the human functional status of the tested individuals.

[0050] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating a method for extracting human functional state parameters based on thermal-kinetic image data, provided in an embodiment of the present invention.

[0052] Figure 2 This is a schematic diagram illustrating the generation process of a target kinetic energy matrix in an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of a neural network structure used in the process of pre-generating thermal-kinetic image data in an embodiment of the present invention.

[0054] Figure 4 This is an example of the effect of combining thermal-kinetic energy images with human functional state parameters extracted from the images in an embodiment of the present invention.

[0055] Figure 5 This is a schematic diagram of the structure of a target population screening device provided in an embodiment of the present invention;

[0056] Figure 6 This is a schematic diagram of another target population screening device provided in an embodiment of the present invention;

[0057] Figure 7 yes Figure 6 Front view of the device shown;

[0058] Figure 8 yes Figure 6 Side view of the actual device shown. Detailed Implementation

[0059] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0060] To obtain a more comprehensive and accurate understanding of human functional status, embodiments of the present invention provide a method for extracting human functional status parameters based on thermal-kinetic imaging data and a method for screening target populations. Both methods provided in these embodiments can be applied to electronic devices, such as computers, tablets, mobile terminals, medical devices, or servers, etc., without specific limitations. Any electronic device with computational processing capabilities can use the two methods provided in these embodiments.

[0061] First, the method for extracting human functional state parameters based on thermal-kinetic image data provided in this embodiment of the invention will be described in detail. (See also...) Figure 1 As shown, the method includes the following steps:

[0062] S1: Acquire thermal-kinetic image data of personnel.

[0063] The thermal-kinetic image data is used to generate thermal-kinetic images; wherein, the thermal-kinetic image data includes: kinetic energy change data, thermal energy change data, and energy change cycle of each point on the head and neck of the person.

[0064] The thermal-kinetic image data is obtained by simultaneously capturing visible light and infrared images of the subject's head and neck, and then processing the acquired visible light video, infrared video, and multiple temperature matrices associated with the infrared video. To clarify the layout of the instruction manual, the specific generation process of the thermal-kinetic image data will be illustrated below. During filming, the subject's head and neck are kept quasi-stationary. "Quasi-stationary" here refers to a relatively static state maintained when the subject is instructed to keep their head and neck still. In this state, very minute movements of the subject's head and neck can be captured using visible light and infrared cameras.

[0065] The vertical balance of the human head is controlled by the vestibular system, a function known as the vestibular reflex. Head movements are among the most frequent human movements, as the head only stops adjusting its posture when it is supported, involving numerous tiny movements on the scale of hundreds of micrometers per second. When a person is tense, anxious, or ill, uncontrollable primary tension occurs, leading to minute head and neck movements and temperature changes in response to the vestibular system. These changes can be collected to create thermal-kinetic imaging data, which can then be used to understand the body's functional state.

[0066] In practice, the acquired kinetic energy change data includes multiple kinetic energy matrices, the thermal energy change data includes multiple thermal energy matrices, and the energy change period of each point on the head and neck of the subject is also contained in a period matrix. These matrices all have the same dimension, the specific size of which depends on the resolution of both cameras, which are set to the same resolution. Specifically, each element in the thermal energy matrix records the thermal energy value of each point on the head and neck of the subject, each element in the kinetic energy matrix records the kinetic energy value of each point on the head and neck of the subject, and each element in the period matrix records the energy change period of each point on the head and neck of the subject during periodic movements.

[0067] S2: Extract at least one human functional state parameter from the thermal-kinetic image data according to the definition formula of the at least one human functional state parameter to be extracted.

[0068] The aforementioned human functional state parameter may include one or more of the following human functional state parameters:

[0069] Aggression index, stress index, anxiety index, balance index, confidence index, vitality index, and inhibition index.

[0070] The above-mentioned definition formula was derived by collecting and analyzing thermal-kinetic image data of a large number of people. The analysis method mainly uses mathematical methods to study the distribution pattern of the data, and then studies the correlation between the data distribution pattern and the known human functional state of the people, thereby establishing the correlation between the data distribution pattern and different human functional state parameters. After establishing the correlation, a mathematical formula for evaluating the data distribution pattern is constructed, which can then be used as the definition formula for evaluating the associated human functional state parameters.

[0071] The following is a detailed explanation of the formulas for defining the various human functional state parameters extracted in the embodiments of the present invention.

[0072] (a) The formula for defining the above-mentioned aggression index is:

[0073]

[0074] Where AGI represents the aggression index; f(t) is the probability density function of the energy change period at each point on the head and neck, t represents the energy change period, and t∈[T] min ,T max ]; T min T is the smallest period among the energy change cycles at various points in the head and neck. max F(t) represents the maximum period among the energy change periods at various points in the head and neck; F(t) represents the number of points in the head and neck with an energy change period of t. Let F(t) be the expected value; F max (t) is the maximum value of F(t); A is a preset constant, 1≤A≤10.

[0075] When calculating the aggression index using the above definition formula, F(t) can be obtained by statistically analyzing the number of points distributed throughout the entire matrix for each specific energy change period t in the periodic matrix. max (t) can be directly extracted from the statistical results. This is obtained by averaging the entire statistical result. The function f(t) can be obtained by calculating a set of energy change period density values ​​using the period matrix, and then fitting the results using the least squares method based on these density values.

[0076] (II) The formula for defining the above pressure index is:

[0077]

[0078] Wherein, PRI represents the pressure index; PRI_1 is the kinetic energy sub-index of the pressure index, PRI_2 is the thermal energy sub-index of the pressure index; N is the number of rows in the kinetic energy matrix, thermal energy matrix and periodic matrix, that is, the number of rows in all three of them is N.

[0079] δ E (n) is the difference in kinetic energy between the left and right sides of the head and neck of a person, calculated based on the elements of the nth row of a target kinetic energy matrix; the target kinetic energy matrix is ​​taken from the above-mentioned multiple kinetic energy matrices.

[0080] Specifically, the elements in the nth row of the target kinetic energy matrix are divided into left and right parts. The difference between each pair of mirror-symmetric elements in the two parts is calculated. The differences are then accumulated or averaged. The resulting value is the kinetic energy difference value calculated using the elements of that row. It can be understood that since the dimension of the kinetic energy difference value is the same as the resolution of the captured image of the person's head and neck, the kinetic energy difference value calculated using the elements of the nth row represents the kinetic energy difference value of the person's head and neck along the nth horizontal line.

[0081] The target kinetic energy matrix can be any one of the multiple kinetic energy matrices mentioned above, such as the last kinetic energy matrix. The last kinetic energy matrix was generated the latest, that is, the video data on which the kinetic energy matrix was based was captured the latest.

[0082] Alternatively, in another implementation, a new matrix can be generated based on multiple kinetic energy matrices to serve as the target kinetic energy matrix; in this new matrix, the element at each position is taken from the kinetic energy value that appears most frequently at the same position in the multiple kinetic energy matrices. For example, see... Figure 2 As shown, among the six kinetic energy matrices, the kinetic energy value that appears most frequently at the top left corner is 1, the kinetic energy value that appears most frequently at the top right corner is 2, the kinetic energy value that appears most frequently at the bottom left corner is 3, and the kinetic energy value that appears most frequently at the bottom right corner is 4. Therefore, the target kinetic energy matrix generated based on these six kinetic energy matrices can be as follows: Figure 2 As shown in the lower middle.

[0083] |δ T(n) represents the energy change period difference between the left and right sides of the head and neck of the person, calculated based on the element in the nth row of the periodic matrix. For the specific calculation method, please refer to the method described above for calculating the kinetic energy difference value based on the element in the nth row of the target kinetic energy matrix; it will not be repeated here.

[0084] The value of the thermal energy difference between the left and right sides of the head and neck of the person is calculated based on the element of the nth row of the m-th thermal energy matrix in [1,M], where M is the total number of thermal energy matrices. The specific calculation method can be found in the method for calculating the kinetic energy difference value based on the element of the nth row of the target kinetic energy matrix above, and will not be repeated here.

[0085] (III) The formula for defining the anxiety index is as follows:

[0086]

[0087] Where ANI represents the anxiety index, f(t) is the probability density function of the energy change cycle at various points in the head and neck, t represents the energy change cycle, and t∈[T] min ,B×T min ], T min B is the smallest period among the energy change cycles at various points in the head and neck, and B is a preset constant, where 1 < B < 5.

[0088] The function f(t) has already been explained above and will not be repeated here. It is understandable that the integral of the probability density function is a value less than 1; therefore, multiplying it by 100 will present the anxiety index as a percentage.

[0089] (iv) The formula for defining the above balance index is:

[0090]

[0091] Here, BAI represents the balance index, which is an index that evaluates the degree to which the body's functional state deviates from the normal state; t∈[T min ,T max N(t) is the standard normal probability density function of the energy change period at each point in the head and neck. The expectation of this standard normal probability density function is the mean of the elements of the periodic matrix, and the variance is the average of the differences between each element in the periodic matrix and the expectation. For the explanation of f(t) and the other parameters, please refer to the above.

[0092] (v) The formula for defining the above self-confidence index is:

[0093]

[0094] Wherein, SCI represents the confidence index, SCI_1 is the kinetic energy sub-index of the confidence index, and SCI_2 is the thermal energy sub-index of the confidence index; E l (n) represents the kinetic energy data of the left half of the nth row of any kinetic energy matrix, E r (n) represents the kinetic energy data of the right half of the nth row of the kinetic energy matrix, H l (n) represents the thermal energy data of the left half of the nth row of any thermal energy matrix, H r (n) represents the thermal energy data of the right half of the nth row of the thermal energy matrix, where n∈[1,N]. The number of rows in both the kinetic energy matrix and the thermal energy matrix is ​​N.

[0095] |E l (n)-E r (n)| represents finding E l (n) and E r The difference of (n) is moduloed again, |H l (n)-H r (n)| represents finding H l (n) and H r The difference (n) is then moduloed; the calculation method of the difference can be found in the method of calculating the kinetic energy difference value based on the nth row element of the target kinetic energy matrix above, and will not be repeated here.

[0096] (vi) The formula for defining the above vitality index is:

[0097]

[0098] Wherein, ENI represents the vitality index, t∈[T] min ,T max ], σ represents the standard deviation of the energy change period at various points on the head and neck of the person; f(t), F(t), and F max The meaning of (t) is explained above.

[0099] (vii) The formula for defining the above inhibition index is:

[0100]

[0101] Wherein, INI represents the inhibition index, which indicates the average minimum reaction time of a person to external stimuli. The smaller the value, the more sensitive the person's reaction and the milder the suppression state; conversely, the larger the inhibition index value, the slower the person's reaction and the more severe the suppression state. Wherein, I represents the energy change period of I points in the energy change cycle of the head and neck of the subject being less than a preset reference threshold, t i This represents the energy change period of the i-th point among these I points.

[0102] Based on the various definitions and formulas shown above, the human functional state parameters of the human subject can be extracted from the thermal-kinetic image data of the human subject, thereby quantifying and outputting the human functional state of the human subject.

[0103] It should be noted that since thermal-kinetic image data contains a variety of information such as kinetic energy, thermal energy, and energy change cycle, the human functional state parameters that can be extracted based on thermal-kinetic image data are not limited to the seven items mentioned above, and there are many more possibilities that need to be further explored and studied.

[0104] The method for extracting human functional state parameters based on thermal-kinetic image data provided in this invention acquires thermal-kinetic image data of a person. This thermal-kinetic image data captures the movement of the head and neck and temperature changes from both thermal and kinetic perspectives, providing more comprehensive information. Furthermore, the two types of data complement each other and can be cross-referenced. Since people experience uncontrollable primary tension during periods of stress and anxiety, the vestibular system reflexes, leading to minute movements of the head and neck and changes in local body temperature. Therefore, this invention, through pre-collection and analysis of thermal-kinetic image data, derives definition formulas for various indicators that accurately reflect human functional state. Based on these formulas, human functional state parameters can be extracted from the thermal-kinetic image data of the person to understand their overall human functional state.

[0105] Furthermore, compared to existing technologies that use electrocardiograms or electroencephalograms to obtain information about human functional status, this invention provides a contactless detection solution. This reduces anxiety caused by physical contact with the detection instrument and minimizes errors introduced by the detection process itself, thereby obtaining more accurate information about the individual's functional status.

[0106] Furthermore, embodiments of the present invention can also visualize thermal-kinetic image data, thereby enabling observers to more intuitively understand the human functional status of the tested individuals.

[0107] The following is a detailed explanation of how thermal-kinetic image data is generated, including the following steps:

[0108] Step 1: Simultaneously capture visible light and infrared images of the head and neck of a person who remains quasi-stationary, obtaining visible light video, infrared video, and multiple temperature matrices associated with the infrared video.

[0109] Preferably, the duration of the video recording can be 30 seconds to 180 seconds, but it is not limited to this.

[0110] In this first step, both a visible light camera and an infrared camera are used to photograph the head and neck of the person. This serves two purposes: firstly, to capture the kinetic energy changes of the head and neck under both visible light and infrared photoelectric sensing conditions, thereby obtaining richer and more detailed motion information; secondly, the use of an infrared camera allows for the further capture of temperature information about the head and neck.

[0111] In practical applications, in order to capture high-quality visible light and infrared video and extract more accurate data from it, we can improve the optical contrast of the background, enhance the quality of ambient lighting, ensure the stability of the camera installation, and set appropriate resolution and frame rate for the camera, etc.

[0112] Step 2: Perform subpixel-level registration of corresponding points between the visible light video and the infrared video.

[0113] Since visible light cameras and infrared cameras have parallax, registering the videos captured by both cameras using corresponding points can eliminate the detection error introduced by the parallax between the two cameras.

[0114] In actual registration, specific registration is performed on each pair of visible light and infrared images that correspond one-to-one on the video timeline.

[0115] In this step, sub-pixel level registration of corresponding points can be achieved using neural networks. For example, it can be achieved using... Figure 3 The improved MatchNet network shown here uses a LIFT (Learned Invariant Feature Transform) network to generate image feature descriptors and a metric network with three fully connected layers to measure the similarity between feature descriptors. Internally, the LIFT network sequentially performs feature point detection, softmax activation, image cropping, orientation estimation, and image rotation on the input image, and finally generates feature descriptors based on the rotated image. The registered image is obtained from the output of the image rotation module of the LIFT network.

[0116] By using LIFT networks to extract feature descriptors from images, we can uncover the invariant features of feature points in images under actual changing factors, and reduce the impact of interference factors such as illumination, background and human intervention on the registration accuracy of optical images.

[0117] The feature descriptors of the two registered images are fed into the metric network, which calculates the similarity between the two images. The activation function of the three fully connected layers of the metric network is ReLU (Linear rectification function).

[0118] The traditional approach to comparing the similarity between feature vectors is to use Euclidean distance. However, in complex manifold learning, Euclidean distance often fails to accurately measure the similarity between feature descriptors. Therefore, using this metric network to compare the similarity between feature descriptors can provide a more accurate similarity assessment of the results of sub-pixel-level registration of corresponding points.

[0119] Optionally, in one implementation, to extract more accurate data from the two types of videos, before performing corresponding point registration on the captured visible light and infrared images, inter-frame subpixel level corresponding point registration can be performed on both images. The implementation method for inter-frame subpixel level corresponding point registration can be found in the corresponding point registration method exemplified above, and will not be repeated in this embodiment.

[0120] Step 3: Based on the registered visible light video and infrared video, generate kinetic energy change data for each point on the head and neck of the person, and generate thermal energy change data for each point on the head and neck of the person based on multiple temperature matrices associated with the infrared video.

[0121] Step 4: Statistically analyze the kinetic energy change data to obtain the energy change cycle at various points on the head and neck of the person.

[0122] Ideally, capturing the movement of a person's head and neck would require knowing the vector displacement of every point on the head and neck in space. However, in reality, achieving this is impossible. This is because: firstly, standard modern computers are not fast enough to track the vector displacement of every point on an object using planar images; secondly, planar images captured by cameras lack depth information; even if depth estimation is achieved by using parallax between multiple cameras for multi-angle shooting, it is still difficult to reconstruct the vector displacement of every point on the object in three-dimensional space based on the coarsely estimated depth information.

[0123] To capture the movement of a person's head and neck, this embodiment of the invention records the position information 'a' of each point on the person's head and neck at multiple moments by shooting two different videos. It is understood that for each pixel in the image, if the person is completely still, the pixel value remains essentially unchanged; if the target point moves, the pixel value changes accordingly. Therefore, the change in pixel value in the video can be considered as the change in position information 'a' of each point on the person's head and neck. Since the frame rate is known, i.e., the frame interval T is known, the change in position information Δa of each point can be divided by T to obtain the velocity v of that point. Then, by assigning a virtual mass m to each pixel, the kinetic energy of each point can be calculated.

[0124] In the process of implementing the embodiments of the present invention, the inventors discovered that knowing the functional state of the human body does not actually require knowing the specific direction and type of movement of each point of the head and neck; therefore, the embodiments of the present invention use kinetic energy to describe the movement of the head and neck of the human object; since the calculation of kinetic energy is a scalar calculation, the computing speed of modern computers can meet the requirements, which provides a hardware foundation for the method provided by the embodiments of the present invention to run in real time.

[0125] Furthermore, since the infrared and visible light videos in this embodiment of the invention are captured under the condition that the person remains quasi-static, the movement of a person in a quasi-static state is determined by unconscious processes, mainly by the work of the vestibular system. Therefore, the changes in the pixel values ​​of the pixels in the video are mainly caused by the movement of the person's internal psychological factors or physiological diseases. This makes the kinetic energy change data obtained in this embodiment of the invention closely related to the person's internal state.

[0126] In step three, based on the registered visible light video and infrared video, kinetic energy change data for various points on the head and neck of the person are generated, including:

[0127] (1) By utilizing the image differences between adjacent frames in the registered visible light video, the visible light kinetic energy matrix of each point on the head and neck of the person at multiple times is estimated;

[0128] (2) By utilizing the image differences between adjacent frames in the registered infrared video, the infrared kinetic energy matrix of each point on the head and neck of the person at multiple times is estimated.

[0129] (3) Referring to the time axis of visible light video and infrared video, the visible light kinetic energy matrix and the infrared kinetic energy matrix are matched one by one, and each pair of visible light kinetic energy matrix and infrared kinetic energy matrix with corresponding relationship is fused to obtain multiple kinetic energy matrices, which serve as the kinetic energy change data of each point on the head and neck of the person.

[0130] For example, suppose a visible light video consists of M frames, sequentially labeled A1, A2, A3…A… M After performing registration with corresponding points, subtracting each pair of adjacent images in the M-frame visible light images yields M-1 difference matrices, namely: D1 = A2 - A1, D2 = A3 - A2, D3 = A4 - A3...D... M-1 =A M -A M-1 The elements in these difference matrices can represent the positional information 'a' of various points on the head and neck of the human object. Therefore, dividing these M-1 difference matrices by the frame interval T yields new matrices whose elements can represent the velocity 'v' of various points on the head and neck of the human object. These new matrices can be called velocity matrices, and are named V1, V2, V3…V M-1 Then, assuming the virtual mass m = 5, the M-1 visible light kinetic energy matrices can be calculated using the kinetic energy calculation formula shown above, denoted as I1, I2, I3…I… M-1 .

[0131] Similarly, by processing the infrared video according to the above process, M-1 infrared kinetic energy matrices can be obtained, denoted as J1, J2, J3…J… M-1 .

[0132] Then, each visible light kinetic energy matrix is ​​fused with its corresponding infrared kinetic energy matrix to obtain M-1 kinetic energy matrices. The specific fusion method is as follows: the element values ​​in the visible light kinetic energy matrix and the infrared kinetic energy matrix are normalized to the same dimension through normalization; then, for the same element location point, when the element values ​​in the two matrices are different, the larger value or the average value is taken as the fusion result.

[0133] In step three, based on multiple temperature matrices associated with infrared video, thermal energy change data for various points on the head and neck of the person are generated, including:

[0134] (1) Based on the time axis of the infrared video, sort the multiple temperature matrices associated with the infrared video;

[0135] (2) By using the difference between every two adjacent temperature matrices, estimate the thermal energy matrix of each point on the head and neck of the person at multiple times, and use it as the thermal energy change data of each point on the head and neck of the person.

[0136] For example, assuming an infrared video consists of M frames, then the infrared video is associated with M temperature matrices, ordered as T along the time axis of the infrared video. mp-1 T mp-2 T mp-3 …T mp-M By subtracting each pair of adjacent temperature matrices, we can obtain M-1 difference matrices, namely: X1 = T mp-2 -T mp-1 X2 = T mp-3 -T mp-2 X3 = T mp-4 -T mp-3 …X M-1 =T mp-M -T mp-M-1 The element values ​​in these difference matrices represent the temperature difference Δt corresponding to each pixel in the infrared image. Therefore, the thermal energy data corresponding to each pixel can be estimated using the thermal energy calculation formula Q=c×m×Δt, which can be used to characterize the thermal energy data of each point on the head and neck of the human object in the infrared image. Here, c is a virtual specific heat capacity assigned to the pixel, m is the virtual mass mentioned above, and Q is the estimated thermal energy.

[0137] In step four, the kinetic energy change data are statistically analyzed to obtain the energy change cycle at various points on the head and neck of the person, including:

[0138] (1) Perform element statistics on the multiple kinetic energy matrices obtained in step three to determine the number of times the most frequent element value appears at each element position of the kinetic energy matrix.

[0139] (2) Calculate the energy change cycle of each point on the head and neck of the person based on the determined number of occurrences of each element position point.

[0140] Specifically, the energy change cycle of a point is obtained by dividing the duration of the video recording by the number of times each point appears.

[0141] It is understandable that head and neck movements in a person under quasi-static conditions are mainly determined by internal psychological factors or physiological diseases. Since these micro-movements are not subject to conscious control, they usually exhibit a regularity. Therefore, this embodiment of the invention selects the energy change cycle of the element value that appears most frequently from multiple kinetic energy matrices to capture the regular movements of various points on the head and neck of the person, which best characterize the person's internal state.

[0142] It should be noted that, since the actual number of pixels in the image processed in the embodiments of the present invention is large, and the number of elements in the matrix processed is also matched with the number of pixels in the image, the size and number of images / matrices have been simplified in the examples above to make the scheme easier to understand.

[0143] The above is a detailed process for generating thermal-kinetic image data used in the embodiments of the present invention.

[0144] Optionally, in one implementation, the human functional state parameter extraction method provided in this embodiment of the invention may further include:

[0145] Based on the correspondence between kinetic energy change data, thermal energy change data, and the energy change cycle of various points on the head and neck of the subject and various points on the head and neck of the subject, these three data are synthesized and displayed to obtain a thermal-kinetic energy image; while displaying the thermal-kinetic energy image, various extracted human functional state parameters are also displayed.

[0146] There are multiple ways to synthesize and display kinetic energy change data, thermal energy change data, and the energy change cycle at various points in the head and neck.

[0147] For example, in one synthetic display method, a basic head and neck contour image can be generated using infrared images. Then, thermal energy change data is used to colorize this image, forming a color patch map or heat map, serving as a thermal image of the head and neck of the person. Next, based on the kinetic energy change data, the energy change period at each point, and the horizontal correspondence between the head and neck thermal image, the kinetic energy change data and the energy change period are plotted on both sides of the head and neck thermal image. For instance, the kinetic energy change data can be plotted as multiple colored straight lines extending horizontally on both sides of the head and neck thermal image. The length of the straight line drawn on the left side of the head and neck thermal image is determined based on the maximum element value of the left half of the same horizontal line in the kinetic energy matrix; here, L can be used. max (i) indicates that i is the row number; correspondingly, the color of the line depends on L. max (i) The corresponding energy change period; similarly, the length of the straight line drawn on the right half is determined by the maximum element value of the right half row on the same horizontal line in the kinetic energy matrix, here using R. max (i) indicates that the color of the corresponding line depends on R. max (i) The corresponding energy change period. Therefore, the thermal-kinetic image obtained based on this synthesis method can be as follows: Figure 4 As shown ( Figure 4 In practice, it is a color image.

[0148] Based on this Figure 4If a person's head and neck show a clear left-right asymmetry in the straight lines, it indicates that their bodily functions are not functioning normally. To determine the precise state of a person's bodily functions, one can check... Figure 4 The right side of the table provides the specific values ​​of various human function status parameters. In a preferred implementation, abnormal human function status parameters can be displayed in a differentiated manner, such as by displaying them in red font or highlighting them against the background.

[0149] Furthermore, since there are multiple kinetic and thermal energy matrices, each kinetic and thermal energy matrix can be plotted sequentially in the composite image, referring to the video's timeline. In this way, the final composite thermal-kinetic image actually presents a dynamic image.

[0150] It should be noted that the above-described synthetic display method is merely an example and does not constitute a limitation on the embodiments of the present invention; any implementation method that synthesizes and displays the data based on the spatial distribution correspondence of kinetic energy change data, thermal energy change data, and energy change cycle, thereby making the displayed data clearer, is applicable to the embodiments of the present invention.

[0151] Based on the human functional state parameter extraction method provided in this embodiment of the invention, this embodiment of the invention also provides a target population screening device, see [link to relevant documentation]. Figure 5 As shown, the device includes a data processing module 501 and a screening module 502.

[0152] The data processing module 501 is used to extract human functional state parameters related to the screening purpose from the thermal-kinetic image data of the person to be screened; the thermal-kinetic image data is the data used to generate thermal-kinetic images; the thermal-kinetic image data includes: kinetic energy change data, thermal energy change data and energy change cycle of each point of the head and neck of the person.

[0153] The human functional status parameters related to the screening purpose may include one or more of the following: aggression index, stress index, anxiety index, balance index, confidence index, vitality index, and inhibition index. Furthermore, they are not necessarily limited to these parameters.

[0154] Regarding the method by which the data processing module 501 extracts human functional state parameters, please refer to the definition formulas of each human functional state parameter in the above method embodiment, which will not be repeated here.

[0155] It is understandable that the relevant human functional status parameters may differ depending on the screening objective. For example, when the screening objective is to screen for ADHD, the relevant human functional status parameters may include: aggression index, stress index, confidence index, and vitality index.

[0156] It should be noted that the screening purpose in the embodiments of the present invention is not limited to those listed above, and may also include screening for other single physiological diseases, etc.

[0157] To make the plan clear, the following examples illustrate the methods for determining relevant human functional status parameters for different screening purposes.

[0158] First, the screening objective is determined, i.e., the target population category is identified. Then, thermal-kinetic imaging data are collected from a large number of both the target and non-target populations, and a complete set of human functional status parameters is extracted. Next, for each human functional status parameter, a T-test is used to analyze the extracted parameters from the two groups. The p-value of the T-test result determines whether the human functional status parameter is related to the target population; specifically, the smaller the p-value, the more significant the difference in that parameter between the two groups, and correspondingly, the stronger the correlation between that parameter and the screening objective.

[0159] For example, assuming the screening objective is to screen for ADHD, thermal-kinetic imaging data is collected from a large number of individuals with ADHD and normal individuals, and a complete set of human functional state parameters is extracted. Then, the extracted parameters are analyzed using a T-test, and the results are shown in the table below:

[0160]

[0161] As shown in the table above, the p-values ​​(shown in bold) for the Aggression Index, Stress Index, Confidence Index, and Vitality Index are all less than 5%, indicating that these four parameters show significant differences between individuals with ADHD and those without ADHD at the 5% significance level. Therefore, these four parameters can be selected as parameters of human functional status related to screening individuals with ADHD.

[0162] The screening module 502 is used to calculate the human function status score of the person based on the difference between the various human function status parameters extracted by the data processing module and the corresponding screening standards, and to determine whether the person belongs to the target population based on the human function status score.

[0163] The calculation method for the human functional status score can be expressed as follows:

[0164]

[0165] In this formula, v i This represents the measured value of the i-th human functional state parameter, which is the value actually extracted from the thermal-kinetic image data. The screening standard for this human functional status parameter is based on the statistical average obtained from extracting this parameter from a large number of target populations; P is the total number of human functional status parameters related to the screening purpose; K is the calculated human functional status score. When K is lower than the reference value, it indicates that the individual is similar to the target population, and the individual can be determined to belong to the target population; otherwise, the individual is determined not to belong to the target population.

[0166] Optionally, such as Figure 6 As shown, the target population screening device provided in this embodiment of the invention may further include: a data acquisition module 500;

[0167] The data acquisition module 500 is used to collect video and temperature data of the head and neck of personnel.

[0168] like Figure 6 As shown, the data acquisition module 500 is associated with a visible light camera and an infrared camera; thus, the data acquisition module 500 can acquire video data and temperature data through these two cameras.

[0169] Correspondingly, the data processing module 501 can also be used to generate thermal-kinetic image data of personnel objects based on video data and temperature data.

[0170] For details on how the data processing module 501 generates thermal-kinetic image data, please refer to the corresponding descriptions in the above method embodiments, which will not be repeated here.

[0171] In practical applications, the data processing module 501 and the screening module 502 can be loaded into a processor. The processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0172] The aforementioned data acquisition module 500 can be a device capable of performing infrared and visible light imaging, such as a commercially available camera or infrared thermal imager.

[0173] Figure 7 and Figure 8 An example is shown Figure 6 A physical diagram of the device shown. Figure 7 This is a front view of the physical device. Figure 8 This is a side view of the physical device. The camera shown in the image could be a visible light camera or an infrared camera. Another type of camera is not shown as it is obscured by this camera. The processor is hidden inside the main unit. The monitor displays thermal-kinetic image data and other information. The control panel may include a keyboard, mouse, and hooks. A supplementary light is mounted on a rotating robotic arm to provide illumination during filming, thereby improving the quality of the captured video.

[0174] Understandable, Figure 7 and Figure 8 The device shown is merely an example of a target population screening device provided in the embodiments of the present invention and does not constitute a limitation on the embodiments of the present invention.

[0175] It should be noted that, for the device embodiment, since it is largely similar to the method embodiment, the relevant parts are described in a relatively simple manner. For specific implementation methods, please refer to the description of the corresponding parts in the method embodiment.

[0176] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0177] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0178] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In this specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. While certain measures are described in different embodiments, this does not mean that these measures cannot be combined to produce good results.

[0179] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for extracting human functional state parameters based on thermal-kinetic image data, characterized in that, include: Acquire thermal-kinetic image data of a person; the thermal-kinetic image data is used to generate thermal-kinetic images; wherein, the thermal-kinetic image data includes: kinetic energy change data, thermal energy change data, and energy change cycle of each point on the head and neck of the person; According to the definition formula of the human functional state parameters to be extracted, the human functional state parameters are extracted from the thermal-kinetic image data. The human body functional state parameters include the following types of human body functional state parameters: Aggression index, stress index, anxiety index, balance index, confidence index, vitality index, and inhibition index; The formula for defining the aggression index is as follows: ； in, This indicates the aggression index; Let be the probability density function of the energy change period at each point in the head and neck. Indicates the period of energy change. ; This is the smallest period among the energy change periods at various points in the head and neck. The maximum period among the energy change periods at each point in the head and neck; For each point in the head and neck, the energy change period is... The number of points; for Expected value; for The maximum value; A is a preset constant, 1≤A≤10.

2. The method according to claim 1, characterized in that, The kinetic energy change data includes multiple kinetic energy matrices, and the thermal energy change data includes multiple thermal energy matrices; the energy change period of each point in the head and neck is contained in a period matrix. The formula for defining the pressure index is: ； in, This indicates the pressure index; This refers to the kinetic energy component of the pressure index. The thermal energy sub-index is the pressure index; the number of rows in the kinetic energy matrix, the thermal energy matrix, and the periodic matrix are all [missing information]. N； To determine the first kinetic energy matrix of a target n The kinetic energy difference between the left and right sides of the head and neck of the person being measured by row element calculation; the target kinetic energy matrix is ​​taken from the plurality of kinetic energy matrices; To be based on the periodic matrix of the first n The energy change cycle difference value between the left and right sides of the head and neck of the person being calculated using row elements; According to the first The first thermal matrix n The difference in thermal energy between the left and right sides of the head and neck of the person being measured by the row element.

3. The method according to claim 1, characterized in that, The formula for defining the anxiety index is: ； in, This indicates the anxiety index. Let be the probability density function of the energy change period at each point in the head and neck. Indicates the period of energy change. , B is the minimum period among the energy change periods at each point in the head and neck, and B is a preset constant, where 1 < B < 5.

4. The method according to claim 1, characterized in that, The formula for defining the balance index is: ； in, This represents the balance index; Let be the probability density function of the energy change period at each point in the head and neck. Indicates the period of energy change. ; This refers to the smallest period among the energy change periods at various points on the head and neck of the described person. The maximum period among the energy change periods at various points on the head and neck of the person being described. Let be the standard normal probability density function of the energy change period at each point in the head and neck.

5. The method according to claim 1, characterized in that, The kinetic energy change data includes multiple kinetic energy matrices; the thermal energy change data includes multiple thermal energy matrices; The formula for defining the confidence index is: ； in, This indicates the confidence index. This refers to the kinetic energy sub-index within the confidence index. This refers to the thermal energy sub-index within the confidence index. Denotes the first of any of the aforementioned kinetic energy matrices n The kinetic energy data for the left half of the row. This represents the first kinetic energy matrix. n The kinetic energy data in the right half of the row. Denotes the first of any of the aforementioned thermal energy matrices n The thermal energy data in the left half of the row. This represents the first element of the thermal energy matrix. n The thermal energy data in the right half of the row. The number of rows in both the kinetic energy matrix and the thermal energy matrix is ​​[number missing]. N。 6. The method according to claim 1, characterized in that, The formula for defining the vitality index is: ； in, This indicates the vitality index. Let be the probability density function of the energy change period at each point in the head and neck. Indicates the period of energy change. ; This is the smallest period among the energy change periods at various points in the head and neck. The maximum period among the energy change periods at each point in the head and neck; For each point in the head and neck, the energy change period is... The number of points; for The maximum value; The standard deviation of the energy change period at each point in the head and neck is represented.

7. The method according to claim 1, characterized in that, The formula for defining the inhibition index is: ； in, This represents the inhibition index. I This indicates that the energy change period at various points on the head and neck of the person being described has... I The energy change period at each point is less than the preset reference threshold. Indicates the I The first point in the nth point i The energy change cycle at each point.

8. A target population screening device, characterized in that, include: Data processing module and screening module; The data processing module is used to extract human functional state parameters related to the screening purpose from the thermal-kinetic image data of the person to be screened; the thermal-kinetic image data is the data used to generate thermal-kinetic images; wherein, the thermal-kinetic image data includes: kinetic energy change data, thermal energy change data, and energy change cycle of each point of the head and neck of the person; The screening module is used to calculate the human function status score of the person based on the difference between the various human function status parameters extracted by the data processing module and the corresponding screening standards, and to determine whether the person belongs to the target population based on the human function status score. The human body functional state parameters include the following types of human body functional state parameters: Aggression index, stress index, anxiety index, balance index, confidence index, vitality index, and inhibition index; The formula for defining the aggression index is as follows: ； in, This indicates the aggression index; Let be the probability density function of the energy change period at each point in the head and neck. Indicates the period of energy change. ; This is the smallest period among the energy change periods at various points in the head and neck. The maximum period among the energy change periods at each point in the head and neck; For each point in the head and neck, the energy change period is... The number of points; for Expected value; for The maximum value; A is a preset constant, 1≤A≤10.

9. The target population screening device according to claim 8, characterized in that, Also includes: Data acquisition module; The data acquisition module is used to acquire video data and temperature data of the head and neck of the person being observed. The data processing module is further configured to generate thermal-kinetic image data of the person based on the video data and the temperature data.

Images