Thermal-energy-kinetic-energy image data generation method and device, and human body state evaluation method

Abstract

The application discloses a kind of thermal energy-kinetic energy image data generation method, device, human state evaluation method and electronic equipment;The generation method includes: the head and neck of personnel object that keeps quasi-stationary is synchronously photographed visible light video and infrared video, simultaneously obtains multiple temperature matrices associated with infrared video;Subpixel level of visible light video and infrared video is registered with the same name point;Based on the visible light video and infrared video after registration, generate the kinetic energy change data of each point of head and neck, and based on multiple temperature matrices, generate the thermal energy change data of each point of head and neck;Kinetic energy change data is statistically processed, and the movement cycle of each point of head and neck is obtained;Kinetic energy change data, thermal energy change data and movement cycle are generated as a group of thermal energy-kinetic energy image data, to generate thermal energy-kinetic energy image based on thermal energy-kinetic energy image data.The application can obtain more comprehensive reference information for psychological state evaluation or physiological state evaluation of human body.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of data generation, and particularly relates to a thermal energy-kinetic energy image data generation method and device and electronic equipment. BACKGROUND

[0002] Traditional systems for detecting the psychological state or physiological state of a person are mostly based on the stability or time variability of specific physiological parameters, such as sports bands and lie detectors. The physiological parameters mentioned include electrocardiogram, electroencephalogram, and galvanic skin response, etc.

[0003] However, the above-mentioned physiological parameters are only detected from the microscopic perspective of biological discharge to detect the human body from a narrow perspective, and the information obtained is one-sided. SUMMARY

[0004] In order to solve the above-mentioned problems in the prior art, the present application provides a thermal energy-kinetic energy image data generation method, device and human state evaluation method. The technical problem to be solved by the present application is solved by the following technical scheme:

[0005] In a first aspect, the present application provides a thermal energy-kinetic energy image data generation method, comprising:

[0006] Synchronously performing visible light shooting and infrared shooting on the head and neck of a person object in a quasi-static state to obtain a visible light video, an infrared video, and a plurality of temperature matrices associated with the infrared video;

[0007] Performing sub-pixel level homonymy point registration on the visible light video and the infrared video;

[0008] Based on the registered visible light video and infrared video, generating kinetic energy change data of each point of the head and neck of the person object, and based on the plurality of temperature matrices, generating thermal energy change data of each point of the head and neck of the person object;

[0009] Statistically analyzing the kinetic energy change data to obtain a motion cycle of each point of the head and neck of the person object;

[0010] Taking the kinetic energy change data, the thermal energy change data, and the motion cycle as a set of generated thermal energy-kinetic energy image data, and generating a thermal energy-kinetic energy image based on the thermal energy-kinetic energy image data.

[0011] Optionally, the method further comprises:

[0012] Before performing sub-pixel level homonymy point registration on the visible light video and the infrared video, performing inter-frame sub-pixel level homonymy point registration on the visible light image and the infrared image.

[0013] Optionally, the homonym point registration is realized based on a pre-trained improved MatchNet network; the improved MatchNet network comprises a first feature extraction module, a second feature extraction module and a metric module; wherein

[0014] The first feature extraction module is configured to extract a first feature descriptor from a reference image;

[0015] The second feature extraction module is configured to perform image transformation on a target image based on the reference image to obtain a transformed target image, and to extract a second feature descriptor from the transformed target image;

[0016] The metric module is configured to calculate a matching score of the first feature descriptor and the second feature descriptor;

[0017] The reference image and the target image are two images to be registered, the transformed target image is matched with the reference image at homonym points, and the matching score is not less than a preset lower limit;

[0018] The second feature extraction module is a LIFT network, and the second feature extraction module is obtained by modifying another LIFT network; the feature dimension of the first feature descriptor is the same as that of the second feature descriptor;

[0019] The metric module is a metric sub-network in the MatchNet network.

[0020] Optionally, based on the registered visible light video and infrared video, the kinetic energy change data of each point of the head and neck of the personnel object is generated, comprising:

[0021] The image difference between adjacent frames in the registered visible light video is used to estimate the first kinetic energy matrix of each point of the head and neck of the personnel object at multiple time points;

[0022] The image difference between adjacent frames in the registered infrared video is used to estimate the second kinetic energy matrix of each point of the head and neck of the personnel object at multiple time points;

[0023] The first kinetic energy matrix and the second kinetic energy matrix are corresponded one by one with reference to the time axis of the visible light video and the infrared video, and each pair of first kinetic energy matrix and second kinetic energy matrix with corresponding relationship is fused to obtain a plurality of third kinetic energy matrices as the kinetic energy change data of each point of the head and neck of the personnel object.

[0024] Optionally, the kinetic energy change data is counted to obtain the motion period of each point of the head and neck of the personnel object, comprising:

[0025] performing element statistics on the plurality of third kinetic energy matrices to determine the frequency of occurrence of the element value with the highest frequency of occurrence at each element position point of the third kinetic energy matrix;

[0026] calculating the motion cycle of each point of the head and neck of the personnel object according to the frequency of occurrence corresponding to each element position point.

[0027] Optionally, based on the plurality of temperature matrices, thermal energy change data of each point of the head and neck of the personnel object is generated, including:

[0028] sorting the plurality of temperature matrices with reference to the time axis of the infrared video;

[0029] estimating thermal energy matrices of each point of the head and neck of the personnel object at multiple time points by using the difference between each two adjacent temperature matrices as the thermal energy change data of each point of the head and neck of the personnel object.

[0030] Optionally, the manner of generating the thermal energy-kinetic energy image based on the thermal energy-kinetic energy image data includes:

[0031] generating a thermal energy image of the head and neck of the personnel object based on the thermal energy change data;

[0032] based on the corresponding relationship in spatial distribution of the kinetic energy change data, the motion cycle and the thermal energy image of the head and neck, the three are synthesized and displayed to obtain a thermal energy-kinetic energy image.

[0033] In a second aspect, the present application provides a thermal energy-kinetic energy image data generation device, including:

[0034] a shooting module for synchronously performing visible light shooting and infrared shooting on the head and neck of the personnel object kept in a quasi-stationary state to obtain a visible light video, an infrared video and a plurality of temperature matrices associated with the infrared video;

[0035] a registration module for performing sub-pixel level same-name point registration on the visible light video and the infrared video;

[0036] a first processing module for generating kinetic energy change data of each point of the head and neck of the personnel object based on the registered visible light video and infrared video, and generating thermal energy change data of each point of the head and neck of the personnel object based on the plurality of temperature matrices;

[0037] a second processing module for performing statistics on the kinetic energy change data to obtain the motion cycle of each point of the head and neck of the personnel object;

[0038] generating a set of thermal-kinetic image data by combining the thermal-kinetic image data with the kinetic data and the thermal data.

[0039] Optionally, the apparatus further comprises an inter-frame registration module.

[0040] The inter-frame registration module is configured to perform inter-frame sub-pixel level co-name point registration on both the visible light images and the infrared images before performing sub-pixel level co-name point registration on the visible light video and the infrared video.

[0041] Optionally, the co-name point registration is implemented based on a pre-trained improved MatchNet network; the improved MatchNet network comprises a first feature extraction module, a second feature extraction module and a measurement module; wherein,

[0042] The first feature extraction module is configured to extract first feature descriptors from a reference image.

[0043] The second feature extraction module is configured to perform image transformation on a target image based on the reference image to obtain a transformed target image, and extract second feature descriptors from the transformed target image.

[0044] The measurement module is configured to calculate a matching score of the first feature descriptors and the second feature descriptors.

[0045] The reference image and the target image are two images to be registered, the transformed target image is co-name point matched with the reference image, and the matching score is not lower than a preset lower limit.

[0046] The second feature extraction module is a LIFT network, and the second feature extraction module is obtained by modifying another LIFT network; the feature dimension of the first feature descriptors is the same as the feature dimension of the second feature descriptors.

[0047] The measurement module is a measurement sub-network in the MatchNet network.

[0048] Optionally, the first processing module generates kinetic data of each point of the head and neck of the personnel object based on the registered visible light video and infrared video, comprising:

[0049] The first kinetic matrix of each point of the head and neck of the personnel object at multiple time points is estimated by using the image difference between adjacent frames in the registered visible light video.

[0050] The second moment of inertia matrix of each point of the head and neck of the personnel object at multiple time points is estimated by using the image difference between adjacent frames in the registered infrared video;

[0051] The first moment of inertia matrix and the second moment of inertia matrix are corresponded one by one with reference to the time axis of the visible light video and the infrared video, and each pair of the first moment of inertia matrix and the second moment of inertia matrix with the corresponding relationship is fused to obtain multiple third moment of inertia matrices as the kinetic energy change data of each point of the head and neck of the personnel object.

[0052] Optionally, the second processing module is specifically configured to:

[0053] The element statistics are performed on the multiple third moment of inertia matrices to determine the occurrence frequency of the element value with the highest occurrence frequency at each element position point of the third moment of inertia matrix;

[0054] The motion cycle of each point of the head and neck of the personnel object is calculated according to the occurrence frequency corresponding to each element position point.

[0055] Optionally, the first processing module generates the thermal energy change data of each point of the head and neck of the personnel object based on the multiple temperature matrices, including:

[0056] The multiple temperature matrices are sorted with reference to the time axis of the infrared video;

[0057] The thermal energy matrix of each point of the head and neck of the personnel object at multiple time points is estimated by using the difference between each two adjacent temperature matrices as the thermal energy change data of each point of the head and neck of the personnel object.

[0058] Optionally, the generation module generates the thermal energy-kinetic energy image in the following manner based on the thermal energy-kinetic energy image data:

[0059] The head and neck thermal energy image of the personnel object is generated based on the thermal energy change data;

[0060] The thermal energy-kinetic energy image is obtained by synthesizing and displaying the thermal energy change data, the motion cycle and the head and neck thermal energy image based on the corresponding relationship among the three in the spatial distribution.

[0061] In a third aspect, the present application provides a human body state evaluation method, including:

[0062] The thermal energy-kinetic energy image data and / or the thermal energy-kinetic image generated by any one of the thermal energy-kinetic image data generation methods are obtained;

[0063] The psychological state or physiological state of the personnel object is evaluated based on the thermal energy-kinetic energy image and / or the thermal energy-kinetic image.

[0064] The application also provides an electronic device, comprising: an infrared camera, a visible light camera, a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0065] The memory is used for storing a computer program, and the processor is used for executing the program stored on the memory to realize the method steps of any one of the above-mentioned heat-energy image data generation methods in cooperation with the infrared camera and the visible light camera.

[0066] The application has the following beneficial effects:

[0067] In the heat-energy image data generation method provided by the application, visible light images and infrared images are synchronously captured in the head and neck of a static personnel object, so that the kinetic energy change data of each point of the head and neck of the personnel object is obtained from the visible light and infrared photoelectric sensing conditions, and more detailed micro-movement information of the head and neck can be captured. In addition, the temperature matrix of the infrared video is used to capture the heat energy change data of each point of the head and neck of the personnel object. Since a person will have uncontrollable autonomic primary tension when nervous and anxious, the movement of the head and neck and the change of the local temperature of the body will be caused under the reflection of the vestibular system. Therefore, the heat-energy image data generated in the application captures the movement of the head and neck and the change of the temperature from the heat energy and the kinetic energy, and the obtained information is more comprehensive. Moreover, the two kinds of information are generated at the same time and can be mutually referenced, so that the accuracy of subsequent human state evaluation and analysis based on the heat-energy image data can be improved.

[0068] In addition, the same point registration of the sub-pixel level is performed on the captured visible light video and infrared video, so that the data error caused by the parallax of the two cameras can be eliminated, and the generated kinetic energy change data is more accurate.

[0069] Compared with the information collection methods such as electrocardiogram, electroencephalogram and galvanic skin response in the prior art, the application realizes a non-contact data extraction scheme, which can reduce the anxiety of the collected object caused by the body contact with the collection instrument and reduce the error introduced by the collection event itself.

[0070] In addition, the kinetic energy change data, the heat energy change data and the movement cycle of each point of the head and neck of the personnel object when performing periodic movement are displayed in the form of images, so that the viewer can more intuitively view the data of the personnel object.

[0071] The application will be further described in detail below in combination with the drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0072] Figure 1 is a flow diagram of a heat-energy kinetic image data generation method provided by an embodiment of the present application;

[0073] Figure 2 is a diagram illustrating an example of a third kinetic matrix according to the method provided by the embodiment of the present application;

[0074] Figure 3 is a diagram illustrating a statistic of the third kinetic matrix according to the method provided by the embodiment of the present application;

[0075] Figure 4 is a diagram illustrating a synthetic image generated by the embodiment of the present application;

[0076] Figure 5 is a diagram illustrating another synthetic image generated by the embodiment of the present application;

[0077] Figure 6 is a real effect diagram of the synthetic image shown in Figure 5

[0078] Figure 7 is a flow diagram of another heat-energy kinetic image data generation method provided by the embodiment of the present application;

[0079] Figure 8 is a structure diagram of an improved MatchNet network used by the embodiment of the present application;

[0080] Figure 9 is a structure diagram of a heat-energy kinetic image data generation device provided by the embodiment of the present application;

[0081] Figure 10 is a structure diagram of an electronic device provided by the embodiment of the present application;

[0082] Figure 11 is a front view of the electronic device shown in Figure 10

[0083] is a side view of the electronic device shown in Figure 12 Figure 10 DETAILED DESCRIPTION

[0084] The present application will be further described in detail below with reference to specific embodiments, but the embodiments of the present application are not limited thereto.

[0085] ​​​To obtain more comprehensive reference information for assessing the psychological or physiological state of the human body, embodiments of the present invention provide a method, apparatus, and electronic device for generating thermal-kinetic image data. The executing entity of the thermal-kinetic image data generation method provided in this embodiment can be a thermal-kinetic image data generation apparatus provided in this embodiment; this apparatus can be applied to an electronic device, specifically as a software function module enabled in the electronic device. In practical applications, any electronic device associated with a visible light camera and an infrared camera capable of simultaneous shooting and possessing computational processing capabilities can serve as the electronic device provided in this embodiment; by loading the aforementioned apparatus onto such an electronic device, the thermal-kinetic image data generation method provided in this embodiment can be implemented within that electronic device.

[0086] The vertical balance of the human head is controlled by the vestibular system, a function known as the vestibular reflex. Head movements are the most frequent repetitive movements throughout a person's life. The head makes continuous, minute movements until it finds support, at which point it stops adjusting. During this adjustment process, numerous minute movements at the scale of hundreds of micrometers are performed every second.

[0087] There is a direct link between human physical state and bodily movement. For example, when a person is calm and at rest, their heart rate and respiration are at their lowest; while when a person is excited, their breathing and heart rate increase. Similarly, when a person is tense, anxious, or suffering from illness, they experience uncontrollable primary tension, which, under the reflex of the vestibular system, leads to subtle movements of the head and neck and changes in temperature. Therefore, by collecting and analyzing data on these changes, we can gain some insight into a person's psychological or physiological state.

[0088] See Figure 1 As shown, the thermal-kinetic image data generation method provided in this embodiment of the invention includes the following steps:

[0089] S1: Simultaneously capture visible light and infrared images of the head and neck of a person who remains quasi-stationary, obtaining visible light video, infrared video, and multiple temperature matrices associated with the infrared video.

[0090] Understandably, "quasi-static" refers to a relatively static state maintained when a person is required to keep their head and neck still. In this state, visible light cameras and infrared cameras can be used to capture the movement of the person's head and neck.

[0091] Preferably, the duration of the video captured in step S1 can be 30 seconds to 180 seconds, but it is not limited to this.

[0092] In this embodiment of the invention, both a visible light camera and an infrared camera are used to photograph the head and neck of a person. This is done to capture the kinetic energy changes of the head and neck under both visible light and infrared photoelectric sensing conditions. It is understood that visible light and infrared light operate in different wavelengths; the former is visible to the naked eye, while the latter is invisible. The photoelectric sensing capabilities of optical systems operating in different wavelengths differ. Therefore, using two different optical cameras to photograph the person allows for the capture of more detailed kinetic energy change data. Furthermore, the use of an infrared camera allows for the capture of temperature information about the head and neck, enabling the detection of both kinetic and thermal energy in the person. This provides richer reference data for subsequent analysis and evaluation based on the detected data.

[0093] In addition, in practical applications, in order to capture high-quality visible light and infrared video and extract more accurate data from them, it is necessary to improve the optical contrast of the background, enhance the quality of ambient lighting, ensure the stability of the camera installation, and set appropriate resolution and frame rate for the camera, etc.

[0094] S2: Perform subpixel-level registration of corresponding points for visible light video and infrared video.

[0095] Understandably, since visible light cameras and infrared cameras have parallax, registering the videos captured by both cameras using corresponding points can eliminate the detection error introduced by the parallax between the two cameras.

[0096] It should be noted that the embodiments of the present invention do not strictly limit the positional relationship between the person and the two types of cameras. It is sufficient that the two types of cameras are placed close to each other, and one of the cameras is directly shooting at the head and neck of the person.

[0097] Since the visible light camera and the infrared camera capture images simultaneously, the multiple frames of visible light images in the visible light video and the multiple frames of infrared images in the infrared video correspond one-to-one on the timeline. Therefore, performing sub-pixel-level registration of corresponding points on the simultaneously captured visible light video and infrared video means registering corresponding points for every two corresponding frames of visible light and infrared images on the timeline.

[0098] In this step, sub-pixel-level registration of corresponding points can be achieved using various neural networks. To ensure clarity in the specification, examples of specific implementation methods for sub-pixel-level registration of corresponding points using neural networks will be provided later.

[0099] S3: Based on the registered visible light video and infrared video, generate kinetic energy change data for each point on the head and neck of the person, and generate thermal energy change data for each point on the head and neck of the person based on multiple temperature matrices associated with the infrared video.

[0100] Ideally, capturing the movement of a person's head and neck would require knowing the vector displacement of every point on the head and neck in space. However, in reality, achieving this is impossible. This is because: firstly, standard modern computers are not fast enough to track the vector displacement of every point on an object using planar images; secondly, planar images captured by cameras lack depth information; even if depth estimation is achieved by using parallax between multiple cameras for multi-angle shooting, it is still difficult to reconstruct the vector displacement of every point on the object in three-dimensional space based on the coarsely estimated depth information.

[0101] To capture the movement of a person's head and neck, this embodiment of the invention records the position information 'a' of each point on the person's head and neck at multiple moments by shooting two different videos. It is understood that for each pixel in the image, if the person is completely still, the pixel value remains essentially unchanged; if the target point moves, the pixel value changes accordingly. Therefore, the change in pixel value in the video can be considered as the change in position information 'a' of each point on the person's head and neck. Since the frame rate is known, i.e., the frame interval T is known, the change in position information Δa of each point can be divided by T to obtain the velocity v of that point. Then, by assigning a virtual mass m to each pixel, the kinetic energy of each point can be calculated.

[0102] In the process of implementing the embodiments of the present invention, the inventors discovered that it is not necessary to know the specific direction and type of movement of each point of the head and neck to analyze and evaluate the human body state. Therefore, the embodiments of the present invention use kinetic energy to describe the movement of the head and neck of the human subject. Furthermore, since the calculation of kinetic energy is a scalar calculation, the computing speed of modern computers can meet the requirements, which provides a hardware foundation for the method provided by the embodiments of the present invention to run in real time.

[0103] Furthermore, since the infrared and visible light videos in this embodiment of the invention are captured under the condition that the person remains quasi-static, the movement of a person in a quasi-static state is determined by unconscious processes, mainly by the work of the vestibular system. Therefore, the changes in the pixel values ​​of the pixels in the video are mainly caused by the movement of the person's internal psychological factors or physiological diseases. This makes the kinetic energy change data obtained in this embodiment of the invention closely related to the person's internal state.

[0104] In step S3, based on the registered visible light video and infrared video, kinetic energy change data for various points on the head and neck of the person are generated, including:

[0105] (1) Using the image differences between adjacent frames in the registered visible light video, estimate the first kinetic energy matrix of each point on the head and neck of the person at multiple times;

[0106] (2) By utilizing the image differences between adjacent frames in the registered infrared video, the second kinetic energy matrix of each point on the head and neck of the person at multiple times is estimated;

[0107] (3) Referring to the time axis of visible light video and infrared video, the first kinetic energy matrix and the second kinetic energy matrix are matched one by one, and each pair of first kinetic energy matrix and second kinetic energy matrix with corresponding relationship is fused to obtain multiple third kinetic energy matrices, which serve as the kinetic energy change data of each point on the head and neck of the person.

[0108] For example, suppose a visible light video consists of M frames, sequentially labeled A1, A2, A3…A… M After performing registration with corresponding points, subtracting each two adjacent frames in the M-frame visible light images yields M-1 difference matrices, namely: D1 = A2 - A1, D2 = A3 - A2, D3 = A4 - A3...D... M-1 =A M -A M-1 The elements in these difference matrices can represent the positional information 'a' of various points on the head and neck of the human object. Therefore, dividing these M-1 difference matrices by the frame interval T yields new matrices whose elements can represent the velocity 'v' of various points on the head and neck of the human object. These new matrices can be called velocity matrices, and are named V1, V2, V3…V M-1 Then, assuming the virtual mass m = 5, the M-1 first kinetic energy matrices can be calculated using the kinetic energy calculation formula shown above, denoted as I1, I2, I3…I… M-1 .

[0109] Similarly, by processing the infrared video according to the above process, M-1 second kinetic energy matrices can be obtained, denoted as J1, J2, J3…J… M-1 .

[0110] Then, by fusing each first kinetic energy matrix with its corresponding second kinetic energy matrix, we can obtain M-1 third kinetic energy matrices, namely: Q1=I1&J1, Q2=I2&J2, Q3=I3&J3, ...Q M-1 =I M-1 &J M-1The symbol “&” here signifies fusion. The specific fusion method is as follows: The element values ​​in the first and second kinetic energy matrices are normalized to the same dimension through normalization; then, for the same element location point, when the element values ​​in the two matrices are different, the larger value or the average value is taken as the fusion result.

[0111] In step S3, based on multiple temperature matrices associated with the infrared video, thermal energy change data for various points on the head and neck of the person are generated, including:

[0112] (1) Based on the time axis of the infrared video, sort the multiple temperature matrices associated with the infrared video;

[0113] (2) By using the difference between every two adjacent temperature matrices, estimate the thermal energy matrix of each point on the head and neck of the person at multiple times, and use it as the thermal energy change data of each point on the head and neck of the person.

[0114] For example, assuming an infrared video consists of M frames, then the infrared video is associated with M temperature matrices, ordered as T along the time axis of the infrared video. mp-1 T mp-2 T mp-3 …T mp-M By subtracting each pair of adjacent temperature matrices, we can obtain M-1 difference matrices, namely: X1 = T mp-2 -T mp-1 X2 = T mp-3 -T mp-2 X3 = T mp-4 -T mp-3 …X M-1 =T mp-M -T mp-M-1 The element values ​​in these difference matrices represent the temperature difference Δt corresponding to each pixel in the infrared image. Therefore, the thermal energy data corresponding to each pixel can be estimated using the thermal energy calculation formula Q=c×m×Δt, which can be used to characterize the thermal energy data of each point on the head and neck of the human object in the infrared image. Here, c is a virtual specific heat capacity assigned to the pixel, m is the virtual mass mentioned above, and Q is the estimated thermal energy.

[0115] After obtaining the kinetic energy change data and thermal energy change data, you can proceed to step S4 to further process the data in order to extract more useful information.

[0116] S4: Statistically analyze the kinetic energy change data to obtain the motion cycle of each point on the head and neck of the person.

[0117] Step S4 specifically includes the following sub-steps:

[0118] (1) Perform element statistics on the multiple third kinetic energy matrices obtained in step S3 to determine the frequency of the most frequent element value at each element position point of the third kinetic energy matrix.

[0119] (2) Calculate the motion cycle of each point on the head and neck of the person based on the frequency of occurrence of each element position.

[0120] It is understandable that head and neck movements in a person under quasi-static conditions are mainly determined by internal psychological factors or physiological diseases. Since these micro-movements are not subject to conscious control, they usually exhibit a regularity. Therefore, this embodiment of the invention selects the frequency of occurrence of the element value with the highest occurrence frequency from multiple third kinetic energy matrices to capture the regular movements of various points on the head and neck of the person, which best characterize the person's internal state.

[0121] After extracting the frequency of the most frequent element value, the motion cycle of each point in the head and neck can be calculated separately. For example, suppose that in step S3, the following was generated: Figure 2 The diagram shows six third kinetic energy matrices; where a, b, and c represent different element values. Then, element statistics are performed on these six third kinetic energy matrices; the statistical results of the most frequent element value at each element location are shown below. Figure 3 As shown in the diagram, the element with the highest frequency of occurrence at row 4, column 3 is value 'b', which appears 4 times. Assuming the entire video recording lasts 5 seconds, then the element value 'b' appears a total of 4 times within those 5 seconds. Therefore, the period of the regular movement corresponding to this element's position is 5 / 4 = 1.25 seconds.

[0122] It should be noted that, since the actual number of pixels in the image processed in the embodiments of the present invention is large, and the number of elements in the matrix processed is also matched with the number of pixels in the image, the size and number of images / matrices have been simplified in the examples above to make the scheme easier to understand.

[0123] S5: Use the kinetic energy change data, thermal energy change data, and calculated motion cycle as a set of thermal-kinetic energy image data to generate a thermal-kinetic energy image based on the thermal-kinetic energy image data.

[0124] The methods for generating thermal-kinetic images based on thermal-kinetic image data may include:

[0125] (1) Generate thermal images of the head and neck of human subjects based on thermal energy change data;

[0126] (2) Based on the spatial distribution of kinetic energy change data, motion cycle and head and neck thermal image, these three are combined and displayed to obtain thermal-kinetic image.

[0127] Specifically, generating a thermal image of the head and neck of a person based on thermal energy change data involves extracting thermal energy data of the head and neck of the person from each thermal energy matrix, and then generating a thermal image of the head and neck of the person based on the extracted data. In practice, the generated thermal image of the head and neck can be a color patch map or a heat map.

[0128] Understandably, there are multiple ways to synthesize and display kinetic energy change data, motion cycle, and head and neck thermal images. For example, kinetic energy change data can be plotted as multiple colored lines extending horizontally on the left and right sides of the head and neck thermal image; the length of the line in the horizontal direction is determined by the value of the element in the third kinetic energy matrix, and the color of the line is determined by the motion cycle corresponding to that element.

[0129] Specifically, the length of the straight line drawn on the left side of the thermal image of the head and neck is determined based on the maximum element value of the left half of the same horizontal line in the third kinetic energy matrix, which can be represented by L. max (i) indicates that i is the row number; correspondingly, the color of the line depends on L. max (i) The corresponding motion period; similarly, the length of the straight line drawn on the right half is determined by the maximum element value of the right half row on the same horizontal line in the third kinetic energy matrix, here denoted as R. max (i) indicates that the color of the corresponding line depends on R. max (i) The corresponding motion cycle.

[0130] Understandably, since there are multiple third kinetic energy matrices and thermal energy matrices, each third kinetic energy matrix and thermal energy matrix can be plotted sequentially in the composite image, referring to the video's timeline. In this way, the final composite thermal-kinetic energy image actually presents a dynamic image.

[0131] For example, in specific drawing, the straight lines on the left and right sides of the thermal image of the head and neck can be as follows: Figure 4 As shown, draw starting with two symmetrically arranged vertical lines; or, as shown... Figure 5 As shown, the drawing starts from a circle surrounding the head, which is more suitable for applications that only focus on the kinetic and thermal energy changes of the head. Figure 6 An example is shown using Figure 5 The composite display method in the image creates a realistic thermal-kinetic image.

[0132] Understandable, Figure 4 ,Figure 5 and Figure 6 In practice, they are all color images. Among them, due to... Figure 4 and Figure 5 These are all simplified schematic diagrams, so the color distribution of the thermal images of the head and neck is not shown in them.

[0133] It should be noted that the above-described synthetic display method is merely an example and does not constitute a limitation on the embodiments of the present invention; any implementation method that synthesizes and displays the data based on the spatial distribution correspondence of kinetic energy change data, thermal energy change data, and motion cycle to make the displayed data clearer is applicable to the embodiments of the present invention.

[0134] In the thermal-kinetic image data generation method provided in this embodiment of the invention, visible light and infrared images are simultaneously captured on the head and neck of a stationary person. This allows for the acquisition of kinetic energy change data at various points on the head and neck of the person under both visible light and infrared photoelectric sensing conditions, thereby capturing more detailed micro-motion information of the head and neck. Furthermore, this embodiment of the invention also utilizes a temperature matrix associated with infrared video to capture thermal energy change data at various points on the head and neck of the person. Since people experience uncontrollable primary tension during periods of stress and anxiety, this tension, reflected by the vestibular system, leads to head and neck movements and changes in local body temperature. Therefore, the thermal-kinetic image data generated in this embodiment of the invention captures changes in both thermal and kinetic energy, including head and neck movements and temperature, providing more comprehensive information. Since the two types of information are generated simultaneously and can be cross-referenced, this improves the accuracy of subsequent human detection and analysis based on the thermal-kinetic image data.

[0135] Furthermore, by performing sub-pixel-level registration of corresponding points on the captured visible light video and infrared video, the embodiments of the present invention can eliminate data errors caused by the parallax of the two cameras, making the generated kinetic energy change data more accurate.

[0136] Compared with existing technologies for collecting information such as electrocardiograms, electroencephalograms, and skin conductance responses, this invention provides a contactless data extraction solution that can reduce the anxiety experienced by the subject due to physical contact with the data collection instrument and reduce errors introduced by the data collection process itself.

[0137] Furthermore, the embodiments of the present invention display the kinetic energy change data, thermal energy change data, and motion cycle of the head and neck of a person in the form of images, which allows viewers to view the data of the person more intuitively.

[0138] Alternatively, in one implementation, see [link to implementation details]. Figure 7As shown, in order to extract more accurate data from the two types of videos captured, the thermal-kinetic image data generation method provided in this embodiment of the invention may further include the following steps before performing corresponding point registration on the captured visible light image and infrared image:

[0139] S11: Perform inter-frame subpixel registration of corresponding points for both the captured visible light and infrared images.

[0140] In this implementation, inter-frame subpixel level registration of corresponding points can also be achieved using neural networks. To save development costs, the same neural network can be used in practice to achieve inter-frame subpixel level registration as well as subpixel level registration between different optical images. Preferably, to improve learning performance and adapt the network to different registration scenarios, two identical networks can be constructed: one trained to achieve inter-frame subpixel level registration, and the other trained to achieve registration between visible light and infrared images.

[0141] Preferably, in this embodiment of the invention, a modified MatchNet network can be used to achieve registration of corresponding points.

[0142] See Figure 8 As shown, the improved MatchNet network includes: a first feature extraction module 10, a second feature extraction module 20, and a metric module 30.

[0143] The first feature extraction module 10 is used to extract a first feature descriptor from the reference image; the second feature extraction module 20 is used to perform image transformation on the target image based on the reference image to obtain the transformed target image, and is also used to extract a second feature descriptor from the transformed target image; the measurement module 30 is used to calculate the matching score of the first feature descriptor and the second feature descriptor.

[0144] The reference image and target image mentioned here refer to the two frames of images to be registered using the same reference points. For example, when performing sub-pixel-level registration of the same reference points between a visible light image and an infrared image, if the visible light image was taken directly at a person, then the visible light image can be used as the reference image and the infrared image as the target image; however, this is not a limitation. When performing inter-frame registration of the same reference points between two adjacent visible light images, the later visible light image can be used as the reference image and the earlier visible light image as the target image; again, this is not a limitation.

[0145] The second feature extraction module 20 is a LIFT (Learned Invariant Feature Transform) network. For the specific network structure, please refer to [link / reference needed]. Figure 8 As shown, it includes: a feature point detection module ( Figure 8DET and Softmax activation modules (in the middle) Figure 8 softargmax and pruning modules (in the text) Figure 8 Crop), orientation estimation module ( Figure 8 ORI in the middle), rotating module ( Figure 8 Rot in the descriptor generation module ( Figure 8 DESC in the middle.

[0146] by Figure 8 Target image P in 2 For example, P 2 A scoremap is generated after DET. 2 In softargmax, S 2 Using the Softmax activation function, we obtain the position x of the center point of the region of interest. 2 Determine the center point x of the region of interest. 2 Afterwards, Crop to P 2 The cropped image patch is then input into the ORI (Organization for Image Retrieval) to estimate the rotation angle θ. 2 And by Rot according to θ 2 Rotate the image patch to obtain the rotated image patch. The input is fed into DESC, and DESC outputs LIFT convolutional features d. 2 .

[0147] The first feature extraction module 10 is derived from a modified LIFT network; the modification method is described in [link to modification details]. Figure 8 As shown, the orientation estimation module and rotation module have been removed; therefore, the feature dimension of the second feature descriptor output by the second feature extraction module 20 is the same as the feature dimension of the first feature descriptor output by the first feature extraction module 10.

[0148] The metric module 30 is the metric subnetwork in the MatchNet network. This metric network consists of three fully connected layers, and the activation function of these three fully connected layers is ReLU (Linear rectification function).

[0149] When training the improved MatchNet network, a large number of sample pairs can be constructed, which can include three types of sample pairs: the first type consists of two images of the same person taken from different angles; the second type consists of two images of different people as targets; and the third type consists of two identical images of people. Furthermore, other sample pairs belonging to the category of negative samples can be constructed to improve the robustness of the network. This embodiment of the invention does not limit the construction method of such augmented negative sample pairs. Then, existing image similarity evaluation methods are used to determine the ground truth match score of each sample pair for subsequent loss calculation. For example, gray-level correlation-based matching methods, mean squared error methods, or methods for calculating structural similarity (SSIM) can be used to evaluate image similarity.

[0150] In each sample pair, one frame serves as the baseline image, and the other as the target image. These sample pairs are sequentially fed into the network for training. During training, a loss function is used to calculate the loss value, and the weights within the network are adjusted according to the loss value until the network converges, resulting in the trained improved MatchNet network. The loss function used is as follows:

[0151]

[0152] In this loss function, This represents the truth value indicating whether two feature descriptors input into the metric network 30 match; This represents the matching score of the two feature descriptors actually output by the network, Q. loss is the loss value, and n is the number of sample pairs.

[0153] Therefore, when the reference image and target image to be registered are fed into the trained improved MatchNet network, the transformed target image output by the rotation module in the second feature extraction module 20 can achieve corresponding point matching with the reference image, and the matching scores of their feature descriptors are high, at least not lower than a preset lower limit. It is understandable that, in actual use of this improved MatchNet network, the matching score output by the metric network 30 can be used as a supervisory parameter to monitor whether the improved MatchNet network is functioning correctly.

[0154] Compared to the existing MatchNet network, this invention uses a LIFT network to achieve feature extraction. This is because the changing factors in optical images include illumination, background, and human factors, making it an extremely complex nonlinear transformation process. This invention leverages the nonlinear representation capabilities of the LIFT network to uncover the invariant features of feature points in the image under actual changing factors, thereby working in conjunction with the improved MatchNet network to achieve high-performance target matching under complex conditions.

[0155] Furthermore, the traditional approach to comparing the similarity between feature vectors is to use Euclidean distance. However, in complex manifold learning, Euclidean distance often fails to accurately measure the similarity between feature descriptors. Therefore, this embodiment of the invention employs a metric network containing three fully connected layers to compare the similarity between feature descriptors, so as to evaluate the similarity of the registration results of corresponding points at the sub-pixel level.

[0156] In summary, utilizing Figure 8 The improved MatchNet network shown can achieve sub-pixel-level registration of corresponding points; thus, the embodiments of the present invention can extract more accurate kinetic energy change data and thermal energy change data from two types of videos.

[0157] Based on the thermal-kinetic image data generation method provided in this embodiment of the invention, this embodiment also provides a human body state assessment method, where human body state includes a person's psychological state or physiological state. This human body state assessment method includes:

[0158] Step 1: Obtain thermal-kinetic image data and / or thermal-kinetic images generated by any of the thermal-kinetic image data generation methods described above;

[0159] Step 2: Based on the thermal-kinetic image data and / or the thermal-kinetic image, assess the psychological or physiological state of the personnel.

[0160] Specifically, taking psychological state assessment as an example, there are various ways to assess the psychological state of individuals based on the aforementioned thermal-kinetic images. For instance, in one implementation, the thermal-kinetic image can be directly displayed to the assessor, allowing the assessor to directly obtain the assessment result based on the left-right symmetry of the color distribution of the head and neck thermal image displayed in the thermal-kinetic image, and / or the left-right symmetry of the kinetic correlation data. It is understood that the kinetic correlation data mentioned here includes the aforementioned kinetic change data and motion cycle; the left-right symmetry of both refers to the symmetry when the head and neck thermal image is used as a central reference point for left-right distribution.

[0161] In one assessment method, when the assessor sees a significant asymmetry in the color distribution of the thermal image of the head and neck, it can be determined that the person's psychological state is abnormal; while when the assessor sees a symmetrical color distribution in the thermal image of the head and neck, it can be determined that the person's psychological state is normal.

[0162] In another assessment method, when the assessor sees that the straight lines on both sides of the thermal image of the head and neck are symmetrical, it can be determined that the psychological state of the person is normal; while when the assessor sees that the straight lines on both sides of the thermal image of the head and neck are asymmetrical, it can be determined that the psychological state of the person is abnormal.

[0163] In another assessment method, when the assessor observes asymmetrical color distribution and asymmetrical straight lines on both sides of the head and neck thermal image, the subject's psychological state can be determined to be abnormal. Conversely, when the assessor observes symmetrical color distribution and symmetrical straight lines on both sides of the head and neck thermal image, the subject's psychological state can be determined to be normal. Since kinetic energy and thermal energy represent two different manifestations of a subject's psychological activity, discrepancies between the two assessment conclusions are rare in actual assessments. In other words, situations where the color distribution of the head and neck thermal image is symmetrical but the straight lines on both sides are asymmetrical, or vice versa, are rare.

[0164] In another implementation, the psychological state of individuals can be assessed based on thermal-kinetic image data. The specific process may include:

[0165] (1) Based on the thermal-kinetic image data, calculate the left-right symmetry index of the kinetic energy change data, thermal energy change data, and motion cycle.

[0166] (2) Calculate the weighted sum of the left and right symmetry indices of the kinetic energy change data, thermal energy change data, and motion period to obtain the comprehensive symmetry index;

[0167] (3) The psychological state of the personnel is assessed based on the comprehensive symmetry index.

[0168] In another implementation, the comprehensive symmetry index calculated in step (2) and the evaluation results obtained in step (3) can be further displayed in the visualized thermal-kinetic image, thereby realizing the evaluation based on both thermal-kinetic image data and thermal-kinetic image.

[0169] The method for calculating the left-right symmetry index of the kinetic energy change data in step (1) includes: calculating the left-right symmetry index of each third kinetic energy matrix and averaging the calculation results.

[0170] Similarly, the method for calculating the left-right symmetry index of thermal energy change data in step (1) includes: calculating the left-right symmetry index of each thermal energy matrix separately and averaging the calculation results.

[0171] It is understandable that since the motion cycles of various points on the head and neck of a person also exist in matrix form in reality, and have the same dimensions (number of rows and columns) as the third kinetic energy matrix and thermal energy matrix, the data composed of motion cycles can be called the motion cycle matrix.

[0172] Therefore, it can be understood that calculating the left-right symmetry indices of the third kinetic energy matrix, the thermal energy matrix, and the motion period matrix is ​​essentially calculating the left-right symmetry indices of the matrices. Thus, the calculation method will be illustrated below using the third kinetic energy matrix as an example; the calculation methods for the remaining two matrices can be similar.

[0173] For example, in one calculation method, the third kinetic energy matrix can be divided into two parts, left and right. Then, the difference between each pair of elements in the left and right parts is calculated, and the number of elements in the difference result that are greater than a preset threshold is counted as the left and right symmetry index of the matrix.

[0174] In another calculation method, the matrix can be divided into two sub-matrices, left and right. Then, these two sub-matrices are flattened into vectors, the product of the two vectors is calculated and then divided by the magnitude of the two vectors. The result is used as an index of left-right symmetry.

[0175] It is understood that other methods for calculating matrix similarity can also be used to calculate the left-right symmetry index in this embodiment of the invention, and will not be described in detail in this embodiment of the invention.

[0176] After calculating the kinetic energy change data, thermal energy change data, and the left-right symmetry index of the motion cycle, a weighted sum of the three can be obtained to obtain a comprehensive symmetry index.

[0177] Then, the comprehensive symmetry index is compared with several preset interval boundary points to determine the interval in which the index falls; thus, the psychological state of the subject is determined based on the psychological state level corresponding to the interval. For example, suppose the preset interval boundary points include 40 and 100 (in percentage); a score below 40 corresponds to a relaxed psychological state, a score within the interval [40, 100] corresponds to a normal psychological state, and a score above 100 corresponds to a tense psychological state. If the calculated comprehensive symmetry index is 50, then the psychological state assessment result for the subject can be determined to be normal.

[0178] Understandably, assuming that the individual has no abnormal psychological state, the psychological assessment process described above can also be applied to the assessment of a person's physiological state to determine whether the person is healthy.

[0179] The above describes a relatively simple scheme for psychological state assessment based on thermal-kinetic image data. More complex assessment methods can involve collecting and analyzing thermal-kinetic image data from a large number of individuals to identify specific patterns within the thermal-kinetic image data of a particular population group. These patterns are not limited to the left-right symmetry pattern shown above. Based on these patterns, a specific population group can be identified from a broad population, enabling more refined psychological or physiological state assessments in various application scenarios. Examples include assessing a person's stress index or the reference probability of having a certain disease. Understandably, these assessment results can provide supplementary reference data or information for the diagnosis of mental or physiological illnesses from a new perspective.

[0180] Corresponding to the above-described method for generating thermal-kinetic image data, this embodiment of the invention also provides a thermal-kinetic image generation apparatus. For example... Figure 9 As shown, the device includes:

[0181] The shooting module 901 is used to simultaneously capture visible light and infrared images of the head and neck of a person who is kept quasi-still, and obtain visible light video, infrared video, and multiple temperature matrices associated with the infrared video.

[0182] Specifically, the electronic device containing the thermal-kinetic image generation device can be associated with a visible light camera and an infrared camera; the shooting module 901 simultaneously sends shooting commands to the visible light camera and the infrared camera to achieve shooting through these two cameras.

[0183] The registration module 902 is used for subpixel-level registration of corresponding points between visible light video and infrared video.

[0184] The first processing module 903 is used to generate kinetic energy change data of each point on the head and neck of the person based on the registered visible light video and infrared video, and to generate thermal energy change data of each point on the head and neck of the person based on multiple temperature matrices associated with the infrared video.

[0185] The second processing module 904 is used to statistically analyze the kinetic energy change data to obtain the motion cycle of each point on the head and neck of the person.

[0186] The generation module 905 is used to take kinetic energy change data, thermal energy change data and motion cycle as a set of thermal-kinetic energy image data to generate thermal-kinetic energy images based on the thermal-kinetic energy image data.

[0187] Optionally, the thermal-kinetic image generation device further includes: an inter-frame registration module;

[0188] This inter-frame registration module is used to perform inter-frame subpixel level registration of corresponding points in both the captured visible light and infrared images before performing corresponding point registration on the captured images.

[0189] Optionally, the above-mentioned registration of corresponding points is implemented based on a pre-trained improved MatchNet network; the improved MatchNet network includes: a first feature extraction module, a second feature extraction module, and a metric module; wherein,

[0190] The first feature extraction module is used to extract a first feature descriptor from the reference image;

[0191] The second feature extraction module is used to perform image transformation on the target image based on the reference image to obtain the transformed target image, and is also used to extract the second feature descriptor from the transformed target image;

[0192] The metrics module is used to calculate the matching score between the first feature descriptor and the second feature descriptor;

[0193] The reference image and the target image are two frames of images to be registered with corresponding points. The transformed target image matches the corresponding points of the reference image, and the matching score is not lower than the preset lower limit.

[0194] The second feature extraction module is a LIFT network, which is obtained by modifying another LIFT network. The feature dimensions of the first feature descriptor are the same as those of the second feature descriptor.

[0195] The metrics module is the metrics subnetwork in the MatchNet network.

[0196] Optionally, the first processing module 903 generates kinetic energy change data for various points on the head and neck of the person based on the registered visible light video and infrared video, including:

[0197] By utilizing the image differences between adjacent frames in the registered visible light video, the first kinetic energy matrix of each point on the head and neck of the human subject is estimated at multiple time points.

[0198] By utilizing the image differences between adjacent frames in the registered infrared video, the second kinetic energy matrix of each point on the head and neck of the human subject is estimated at multiple time points.

[0199] Referring to the timelines of visible light and infrared video, the first kinetic energy matrix and the second kinetic energy matrix are matched one-to-one, and each pair of first and second kinetic energy matrices with corresponding relationships is fused to obtain multiple third kinetic energy matrices, which serve as the kinetic energy change data of various points on the head and neck of the human subject.

[0200] Optionally, the second processing module 904 is specifically used for:

[0201] Element statistics are performed on multiple third kinetic energy matrices to determine the frequency of the most frequent element value at each element location point of the third kinetic energy matrix.

[0202] The motion cycle of each point on the head and neck of the human subject is calculated based on the frequency of occurrence of each element's position.

[0203] Optionally, the first processing module 903 generates thermal energy change data for various points on the head and neck of the human object based on multiple temperature matrices, including:

[0204] The multiple temperature matrices are sorted by referring to the timeline of the infrared video;

[0205] By utilizing the difference between every two adjacent temperature matrices, the thermal energy matrix of each point on the head and neck of the human subject at multiple times is estimated, which serves as the thermal energy change data of each point on the head and neck of the human subject.

[0206] Optionally, the generation module 905 generates thermal-kinetic images based on thermal-kinetic image data in the following ways:

[0207] Based on thermal energy change data, generate thermal images of the head and neck of human subjects;

[0208] Based on the spatial correspondence between kinetic energy change data, motion cycle, and head and neck thermal images, the three are synthesized and displayed to obtain thermal-kinetic images.

[0209] The thermal-kinetic image data generation device provided in this invention simultaneously captures visible light and infrared images of the head and neck of a stationary person. This allows for the acquisition of kinetic energy change data at various points on the head and neck from both visible light and infrared photoelectric sensing conditions, thereby capturing more detailed micro-movement information of the head and neck. Furthermore, the device utilizes a temperature matrix associated with the infrared video to capture thermal energy change data at various points on the head and neck. Since people experience uncontrollable primary tension during periods of stress and anxiety, this tension, reflected by the vestibular system, leads to head and neck movements and changes in local body temperature. Therefore, the thermal-kinetic image data generated by this device captures changes in both thermal and kinetic energy, including head and neck movement and temperature, providing more comprehensive information. Since the two types of information are generated simultaneously and can be cross-referenced, the accuracy of subsequent human detection and analysis based on this thermal-kinetic image data can be improved.

[0210] Furthermore, by performing sub-pixel-level registration of corresponding points on the captured visible light and infrared videos, the device can eliminate data errors caused by the parallax of the two cameras, making the generated kinetic energy change data more accurate.

[0211] Compared to existing technologies for collecting information such as electrocardiograms, electroencephalograms, and skin conductance responses, this device achieves a contactless data extraction solution, which can reduce the anxiety of the subjects being collected due to physical contact with the collection instrument and reduce errors introduced by the collection event itself.

[0212] In addition, the device displays the kinetic energy change data, thermal energy change data, and motion cycle of the head and neck of the person in the form of images, which allows viewers to view the data of the person more intuitively.

[0213] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, such as... Figure 10 As shown, it includes: a processor 1001, a communication interface 1002, a memory 1003, a communication bus 1004, an infrared camera 1005, and a visible light camera 1006.

[0214] The processor 1001, communication interface 1002, and memory 1003 communicate with each other through communication bus 1004.

[0215] Memory 1003 is used to store computer programs;

[0216] When the processor 1001 executes the program stored in the memory 1003, it works in conjunction with the infrared camera 1005 and the visible light camera 1006 to implement any of the above-mentioned thermal-kinetic image generation methods.

[0217] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0218] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0219] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0220] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0221] Figure 11 and Figure 12 An example is shown Figure 10 A physical image of the electronic device shown. Figure 11 This is a front view of the actual electronic device. Figure 12 This is a side view of the physical electronic device. The camera shown in the figure can be a visible light camera 1006 or an infrared camera 1005. In addition to this camera, another type of camera is also present. Figure 12The image is not shown as it is obscured by the camera. The processor 1001, communication interface 1002, memory 1003, and communication bus 1004 are hidden within the main unit chassis and mechanical structure. The display shows thermal-kinetic images and allows users of the electronic device to view thermal-kinetic image data; the control panel may include a keyboard, mouse, and hooks. A supplementary light is mounted on a rotating robotic arm for providing illumination during filming, thereby improving the quality of the captured video.

[0222] Understandable, Figure 11 and Figure 12 The electronic devices shown are merely examples of electronic devices provided in the embodiments of the present invention and do not constitute a limitation on the embodiments of the present invention.

[0223] It should be noted that, for the device / electronic device embodiment, since it is basically similar to the thermal-kinetic image generation method embodiment, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiment.

[0224] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0225] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0226] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In this specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. While certain measures are described in different embodiments, this does not mean that these measures cannot be combined to produce good results.

[0227] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus (devices), or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects, all of which are collectively referred to herein as "modules" or "systems." Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The computer program may be stored / distributed in a suitable medium, provided with or as part of other hardware, or may take other distribution forms, such as via the Internet or other wired or wireless telecommunications systems.

[0228] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for generating thermal-kinetic image data, characterized in that, include: Simultaneous light and infrared imaging of the head and neck of a person who remains quasi-static can be performed to obtain visible light video, infrared video, and multiple temperature matrices associated with the infrared video; Subpixel-level registration of corresponding points is performed on the visible light video and the infrared video; Based on the registered visible light video and infrared video, kinetic energy change data of each point on the head and neck of the person are generated, and thermal energy change data of each point on the head and neck of the person are generated based on the multiple temperature matrices. By statistically analyzing the kinetic energy change data, the motion cycle of each point on the head and neck of the person is obtained; The kinetic energy change data, the thermal energy change data, and the motion cycle are used as a set of thermal-kinetic energy image data to generate a thermal-kinetic energy image.

2. The method according to claim 1, characterized in that, The method further includes: Before performing subpixel-level registration of corresponding points on the visible light video and the infrared video, inter-frame subpixel-level registration of corresponding points is performed on both the visible light video and the infrared video.

3. The method according to claim 1 or 2, characterized in that, The corresponding point registration is implemented based on a pre-trained improved MatchNet network; the improved MatchNet network includes: a first feature extraction module, a second feature extraction module, and a metric module; wherein, The first feature extraction module is used to extract a first feature descriptor from the reference image; The second feature extraction module is used to perform image transformation on the target image based on the reference image to obtain the transformed target image, and is also used to extract the second feature descriptor from the transformed target image; The metric module is used to calculate the matching score between the first feature descriptor and the second feature descriptor; Wherein, the reference image and the target image are two frames of images to be registered with corresponding points, the transformed target image matches the corresponding points of the reference image, and the matching score is not lower than a preset lower limit; The second feature extraction module is a LIFT network, which is obtained by modifying another LIFT network. The feature dimension of the first feature descriptor is the same as the feature dimension of the second feature descriptor. The metric module is the metric subnetwork in the MatchNet network.

4. The method according to claim 1, characterized in that, Based on the registered visible light video and infrared video, kinetic energy change data for each point on the head and neck of the person are generated, including: By utilizing the image differences between adjacent frames in the registered visible light video, the first kinetic energy matrix of each point on the head and neck of the person is estimated at multiple times. By utilizing the image differences between adjacent frames in the registered infrared video, the second kinetic energy matrix of each point on the head and neck of the person is estimated at multiple times. Referring to the timelines of the visible light video and the infrared video, the first kinetic energy matrix and the second kinetic energy matrix are mapped one-to-one, and each pair of first and second kinetic energy matrices with a corresponding relationship is fused to obtain multiple third kinetic energy matrices, which serve as the kinetic energy change data of each point on the head and neck of the person.

5. The method according to claim 4, characterized in that, Statistical analysis of the kinetic energy change data yields the motion cycle of various points on the head and neck of the person, including: Element statistics are performed on the multiple third kinetic energy matrices to determine the frequency of the most frequent element value at each element position of the third kinetic energy matrix. The motion cycle of each point on the head and neck of the person is calculated based on the frequency of occurrence corresponding to each of the element locations.

6. The method according to claim 1, characterized in that, Based on the multiple temperature matrices, thermal energy change data for various points on the head and neck of the person are generated, including: The multiple temperature matrices are sorted with reference to the time axis of the infrared video; By utilizing the difference between every two adjacent temperature matrices, the thermal energy matrix of each point on the head and neck of the person is estimated at multiple times, which serves as the thermal energy change data of each point on the head and neck of the person.

7. The method according to claim 1, characterized in that, The method of generating the thermal-kinetic image based on the thermal-kinetic image data includes: Based on the thermal energy change data, a thermal image of the head and neck of the person is generated; Based on the spatial correspondence between the kinetic energy change data, the motion cycle, and the head and neck thermal image, the three are synthesized and displayed to obtain a thermal-kinetic image.

8. A thermal-kinetic image data generation device, characterized in that, include: The shooting module is used to simultaneously capture light and infrared images of the head and neck of a person who is kept quasi-still, and obtain visible light video, infrared video, and multiple temperature matrices associated with the infrared video; The registration module is used to perform sub-pixel-level matching of corresponding points between the visible light video and the infrared video; The first processing module is used to generate kinetic energy change data of each point on the head and neck of the person based on the registered visible light video and infrared video, and to generate thermal energy change data of each point on the head and neck of the person based on the multiple temperature matrices. The second processing module is used to statistically analyze the kinetic energy change data to obtain the motion cycle of each point on the head and neck of the person. The generation module is used to take the kinetic energy change data, the thermal energy change data, and the motion cycle as a set of thermal-kinetic energy image data to generate a thermal-kinetic energy image based on the thermal-kinetic energy image data.

9. A method for assessing human condition, characterized in that, include: Obtain thermal-kinetic image data and / or thermal-kinetic images generated in the thermal-kinetic image data generation method according to any one of claims 1-7; The psychological or physiological state of the subjects is assessed based on the thermal-kinetic image data and / or the thermal-kinetic images.

10. An electronic device, characterized in that, include: The system includes an infrared camera, a visible light camera, a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. The memory is used to store computer programs; the processor is used to execute the programs stored in the memory, and in conjunction with the infrared camera and the visible light camera, to implement the method steps of the thermal-kinetic image data generation method according to any one of claims 1-7 and 9.

Images