A thermal-energy-mechanical-energy image data generation method and a human body state detection method

Abstract

This invention discloses a method for generating thermal-kinetic image data and a method for detecting human body state. The former includes: simultaneously acquiring visible light images and infrared images with temperature values ​​of the head and neck of a subject in a quasi-stationary state at three time points before and after the same time interval; registering corresponding points of the two images at the same time; obtaining a thermal energy change index matrix representing the spatial distribution of thermal energy changes at various points on the head and neck for the infrared image in all registered image groups; obtaining kinetic energy change index matrices representing the spatial distribution of kinetic energy changes at various points on the head and neck for the two images respectively; using the two obtained kinetic energy change index matrices to obtain a target kinetic energy change index matrix; and obtaining thermal-kinetic image data based on the target kinetic energy change index matrix and the thermal energy change index matrix to generate a thermal-kinetic image. This invention does not involve contact with the human body, does not rely on operational expertise, can comprehensively describe human body characteristics, and accurately and quickly detect human body state.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to a method for generating thermal-kinetic image data and a method for detecting human body state. Background Technology

[0002] In medical research and the auxiliary diagnosis of related diseases, instruments are often used to measure physiological parameters to detect a person's state. For example, electrocardiograms (ECGs) and electroencephalograms (EEGs) are used to detect a person's physiological state; and lie detectors are used to record changes in stress-related physiological parameters when a subject is questioned to determine whether the subject is lying, thus detecting psychological state.

[0003] However, existing methods for human condition detection usually require human contact, and the detection process often relies on the operator's professional knowledge. In addition, the data information describing human characteristics obtained is relatively one-sided, thus having certain limitations. Summary of the Invention

[0004] To address the aforementioned problems in the prior art, this invention provides a method for generating thermal-kinetic image data, a method for detecting human body state, and a device for detecting human body state. The technical problem to be solved by this invention is achieved through the following technical solution:

[0005] In a first aspect, embodiments of the present invention provide a method for generating thermal-kinetic image data, the method comprising:

[0006] Visible light and infrared images of the head and neck of the subject in a quasi-static state are acquired synchronously at three time points before and after the same time interval; wherein each infrared image carries the temperature value of each point on the head and neck.

[0007] Register corresponding points on the visible light image and infrared image at the same time to obtain the registered image group corresponding to each time.

[0008] In the registered image group corresponding to all times, for the infrared image type, a thermal energy change index matrix is ​​obtained to characterize the spatial distribution of thermal energy change at each location point of the head and neck;

[0009] In the registered image group corresponding to all time points, for the two image types, a kinetic energy change index matrix representing the spatial distribution of kinetic energy change at each position point of the head and neck is obtained, and a target kinetic energy change index matrix is ​​obtained by using the two obtained kinetic energy change index matrices.

[0010] Thermal-kinetic image data is obtained based on the thermal energy change index matrix and the target kinetic energy change index matrix, and a thermal-kinetic image is generated using the thermal-kinetic image data.

[0011] In one embodiment of the present invention, before performing corresponding point registration on the visible light image and infrared image at the same time, the method further includes:

[0012] For each of the three consecutive time points, the visible light images of the subject's head and neck in a quasi-static state acquired at every two adjacent time points are registered using corresponding points; and for each of the three consecutive time points, the infrared images of the subject's head and neck in a quasi-static state acquired at every two adjacent time points are registered using corresponding points.

[0013] In one embodiment of the present invention, the process of registering corresponding points includes:

[0014] The first image, serving as the reference image, and the second image, serving as the image to be registered, are respectively input into a pre-trained first LIFT network and a second LIFT network. The result is a registered image output by the second LIFT network, which registers the second image with the first image as a reference. The first image and the registered image are then used as the result of the registration of the corresponding points. The first LIFT network and the second LIFT network are trained using a pre-built improved MatchNet network based on multiple sample reference images and sample images to be registered.

[0015] In one embodiment of the present invention, the improved MatchNet network includes:

[0016] The system comprises a first LIFT network, a second LIFT network, and a metric network; wherein the first LIFT network and the second LIFT network are connected in parallel, and their outputs are connected to the metric network; during the training phase, the first LIFT network and the second LIFT network are used to extract feature vectors from the input image, and the metric network is used to measure the similarity between the two feature vectors.

[0017] In one embodiment of the present invention, the step of obtaining a thermal energy change index matrix characterizing the spatial distribution of thermal energy changes at various locations of the head and neck, for the infrared image type, from the registered image group corresponding to all times, includes:

[0018] The temperature values ​​of each position point of the head and neck carried by each infrared image in the registered image group corresponding to all time moments are obtained respectively.

[0019] For each location point, the thermal energy change index of that location point is calculated based on the thermal energy using the temperature values ​​of that location point at the three times.

[0020] The thermal energy change indexes obtained at each location point of the head and neck are arranged spatially to obtain the thermal energy change index matrix of the head and neck.

[0021] In one embodiment of the present invention, in the registered image group corresponding to all times, for two image types, a kinetic energy change index matrix characterizing the spatial distribution of kinetic energy change at each location point of the head and neck is obtained, including:

[0022] In each image type of the registered image group corresponding to all times, for each position point of the head and neck, the image data of that position point is calculated based on kinetic energy to obtain the kinetic energy value of that position point in the last two times under that image type.

[0023] Based on the difference in kinetic energy values ​​at the next two times, the kinetic energy change index at that location is obtained.

[0024] The kinetic energy change indexes obtained at each position point of the head and neck are arranged according to the spatial distribution of the position points to obtain the kinetic energy change index matrix of the head and neck under this image type.

[0025] In one embodiment of the present invention, obtaining a target kinetic energy change index matrix using the two obtained kinetic energy change index matrices includes:

[0026] The kinetic energy change index matrices obtained from the two image types are fused together to obtain the target kinetic energy change index matrix.

[0027] In one embodiment of the present invention, generating a thermal-kinetic image using the thermal-kinetic image data includes:

[0028] The thermal energy change index matrix is ​​used to generate a thermal energy change image of the head and neck;

[0029] The values ​​in the target kinetic energy change index matrix are displayed visually around the head and neck in the head and neck thermal energy change image to obtain the thermal energy-kinetic energy image.

[0030] Secondly, embodiments of the present invention provide a method for detecting human body status, including:

[0031] Using the thermal-kinetic image data generation method described in the first aspect, thermal-kinetic image data and / or thermal-kinetic images of the head and neck of the subject are obtained;

[0032] The physiological or psychological state of the subject is detected using the thermal-kinetic image data and / or the thermal-kinetic image.

[0033] Thirdly, embodiments of the present invention provide a human body state detection device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0034] Memory, used to store computer programs;

[0035] The processor, when executing a program stored in memory, implements the steps of the human state detection method described in the second aspect.

[0036] The thermal-kinetic image data generation method provided in this invention combines three consecutive visible light and infrared images captured when the subject's head and neck are in a quasi-static state. This allows for the capture of more detailed head and neck movement information by integrating the spatial distribution data of kinetic energy changes from different image types. Simultaneously, by combining the spatial distribution data of thermal energy changes obtained from the three consecutive infrared images, thermal-kinetic image data is generated, combining kinetic and thermal energy data to obtain more comprehensive and detailed information for describing the human body's state. Furthermore, since the subject only needs to maintain a quasi-static state without any other cooperation, it avoids psychological or physiological changes caused by contact biometric extraction or other cooperation, thus better reflecting the subject's true physical condition. Moreover, the thermal-kinetic image data generation process in this invention is based on image processing techniques, requiring simple equipment, no operators, and no specialized operational requirements. Furthermore, by registering corresponding points in visible light and infrared images taken at the same time, data errors caused by differences in the shooting angles of the two image acquisition devices can be eliminated, improving the accuracy of subsequent thermal-kinetic image data. Additionally, because image acquisition time is short and image processing analyzes and processes data differences between frames of the same type, the amount of data processed is far less than the amount of data in a single frame, thus reducing computational load and increasing processing speed.

[0037] This invention provides a human body state detection scheme based on the thermal-kinetic image data generation method of this invention. It can combine kinetic and thermal energy to obtain relatively comprehensive data information for human body state detection without human contact and without requiring specialized operational skills. Furthermore, since the subject does not need to cooperate, it can preserve the subject's true state to the greatest extent possible, meeting the needs of human body state detection in concealed situations. Moreover, the detection is simple and fast, enabling real-time detection.

[0038] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0039] Figure 1A schematic flowchart illustrating a method for generating thermal-kinetic image data according to an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the structure of the MatchNet network in the prior art;

[0041] Figure 3 This is a schematic diagram of the structure of the improved MatchNet network provided in an embodiment of the present invention;

[0042] Figure 4 The training structure diagram based on the improved MatchNet network provided in the embodiments of the present invention is shown.

[0043] Figure 5 This is a schematic diagram illustrating the realistic effect of a thermal-kinetic energy image provided in an embodiment of the present invention;

[0044] Figure 6 This is a schematic diagram illustrating the effect of another thermal-kinetic energy image provided in an embodiment of the present invention;

[0045] Figure 7 This is a schematic diagram of the structure of a human body status detection device provided in an embodiment of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] The inventors discovered a direct link between a person's physiological and psychological states and physical movement. When a person is calm and resting, their heart rate and respiration are at their lowest; when a person is excited, their breathing and heart rate increase. The vestibular emotional reflex theory indicates that when a person is tense or anxious, uncontrollable primary tension occurs, leading to head and neck movements under the reflex of the vestibular system. Furthermore, under different emotional states (such as excitement, stillness, and inhibition), the vestibular discharge rate varies, resulting in differences in head and neck movements and changes in kinetic energy. Simultaneously, under the influence of certain pathologies, uncontrollable head and neck movements may occur, resulting in changes in kinetic energy, such as neurological convulsions. Therefore, representing changes in head and neck movement through changes in kinetic energy to detect a person's psychological and physiological states—that is, to achieve human state detection—is feasible.

[0048] Furthermore, because humans are capable of thermal radiation, the thermal energy in different areas of the head and neck can change under the influence of different psychological and physiological states. For example, under stress, the nose may sweat or twitch, or a fever may cause a local temperature increase, all of which will cause changes in thermal energy. Therefore, it is feasible to detect a person's psychological or physiological state by observing changes in thermal energy.

[0049] Based on this concept, and in order to achieve non-contact operation, without relying on operational expertise, and to obtain relatively comprehensive data information describing human characteristics, embodiments of the present invention provide a method for generating thermal-kinetic image data. Furthermore, to enable simple and effective human state detection, embodiments of the present invention also provide a method and device for human state detection.

[0050] In a first aspect, embodiments of the present invention provide a method for generating thermal-kinetic image data, such as... Figure 1 As shown, the following steps may be included:

[0051] S1, at three time points before and after the same time interval, simultaneously acquire visible light and infrared images of the head and neck of the subject in a quasi-stationary state.

[0052] The inventors discovered that a person's movement in a quasi-static state (standing or sitting) is determined by unconscious processes, primarily by the vestibular system, and that this quasi-static state reflects their emotional state. Therefore, to avoid interference from voluntary human movement, in this embodiment of the invention, the subjects are all placed in a quasi-static state, i.e., their head and neck remain still.

[0053] From a physics perspective, head movement can be understood as a vibration process, which generates kinetic energy. In this embodiment of the invention, for the same type of image (infrared image or visible light image), images at three different times are acquired. The main purpose is to determine the changes in kinetic energy, thereby representing the changes in the movement of the subject's head and neck at various points during this period. This part will be described in detail later.

[0054] In this embodiment of the invention, the time interval between the first two moments and the time interval between the last two moments are the same. Synchronous acquisition means that at each moment, a visible light image and an infrared image of the subject's head and neck in a quasi-stationary state are acquired simultaneously.

[0055] In this embodiment of the invention, a visible light image acquisition device and an infrared image acquisition device, which are positioned side by side facing the subject, can be used to capture images at three consecutive moments. Alternatively, images at three consecutive moments can be extracted from a video of a certain duration captured by the visible light image acquisition device and the infrared image acquisition device, respectively.

[0056] In this embodiment of the invention, the visible light image acquisition device can be a conventional camera, camera, etc. Each infrared image carries the temperature values ​​of various points on the head and neck. Specifically, the infrared image acquisition device can be an infrared thermal imager, an infrared thermal imaging camera, an infrared thermal imaging webcam, or other infrared thermal imaging devices. These devices can acquire the temperature data of the target and generate a thermal image representing the temperature distribution of the target surface; the image data is the temperature value at each point. Of course, in this embodiment of the invention, the temperature value carried by the infrared image can also be additional information matched to pixels at each position in the image. For example, the infrared image in this embodiment can be a traditional infrared image where each pixel is accompanied by corresponding position data in a temperature matrix. The temperature matrix can be obtained using infrared imaging technology or by image measurement and estimation using deep learning or other technologies. In this case, a point in the infrared image has both a pixel value and a temperature value. In summary, the infrared image acquisition device in this embodiment of the invention can include, but is not limited to, infrared thermal imaging devices. Any device or combination of devices capable of obtaining infrared images carrying the temperature values ​​of various points on the head and neck can be included within the protection scope of this embodiment of the invention.

[0057] To achieve good image capture results and simplify subsequent image processing, the height and distance from the subject can be appropriately set for both visible light and infrared image acquisition devices to ensure that the captured images are of the same size and that the subject's head and neck are fully displayed. Furthermore, the proportion of the head and neck area in the image can be set to be higher than a preset proportion to ensure pixel clarity; for example, the preset proportion could be 2 / 3.

[0058] Those skilled in the art will understand that visible light images typically possess high spatial resolution and a considerable amount of detail and contrast. Therefore, they are well-suited for human visual perception. However, these images are easily affected by adverse conditions, such as poor lighting, fog, and other effects of inclement weather, which may lead to problems such as missing pixels. Infrared images, with wavelengths longer than visible light, capture thermal radiation, unlike visible light images which capture reflected light. They can represent the temperature information of objects and are resistant to the interference experienced by visible light images. Infrared images can distinguish targets from backgrounds based on radiation differences, which works well in all weather conditions and throughout the day / night. Therefore, embodiments of the present invention acquire both types of images, aiming to combine the advantages of thermal radiation information in infrared images and detailed texture information in visible light images to obtain richer image information and capture more detailed motion data and other data.

[0059] S2, perform corresponding point registration on the visible light image and infrared image at the same time to obtain the corresponding registered image group at each time.

[0060] Image registration is the process of optimally matching two or more images obtained from different sensors, at different times, and from different angles.

[0061] It is understandable that even if a visible light image acquisition device and an infrared image acquisition device are placed side by side facing the subject, the shooting angles of the two devices will differ at any given moment. Therefore, image registration is necessary.

[0062] In this embodiment of the invention, corresponding point registration involves registering one of the two images to be processed with the other image to be processed as a reference image. A corresponding point is the image point of the same point on the head and neck of the subject in different images. In this embodiment of the invention, the reference image in corresponding point registration is the one of the two images to be processed whose shooting angle is directly opposite the subject.

[0063] Specifically, in this embodiment of the invention, one image acquisition device shoots directly at the subject, and its image serves as the reference image. The other image acquisition device may have a small angular difference from the subject's facing direction, resulting in an angular difference between its image and the reference image; this image serves as the image to be registered. Registration using corresponding points involves analyzing the geometric distortion on the image to be registered, and then employing a geometric transformation to normalize the image to a unified coordinate system referenced to the reference image. Specifically, methods such as rotation and translation can be used to bring the coordinates of the two images at the same position as close as possible, or even make them equal, to achieve the registration effect.

[0064] In this embodiment of the invention, any existing image registration method applicable to corresponding points can be used, such as image grayscale-based registration algorithms, image feature-based registration algorithms, and registration algorithms based on image understanding and interpretation, etc.

[0065] For example, one possible implementation is to use an image feature-based registration algorithm. This typically uses certain salient features (points, lines, regions) in the image as registration primitives, and the algorithm process can be divided into two steps: feature extraction and feature matching. First, features such as points, lines, and regions with significant grayscale changes are extracted from the two images to form a feature set. Then, a feature matching algorithm is used to select as many corresponding feature pairs as possible from the feature sets corresponding to the two images. For non-feature pixels, interpolation and other methods are used to calculate the corresponding matching relationship, thereby achieving pixel-by-pixel registration between the two images.

[0066] In this embodiment of the invention, the specific processing procedure of the image feature-based registration algorithm may be as follows: extract SIFT (Scale-invariant feature transform) features of two images using any existing neural network model, and perform feature matching on the SIFT features of the two images using a similarity metric based on Euclidean distance, etc. The specific process will not be described in detail here.

[0067] Optionally, in this embodiment of the invention, the visible light image acquisition device can be positioned facing the subject, and the visible light image at each moment can be used as the reference image at that moment. Corresponding points are registered for both the visible light image and the infrared image at each moment to obtain a registered image group for that moment, which includes the visible light image and the registered infrared image at that moment.

[0068] S3, in the registered image group corresponding to all times, for the infrared image type, obtain a thermal energy change index matrix that characterizes the spatial distribution of thermal energy change at each location point of the head and neck.

[0069] As described earlier, in this embodiment of the invention, for the same type of image (infrared image or visible light image), acquiring images at three different times before and after it is primarily aimed at determining the changes in kinetic energy. However, for infrared images, acquiring images at three different times before and after them also serves another purpose: to determine the changes in thermal energy. It is understood that thermal energy is related to temperature.

[0070] Since each infrared image in this embodiment carries the temperature value of each location point in the head and neck, the temperature value of each location point in the head and neck can be obtained for the infrared image at each time point in the registered image group corresponding to all times. Based on the changes in temperature values, the thermal energy change of each location point can be obtained. Then, by integrating the thermal energy changes of all location points, the spatial distribution of thermal energy changes at each location point in the head and neck can be obtained.

[0071] For S3, one possible implementation may include the following steps:

[0072] S31, extract the temperature values ​​of each position point on the head and neck from each infrared image in the registered image group corresponding to all times.

[0073] Specifically, for each location point, the corresponding temperature values ​​T1, T2, and T3 can be obtained from the infrared images of three consecutive times t1, t2, and t3 within all registered image groups.

[0074] S32, for each location point, using the temperature values ​​of that location point at three times, calculate the thermal energy change index of that location point based on thermal energy.

[0075] Among them, the thermal energy change index of a location point represents the thermal energy change of that location point in the latter two time periods.

[0076] The known formula for calculating thermal energy is:

[0077] Q = c × m × ΔT

[0078] Where Q represents thermal energy (or heat); c represents the specific heat capacity of the object; m represents the mass of the object; and ΔT represents the temperature difference.

[0079] Therefore, in one possible implementation of S32, the following steps may be included:

[0080] 1) For each location point, use the thermal energy calculation formula to calculate the thermal energy value corresponding to the middle time and the thermal energy value corresponding to the last time of that location point.

[0081] The thermal energy value corresponding to the intermediate time is Q1 = c × m × (T2 - T1); the thermal energy value corresponding to the final time is Q2 = c × m × (T3 - T2). During the calculation, c and m are generally fixed for the same location point and can be substituted with their respective preset values. For example, c can be the specific heat capacity of water or other objects, and m can be a preset molecular mass, etc.

[0082] 2) Calculate the absolute value of the difference between the two thermal energy values ​​at this location.

[0083] Specifically, we calculate ΔQ = |Q2 - Q1|.

[0084] 3) The heat energy change index at that location is obtained by using the absolute value of the difference between the two heat energy values ​​at that location.

[0085] For example, ΔQ at that location can be directly used as the thermal energy change index at that location.

[0086] Alternatively, to ensure standardized operation at the same location point, a corresponding normalized value can be preset for each location point to serve as the denominator for normalizing ΔQ at that location point. For example, the normalized value for a location point could be the maximum, minimum, or average value of ΔQ determined based on a large number of data samples from test subjects at similar locations.

[0087] In one alternative implementation, the normalized value used at the same location point can be achieved using the thermal energy value currently obtained at that location point. For example, Q1, Q2, or the average of the two can all be used, all of which are reasonable.

[0088] For example, taking normalization using Q1 as an example, the thermal energy change index at a location point can be calculated using the following formula:

[0089]

[0090] Among them, Index EQ This indicates the thermal energy change index at that location.

[0091] Furthermore, by expanding and rearranging the above formula, we can obtain:

[0092]

[0093] Therefore, in one possible implementation, in order to reduce the amount of calculation and improve the calculation speed, for each location point, the specific heat capacity and mass can be ignored, and the temperature values ​​at the three times can be directly substituted into the above formula to calculate the thermal energy change index at that location point.

[0094] S33, the thermal energy change indexes obtained at each location point of the head and neck are arranged according to the spatial distribution of the location points to obtain the thermal energy change index matrix of the head and neck.

[0095] For each location point in the head and neck, a corresponding thermal energy change index can be obtained. Arranging the thermal energy change indices according to the spatial position of the location point in the head and neck and in the image yields a thermal energy change index matrix.

[0096] S4. In the registered image group corresponding to all times, for the two image types, a kinetic energy change index matrix representing the spatial distribution of kinetic energy change at each position point of the head and neck is obtained, and a target kinetic energy change index matrix is ​​obtained by using the two obtained kinetic energy change index matrices.

[0097] For ease of subsequent description, S4 will be divided into two steps: S41 and S42.

[0098] S41, for the two image types, in the registered image group corresponding to all times, a kinetic energy change index matrix is ​​obtained to represent the spatial distribution of kinetic energy change at each position point of the head and neck.

[0099] S42, using the two obtained kinetic energy change index matrices, a target kinetic energy change index matrix is ​​obtained.

[0100] The following is a detailed explanation:

[0101] It is understandable that, for the same image type, if head and neck movement occurs between adjacent time points, the position of the same point will change in the images at adjacent time points, thus generating kinetic energy. Therefore, there will be a kinetic energy value for the intermediate time point and the final time point. If these two kinetic energy values ​​change, it can reflect the change in motion during this time period. Therefore, in one possible implementation, S41 may include S411 to S413:

[0102] S411, for each image type in the registered image group corresponding to all times, for each position point of the head and neck, the image data of that position point is calculated based on kinetic energy to obtain the kinetic energy value corresponding to that position point in the last two times under that image type.

[0103] The known formula for calculating kinetic energy is:

[0104]

[0105] Where E represents kinetic energy; m represents the mass of the object; and v represents the velocity of the object.

[0106] In this embodiment of the invention, for two adjacent frames of the same image type, the difference between the image data of the same location point in the two adjacent frames can represent the movement distance of that location point; the time interval between the two adjacent frames can represent the movement duration of that location point, and the ratio of these two can be used to obtain the movement speed of that location point.

[0107] For visible light images, the image data for a given location is simply the pixel value in the image. After calculating the corresponding velocity using the method described above, the kinetic energy value at that location at an intermediate time point can be obtained. At the final moment, the kinetic energy at that point is

[0108] When calculating, for the same location point, m is generally fixed and can be substituted with a preset value.

[0109] For infrared images, if a traditional infrared image is used with an added temperature matrix, the image data for a single location can be the pixel values ​​from the traditional infrared image. Therefore, similar to visible light images, for infrared images, the corresponding E1 and E2 values ​​can also be calculated for each location.

[0110] For infrared images, if it's a thermal image generated by an infrared thermal imaging device, the image data consists of temperature values. Therefore, for the same location point, the difference in temperature values ​​between two adjacent frames can be used to represent the distance traveled at that location, thus allowing the calculation of the velocity. Similarly, E1 and E2 can be calculated for each location point, but this will not be discussed in detail here.

[0111] However, it is understandable that the dimensions of the thermal energy values ​​obtained by the two methods mentioned above may not be the same under infrared image types.

[0112] S412, based on the difference in kinetic energy values ​​at the next two times, the kinetic energy change index at that location is obtained.

[0113] First, the difference in kinetic energy at the location point at the last two moments can be calculated, and the absolute value of the difference can be obtained.

[0114] In other words, for this location point, we calculate ΔE = |E2 - E1|.

[0115] Secondly, the absolute value of this difference is used to obtain the kinetic energy change index at that location.

[0116] For example, ΔE at that location can be directly used as the kinetic energy change index at that location.

[0117] Alternatively, to ensure standardized operation at the same location point, a corresponding normalized value can be preset for each location point to serve as the denominator for normalizing ΔE at that location point. For example, the normalized value for a location point could be the maximum, minimum, or average value of ΔE determined based on a large number of data samples from test subjects at similar locations.

[0118] In one alternative implementation, the normalized value used at the same location point can be achieved using the kinetic energy value currently obtained at that location point. For example, it can use E1, E2, or the average of the two, all of which are reasonable.

[0119] For example, taking E1 as a normalization method, the exponent of kinetic energy change at a given location can be calculated using the following formula:

[0120]

[0121] Among them, Index EK This represents the index of kinetic energy change at that location.

[0122] Furthermore, by expanding and rearranging the above formula, we can obtain:

[0123]

[0124] Therefore, in one possible implementation, in order to reduce the amount of calculation and improve the calculation speed, for each position point, the mass can be ignored, and the two calculated velocity values ​​can be directly substituted into the above formula to calculate the kinetic energy change index of that position point.

[0125] Regardless of the method used to calculate the kinetic energy change index in this invention embodiment, it is based on the principle of kinetic energy. Ideally, to capture the movement of the subject's head and neck, it is necessary to know the trajectory of every point on the head and neck in space, so that it is possible to determine the true displacement of each point on the head and neck. However, in reality, it is impossible to achieve this effect. This is because: on the one hand, about 80% of the points on the human face have only slight differences from their neighbors, and there is a very high spatial correlation between adjacent points. Therefore, even if a planar image is obtained in real time from a single camera, the speed of a standard modern computer is insufficient to track the vector displacement of every point on the object using the planar image. On the other hand, the camera captures a planar image without depth information. Even if multi-angle shooting using the parallax between multiple cameras can achieve depth information estimation, it is difficult to process and fuse the planar image with the depth information to obtain accurate three-dimensional spatial displacement.

[0126] In this embodiment of the invention, the movement of the subject's head and neck is captured not by tracking the actual displacement of each point on the head and neck with a camera, but primarily by utilizing the kinetic energy obtained from the differences in image data between adjacent frames. In other words, the physical quantity used to measure the motion of an object in this embodiment primarily uses the kinetic energy obtained from the inter-frame differences, rather than the actual displacement of each point. This approach has two advantages: firstly, the amount of data generated by the inter-frame differences is significantly reduced compared to acquiring the actual displacement of each point; therefore, using the differences in image data between adjacent frames reduces the computational load. Secondly, since kinetic energy is a scalar while displacement is a vector, the inter-frame data differences obtained in this embodiment yield scalar kinetic energy rather than vector displacement differences. This avoids considering the influence of the direction of motion during calculation, further reducing the computational load and better aligning with the physics-based description of motion.

[0127] S413, the kinetic energy change indexes obtained at each position point of the head and neck are arranged according to the spatial distribution of the position points to obtain the kinetic energy change index matrix of the head and neck under this image type.

[0128] For each location point in the head and neck, a corresponding kinetic energy change index can be obtained. Arranging the various kinetic energy change indices according to the spatial position of the location point in the head and neck and in the image yields a kinetic energy change index matrix.

[0129] In one optional implementation of S42, it may include:

[0130] The kinetic energy change index matrices obtained from the two image types are fused together to obtain the target kinetic energy change index matrix.

[0131] In principle, the kinetic energy captured by visible light image acquisition equipment and infrared image acquisition equipment should be consistent at the same time. However, due to factors such as the possibility of missing pixels in visible light images, the data at corresponding positions in the kinetic energy change index matrices obtained from the two image types may not be the same. Therefore, for each position in the kinetic energy change index matrix, the values ​​at that position in the two kinetic energy change index matrices can be fused to obtain more complete, richer, and higher-quality data information.

[0132] Based on the preceding description, if the image data used to calculate the kinetic energy change index matrix is ​​pixel values, then the data dimensions in the kinetic energy change index matrix obtained from visible light images and the kinetic energy change index matrix obtained from infrared images are the same. Therefore, the two data points at corresponding positions in the two kinetic energy change index matrices can be directly fused to obtain a fused data set. All fused data points then constitute the target kinetic energy change index matrix. Specifically, for a given point in the target kinetic energy change index matrix, the fused data can be the maximum, average, or weighted average of the two data points to be fused.

[0133] If the image data used to calculate the kinetic energy change index matrix is ​​temperature value when using infrared image type, the data dimensions in the kinetic energy change index matrix obtained from visible light image type and the kinetic energy change index matrix obtained from infrared image type may be different. In this case, it may not be possible to directly perform fusion processing. In this case, the dimensions of the two types of data can be unified first according to the pre-determined multiplier factor, divisor factor, etc., and then fusion processing can be performed in the manner described above.

[0134] In this approach, the multiplier and divisor factors can be determined by comparing a large number of first kinetic energy change indices obtained from the same location point in visible light images with a large number of second kinetic energy change indices obtained in infrared images. Using statistical methods, the numerical levels (dimensions) of these two kinetic energy change indices are determined. Then, one or all of these indices can be multiplied by a corresponding value (as a multiplier factor); or one or all of these indices can be divided by a corresponding value (as a divisor factor); or one of these indices can be multiplied by a corresponding value and the other divided by a corresponding value, etc. This determines the unified data dimension method to be used for unknown test subjects, as well as the corresponding multiplier and divisor factors, etc., so that the two kinetic energy change index matrices of the test subject can be uniformly processed in this step.

[0135] In summary, through fusion processing, a fused kinetic energy change index matrix is ​​obtained, which is referred to here as the target kinetic energy change index matrix.

[0136] S5. Obtain thermal-kinetic image data based on the thermal energy change index matrix and the target kinetic energy change index matrix, and generate a thermal-kinetic image using the thermal-kinetic image data.

[0137] As described above, the embodiments of the present invention aim to combine the advantages of thermal radiation information in infrared images and detailed texture information in visible light images to obtain richer and more detailed image information and motion data.

[0138] In one alternative approach, thermal-kinetic image data can be directly constructed from the thermal energy change index matrix and the target kinetic energy change index matrix.

[0139] It is understandable that the thermal energy change index matrix reflects the spatial distribution of thermal energy changes at various points in the head and neck, while the target kinetic energy change index matrix reflects the spatial distribution of kinetic energy changes at various points in the head and neck. Therefore, the thermal-kinetic image data in this embodiment combines data information from both kinetic and thermal aspects. Compared to existing technologies that mostly rely on bioelectrical signals to obtain biological characteristics, such as electroencephalograms, this embodiment combines thermal and kinetic energy data information, resulting in a more comprehensive description of human characteristics.

[0140] After obtaining the thermal-kinetic energy image data, it can be used to generate thermal-kinetic energy images for intuitive display, observation, and analysis and detection of human body conditions. This part will be explained later.

[0141] The thermal-kinetic image data generation method provided in this invention combines three consecutive visible light and infrared images captured when the subject's head and neck are in a quasi-static state. This allows for the capture of more detailed head and neck movement information by integrating the spatial distribution data of kinetic energy changes from different image types. Simultaneously, by combining the spatial distribution data of thermal energy changes obtained from the three consecutive infrared images, thermal-kinetic image data is generated, combining kinetic and thermal energy data to obtain more comprehensive and detailed information for describing the human body's state. Furthermore, since the subject only needs to maintain a quasi-static state without any other cooperation, it avoids psychological or physiological changes caused by contact biometric extraction or other cooperation, thus better reflecting the subject's true physical condition. Moreover, the thermal-kinetic image data generation process in this invention is based on image processing techniques, requiring simple equipment, no operators, and no specialized operational requirements. Furthermore, by registering corresponding points in visible light and infrared images taken at the same time, data errors caused by differences in the shooting angles of the two image acquisition devices can be eliminated, improving the accuracy of subsequent thermal-kinetic image data. Additionally, because image acquisition time is short and image processing analyzes and processes data differences between frames of the same type, the amount of data processed is far less than the amount of data in a single frame, thus reducing computational load and increasing processing speed.

[0142] The following describes the optional implementation methods of the present invention.

[0143] Since images of the same type (visible light or infrared) acquired at two different times may have certain angular changes, misalignment of the same location points in the two images may occur. Therefore, in one optional implementation, before performing corresponding point registration on the visible light and infrared images at the same time, the thermal-kinetic image data generation method further includes:

[0144] For each of the three consecutive time points, the visible light images of the subject's head and neck in a quasi-stationary state acquired at every two adjacent time points are registered using corresponding points; and for each of the three consecutive time points, the infrared images of the subject's head and neck in a quasi-stationary state acquired at every two adjacent time points are registered using corresponding points.

[0145] In other words, before performing step S2, the three visible light images obtained in step S1 are registered with corresponding points, and the three infrared images obtained in step S1 are also registered with corresponding points. This can be simply referred to as inter-frame corresponding point registration.

[0146] For images of the same type, the objects of each registration of corresponding points are images from two consecutive time points. Optionally, the reference image in each registration can be the image from the later of these two time points. That is, for visible light or infrared images, the process of registering corresponding points for three frames is as follows: the images at time t1 and t2 are registered with the image at time t2; the images at time t2 and t3 are registered with the image at time t3. The process of registering corresponding points between frames is similar to the process of registering corresponding points for visible light and infrared images at the same time point described above, and will not be repeated here.

[0147] However, it should be noted that in this implementation, since the process of inter-frame registration is added, when S2 is performed after the inter-frame registration is completed, the visible light image and infrared image at the same moment in S2 refer to the visible light image and infrared image at that moment after the inter-frame registration has been performed.

[0148] By adding a corresponding point registration process between visible light and infrared images, two consecutive frames of the same type can be aligned at the same position, which can improve the image quality of visible light and infrared images at each moment. At the same time, it can also reduce the workload of corresponding point registration between visible light and infrared images at the same moment in subsequent S2.

[0149] As previously stated, embodiments of the present invention can employ a registration algorithm based on image features to achieve corresponding point registration. In one optional implementation, the corresponding point registration process includes:

[0150] The first image, which serves as the reference image, and the second image, which serves as the image to be registered, are input into the first and second LIFT networks, which have been pre-trained, respectively. The result is the registered image output by the second LIFT network, which is the second image registered with the first image as a reference. The first image and the registered image are then used as the result of the registration of corresponding points.

[0151] The first LIFT network and the second LIFT network were trained using a pre-built improved MatchNet network based on multiple sample reference images and sample images to be registered.

[0152] The corresponding point registration in this embodiment of the invention includes corresponding point registration between visible light images and infrared images at the same time, inter-frame corresponding point registration between visible light images at every two adjacent time points, and inter-frame corresponding point registration between infrared images at every two adjacent time points. All three types of corresponding point registration are trained using a pre-built improved MatchNet network, but the samples used during training and the network parameters obtained are not the same, and the network parameters are not the same in actual use.

[0153] As is known to those skilled in the art, the existing MatchNet network consists of two deep convolutional networks and one metric network. For a schematic diagram of the specific network structure, please refer to [link to schematic diagram]. Figure 2 The metric network consists of three fully connected layers, all of which use ReLU (Linear Rectification Function) as their activation function. Two deep convolutional networks (Deep Convolutional Network 1 and Deep Convolutional Network 2) are used to extract deep convolutional features from the input image, and the metric network is used to compare the similarity between these two deep convolutional features.

[0154] In this embodiment of the invention, an improved MatchNet network is obtained by replacing the deep convolutional network in the existing MatchNet network with a LIFT (Learned Invariant Feature Transform) network. For example... Figure 3 As shown in the embodiment of the present invention, the improved MatchNet network includes:

[0155] The first LIFT network, the second LIFT network, and the metric network.

[0156] In this system, the first and second LIFT networks are connected in parallel, with their outputs connected to a metric network. During training, the first and second LIFT networks are used to extract feature vectors from the input image, resulting in LIFT convolutional features in vector form. The metric network is used to measure the similarity between the two feature vectors.

[0157] Specifically, compared to the existing MatchNet network, this embodiment of the invention uses a LIFT network to generate feature descriptors. A metric network is used to measure the similarity between two feature vectors.

[0158] Traditional SIFT features remain invariant to basic image transformations such as rotation and scaling. When the changes between two images only involve basic transformations like rotation and scaling, SIFT features can achieve efficient and accurate registration of corresponding points. However, optical images are subject to changes in factors such as illumination, season, cloud cover, and human factors, resulting in an extremely complex nonlinear transformation process. This invention utilizes a LIFT network for feature extraction, thereby leveraging the nonlinear representation capabilities of the LIFT network to uncover the invariant features of feature points in the image under actual changing factors. This overcomes the limitations of SIFT features and, in conjunction with the improved MatchNet network, achieves high-performance target matching under complex conditions.

[0159] The traditional approach to comparing the similarity between feature vectors is to use Euclidean distance. However, manifold learning assumes that high-dimensional data is a high-dimensional embedding of low-dimensional features. On complex manifolds, Euclidean distance often fails to accurately measure the similarity between feature descriptors. This invention's improved MatchNet utilizes a LIFT deep neural network to mine LIFT convolutional features from large datasets. It employs a metric network containing three fully connected layers to compare the similarity between feature descriptors, achieving sub-pixel-level precise matching of corresponding points.

[0160] To facilitate understanding of the corresponding point registration process in the embodiments of the present invention, the training process using the improved MatchNet network is briefly described below. For any of the above corresponding point registration methods, the network architecture used in the training phase is built based on the improved MatchNet network. For details of the structure, please refer to [link to relevant documentation]. Figure 4 , Figure 4 This is a training structure diagram based on the improved MatchNet network provided in an embodiment of the present invention. During the training phase... Figure 4 The upper dashed box 10 corresponds to the first LIFT network, used to extract the first feature descriptor from the reference image. The lower dashed box 20 corresponds to the second LIFT network, used to perform image transformation on the target image (i.e., the image to be registered) based on the reference image, obtaining the transformed target image. Dashed box 20 is also used to extract the second feature descriptor from the transformed target image. The metric network 30 is used to calculate the matching score of the first and second feature descriptors. The reference image and the target image are a pair of images to be processed for corresponding point registration.

[0161] The second LIFT network corresponding to dashed box 20 includes a feature point detection module ( Figure 4 DET and Softmax activation modules (in the middle) Figure 4 softargmax and pruning modules (in the text) Figure 4 Crop), orientation estimation module ( Figure 4 ORI in the middle), rotating module ( Figure 4 Rot in the descriptor generation module ( Figure 4 (DESC in the model). Deep convolutional networks are used to learn image feature representations unsupervised during the Structure-from-Motion (SFM) process.

[0162] by Figure 4 Image P in 2 For example, P 2 A scoremap is generated after DET. 2 In softargmax, S 2Using the Softmax activation function, we obtain the position x of the center point of the region of interest. 2 Here, the Softmax function acts as a maximum suppression function, but because it is differentiable, the backpropagation algorithm can be used for end-to-end training of the network. Determine the center point x of the region of interest. 2 Afterwards, Crop to P 2 The image is cropped, and the cropped image patch is then input into ORI to estimate the rotation angle θ. 2 And by Rot according to θ 2 Rotate the image patch to obtain the rotated image patch. Will The input is fed into DESC, and DESC outputs LIFT convolutional features d. 2 .

[0163] The first LIFT network corresponding to dashed box 10 is obtained by modifying the second LIFT network. See [link to modification details] for details. Figure 4 As shown, the orientation estimation module and rotation module have been removed; the feature descriptors output by these two LIFT networks have the same feature dimension.

[0164] The metric module 30 is the metric subnetwork in the MatchNet network. This metric network consists of three fully connected layers, and the activation function of these three fully connected layers is ReLU (Linear rectification function).

[0165] When training the improved MatchNet network, a large number of sample pairs can be constructed, which can include three types of sample pairs: the first type consists of two images of the same person taken from different angles, along with their matching scores; the second type consists of two images of different people as targets; and the third type consists of two identical images of the same person. Furthermore, other sample pairs belonging to the category of negative samples can be constructed to improve the network's robustness. This embodiment of the invention does not limit the construction method of these augmented negative sample pairs. Then, existing image similarity evaluation methods are used to determine the ground truth value of the matching score for each sample pair for subsequent loss calculation. For example, gray-level correlation-based matching methods, mean squared error methods, or methods for calculating structural similarity (SSIM) can be used to evaluate image similarity.

[0166] In each sample pair, one frame serves as the baseline image, and the other as the target image. These sample pairs are sequentially fed into the network for training. During training, a loss function is used to calculate the loss value, and the weights within the network are adjusted according to the loss value until the network converges, resulting in the trained improved MatchNet network. The loss function used is as follows:

[0167]

[0168] In this loss function, This represents the truth value indicating whether two feature descriptors input into the metric network 30 match; This represents the matching score of the two feature descriptors actually output by the network, Q. loss is the loss value, and n is the number of sample pairs.

[0169] Therefore, when the reference image and target image to be registered are fed into the trained improved MatchNet network, the transformed target image output by the rotation module in the second LIFT network can achieve corresponding point matching with the reference image, and the matching scores of their feature descriptors are high, at least not lower than a preset lower limit. It is understandable that, in actual use of this improved MatchNet network, the matching score of the network output can be used as a supervisory parameter to monitor whether the improved MatchNet network is functioning correctly.

[0170] Compared to the existing MatchNet network, this invention uses a LIFT network to achieve feature extraction. This is because the changing factors in optical images include illumination, background, and human factors, making it an extremely complex nonlinear transformation process. This invention leverages the nonlinear representation capabilities of the LIFT network to uncover the invariant features of feature points in the image under actual changing factors, thereby working in conjunction with the improved MatchNet network to achieve high-performance target matching under complex conditions.

[0171] Furthermore, the traditional approach to comparing the similarity between feature vectors is to use Euclidean distance. However, in complex manifold learning, Euclidean distance often fails to accurately measure the similarity between feature descriptors. Therefore, this embodiment of the invention employs a metric network containing three fully connected layers to compare the similarity between feature descriptors, so as to evaluate the similarity of the registration results of corresponding points at the sub-pixel level.

[0172] In summary, utilizing Figure 4 The improved MatchNet network shown can achieve subpixel-level registration of corresponding points; thus, the embodiments of the present invention can extract more accurate kinetic energy change data and thermal energy change data from two types of images captured.

[0173] In one possible implementation of generating a thermal-kinetic image using thermal-kinetic image data, the data in the thermal energy change index matrix of the thermal-kinetic image data can be displayed to obtain a thermal energy change map; the data in the target kinetic energy change index matrix of the thermal-kinetic image data can be displayed to obtain a kinetic energy change map; and the thermal energy change map and the kinetic energy change map can be displayed as a single thermal-kinetic image using a side-by-side or top-bottom arrangement.

[0174] In one possible implementation of the thermal energy change graph, the values ​​in the thermal energy change index matrix can be directly displayed as a thermal energy change graph using corresponding colors. This employs a display method similar to a "heatmap," where different colors correspond to different values, allowing for the differentiation of the head and neck contours and the temperature levels of various parts of the body.

[0175] In another optional implementation, the values ​​in the thermal energy change index matrix can be graphically displayed to obtain a thermal energy change map. For example, the thermal energy change map can be a grid-like graph matching the number of elements in the thermal energy change index matrix, with the number of grids being the same as the number of elements in the thermal energy change index matrix. The i-th row and j-th grid of the thermal energy change map corresponds to the i-th row and j-th element in the thermal energy change index matrix. In the i-th row and j-th grid of the thermal energy change map, the length of the vertical rectangle extending from bottom to top represents the magnitude of the i-th row and j-th element in the thermal energy change index matrix. The correspondence between the length of the vertical rectangle and the value of the element in the thermal energy change index matrix can be preset. For example, a vertical rectangle length of 0 corresponds to a magnitude of 0 in the thermal energy change index matrix, and a vertical rectangle length equal to the height of a single grid corresponds to the maximum magnitude of the element in the thermal energy change index matrix. Multiple numerical levels can be divided between these two values, and different vertical rectangle lengths can be displayed with corresponding colors for easy subsequent observation. All of these are reasonable.

[0176] Furthermore, for easier observation, in the above embodiments, the left and right halves of the thermal energy change graph can be displayed alternately, for example, by leaving a certain gap or dividing it with a vertical center line that is easy to distinguish, so as to distinguish the left and right regions of the thermal energy change graph.

[0177] Similarly, the kinetic energy change graph can also be obtained using the above method. Accordingly, the thermal energy change graph and kinetic energy change graph in the thermal energy-kinetic energy image can be displayed in combination using the different methods described above.

[0178] In one possible implementation of generating thermal-kinetic images from thermal-kinetic image data, the following steps may be included:

[0179] (1) Generate thermal energy change images of the head and neck using thermal energy change index matrix.

[0180] Specifically, the values ​​in the thermal energy change index matrix can be assigned corresponding colors, creating a thermal energy change image of the head and neck that displays the contour and temperature information of the head and neck area, similar to a "heatmap." This thermal energy change image of the head and neck can distinguish the temperature levels of different regions and even individual pixels using color.

[0181] (2) The values ​​in the target kinetic energy change index matrix are displayed around the head and neck in the head and neck thermal energy change image through visualization, thus obtaining a thermal energy-kinetic energy image.

[0182] Specifically, a bounding box covering the head and neck region can be first determined in the thermal energy change image of the head and neck using detection methods. This bounding box is used to select the head and neck region and distinguish it from the background region. Optionally, the shape of the bounding box can be a circle, ellipse, rectangle, or other symmetrical shapes.

[0183] Then, the values ​​of the left and right regions of each row in the target kinetic energy change index matrix are displayed outside the corresponding region box in the corresponding row of the head and neck thermal energy change image using horizontal bars of a certain length.

[0184] Specifically, for the left and right regions of the target kinetic energy change index matrix, the horizontal bars within each region extend in the same direction as the region's orientation. For example, the horizontal bars in the left half of the region extend outwards to the left of the region frame. The region frame may not be clearly displayed in the thermal-kinetic energy image, while the horizontal bars can have a distinct color to differentiate them from the rest of the region for easier observation.

[0185] In this method, the subject's head and neck are displayed in the central area, with different colored "spots" representing the temperature levels of the head and neck. For a row of horizontal bars, from the left edge of the region frame to the left, the horizontal bars corresponding to the positions in the left half of that row of the target kinetic energy change index matrix are displayed; from the right edge of the region frame to the right, the horizontal bars corresponding to the positions in the right half of that row of the target kinetic energy change index matrix are displayed. At this point, the overall display effect of the thermal-kinetic energy image is that multiple rows of horizontal bars extend outward from the subject's head and neck region frame, forming a "halo".

[0186] In one alternative implementation, the length of the horizontal column corresponding to each row of the target kinetic energy change index matrix is ​​matched with the value.

[0187] In other words, if the value in the target kinetic energy change index matrix is ​​large, then the length of its horizontal column will be long. The correspondence between the value in the target kinetic energy change index matrix and the length of the horizontal column can be determined through prior experiments.

[0188] In one alternative implementation, for a single-sided area, the starting points of each horizontal column in the same row can be the same, for example, all located at the edge of the area frame. Thus, for example, for the left half of the same row, each horizontal column extends to the left from the left edge of the area frame, but the endpoints may be different. In this way, the length of the sum of the lengths of these multiple horizontal columns is the length of the longest horizontal column.

[0189] Alternatively, for a single-sided region, the starting point of each horizontal column in the same row can be based on the edge of the region box, extending out according to pixel intervals. For example, for the left half of the same row of the target kinetic energy change index matrix, the horizontal column for the rightmost value in this region extends leftward from the left edge of the region box. Then, the starting point of the first value to the left of that value is spaced a certain distance to the left of the region box edge, such as one pixel; the starting point of the second value to the left of that value is spaced two pixels to the left of the region box edge, and so on. In this method, the total length of the multiple horizontal columns still primarily depends on the length of the longest horizontal column.

[0190] The inventors discovered that if a point on the head and neck moves under the control of body or facial muscles, adjacent points will also make similar movements, but with a lower amplitude. This movement of adjacent points is not determined by psychophysiological reasons, but by the anatomical characteristics of the human body. Therefore, the maximum values ​​of the left and right regions in each row of the target kinetic energy change index matrix are the most valuable. In previous implementations, a row of the thermal-kinetic energy image showed the effect of multiple horizontal columns appearing on one side, with the starting points of these horizontal columns being the same. The length of the superimposed lengths of these horizontal columns is the length of the longest horizontal column. Therefore, the maximum values ​​of the left and right regions in each row of the target kinetic energy change index matrix can be used to generate the respective horizontal columns. The maximum values ​​of each row of the target kinetic energy change index matrix can be obtained by scanning line by line.

[0191] It is understandable that, in the above-described embodiments, for the same row, the symmetry of the spatial distribution of head and neck kinetic energy information in that row can be determined by the difference in length between the two transverse columns.

[0192] Regarding the effects of the above-mentioned thermal-kinetic energy images, taking the maximum value selected from the left and right regions of each row of the target kinetic energy change index matrix as an example, please refer to [link to example]. Figure 5 , Figure 5 This is a realistic rendering of a thermal-kinetic energy image provided by an embodiment of the present invention. The original image is displayed in grayscale. In this image, the region is bounded by a circle, with multiple rows of horizontal columns extending outward from the edge of the circle to form a "aperture". The temperature information distribution in the central area is represented by light spots of different colors.

[0193] In an alternative implementation, the horizontal bars corresponding to each row of values ​​in the target kinetic energy change index matrix have the same length, but their colors match the numerical values. The correspondence between the numerical values ​​and the colors of the horizontal bars in the target kinetic energy change index matrix can be determined through prior experiments.

[0194] In this implementation, the starting point of the horizontal columns can be similar to any of the methods described above. However, all horizontal columns can have the same length, and colors are used to distinguish the magnitude of the values ​​in the target kinetic energy change index matrix, similar to the display method of a heat map, with darker colors indicating larger values, and so on.

[0195] For simplification, optionally, the maximum value can be selected for the left and right regions of each row in the target kinetic energy change index matrix to generate their respective horizontal columns.

[0196] It is understandable that, in the above implementation, for the same row, the symmetry of the spatial distribution of head and neck kinetic energy information in that row can be known by the color difference of the horizontal columns on both sides.

[0197] Regarding the effect of the thermal-kinetic energy image of this implementation method, taking the maximum value selected from the left and right regions of each row of the target kinetic energy change index matrix as an example, please refer to [link to example]. Figure 6 , Figure 6 This is a schematic diagram illustrating another thermal-kinetic energy image provided in an embodiment of the present invention, wherein all horizontal columns have the same length, and colors are used to distinguish numerical values. Figure 6 Different shades of gray are used to represent different areas. Therefore, for a single row, the color comparison can be used to visually determine whether there is asymmetry between the left and right areas of the same row.

[0198] Of course, in the embodiments of the present invention, the generation and display methods of thermal-kinetic images are not limited to those described above.

[0199] Secondly, embodiments of the present invention provide a method for detecting human body status, comprising the following steps:

[0200] ①Using the thermal-kinetic image data generation method of the first aspect, obtain thermal-kinetic image data and / or thermal-kinetic images of the head and neck of the subject.

[0201] For information on how thermal-kinetic image data is acquired and how thermal-kinetic images are generated, please refer to the first section; it will not be elaborated upon here.

[0202] ② Detect the physiological or psychological state of the subject using thermal-kinetic image data and / or thermal-kinetic images.

[0203] In this embodiment of the invention, physiological state detection or psychological state detection can be performed. The detection principles of the two are similar, both utilizing the left-right symmetry of the spatial distribution of kinetic energy change and / or the spatial distribution of thermal energy change to achieve detection.

[0204] Specifically, viewed from the front, a person's head and neck can be considered relatively symmetrical. Therefore, observed asymmetry in movement indicates irregularity in head and neck movements. The inventors' research on electroencephalograms (EEGs) revealed significant differences in the symmetry of brain electrical activity in individuals in an active state. Therefore, in this embodiment of the invention, the asymmetry of elements at corresponding positions on the left and right sides of the thermal energy change index matrix and the target kinetic energy change index matrix can, to some extent, be used to analyze the psychological or physiological state of the test subject. For example, the presence of asymmetry may indicate a problem or change in the test subject's psychological state; it may also indicate a problem in the test subject's physiological state, i.e., the possible occurrence of a disease, and so on. Therefore, this data asymmetry can be used to detect psychological and physiological states to a certain extent.

[0205] When performing any human body state detection, it can be achieved using thermal-kinetic energy image data of the subject's head and neck, or using thermal-kinetic energy images of the subject's head and neck, or using thermal-kinetic energy image data combined with thermal-kinetic energy images.

[0206] In other words, this data asymmetry can be achieved by extracting and comparing data from thermal-kinetic image data, or by visually observing and comparing thermal-kinetic images, or by combining the two methods. But at its core, it mainly utilizes the numerical differences in thermal and kinetic energy information at corresponding positions in the left and right regions of each row of thermal-kinetic image data / images.

[0207] Therefore, in one optional implementation, step ②, using thermal-kinetic image data and / or thermal-kinetic images to detect the physiological or psychological state of the subject, includes:

[0208] By utilizing thermal-kinetic image data or the left-right symmetry of the spatial distribution of kinetic energy change and / or thermal energy change in thermal-kinetic images, the physiological or psychological state of the subject can be detected.

[0209] Since the principles of physiological state detection and psychological state detection are similar in the embodiments of the present invention, the following explanation will take psychological state detection as an example.

[0210] This invention allows for pre-conducting experiments with a large sample of test subjects to observe changes in the symmetry of the spatial distribution of thermal and kinetic information in thermal-kinetic image data and / or thermal-kinetic images under different psychological states. Relevant indicators for detecting psychological states can be determined. For example, regarding kinetic or thermal information, it can be determined that when the number of times asymmetric information distribution occurs in the corresponding positions of the left and right regions of a row of thermal-kinetic image data (or thermal-kinetic image, hereinafter similar) exceeds a certain value, the test subject exhibits a certain psychological state. Alternatively, it can be determined that when the total number of times asymmetric information distribution occurs in the corresponding positions of the left and right regions across all rows of thermal-kinetic image data exceeds a certain value, the test subject exhibits a certain psychological state. Alternatively, by discovering the relationship between movement in specific locations, such as the eyes, eyebrows, and mouth, and specific psychological states, the pixels at these specific locations in the thermal-kinetic image data can be determined. If there is an asymmetry in the spatial distribution of information in the corresponding locations of the left and right regions, or if the number of times the asymmetry in the spatial distribution of information occurs exceeds a certain value, the sample test subject is experiencing a specific psychological state, such as doubt, frustration, anger, etc. Alternatively, the frequency of asymmetric information distribution in the left and right regions of specific rows, all rows, and specific locations in the thermal-kinetic image data can be divided into different numerical intervals to correspond to different psychological states. For example, if the frequency of asymmetric information distribution in the left and right regions of a certain row in the thermal-kinetic image data falls within the first interval, it indicates that the sample test subject is in a state of dissatisfaction; if the frequency of asymmetric information distribution falls within the second interval with higher values, it indicates that the sample test subject is in a state of anger. Thus, the relationship between different interval values ​​and the degree of a certain psychological state can be determined, and so on. Kinetic energy information or thermal energy information can be used alone or in combination. Other methods will not be illustrated here.

[0211] In the process of conducting experiments on sample test subjects, other physiological and psychological testing methods can be used to help determine the psychological state of the sample test subjects and help establish the relationship between exercise and specific psychological states. For example, the respiratory rate, heart rate, electroencephalogram, etc. of the sample test subjects can be detected.

[0212] Based on the above experiments, some indicators for detecting human psychological states can be identified. In actual psychological state testing, the symmetry of the spatial distribution of head and neck kinetic and / or thermal energy information can be statistically analyzed from the subject's thermal-kinetic image data. This allows for the detection of the subject's psychological state using existing experimental conclusions. The process for detecting physiological states is similar to that for detecting psychological states and will not be repeated here.

[0213] It is understood that, through the above processing, the embodiments of the present invention can obtain a human body state detection result of the subject. The human body state detection result can be a physiological state detection result, a psychological state detection result, or both.

[0214] In one optional implementation, the present invention can acquire visible light video and infrared video of the head and neck of the subject in a quasi-static state over a period of time, extract three consecutive visible light images from the visible light video, and extract three consecutive infrared images from the infrared video, to perform S1 and subsequent steps, and obtain a human body state detection result of the subject.

[0215] Therefore, in this implementation method, multiple human state detection results can be obtained over a period of time for both visible light and infrared video. Furthermore, statistical analysis can be performed on the sequentially obtained human state detection results to obtain the final human state detection result of the subject within that period, such as calculating the average state of any one state detection result, plotting the state change curve, and so on.

[0216] As described above, embodiments of the present invention utilize the numerical differences in thermal energy information and kinetic energy information at corresponding positions in the left and right regions of each row of thermal-kinetic image data / images to achieve human state detection. In one optional implementation, the thermal-kinetic image data and the thermal-kinetic image can be directly constructed using the data differences at corresponding positions. For example, in one optional implementation, obtaining the thermal-kinetic image data based on the thermal energy change index matrix and the target kinetic energy change index matrix may include:

[0217] A1. Calculate the difference between the values ​​at corresponding positions in the left and right regions of the thermal energy change index matrix to obtain the thermal energy change index difference matrix.

[0218] A2, calculate the difference between the values ​​at corresponding positions in the left and right regions of the target kinetic energy change index matrix to obtain the target kinetic energy change index difference matrix.

[0219] A3 consists of thermal energy-kinetic energy image data composed of the thermal energy change index difference matrix and the target kinetic energy change index difference matrix.

[0220] Specifically, the elements in the aforementioned thermal energy change index difference matrix and target kinetic energy change index difference matrix can be used to calculate the modulus values ​​to construct thermal energy-kinetic energy image data.

[0221] It is understandable that in this thermal-kinetic image data, the thermal energy change index difference matrix or the target kinetic energy change index difference matrix can already reflect the difference in the corresponding position data of the left and right regions. Therefore, this difference data can be used to perform detection according to the aforementioned human body state detection method, which will not be explained in detail here.

[0222] After obtaining the aforementioned thermal-kinetic energy image data, thermal-kinetic energy images can be generated using some graphical display techniques.

[0223] Specifically, a thermal energy change difference image of the head and neck can be generated first using a thermal energy change index difference matrix as a "base" central region image, and then displayed in the aforementioned "heatmap" manner. It is understandable that for the thermal energy change difference image of the head and neck, the temperature difference between corresponding positions on the left and right sides can be displayed in different colors, clearly showing the locations of areas with large temperature differences.

[0224] Then, the values ​​in the target kinetic energy change index difference matrix can be displayed around the head and neck in the head and neck thermal energy change difference image through visualization, thus obtaining a thermal energy-kinetic energy image.

[0225] Alternatively, in one optional implementation, obtaining thermal-kinetic image data based on the thermal energy change index matrix and the target kinetic energy change index matrix may include:

[0226] B1. Calculate the difference between the values ​​at corresponding positions in the left and right regions of the target kinetic energy change index matrix to obtain the target kinetic energy change index difference matrix.

[0227] B2 consists of thermal energy change index matrix and target kinetic energy change index difference matrix, forming thermal energy-kinetic energy image data.

[0228] It is understandable that in this thermal-kinetic image data, the target kinetic energy change index difference matrix can reflect the difference in data at corresponding positions in the left and right regions, while the thermal energy change index matrix still reflects the original temperature information at each position point of the head and neck. Using the above thermal-kinetic image data, detection can also be performed according to the aforementioned human body state detection method, which will not be explained in detail here.

[0229] After obtaining the aforementioned thermal-kinetic image data, a thermal-kinetic image can be generated using graphical display techniques. Specifically, the original thermal change index matrix is ​​used to generate a head and neck thermal change image as a "base" central region image, which is then displayed in the aforementioned "heatmap" manner. As described earlier, for the head and neck thermal change image, the temperature of each region, and even individual pixel locations, can be distinguished by color. Then, the values ​​in the target kinetic change index difference matrix are displayed visually around the head and neck in the head and neck thermal change image, resulting in a thermal-kinetic image.

[0230] In any of the above methods, utilizing the horizontal columns described earlier, or one or a combination of different colors displayed on the columns, is reasonable. Therefore, the thermal-kinetic energy image obtained through intuitive observation can be used to detect human body conditions. The specific process will not be described in detail here.

[0231] The human body state detection method provided in this invention first obtains thermal-kinetic image data using a thermal-kinetic image data generation method, and further generates thermal-kinetic images. During the acquisition of thermal-kinetic image data, this invention combines three consecutive visible light and infrared images captured when the subject's head and neck are in a quasi-static state. This allows for the capture of more detailed head and neck movement information by combining the spatial distribution data of kinetic energy changes from different types of images. Simultaneously, by combining the thermal energy change spatial distribution data obtained from the three consecutive infrared images, thermal-kinetic image data is obtained, combining kinetic and thermal energy data to obtain more comprehensive and detailed data for describing the human body state. Furthermore, throughout the entire human body state detection process, the subject only needs to maintain a quasi-static state; no further cooperation is required. This avoids psychological or physiological changes caused by tension during contact biometric extraction or other cooperation, which could affect the detection results. Since the subject does not need to cooperate, their true state can be preserved to the greatest extent possible, meeting the needs of human body state detection under concealed conditions. Furthermore, the human body state detection process in this embodiment of the invention is based on image processing techniques, requiring simple equipment, no operators, and no specialized operational skills. Additionally, during the generation of thermal-kinetic image data, registration of corresponding points between visible light and infrared images taken at the same time eliminates data errors caused by differences in the shooting angles of the two image acquisition devices, improving the accuracy of subsequent thermal-kinetic image data. Moreover, because image acquisition time is short and the image processing analyzes and processes data differences between frames of the same type, the data processing volume is far less than the data volume of a single frame image, thus reducing computational load, increasing detection speed, and enabling real-time detection.

[0232] Thirdly, embodiments of the present invention provide a thermal-kinetic image data generation device. For example... Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703, and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0233] Memory 703 is used to store computer programs;

[0234] The processor 701 is used to implement the steps of the human state detection method in the second aspect when executing the program stored in the memory.

[0235] The device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made herein; any electronic device that can implement this invention falls within the protection scope of this invention.

[0236] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0237] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0238] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0239] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0240] Since this device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0241] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0242] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for generating thermal-kinetic image data, characterized in that, include: Visible light and infrared images of the head and neck of the subject in a quasi-static state are acquired synchronously at three time points before and after the same time interval; wherein each infrared image carries the temperature value of each point on the head and neck. Register corresponding points on the visible light image and infrared image at the same time to obtain the registered image group corresponding to each time. The temperature values ​​of each position point of the head and neck are obtained from each infrared image in the registered image group corresponding to all time points. For each position point, the thermal energy change index of the position point is calculated based on the thermal energy using the temperature values ​​of the position point at the three time points. The thermal energy change indices obtained for each position point of the head and neck are arranged according to the spatial distribution of the position points to obtain a thermal energy change index matrix that characterizes the spatial distribution of thermal energy change at each position point of the head and neck. In each image type within the registered image group corresponding to all time points, for each location point of the head and neck, the image data of that location point is calculated based on kinetic energy to obtain the kinetic energy value corresponding to that location point at the last two time points under that image type. Based on the difference between the kinetic energy values ​​corresponding to that location point at the last two time points, the kinetic energy change index of that location point is obtained. The kinetic energy change indices obtained for each location point of the head and neck are arranged according to the spatial distribution of the location points to obtain the kinetic energy change index matrix of the head and neck under that image type to characterize the spatial distribution of kinetic energy change at each location point of the head and neck. The kinetic energy change index matrices obtained from the two image types are fused to obtain a target kinetic energy change index matrix. Thermal-kinetic image data is obtained based on the thermal energy change index matrix and the target kinetic energy change index matrix, and a thermal-kinetic image is generated using the thermal-kinetic image data.

2. The method for generating thermal-kinetic image data according to claim 1, characterized in that, Before performing corresponding point registration on the visible light image and infrared image at the same time, the method further includes: For each of the three consecutive time points, the visible light images of the subject's head and neck in a quasi-static state acquired at every two adjacent time points are registered using corresponding points; and for each of the three consecutive time points, the infrared images of the subject's head and neck in a quasi-static state acquired at every two adjacent time points are registered using corresponding points.

3. The method for generating thermal-kinetic image data according to claim 1 or 2, characterized in that, The process of registering corresponding points includes: The first image, serving as the reference image, and the second image, serving as the image to be registered, are respectively input into a pre-trained first LIFT network and a second LIFT network. The result is a registered image output by the second LIFT network, which registers the second image with the first image as a reference. The first image and the registered image are then used as the result of the registration of the corresponding points. The first LIFT network and the second LIFT network are trained using a pre-built improved MatchNet network based on multiple sample reference images and sample images to be registered.

4. The method for generating thermal-kinetic image data according to claim 3, characterized in that, The improved MatchNet network includes: The system comprises a first LIFT network, a second LIFT network, and a metric network; wherein the first LIFT network and the second LIFT network are connected in parallel, and their outputs are connected to the metric network; during the training phase, the first LIFT network and the second LIFT network are used to extract feature vectors from the input image, and the metric network is used to measure the similarity between the two feature vectors.

5. The method for generating thermal-kinetic image data according to claim 1, characterized in that, The process of generating a thermal-kinetic image using the thermal-kinetic image data includes: The thermal energy change index matrix is ​​used to generate a thermal energy change image of the head and neck; The values ​​in the target kinetic energy change index matrix are displayed visually around the head and neck in the head and neck thermal energy change image to obtain the thermal energy-kinetic energy image.

6. A method for detecting human body status, characterized in that, include: Using the thermal-kinetic image data generation method according to any one of claims 1 to 5, thermal-kinetic image data and / or thermal-kinetic images of the head and neck of the subject are obtained; The physiological or psychological state of the subject is detected using the thermal-kinetic image data and / or the thermal-kinetic image.

7. A human body condition detection device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the method described in claim 6.

Images