A method for generating three-dimensional dynamic thermal-kinetic image data

Abstract

This invention discloses a method for generating three-dimensional dynamic thermal-kinetic image data, comprising: simultaneously capturing images of the head and neck using a three-channel imaging module to obtain three sets of imaging data; each set of data includes visible light video, thermal infrared video, and time-series elevation data; based on the three channels of visible light video and three channels of thermal infrared video, extracting and fusing the first and second dynamic data to obtain the third dynamic data, which characterizes the rate of change of kinetic energy at each point of the head and neck on the imaging plane; based on the three channels of time-series elevation data, extracting the fourth dynamic data, which characterizes the rate of change of kinetic energy at each point of the head and neck in the depth direction; and forming three-dimensional dynamic thermal-kinetic image data based on the third and fourth dynamic data and the thermal infrared video. This invention can more comprehensively and accurately characterize and capture the head and neck movement information of a person, thereby improving the accuracy of subsequent assessment or analysis of human functional status based on the extracted information.

Description

Technical Field

[0001] This invention belongs to the field of data generation, specifically relating to a method for generating three-dimensional dynamic thermal-kinetic image data. Background Technology

[0002] Traditional systems for detecting human functional status are mostly based on the stability or time variability of specific physiological parameters, such as fitness trackers and lie detectors. These physiological parameters include electrocardiograms (ECGs), electroencephalograms (EEGs), and skin conductance responses.

[0003] However, the above physiological parameters are only used to detect the human body from the microscopic perspective of bio-discharge, which is a narrow perspective and the information obtained is rather one-sided. Summary of the Invention

[0004] To address the aforementioned problems in the existing technology, this invention provides a method for generating three-dimensional dynamic thermal-kinetic image data. The technical problem to be solved by this invention is achieved through the following technical solution:

[0005] A method for generating three-dimensional dynamic thermal-kinetic image data includes:

[0006] Three-channel shooting modules were used to simultaneously capture images of the head and neck of the person from different angles, resulting in three sets of shooting data. Each set of shooting data included: visible light video, thermal infrared video, and time-series elevation data.

[0007] Based on the three visible light video streams, the kinetic energy change rate information of each point in the head and neck over time is extracted to obtain the first dynamic data. Based on the three thermal infrared video streams, the kinetic energy change rate information of each point in the head and neck over time is extracted to obtain the second dynamic data.

[0008] The first dynamic data and the second dynamic data are fused to obtain the third dynamic data, which characterizes the rate of change of kinetic energy of each point on the head and neck on the shooting plane over time.

[0009] Based on the three-channel time-series elevation data, the kinetic energy change rate information of each point in the head and neck in the depth direction over time is extracted to obtain the fourth dynamic data.

[0010] Three-dimensional dynamic thermal-kinetic image data is formed based on the third dynamic data, the fourth dynamic data, and the thermal infrared video, and a three-dimensional dynamic thermal-kinetic image is generated based on the three-dimensional dynamic thermal-kinetic image data.

[0011] Optionally, the step of extracting the kinetic energy change rate information of each point in the head and neck over time based on the three visible light video streams to obtain the first dynamic data includes:

[0012] For each of the visible light video streams, inter-frame subpixel level registration of corresponding points is performed;

[0013] For each registered visible light video, based on every two adjacent frames in the video, calculate multiple first kinetic energy change rate matrices for each point in the head and neck over time.

[0014] The first dynamic data is formed by splicing together every three first kinetic energy change rate matrices that are time-corresponding from the three visible light video streams respectively.

[0015] Based on the three channels of thermal infrared video, the kinetic energy change rate information of each point in the head and neck over time is extracted to obtain the second dynamic data, including:

[0016] For each of the aforementioned thermal infrared video streams, inter-frame subpixel level registration of corresponding points is performed.

[0017] For each registered thermal infrared video, based on every two adjacent frames in the video, calculate multiple second kinetic energy change rate matrices for each point in the head and neck over time.

[0018] The second dynamic data is formed by splicing together every three second kinetic energy change rate matrices that are time-corresponding from the three thermal infrared videos respectively.

[0019] Optionally, based on the three channels of time-series elevation data, the kinetic energy change rate information of each point in the head and neck along the depth direction over time is extracted to obtain the fourth dynamic data, including:

[0020] Each set of three elevation data matrices, which are derived from the three time-series elevation data sources and have a time correspondence, is spliced ​​together to obtain multiple spliced ​​data matrices.

[0021] Using a spliced ​​data matrix of two adjacent data measurement times, multiple third kinetic energy change rate matrices of the head and neck in the depth direction are calculated over time.

[0022] The fourth dynamic data is obtained based on the plurality of third kinetic energy change rate matrices.

[0023] Optionally, obtaining the fourth dynamic data based on the plurality of third kinetic energy change rate matrices includes:

[0024] Each of the third kinetic energy change rate matrices is expanded using a cubic spline interpolation algorithm to obtain multiple expanded matrices;

[0025] Data is sampled from each of the expanded matrices according to the preset sampling coordinates to obtain the fourth dynamic data;

[0026] The sampling coordinates are calculated in advance based on the ranging coverage area of ​​the test module and the video framing coordinates.

[0027] Optionally, the step of concatenating every three elevation data matrices that are time-corresponding and respectively derived from the three channels of time-series elevation data includes:

[0028] For every two adjacent elevation data matrices, the row-to-row or column-to-column similarity of the two elevation data matrices is calculated based on the deployment of the three-channel shooting modules relative to the personnel object, and the splicing boundary is determined based on the similarity calculation results; based on the splicing boundary, the two adjacent elevation data matrices are spliced ​​together.

[0029] Among them, adjacent elevation data matrices refer to the distance measuring modules that measured the elevation data matrices being physically adjacent.

[0030] Optionally, the third dynamic data includes multiple planar kinetic energy change rate matrices sorted over time on the shooting plane for each point of the head and neck, and the fourth dynamic data includes multiple depth kinetic energy change rate matrices sorted over time in the depth direction for each point of the head and neck, wherein the planar kinetic energy change rate matrix and the depth kinetic energy change rate matrix have the same dimension.

[0031] The process of forming three-dimensional dynamic thermal-kinetic image data based on the third dynamic data, the fourth dynamic data, and the thermal infrared video includes:

[0032] Summing is performed on each pair of planar kinetic energy rate of change matrices and depth kinetic energy rate of change matrices that have a time correspondence, resulting in multiple three-dimensional kinetic energy rate of change matrices sorted over time.

[0033] Based on the multiple three-dimensional kinetic energy change rate matrices and the thermal infrared video, three-dimensional dynamic thermal-kinetic energy image data is formed.

[0034] Optionally, the third dynamic data includes multiple planar kinetic energy change rate matrices sorted over time on the shooting plane for each point of the head and neck, and the fourth dynamic data includes multiple depth kinetic energy change rate matrices sorted over time in the depth direction for each point of the head and neck, wherein the planar kinetic energy change rate matrix and the depth kinetic energy change rate matrix have the same dimension.

[0035] The process of forming three-dimensional dynamic thermal-kinetic image data based on the third dynamic data, the fourth dynamic data, and the thermal infrared video includes:

[0036] For each pair of planar kinetic energy change rate matrices and depth kinetic energy change rate matrices that have a time correspondence, the modulus of the planar vector is calculated by using the pairs of elements with the same position in the matrix as the horizontal and vertical projection coordinates of the planar vector, thus obtaining a modulus matrix.

[0037] Three-dimensional dynamic thermal-kinetic image data is formed based on the obtained multiple mode matrices and the thermal infrared video.

[0038] Optionally, the step of forming three-dimensional dynamic thermal-kinetic image data based on the third dynamic data, the fourth dynamic data, and the thermal infrared video includes:

[0039] Obtain multiple raw temperature data matrices associated with each of the thermal infrared video streams;

[0040] Each set of three original temperature data matrices, which are associated with the three thermal infrared videos and have a time-corresponding relationship, is spliced ​​together to obtain multiple spliced ​​temperature data matrices.

[0041] Based on the spliced ​​temperature data matrix of two adjacent temperature acquisition times, calculate multiple thermal energy change rate matrices of each point on the head and neck of the person subject over time.

[0042] Three-dimensional dynamic thermal-kinetic image data is formed based on the third dynamic data, the fourth dynamic data, and the multiple thermal energy change rate matrices.

[0043] Optionally, the method of generating a three-dimensional dynamic thermal-kinetic image based on the three-dimensional dynamic thermal-kinetic image data includes:

[0044] A three-dimensional thermal image of the head and neck of the person is generated based on the thermal infrared video;

[0045] Based on the correspondence between the third dynamic data, the fourth dynamic data, and the three-dimensional head and neck thermal image and each point on the head and neck, the three data are synthesized and displayed to obtain a three-dimensional dynamic thermal-kinetic image.

[0046] Optionally, the method further includes:

[0047] By statistically analyzing the third dynamic data, the period of change of the kinetic energy change rate of each point on the head and neck on the shooting plane is obtained;

[0048] By statistically analyzing the fourth dynamic data, the period of change of the kinetic energy change rate of each point on the head and neck on the shooting plane is obtained;

[0049] The intersection of the two obtained change periods is taken as the change period of the kinetic energy change rate of each point in the head and neck in three-dimensional space.

[0050] The period of change of the kinetic energy change rate of each point in the head and neck in three-dimensional space is added to the three-dimensional dynamic thermal-kinetic image data.

[0051] The beneficial effects of this invention are:

[0052] The method for generating three-dimensional dynamic thermal-kinetic image data provided by this invention employs a three-channel shooting module to simultaneously capture images of the head and neck of a person from different angles, obtaining three sets of shooting data. Each set of shooting data includes visible light video, thermal infrared video, and time-series elevation data. Specifically, this invention extracts the rate of change of kinetic energy of each point on the head and neck under two different photoelectric sensing conditions based on the three channels of visible light video and the three channels of thermal infrared video, and fuses the two types of information to obtain the rate of change of kinetic energy of the person on the shooting plane over time (i.e., the third dynamic data). Furthermore, this invention also extracts the rate of change of kinetic energy of each point on the head and neck in the depth direction over time based on the three channels of time-series elevation data (i.e., the fourth dynamic data). Therefore, this invention, based on the third dynamic data, the fourth dynamic data, and the thermal infrared video, forms three-dimensional dynamic thermal-kinetic image data, comprehensively and meticulously extracting the head and neck movement information of the person from multiple dimensions including kinetic energy, thermal energy, time, and three-dimensional space. Thus, even when a person is under stress, anxiety, or suffering from illness, the present invention can accurately capture and characterize head and neck movements caused by the reflexes of the vestibular system, thereby improving the effectiveness and accuracy of subsequent assessments or analyses of human functional status based on the three-dimensional dynamic thermal-kinetic image data.

[0053] Compared with existing technologies for collecting information such as electrocardiograms, electroencephalograms, and skin conductance responses, this invention achieves a contactless data extraction scheme, which can reduce the anxiety of the subjects being collected due to physical contact with the collection instrument and reduce the errors introduced by the collection event itself.

[0054] Furthermore, when the present invention displays the three-dimensional dynamic thermal-kinetic image data of a person in the form of an image, viewers can more intuitively view the data of the person and thus quickly obtain some human functional status information of the person based on the image data.

[0055] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0056] Figure 1 This is a flowchart illustrating a method for generating three-dimensional dynamic thermal-kinetic image data according to an embodiment of the present invention;

[0057] Figures 2(a) to 2(c) This is a schematic diagram illustrating three ways of deploying three sets of shooting modules in an embodiment of the present invention;

[0058] Figure 3 This is a schematic diagram of the time correspondence mentioned in the embodiments of the present invention;

[0059] Figure 4 This is a schematic diagram illustrating the splicing of elevation data matrices in an embodiment of the present invention;

[0060] Figure 5 This is a schematic diagram of the planar vector mentioned in the embodiments of the present invention;

[0061] Figure 6 This is a schematic diagram of the three-dimensional head and neck model used in the embodiments of the present invention;

[0062] Figure 7 This is a schematic diagram illustrating the visualization of three-dimensional dynamic thermal-kinetic image data in an embodiment of the present invention;

[0063] Figure 8 This is another schematic diagram illustrating the visualization of three-dimensional dynamic thermal-kinetic image data in an embodiment of the present invention;

[0064] Figure 9 This is another schematic diagram illustrating the visualization of three-dimensional dynamic thermal-kinetic image data in an embodiment of the present invention;

[0065] Figure 10 and Figure 11 This constitutes a schematic diagram of the process of generating three-dimensional dynamic thermal-kinetic image data in a preferred embodiment of the present invention;

[0066] Figure 12 This is a schematic diagram of the structure of the improved MatchNet network used in the embodiments of the present invention;

[0067] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0068] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0069] The vertical balance of the human head is controlled by the vestibular system, a function known as the vestibular reflex. Head movements are the most frequent repetitive movements throughout a person's life. The head makes continuous, minute movements until it finds support, at which point it stops adjusting. During this adjustment process, numerous minute movements at the scale of hundreds of micrometers are performed every second.

[0070] There is a direct link between bodily function and physical movement. For example, when a person is calm and at rest, their heart rate and respiration are at their lowest; while when a person is excited, their breathing and heart rate increase. Similarly, when a person is tense, anxious, or suffering from illness, they experience uncontrollable primary tension, which, under the reflex of the vestibular system, leads to subtle movements of the head and neck and changes in temperature. By collecting and analyzing data on these changes, we can gain some insight into the body's functional state.

[0071] In order to more comprehensively and accurately characterize and capture the head and neck movement information of human subjects, thereby improving the accuracy of subsequent assessment or analysis of human functional status based on the extracted information, this invention provides a method for generating three-dimensional dynamic thermal-kinetic image data.

[0072] See Figure 1 As shown, the three-dimensional dynamic thermal-kinetic image data generation method provided in this embodiment of the invention includes the following steps:

[0073] S1: A 3-channel shooting module is used to simultaneously shoot the head and neck of the person from different angles, resulting in 3 sets of shooting data; each set of shooting data includes: visible light video, thermal infrared video, and time-series elevation data.

[0074] During the filming process, the subject's head and neck are kept in a quasi-static state. Quasi-static refers to a relatively static state, rather than an absolutely static state, where the subject is required to keep their head and neck still.

[0075] In a quasi-static state, head and neck movements are determined by unconscious processes, primarily by the vestibular system. This manifests as subtle head and neck movements that are outwardly related to internal psychological factors or physiological conditions. In other words, head and neck movement information captured in a quasi-static state is closely related to the individual's internal state.

[0076] In practical applications, each imaging module includes a visible light camera, a thermal infrared camera, and a ranging module. The visible light camera captures visible light video; the thermal infrared camera captures thermal infrared video. It's understood that thermal infrared video is typically associated with multiple raw temperature data matrices, and the image presented in the thermal infrared video is obtained after pixel visualization processing based on these raw temperature data matrices. The ranging module can be a miniature short-range ranging radar, but it is not limited to this. The data actually measured by the ranging module includes multiple elevation data matrices ordered over time, hence it is called time-series elevation data.

[0077] In actual filming, the focus is primarily on filming the front side of the subject's head and neck, as facial features contain more useful information, while the back of the head contains less. The three-way camera module can be arranged in the following manner: Figures 2(a) to 2(c) The image shows any of the camera setups in the shooting environments shown. It can be seen that the framing of adjacent shooting modules overlaps to some extent in these different shooting environments.

[0078] Preferably, the shooting duration in step S1 can be 30 seconds to 180 seconds, but it is not limited to this.

[0079] In addition, in practical applications, in order to capture high-quality video and elevation data and extract more accurate information from them, it is necessary to improve the optical contrast of the background, enhance the quality of ambient lighting, ensure the stability of the camera installation, and set appropriate resolution and frame rate for the camera, etc.

[0080] S2: Based on 3 channels of visible light video, extract the kinetic energy change rate information of each point in the head and neck over time to obtain the first dynamic data, and based on 3 channels of thermal infrared video, extract the kinetic energy change rate information of each point in the head and neck over time to obtain the second dynamic data.

[0081] Specifically, based on three visible light video streams, the kinetic energy change rate information of each point in the head and neck over time is extracted to obtain the first dynamic data, which may include:

[0082] (1) For each visible light video, based on every two adjacent frames in the video, calculate multiple first kinetic energy change rate matrices of each point in the head and neck sorted over time.

[0083] (2) The first kinetic energy change rate matrices, which are derived from three visible light video streams and have a time correspondence, are spliced ​​together to form the first dynamic data.

[0084] For example, suppose a visible light video consists of M frames, sequentially labeled A1, A2, A3…A… M The difference between any two adjacent frames in the M-frame visible light image dataset yields M-1 difference matrices, namely: D1 = A2 - A1, D2 = A3 - A2, D3 = A4 - A3…D… M-1 =A M -A M-1The elements in these difference matrices represent the positional changes of various points on the head and neck of the person on the shooting plane. Therefore, dividing each of these M-1 difference matrices by the video frame interval yields new matrices whose elements represent the velocity v of the head and neck of the person on the shooting plane. Then, a virtual mass m is assigned to each point; for example, m = 100. The kinetic energy formula E = mv is then used to calculate the velocity v. 2 By dividing by 2, the kinetic energy of each point can be calculated, resulting in M-1 kinetic energy matrices. Then, using the kinetic energy of each point at adjacent time points, the rate of change of kinetic energy at that point can be calculated, forming M-2 first kinetic energy rate of change matrices. The calculation method for the rate of change of kinetic energy is shown in the following formula:

[0085]

[0086] In this formula, E i E represents the element in the second kinetic energy matrix among two adjacent kinetic energy matrices ordered over time. i-1 This indicates the element at the same position in the preceding kinetic energy matrix; Index Ek This represents the calculated rate of change of kinetic energy at the point corresponding to the element.

[0087] After calculating the matrices of the first rate of change of kinetic energy, it can be done as follows: Figure 3 As shown, every three first kinetic energy change rate matrices, which are time-corresponding and originate from three visible light video streams, are stitched together to form the first dynamic data. There are multiple ways to stitch the three first kinetic energy change rate matrices. To ensure clarity in the specification, specific stitching methods will be illustrated later.

[0088] In another implementation, to improve data accuracy, before calculating the first kinetic energy change rate matrix based on step (1) above, inter-frame subpixel level registration of corresponding points can be performed on each visible light video. Then, for each registered visible light video, multiple first kinetic energy change rate matrices sorted over time for each point in the head and neck are calculated in the manner described in step (1) above.

[0089] Subpixel-level registration of corresponding points can be achieved using neural networks. To ensure clarity in the specification, examples of how to implement subpixel-level registration of corresponding points using neural networks will be provided later.

[0090] The specific method for forming the second dynamic data can refer to the specific method for forming the first dynamic data. For example, based on three thermal infrared videos, the kinetic energy change rate information of each point in the head and neck over time is extracted to obtain the second dynamic data. This can include: performing sub-pixel-level registration of corresponding points for each thermal infrared video; for each registered thermal infrared video, calculating multiple second kinetic energy change rate matrices sorted over time for each point in the head and neck based on every two adjacent frames in the video; and stitching together every three second kinetic energy change rate matrices that are time-corresponding from the three thermal infrared videos to form the second dynamic data.

[0091] Understandably, for a single test module, the visible light camera, infrared camera, and ranging module are very closely spaced, so the shooting angles of the three relative to the person are not significantly different. Therefore, by setting the resolution of the visible light camera and the infrared camera to be the same, the dimensions of the stitched first kinetic energy change rate matrix and the stitched second kinetic energy change rate matrix can be matched.

[0092] S3: The first dynamic data and the second dynamic data are fused to obtain the third dynamic data, which represents the rate of change of kinetic energy of each point on the head and neck on the shooting plane over time.

[0093] Specifically, as mentioned above, the first dynamic data includes multiple stitched first kinetic energy change rate matrices, and the second dynamic data includes multiple stitched second kinetic energy change rate matrices. Here, the stitched first kinetic energy change rate matrix is ​​denoted as T2, and the stitched second kinetic energy change rate matrix is ​​denoted as T3. The formation process of the third dynamic data includes: normalizing the element values ​​in T2 and T3 to the same dimension. Then, according to the chronological order, each pair of T2 and T3 with a time correspondence is fused to obtain multiple planar kinetic energy change rate matrices of each point on the head and neck, ordered over time on the imaging plane, which serve as the third dynamic data. The specific fusion method includes: for each identical element position point in T2 and T3, when the element values ​​of T2 and T3 are not equal, the larger value or the average value is taken as the fusion result for that point; when the element values ​​of T2 and T3 are equal, the element value is retained.

[0094] S4: Based on three-channel time-series elevation data, extract the kinetic energy change rate information of each point in the head and neck in the depth direction over time to obtain the fourth dynamic data.

[0095] Each time-series elevation data stream includes multiple elevation data matrices; therefore, step S4 specifically includes the following sub-steps:

[0096] (1) Each of the three elevation data matrices that are derived from three time-series elevation data and have a time correspondence is spliced ​​together to obtain multiple spliced ​​data matrices.

[0097] When the physical locations of the ranging modules are adjacent, the elevation data matrices measured by these two ranging modules can be called adjacent elevation data matrices. Therefore, when stitching together three elevation data matrices with a time correspondence, for each pair of adjacent elevation data matrices, the inter-row similarity or inter-column similarity is calculated based on the arrangement of the three shooting modules relative to the personnel, to determine the stitching boundary. Specifically, when the three shooting modules are arranged horizontally relative to the personnel as shown in Figure 2(a) or Figure 2(b), the inter-column similarity of the two elevation data matrices is calculated; when the three shooting modules are arranged vertically relative to the personnel as shown in Figure 2(c), the inter-row similarity of the two elevation data matrices is calculated. Then, the two rows or two columns with the highest similarity are used as the stitching boundary for these two adjacent elevation data matrices, and the matrices are stitched together. During the splicing process, for every two elements that need to be merged, the average value of the two elements is taken.

[0098] For example, such as Figure 4 As shown, the three elevation data matrices B1, B2, and B3 have a temporal correspondence, with B1 and B2 being adjacent in position, and B2 and B3 being adjacent in position. Therefore, the column similarity between B1 and B2 is calculated, and the two columns with the highest similarity are selected as the splicing boundary (e.g., ...). Figure 4 As shown by the dotted line in the diagram, similarly, the column similarity between B2 and B3 is calculated, and the two columns with the highest similarity are selected as the splicing boundary for splicing.

[0099] In one alternative implementation, when the splicing boundary is determined based on the three elevation data matrices of the first group, subsequent groups can be spliced ​​according to this boundary, thereby reducing computational load. Alternatively, several groups of elevation data matrices from earlier times can be selected to attempt to determine the splicing boundary, and the splicing boundary with the highest similarity can be used as the official splicing boundary. This boundary can then be used for splicing the elevation data matrices corresponding to every three time periods.

[0100] Similarly, in step S2 above, the method of stitching together the first kinetic energy change rate matrix from the visible light video and the method of stitching together the second kinetic energy change rate matrix from the infrared video can refer to the method of stitching together the elevation data matrix here.

[0101] (2) Using the spliced ​​data matrix of two adjacent data measurement times, calculate multiple third kinetic energy change rate matrices of the head and neck in the depth direction over time.

[0102] For example, suppose that in step (1) a total of M spliced ​​data matrices are obtained, and they are ordered in chronological order as G1, G2, G3...G M By subtracting the concatenated data matrices from each pair of adjacent time points, we can obtain M-1 difference matrices, namely: △1=G2-G1, △2=G3-G2, △3=G4-G3…△ M-1 =G M -G M-1 The elements in these difference matrices can represent the positional changes of various points on the head and neck of a person in the depth direction. Therefore, by dividing each of these M-1 difference matrices by the elevation data acquisition interval, the elements in the resulting new matrices can represent the velocity v of the head and neck of the person in the depth direction. Similarly, if the virtual mass of each point is m, then the kinetic energy can be calculated using the formula E = mv. 2 By dividing by 2, the kinetic energy of each point in the depth direction can be calculated, thus obtaining M-1 kinetic energy matrices in the depth direction. Then, using the kinetic energy of each point at adjacent time points and the formula for calculating the rate of change of kinetic energy given above, the rate of change of kinetic energy of that point in the depth direction can be calculated, thus forming M-2 third kinetic energy rate of change matrices.

[0103] (3) Based on multiple third kinetic energy change rate matrices, the fourth dynamic data is obtained.

[0104] Here, there are multiple ways to implement step (3). For example, in one implementation, the fourth dynamic data is obtained based on multiple third kinetic energy change rate matrices, which may include:

[0105] (3-1) The cubic spline interpolation algorithm is used to expand the elements of each third kinetic energy rate matrix to obtain multiple expanded matrices;

[0106] (3-2) Data is sampled for each extended matrix according to the preset sampling coordinates to obtain the fourth dynamic data.

[0107] Therefore, the fourth dynamic data obtained includes multiple depth kinetic energy change rate matrices sorted by time at various points in the head and neck along the depth direction.

[0108] The sampling coordinates are pre-calculated based on the ranging coverage area of ​​the test module and the video view coordinates. Specifically, each test module corresponds to a set of sampling coordinates. Since the visible light camera, infrared camera, and ranging module within a single test module are very closely spaced and their positions are fixed, once the video view coordinates are known, they can be transformed to the ranging coverage area of ​​the ranging module based on the positional relationship between the ranging module and the camera, thus obtaining the sampling coordinates from the ranging module's perspective.

[0109] Understandably, if the ranging module's measurement resolution is lower than the video resolution, this implementation method expands the data volume, which is equivalent to increasing the ranging module's measurement resolution. Then, through sampling operations, the obtained depth kinetic energy change rate matrix is ​​made to match the data resolution of the aforementioned planar kinetic energy change rate matrix, that is, to make the dimensions of the two matrices the same, thus facilitating their fusion.

[0110] In another implementation, if the measurement resolution of the ranging module can match the video resolution, multiple third kinetic energy change rate matrices can be directly used as fourth dynamic data, that is, the third kinetic energy change rate matrix can be directly used as the depth kinetic energy change rate matrix.

[0111] S5: Based on the third dynamic data, the fourth dynamic data, and the thermal infrared video, form three-dimensional dynamic thermal-kinetic image data, and generate a three-dimensional dynamic thermal-kinetic image based on the three-dimensional dynamic thermal-kinetic image data.

[0112] There are various specific ways to generate three-dimensional dynamic thermal-kinetic image data based on third dynamic data, fourth dynamic data, and thermal infrared video.

[0113] For example, in the first implementation, forming three-dimensional dynamic thermal-kinetic image data based on the third dynamic data, the fourth dynamic data, and the thermal infrared video may include:

[0114] (1) Summing is performed on each pair of planar kinetic energy change rate matrices and depth kinetic energy change rate matrices that have a time correspondence, resulting in multiple three-dimensional kinetic energy change rate matrices sorted over time.

[0115] (2) Based on multiple three-dimensional kinetic energy change rate matrices and thermal infrared video, three-dimensional dynamic thermal-kinetic energy image data is formed.

[0116] Specifically, step (2) may include: extracting the video frames used to generate the planar kinetic energy change rate matrix from the three thermal infrared videos, for example, always taking the last frame among the multiple frames required to generate the planar kinetic energy change rate matrix, but not limited to this; then, stitching together every three video frames from the three thermal infrared videos that have a time correspondence to obtain a stitched image. When stitching the video frames, the stitching boundary can refer to the stitching boundary of the second kinetic energy change rate matrix. Then, the obtained multi-frame stitched image is matched one-to-one with multiple three-dimensional kinetic energy change rate matrices in chronological order, thus obtaining three-dimensional dynamic thermal-kinetic energy image data.

[0117] In the second implementation, the three-dimensional dynamic thermal-kinetic image data formed based on the third dynamic data, the fourth dynamic data, and the thermal infrared video can include:

[0118] (1) For each pair of planar kinetic energy change rate matrices and depth kinetic energy change rate matrices that have a time correspondence, the elements with the same position in the pair of matrices are used as the horizontal and vertical projection coordinates of the planar vector, and the modulus of the planar vector is calculated to obtain a modulus matrix.

[0119] (2) Three-dimensional dynamic thermal-kinetic image data are formed based on the obtained multiple mode matrices and thermal infrared video.

[0120] For example, suppose we have a pair of plane kinetic energy change rate matrices and depth kinetic energy change rate matrices, respectively:

[0121]

[0122] Using the pairs of elements with the same position in the matrix as the horizontal and vertical projection coordinates of the planar vectors, we can obtain four sets of planar coordinates: (x1, y1), (x2, y2), (x3, y3), and (x4, y4). The corresponding planar vectors can be represented as follows: Figure 5 As shown; then, the modulus of these four vectors is calculated, and the resulting modulus matrix is ​​as follows:

[0123]

[0124] Then, referring to the first implementation method, several frames of stitched images are generated based on thermal infrared video, which are then combined with the obtained multiple mode matrices to form three-dimensional dynamic thermal-kinetic image data.

[0125] In the third implementation, the three-dimensional dynamic thermal-kinetic image data formed based on the third dynamic data, the fourth dynamic data, and the thermal infrared video can include:

[0126] (1) Obtain multiple raw temperature data matrices associated with each thermal infrared video stream.

[0127] (2) The original temperature data matrices that are associated with the three thermal infrared videos and have a time correspondence are spliced ​​together to obtain multiple spliced ​​temperature data matrices.

[0128] Here, the method for stitching together the original temperature data matrix can be found in the above description of stitching together the first kinetic energy change rate matrix, the second kinetic energy change rate matrix, or the elevation data matrix, and will not be repeated here.

[0129] (3) Based on the spliced ​​temperature data matrix of every two adjacent temperature acquisition times, calculate multiple thermal energy change rate matrices of each point on the head and neck of the person as sorted over time.

[0130] For example, if there are M spliced ​​temperature data matrices, sorted along time as T mp-1 T mp-2 T mp-3 …T mp-M By subtracting each pair of adjacent spliced ​​temperature data matrices, we can obtain M-1 difference matrices, namely: X1 = T mp-2 -T mp-1 X2 = T mp-3 -T mp-2 X3 = T mp-4 -T mp-3 …X M-1 =T mp-M -T mp-M-1 The elements in these difference matrices represent the temperature difference Δt corresponding to each pixel in the thermal infrared image. Therefore, using the thermal energy calculation formula Q = c × m × Δt, the thermal energy data corresponding to each pixel can be estimated, resulting in M-1 thermal energy matrices to characterize the thermal energy data of each point on the head and neck of a person in the thermal infrared image as a function of time. Here, c is a virtual specific heat capacity assigned to the pixel, m is the aforementioned virtual mass, and Q is the calculated thermal energy. Then, based on the thermal energy of each pixel at adjacent time points, the rate of change of thermal energy at that point can be calculated using the formula for calculating the rate of change of thermal energy, thus forming M-2 thermal energy rate of change matrices. The formula for calculating the rate of change of thermal energy is as follows:

[0131]

[0132] Among them, Q i Q represents the element in the second-to-last thermal energy matrix among two adjacent thermal energy matrices sorted over time. i-1 This indicates the element at the same position in the preceding heat energy matrix; Index EQ This represents the calculated rate of change of thermal energy at the point corresponding to the element.

[0133] (4) Three-dimensional dynamic thermal-kinetic image data are formed based on the third dynamic data, the fourth dynamic data and multiple thermal energy change rate matrices.

[0134] Specifically, you can refer to the method of fusing the third dynamic data and the fourth dynamic data in the first or second implementation method above, so that the fusion result is combined with the multiple thermal energy change rate matrices obtained in step (4) to form three-dimensional dynamic thermal energy-kinetic energy image data.

[0135] In a preferred embodiment, the process of generating three-dimensional dynamic thermal-kinetic image data can be found in [reference needed]. Figure 10 and Figure 11 As shown.

[0136] In the method for generating three-dimensional dynamic thermal-kinetic image data provided by this invention, a three-channel shooting module is used to simultaneously capture images of the head and neck of a person from different angles, resulting in three sets of shooting data. Each set of shooting data includes visible light video, thermal infrared video, and time-series elevation data. Specifically, this invention extracts the kinetic energy change rate information of each point on the head and neck under two different photoelectric sensing conditions based on the three channels of visible light video and the three channels of thermal infrared video, and fuses the two types of information to obtain the kinetic energy change rate information of the person on the shooting plane over time (i.e., the third dynamic data). Furthermore, this invention also extracts the kinetic energy change rate information of each point on the head and neck in the depth direction over time based on the three channels of time-series elevation data (i.e., the fourth dynamic data). Therefore, this invention, based on the third dynamic data, the fourth dynamic data, and the thermal infrared video, forms three-dimensional dynamic thermal-kinetic image data, comprehensively and meticulously extracting the head and neck movement information of the person from multiple dimensions including kinetic energy, thermal energy, time, and three-dimensional space. Thus, even when a person is under stress, anxiety, or suffering from illness, the embodiments of the present invention can accurately capture and characterize head and neck movements caused by the reflexes of the vestibular system, thereby improving the effectiveness and accuracy of subsequent assessment or analysis of human functional status based on the three-dimensional dynamic thermal-kinetic image data.

[0137] Compared with existing technologies for collecting information such as electrocardiograms, electroencephalograms, and skin conductance responses, this invention provides a contactless data extraction solution that can reduce the anxiety experienced by the subject due to physical contact with the data collection instrument and reduce errors introduced by the data collection process itself.

[0138] Optionally, in one implementation, the method for generating three-dimensional dynamic thermal-kinetic image data provided in this embodiment of the invention may further include the following steps:

[0139] Step 1: Statistically analyze the third dynamic data to obtain the period of change of the kinetic energy rate of each point on the shooting plane.

[0140] Specifically, multiple three-dimensional kinetic energy change rate matrices are statistically analyzed. The statistical method includes: for each element position point of the three-dimensional kinetic energy change rate matrix, counting the occurrence frequency of the element value that appears most frequently at that position point, and dividing the shooting duration by that number of occurrences to obtain the change period of the kinetic energy change rate corresponding to that position point. This yields the change period of the kinetic energy change rate of each point on the shooting plane. In practice, the statistical results obtained after performing this step can form a change period matrix.

[0141] Step 2: Statistically analyze the fourth dynamic data to obtain the period of change of the kinetic energy rate of each point on the shooting plane.

[0142] For details on how to implement step two, please refer to step one.

[0143] Step 3: Find the intersection of the two obtained change periods, and use it as the change period of the kinetic energy change rate of each point in the head and neck in three-dimensional space.

[0144] Specifically, the intersection of the periodicity matrix obtained in step one and the periodicity matrix obtained in step two is calculated. In the intersection, for each element location, if the two periods are equal within a preset precision range, that period is retained; if the two periods are unequal within the precision range, the intersection result for that point is set to empty or a preset value representing a maximum period. For example, for a certain element location, the two periods are 20 milliseconds and 22 milliseconds respectively. If the precision range is set to 5 milliseconds, these two periods are considered equal. If the precision range is set to 1 millisecond, these two periods are considered unequal.

[0145] It is understandable that for the same type of movement (e.g., nystagmus), the change period captured from the plane and depth directions should be consistent. Therefore, in this embodiment of the invention, the intersection of the change periods extracted from the plane and depth directions is used to characterize the change period of the kinetic energy change rate of each point in the head and neck in three-dimensional space.

[0146] Step 4: Add the period of change of the kinetic energy change rate of each point in the head and neck in three-dimensional space to the three-dimensional dynamic thermal-kinetic image data.

[0147] Therefore, the three-dimensional dynamic thermal-kinetic image data also includes temporal statistical data, making the content richer.

[0148] The present invention can display three-dimensional dynamic thermal-kinetic image data of a person in the form of images, so that viewers can view the data of the person more intuitively and quickly obtain some human functional status information of the person based on the image data.

[0149] The method of generating three-dimensional dynamic thermal-kinetic images based on three-dimensional dynamic thermal-kinetic image data may include:

[0150] (1) Generating three-dimensional head and neck thermal images of human objects based on thermal infrared video.

[0151] Among them, generating a 3D head and neck thermal image of a person based on thermal infrared video can be achieved by displaying the stitched image generated from thermal infrared video (as mentioned above) in 3D; specifically, the pixels in the stitched image are mapped into a preset 3D head and neck model (such as...). Figure 6 As shown, the head and neck model is made to display the image texture in the stitched image, thereby obtaining a three-dimensional thermal image of the head and neck.

[0152] Alternatively, the thermal energy change rate matrix can be mapped into a preset three-dimensional head and neck model to obtain a three-dimensional thermal image of the head and neck.

[0153] (2) Based on the correspondence between the third dynamic data, the fourth dynamic data and the thermal image of the head and neck and each point of the head and neck, these three data are combined and displayed to obtain a three-dimensional dynamic thermal-kinetic image.

[0154] The third and fourth dynamic data are fused into multiple three-dimensional kinetic energy change rate matrices. These matrices can then be plotted as multiple horizontally extending straight lines on both sides of the three-dimensional head and neck thermal image. The length of these horizontal lines is determined by the values ​​of the elements in the three-dimensional kinetic energy change rate matrix. Since there are multiple three-dimensional kinetic energy change rate matrices, multiple stitched images generated from thermal infrared video, and multiple thermal energy change rate matrices, the synthesized three-dimensional dynamic thermal-kinetic image actually presents a dynamic image.

[0155] Specifically, the length of the straight line drawn on the left side of the three-dimensional head and neck thermal image is determined based on the element values ​​of the left half of the three-dimensional kinetic energy change rate matrix; similarly, the length of the straight line drawn on the right side is determined based on the element values ​​of the right half of the three-dimensional kinetic energy change rate matrix.

[0156] For example, in specific rendering, the straight lines on the left and right sides of the three-dimensional head and neck thermal image can be as follows: Figure 7 As shown, the drawing is initiated from two planes symmetrically positioned on the left and right sides of the 3D head and neck thermal image. Alternatively, it can be done as follows: Figure 8As shown, straight lines on the left and right sides are wrapped around the sides of the three-dimensional head and neck thermal image.

[0157] Therefore, when viewers see such Figure 7 or Figure 8 When viewing the 3D dynamic thermal-kinetic image, if the straight lines on both sides of the 3D head and neck thermal image show a significantly asymmetrical distribution, it indicates that the subject's bodily function is in an abnormal state. Conversely, if the straight lines on both sides of the 3D head and neck thermal image show a significantly symmetrical distribution, it indicates that the subject's bodily function is normal.

[0158] Optionally, when the three-dimensional dynamic thermal-kinetic image data also includes the change period of the kinetic energy change rate of each point in the head and neck in three-dimensional space, the color of the straight line drawn on the left side of the three-dimensional head and neck thermal image can be further set according to the value of the elements in the change period matrix.

[0159] In another synthetic display method, it is possible to... Figure 9 As shown, two colored surfaces are drawn on the left and right sides of the three-dimensional head and neck thermal image, respectively. Each point on the surface corresponds to an element in the three-dimensional kinetic energy change rate matrix, and the color of the point is determined according to the value of the element.

[0160] Therefore, when viewers see such Figure 9 When viewing the 3D dynamic thermal-kinetic image, if the colors of the curved surfaces on the left and right sides of the 3D head and neck thermal image are significantly asymmetrical, it indicates that the human functional state of the person being tested is abnormal. Conversely, if the colors of the curved surfaces on the left and right sides of the 3D head and neck thermal image are basically symmetrical, it indicates that the person's human functional state is normal.

[0161] It should be noted that the synthetic display method exemplified above is merely a principle-based example and does not constitute an actual limitation on the embodiments of the present invention; any implementation method that synthesizes and displays the data based on the correspondence between the third dynamic data, the fourth dynamic data, and the thermal image of the head and neck and various points of the head and neck, thereby making the displayed data clearer, is applicable to the embodiments of the present invention.

[0162] The following is an example illustrating the neural network used in this embodiment of the invention to achieve sub-pixel level registration of corresponding points between frames.

[0163] For example, in this embodiment of the invention, inter-frame sub-pixel level registration of corresponding points can be achieved using methods such as... Figure 12This illustrates an improved MatchNet network. This improved MatchNet network uses a LIFT (Learned Invariant Feature Transform) network to generate image feature descriptors and a metric network with three fully connected layers to measure the similarity between feature descriptors. Internally, the LIFT network sequentially performs feature point detection, softmax activation, image cropping, orientation estimation, and image rotation on the input image, and finally generates feature descriptors based on the rotated image. The registered image is obtained from the output of the image rotation module of the LIFT network. Furthermore, the feature descriptors of the two registered images are fed into the metric network, which calculates the similarity between the two images. The activation function of the three fully connected layers of this metric network is ReLU (Linear Rectification function).

[0164] In practical applications, every two adjacent frames in a video form a set of images to be registered and are fed into the improved MatchNet network for registration.

[0165] In this embodiment of the invention, a LIFT network is used to extract feature descriptors from images. This can uncover the invariant features of feature points in the image under actual changing factors, reducing the impact of interference factors such as illumination, background, and human intervention on registration accuracy in optical images. Furthermore, the traditional method for comparing the similarity between feature vectors uses Euclidean distance; however, in complex manifold learning, Euclidean distance often fails to accurately measure the similarity between feature descriptors. Therefore, this embodiment of the invention uses this metric network to compare the similarity between feature descriptors, enabling a more accurate similarity assessment of the registration results for sub-pixel-level corresponding points.

[0166] It should be noted that the neural network for achieving sub-pixel level registration of corresponding points is not limited to the improved MatchNet network used in the embodiments of this invention. For example, the existing MatchNet network can achieve similar results; the difference between the existing MatchNet network and the improved MatchNet network used in the embodiments of this invention is that the existing MatchNet network uses a deep convolutional network instead of a LIFT network, so its registration accuracy is slightly worse when there are interference factors such as lighting, background, and human intervention.

[0167] Based on the method for generating three-dimensional dynamic thermal-kinetic image data provided in this embodiment of the invention, there are various human functional state parameters that can be extracted from the three-dimensional dynamic thermal-kinetic image data. Examples of these extractable human functional state parameters are given below.

[0168] For example, a pressure index representing the magnitude of human stress can be extracted from three-dimensional dynamic thermal-kinetic image data. The formula for this pressure index is as follows:

[0169]

[0170] Wherein, PRI represents the pressure index; the number of rows in both the three-dimensional kinetic energy rate of change matrix and the thermal energy rate of change matrix is ​​N. The number of elements in both the three-dimensional kinetic energy rate of change matrix and the thermal energy rate of change matrix is ​​M-2. The difference in kinetic energy between the left and right sides of the head and neck of a person is calculated based on the nth row element of the m-th three-dimensional kinetic energy change rate matrix in [1,M]. The difference in thermal energy between the left and right sides of the head and neck of a person is calculated based on the nth row element of the m-th thermal energy matrix in [1,M].

[0171] In addition, an anxiety index representing a person's level of anxiety can be extracted from three-dimensional dynamic thermal-kinetic image data. The formula for this anxiety index is as follows:

[0172]

[0173] Where ANI represents the anxiety index, and the function f(t) is a probability density function used to calculate the period of change of the rate of change of kinetic energy at various points in the head and neck in three-dimensional space, where t represents the period of change; t∈[T] min ,B×T min ], T min Let B be the minimum period of change of the rate of change of kinetic energy of each point in the head and neck in three-dimensional space, and let B be a preset constant, where 1 < B < 5. The function f(t) can be obtained by calculating a set of probability density values ​​of change periods using the actual statistical change period matrix, and then fitting the set of density values ​​using the least squares method.

[0174] It should be noted that the human functional state parameters that can be extracted from 3D dynamic thermal-kinetic image data are not limited to the stress index and anxiety index mentioned above. In practical applications, by collecting and analyzing 3D dynamic thermal-kinetic image data of a large number of people, specific patterns can be found in the 3D dynamic thermal-kinetic image data of a specific population group. Then, by summarizing and representing these specific patterns using mathematical methods, more defining formulas for human functional state parameters can be obtained. Based on these defining formulas, human functional state parameters of individuals can be extracted, thereby enabling the assessment and analysis of the individuals' human functional state.

[0175] Since more specific parameter extraction methods are not the key points of this invention, they will not be exemplified one by one in this embodiment.

[0176] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, such as... Figure 13 As shown, it includes: a thermal infrared camera 1305, a visible light camera 1306, a ranging module 1302, a processor 1301, and a memory 1303; wherein, the thermal infrared camera 1305, the visible light camera 1306, and the ranging module 1302 are electrically connected to the processor 1301; the memory 1303 is used to store computer programs; the processor 1301 is used to execute the program stored in the memory 1303 to implement any of the above-mentioned methods for generating three-dimensional dynamic thermal-kinetic image data.

[0177] In practical applications, any electronic device that is associated with a visible light camera, a thermal infrared camera, and a ranging module capable of simultaneous shooting, and that has computational processing capabilities, can be used as the electronic device provided in the embodiments of this invention.

[0178] The aforementioned memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0179] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), graphics processing units (GPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0180] It should be noted that, for the electronic device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiment.

[0181] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0182] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0183] Although embodiments of the invention have been described herein in conjunction with various examples, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims. In this specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. While certain measures are described in different embodiments, this does not mean that these measures cannot be combined to produce good results.

[0184] Embodiments of the present invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects, all of which are collectively referred to herein as a "module" or a "system". Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The computer program may be stored / distributed in a suitable medium, provided with or as part of other hardware, or may take other distribution forms, such as via the Internet or other wired or wireless telecommunications systems.

[0185] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for generating three-dimensional dynamic thermal-kinetic image data, characterized in that, include: Three-channel shooting modules were used to simultaneously capture images of the head and neck of the person from different angles, resulting in three sets of shooting data. Each set of shooting data included: visible light video, thermal infrared video, and time-series elevation data. Based on the three visible light video streams, the kinetic energy change rate information of each point in the head and neck over time is extracted to obtain the first dynamic data. Based on the three thermal infrared video streams, the kinetic energy change rate information of each point in the head and neck over time is extracted to obtain the second dynamic data. The first dynamic data and the second dynamic data are fused to obtain the third dynamic data, which characterizes the rate of change of kinetic energy of each point on the head and neck on the shooting plane over time. Based on the three-channel time-series elevation data, the kinetic energy change rate information of each point in the head and neck in the depth direction over time is extracted to obtain the fourth dynamic data. Three-dimensional dynamic thermal-kinetic image data is formed based on the third dynamic data, the fourth dynamic data, and the thermal infrared video, and a three-dimensional dynamic thermal-kinetic image is generated based on the three-dimensional dynamic thermal-kinetic image data.

2. The method according to claim 1, characterized in that, Based on the three visible light video streams, the kinetic energy change rate information of each point in the head and neck over time is extracted to obtain the first dynamic data, including: For each of the visible light video streams, inter-frame subpixel level registration of corresponding points is performed; For each registered visible light video, based on every two adjacent frames in the video, calculate multiple first kinetic energy change rate matrices for each point in the head and neck over time. The first dynamic data is formed by splicing together every three first kinetic energy change rate matrices that are time-corresponding from the three visible light video streams respectively. Based on the three channels of thermal infrared video, the kinetic energy change rate information of each point in the head and neck over time is extracted to obtain the second dynamic data, including: For each of the aforementioned thermal infrared video streams, inter-frame subpixel level registration of corresponding points is performed. For each registered thermal infrared video, based on every two adjacent frames in the video, calculate multiple second kinetic energy change rate matrices for each point in the head and neck over time. The second dynamic data is formed by splicing together every three second kinetic energy change rate matrices that are time-corresponding from the three thermal infrared video streams.

3. The method according to claim 1, characterized in that, Based on the three channels of time-series elevation data, the kinetic energy change rate information of each point in the head and neck along the depth direction over time is extracted to obtain the fourth dynamic data, including: Each set of three elevation data matrices, which are derived from the three time-series elevation data sources and have a time correspondence, is spliced ​​together to obtain multiple spliced ​​data matrices. Using a spliced ​​data matrix of two adjacent data measurement times, multiple third kinetic energy change rate matrices of the head and neck in the depth direction are calculated over time. The fourth dynamic data is obtained based on the plurality of third kinetic energy change rate matrices.

4. The method according to claim 3, characterized in that, The fourth dynamic data is obtained based on the plurality of third kinetic energy change rate matrices, including: Each of the third kinetic energy change rate matrices is expanded using a cubic spline interpolation algorithm to obtain multiple expanded matrices; Data is sampled from each of the expanded matrices according to the preset sampling coordinates to obtain the fourth dynamic data; The sampling coordinates are calculated in advance based on the ranging coverage of the three shooting modules and the video framing coordinates.

5. The method according to claim 3, characterized in that, The step of concatenating every three elevation data matrices, which are derived from the three time-series elevation data sources and have a time correspondence, includes: For every two adjacent elevation data matrices, the row-to-row or column-to-column similarity of the two elevation data matrices is calculated based on the deployment of the three-channel shooting modules relative to the personnel object, and the splicing boundary is determined based on the similarity calculation results; based on the splicing boundary, the two adjacent elevation data matrices are spliced ​​together. Among them, adjacent elevation data matrices refer to the distance measuring modules that measured the elevation data matrices being physically adjacent.

6. The method according to claim 1, characterized in that, The third dynamic data includes multiple planar kinetic energy change rate matrices of each point of the head and neck sorted over time on the shooting plane, and the fourth dynamic data includes multiple depth kinetic energy change rate matrices of each point of the head and neck sorted over time in the depth direction. The planar kinetic energy change rate matrix and the depth kinetic energy change rate matrix have the same dimension. The process of forming three-dimensional dynamic thermal-kinetic image data based on the third dynamic data, the fourth dynamic data, and the thermal infrared video includes: Summing is performed on each pair of planar kinetic energy rate of change matrices and depth kinetic energy rate of change matrices that have a time correspondence, resulting in multiple three-dimensional kinetic energy rate of change matrices sorted over time. Based on the multiple three-dimensional kinetic energy change rate matrices and the thermal infrared video, three-dimensional dynamic thermal-kinetic energy image data is formed.

7. The method according to claim 1, characterized in that, The third dynamic data includes multiple planar kinetic energy change rate matrices of each point of the head and neck sorted over time on the shooting plane, and the fourth dynamic data includes multiple depth kinetic energy change rate matrices of each point of the head and neck sorted over time in the depth direction. The planar kinetic energy change rate matrix and the depth kinetic energy change rate matrix have the same dimension. The process of forming three-dimensional dynamic thermal-kinetic image data based on the third dynamic data, the fourth dynamic data, and the thermal infrared video includes: For each pair of planar kinetic energy change rate matrices and depth kinetic energy change rate matrices that have a time correspondence, the modulus of the planar vector is calculated by using the pairs of elements with the same position in the matrix as the horizontal and vertical projection coordinates of the planar vector, thus obtaining a modulus matrix. Three-dimensional dynamic thermal-kinetic image data is formed based on the obtained multiple mode matrices and the thermal infrared video.

8. The method according to claim 1, characterized in that, The process of forming three-dimensional dynamic thermal-kinetic image data based on the third dynamic data, the fourth dynamic data, and the thermal infrared video includes: Obtain multiple raw temperature data matrices associated with each of the thermal infrared video streams; Each set of three original temperature data matrices, which are associated with the three thermal infrared videos and have a time-corresponding relationship, is spliced ​​together to obtain multiple spliced ​​temperature data matrices. Based on the spliced ​​temperature data matrix of two adjacent temperature acquisition times, calculate multiple thermal energy change rate matrices of each point on the head and neck of the person subject over time. Three-dimensional dynamic thermal-kinetic image data is formed based on the third dynamic data, the fourth dynamic data, and the multiple thermal energy change rate matrices.

9. The method according to claim 1, characterized in that, The methods for generating three-dimensional dynamic thermal-kinetic images based on the aforementioned three-dimensional dynamic thermal-kinetic image data include: A three-dimensional thermal image of the head and neck of the person is generated based on the thermal infrared video; Based on the correspondence between the third dynamic data, the fourth dynamic data, and the three-dimensional head and neck thermal image and each point on the head and neck, the three data are synthesized and displayed to obtain a three-dimensional dynamic thermal-kinetic image.

10. The method according to claim 1, characterized in that, The method further includes: By statistically analyzing the third dynamic data, the period of change of the kinetic energy change rate of each point on the head and neck on the shooting plane is obtained; By statistically analyzing the fourth dynamic data, the period of change of the kinetic energy change rate of each point on the head and neck on the shooting plane is obtained; The intersection of the two obtained change periods is taken as the change period of the kinetic energy change rate of each point in the head and neck in three-dimensional space. The period of change of the kinetic energy change rate of each point in the head and neck in three-dimensional space is added to the three-dimensional dynamic thermal-kinetic image data.

Images