Thermal infrared face recognition system combining scene diversity and complexity

Abstract

The invention relates to the technical field of thermal infrared face recognition, in particular to a thermal infrared face recognition system combining scene diversity and complexity. A data acquisition module of the system comprises thermal infrared face images acquired under different camouflage shielding, time, equipment and physiological state conditions, and the complexity in real life is reflected. A multi-device data synchronization method based on time and space calibration is designed in the system, and data consistency is ensured. And efficient storage and access of large-scale data are ensured through a dynamic compression and storage optimization strategy. The method is based on rich data resources, can support powerful thermal infrared face recognition model training, and provides a new method for a thermal infrared face recognition system under real conditions. By increasing the scale of the data set and the scene complexity, wider data support is provided for thermal infrared face recognition research.

Description

Technical Field

[0001] The present application relates to the field of thermal infrared face recognition technology, and in particular to a thermal infrared face recognition system that combines scene diversity and complexity. Background Art

[0002] With the rapid development of biometric technology, thermal infrared face recognition has attracted more and more attention due to its unique advantages in complex lighting conditions. Traditional visible light face recognition performs poorly in low-light or no-light environments, while thermal infrared imaging technology does not rely on ambient light and can work effectively in dark environments, which makes it have broad application prospects in security monitoring, medical health and military fields.

[0003] However, the commonly used public thermal infrared face datasets include SpeakingFaces, TUFTS, Terravic, IRIS, and Charlotte. Although they provide a certain amount of thermal infrared face data, these datasets often lack diversity, especially in some real-world scenarios. This leads to poor performance of the trained models in practical applications and inability to effectively cope with complex and changing real-world environments, limiting the widespread application of thermal infrared face recognition technology. There are still some shortcomings in the field of thermal infrared face recognition: (1) The number of thermal infrared face datasets is limited and most of them are not public, which makes it difficult to meet the training requirements of large-scale deep learning models. (2) Existing datasets are usually collected under simple recording conditions and lack diverse scenarios such as changes in equipment and time. (3) Due to the limitations of dataset size and complexity, the recognition accuracy reported in many studies may be overestimated and difficult to reflect the performance in real applications.

[0004] In order to solve these problems, a thermal infrared face recognition system that combines diversity and complexity is needed to provide more extensive data support for thermal infrared face recognition research.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide a thermal infrared face recognition system that combines scene diversity and complexity, aiming to provide more extensive data support for thermal infrared face recognition research by increasing the size of the data set and the complexity of the scene.

[0007] To achieve the above objectives, the present application provides a thermal infrared face recognition system that combines scene diversity and complexity, the system comprising:

[0008] A data collection module, including a plurality of thermal infrared cameras, which collects face videos of participants at a plurality of different head posture positions through the plurality of thermal infrared cameras to form a data set, wherein the data set includes features of facial occlusions, time variation features, and physiological state variation features;

[0009] A data processing module, used to extract image frames, perform face detection and cropping, and optimize the quality of the data set, and synchronize the face videos collected by different thermal infrared cameras in the data set;

[0010] A data annotation module, used for annotating the facial image data in the data set after data synchronization;

[0011] A data storage module, used for classifying and compressing the face image data for storage based on the importance and access frequency of the data;

[0012] Data application module, including software platforms for data management and processing, development of data visualization tools and standardized interfaces.

[0013] Optionally, the data processing module corrects the inter-frame differences between the thermal infrared cameras based on a correction algorithm of frame numbers, so as to achieve time alignment of acquisition frames based on the inter-frame differences.

[0014] Optionally, the steps of the correction algorithm based on frame number include:

[0015] Select an image frame of a thermal infrared camera at a target time as a reference frame;

[0016] Calculate the frame difference between the reference frame and the image frames captured by other thermal infrared cameras at the same target time;

[0017] Based on the inter-frame difference, selecting an image frame that is closest to the reference frame in terms of acquisition time and the target time as a synchronization frame corresponding to the other thermal infrared camera;

[0018] The synchronization frame is used as an image frame acquired at the same time as the reference frame, thereby achieving time alignment of the acquired frames.

[0019] Optionally, the data processing module uses a homography matrix transformation algorithm to spatially align images of multiple devices.

[0020] Optionally, the data annotation module combines manual and machine learning techniques to perform data annotation, and the annotation information includes number, collection time, head posture, accessory type and device model.

[0021] Optionally, the data storage module divides the image data into three storage types: real-time access data, near-line storage data and cold data based on the importance and access frequency of the data, wherein the real-time access data is stored in a solid-state drive, the near-line storage data is stored in a mechanical hard drive, and the cold data is stored in a large-capacity tape library or cloud storage space.

[0022] Optionally, the data storage module compresses the image data using a lossless compression technology.

[0023] Optionally, the data storage module uses an algorithm for merging adjacent data blocks to perform data block optimization on the written data during the data writing process.

[0024] This application has at least the following beneficial effects:

[0025] 1. By constructing a data set covering a variety of scene conditions, a more comprehensive and complex data set is constructed, which better reflects the complexity of the actual application environment and provides a more realistic and rich data foundation for the training and evaluation of thermal infrared face recognition models;

[0026] 2. It can enhance the stability and generalization ability of thermal infrared face recognition models in various environments and improve the performance of the model in real life. It can serve as a new benchmark for evaluating the performance of existing thermal infrared face recognition models and systems. Through this benchmark, the actual performance of existing technologies in real-world applications can be better revealed, promoting research progress in this field. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of the architecture of the hardware operating environment of a thermal infrared face recognition system that combines scene diversity and complexity involved in an embodiment of the present application;

[0028] Figure 2 A schematic diagram of the architecture of a thermal infrared face recognition system that combines scene diversity and complexity involved in the first embodiment of the present application;

[0029] Figure 3 A schematic diagram of a data set collection scenario involved in an embodiment of the present application.

[0030] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0031] In order to better understand the above technical solution, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0032] With the rapid development of biometric technology, thermal infrared face recognition has attracted more and more attention due to its unique advantages in complex lighting conditions. Traditional visible light face recognition performs poorly in low-light or no-light environments, while thermal infrared imaging technology does not rely on ambient light and can work effectively in dark environments, which makes it have broad application prospects in security monitoring, medical health and military fields.

[0033] However, the most commonly used public thermal infrared face datasets are SpeakingFaces, TUFTS, Terravic, IRIS, and Charlotte, as shown in Table 1:

[0034] Table 1. Currently available infrared face datasets for download

[0035]

[0036] Although a certain amount of thermal infrared face data is available, these data sets often lack diversity, especially in some real-world scenarios. This results in poor performance of the trained models in practical applications and inability to effectively cope with complex and changing real-world environments, limiting the widespread application of thermal infrared face recognition technology. Therefore, the following deficiencies still exist in the field of thermal infrared face recognition:

[0037] 1. Data scarcity: The number of thermal infrared face datasets is limited and most of them are not public, which makes it difficult to meet the training needs of large-scale deep learning models.

[0038] 2. Lack of complexity: Existing datasets are usually collected under simple recording conditions and lack diverse scenarios such as changes in equipment and time.

[0039] 3. Overestimation of performance: Due to the limitations of dataset size and complexity, the recognition accuracy reported in many studies may be overestimated and difficult to reflect the performance in real applications.

[0040] To address these problems, this application proposes a new thermal infrared and visible light face database ComplexFace that combines diversity and complexity, aiming to provide more extensive data support for thermal infrared face recognition research by increasing the scale of the dataset and the complexity of the scenes.

[0041] First embodiment

[0042] As an implementation plan, Figure 1 This is a schematic diagram of the hardware operating environment of a thermal infrared face recognition system that combines the diversity and complexity of scenarios involved in the embodiments of the present application. Figure 1 As shown, the thermal infrared face recognition system combining scene diversity and complexity may include: a processor 1001, such as a CPU, a memory 1005, a user interface 1003, a network interface 1004, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0043] Those skilled in the art will understand that Figure 1 The thermal infrared face recognition system architecture combining scene diversity and complexity shown in the figure does not constitute a limitation on the thermal infrared face recognition system combining scene diversity and complexity, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0044] In addition, refer to Figure 2 In this embodiment, the thermal infrared face recognition system combining scene diversity and complexity includes:

[0045] The data acquisition module 100 includes a plurality of thermal infrared cameras, which collect face videos of participants at a plurality of different head posture positions through the plurality of thermal infrared cameras to form a data set, wherein the data set includes facial occlusion features, time change features, and physiological state change features;

[0046] Exemplarily, based on the architecture in the data acquisition module 100, the data acquisition process in the system is as follows:

[0047] Step 1: First, prepare at least three thermal infrared cameras and one visible light camera. Before the formal recording, we first use these four devices to shoot a video of a calibration board to facilitate the spatial alignment between the calibration devices later.

[0048] For each volunteer who participates in data collection, we will record the following steps:

[0049] Place a piece of cardboard in front of the four cameras as a reference for time synchronization; quickly remove the cardboard to ensure that all cameras can capture the moment when the cardboard is completely removed; define the four frames captured by the four cameras when the cardboard is completely removed as synchronization frames, and start formal recording from this point.

[0050] Step 2: During the data collection process, take photos according to the different states of the volunteers. The states of the volunteers include the following:

[0051] 1) Natural state: At the beginning of data collection, we first focus on the natural state of the volunteers. To this end, we ask the volunteers to take a full rest before recording to ensure that their facial expressions and physiological states are stable. In this stage, the volunteers do not wear any accessories to ensure that their faces are not blocked by any objects so that we can accurately capture their original facial features without external interference. The purpose of this step is to obtain facial video data of volunteers in the most natural and unmodified state, providing a benchmark reference for subsequent analysis and research. In this way, we can more accurately evaluate the impact of different accessories and conditions on the performance of the facial recognition system.

[0052] 2) Wearing glasses: Then, let the volunteers wear ordinary glasses and record their face videos. This step helps to study the impact of glasses on facial recognition systems.

[0053] 3) Wearing sunglasses: Next, the volunteers will be photographed wearing sunglasses to evaluate the potential impact of sunglasses on the performance of the recognition system.

[0054] 4) Wearing masks: In the current global health environment, masks have become a daily necessity. Therefore, we will also record the face videos of volunteers wearing masks to study the challenges that masks pose to facial recognition technology.

[0055] 5) Physiological state changes: In order to explore the impact of physiological state changes on thermal infrared face recognition, we chose to shoot the state after exercise. In the current research, thermal infrared face images after exercise have not been collected. We asked the volunteers to record their face videos immediately after doing a certain amount of exercise (such as skipping rope). This step helps to understand how physiological changes after exercise affect the recognition of thermal infrared facial features.

[0056] In the above five different states, volunteers need to follow the instructions and look at the nine different head posture markers in front of them in a clockwise direction, such as Figure 3As shown, each point is kept for several seconds. This process is designed to simulate the diversity of head poses in the real world. At the same time, in order to evaluate the impact of time factors on data consistency, we will invite volunteers to do the same recording process again three months later. This long-term tracking will provide us with valuable data to analyze and understand the impact of time span on the performance of facial recognition technology.

[0057] The data processing module 200 is used to extract image frames, perform face detection and cropping, and optimize the quality of the data set, and synchronize the face videos collected by different thermal infrared cameras in the data set;

[0058] Exemplarily, based on the architecture in the data acquisition module 200, the data processing process in the system is as follows:

[0059] 1) Image frame extraction: First, all videos are extracted into image frames. Then we select a device to select key image frames with good quality and meet the requirements from the video. For each head posture, 3 face images are selected. For 9 head postures plus 5 rounds of changes, one person and one device collect 3x 9x 5, which is 135 image frames. These frames will be used as references for selecting the same time frames from other devices in the next step.

[0060] 2) Frame selection and time alignment: Considering the time synchronization issues that may exist between devices during multi-device acquisition, we implemented an efficient frame selection and time alignment process. The core idea is to select the image frames of one device as the reference frames and calculate the inter-frame differences between them and the synchronization frames. Using this inter-frame difference, we select the image frames that are closest in time to those acquired by other devices based on the synchronization frames of other devices. Based on this concept, we designed and implemented a correction algorithm based on frame numbering to ensure that the image frames closest in time to the reference frames are accurately selected from other devices for time alignment.

[0061] 3) Image alignment: To ensure the spatial consistency of images captured by multiple devices, we used homography matrix transformation to align images. First, we used four cameras to simultaneously capture the calibration board and select the same positioning markers from the same frame to calculate the homography matrix for spatial alignment. Then, we applied these matrices to subsequent face images to achieve accurate spatial alignment between images of different devices.

[0062] 4) Face detection and cropping: After selecting and aligning the frames from multiple devices, we use the Thermal Face model to detect faces and crop the images based on the detection results. After cropping, we manually check and manually correct any images that were not cropped accurately.

[0063] A data annotation module 300 is used to annotate the facial image data in the data set after data synchronization;

[0064] Exemplarily, based on the architecture in the data annotation module 300, the pre-processed image data is annotated. The annotation content includes the following dimensions:

[0065] 1) Person information: uniquely identify the person in the image to ensure individual distinction of the data. The annotation format is such as Person:[ID, Name, Gender].

[0066] 2) Accessory information: mark whether the volunteer wears glasses, sunglasses or masks, etc. The marking format is such as Glasses: Yes / No, Sunglasses: Yes / No, Mask: Yes / No.

[0067] 3) Time information: record the specific time point of image acquisition. The marking format is Time:YYYY-MM-DD HH:MM.

[0068] 4) Device information: record the acquisition device used, including the device model and parameters. The notation format is Device: [Device Model, Resolution, Focal Length].

[0069] 5) Physiological state information: record the changes in the volunteer's physiological state, such as static or after sport. The notation format is State:Static / After Sport.

[0070] It should be noted that the data set used in this embodiment has 70 people participating in two data collections, providing three facial images for each of the nine head postures, and there are 5 rounds of recording to reflect scene changes, and 4 cameras represent equipment changes. This produces a total of 70×9×3×5×4×2=75,600 images. In the process of building an efficient face recognition system, we store the pre-processed image data and its detailed annotation information in the database. These annotation information covers multi-dimensional data such as personnel number, accessories, acquisition time, physiological state, and equipment type, providing us with a comprehensive data background. Subsequently, we use these annotation information to conduct in-depth analysis and processing of the image data and further extract key feature vectors. This step is crucial because it allows us to convert raw image data into numerical features that can be used for pattern recognition, thereby providing a solid foundation for the training and optimization of face recognition algorithms. By combining the annotation information with the extracted feature vectors, we construct a structured multi-dimensional face model database. This database not only contains rich individual features, but also enhances the adaptability and recognition accuracy of the model to different scene changes through the fusion of multi-dimensional data.

[0071] The data storage module 400 is used to classify and compress the face image data based on the importance and access frequency of the data and then store them;

[0072] In this embodiment, the data storage module 400 includes three functions: data classification storage, image data compression and storage optimization.

[0073] For data classification storage, image data is divided into three levels: real-time access data, near-line storage data, and cold data storage according to the importance and access frequency of the data. Real-time access data mainly includes the latest collected data and frequently accessed data, which are stored in high-performance storage media (such as NVMe SSD) to ensure rapid response to needs. Near-line storage data includes historical data and data with medium access frequency, which are stored in high-performance HDDs. Cold data storage is used for long-term archiving of historical data, which is stored in large-capacity tape libraries or cloud storage.

[0074] For image data compression, lossless compression technology is used to compress image data to reduce storage space while ensuring data integrity and accuracy. Targeted compression algorithms are used for different image types (such as thermal infrared images and visible light images) to maximize compression efficiency.

[0075] For storage optimization, during the data writing process, data block optimization technology is used to improve storage efficiency by merging adjacent data blocks and eliminating redundant data.

[0076] In addition, perform data defragmentation regularly to ensure data continuity and access efficiency.

[0077] The data application module 500 includes a software platform for data management and processing, development of data visualization tools and standardized interfaces.

[0078] The data application module 500 includes two types of applications: hardware level and software level.

[0079] At the hardware level, multiple high-precision camera devices are deployed and installed, including fixed and mobile devices, to capture facial images in different scenarios.

[0080] At the software level, we will develop a software platform for data management and processing to support the analysis and application of large-scale data. We will develop data visualization tools to facilitate users to quickly browse and analyze data. We will provide standardized interfaces to support the integration and testing of different face recognition algorithms.

[0081] Further, in this embodiment, the data processing module corrects the inter-frame differences between the thermal infrared cameras based on the frame number correction algorithm to achieve acquisition frame time alignment based on the inter-frame differences. Further, the steps of the frame number-based correction algorithm include:

[0082] Select an image frame of a thermal infrared camera at a target time as a reference frame;

[0083] Calculate the frame difference between the reference frame and the image frames captured by other thermal infrared cameras at the same target time;

[0084] Based on the inter-frame difference, selecting an image frame that is closest to the reference frame in terms of acquisition time and the target time as a synchronization frame corresponding to the other thermal infrared camera;

[0085] The synchronization frame is used as an image frame acquired at the same time as the reference frame, thereby achieving time alignment of the acquired frames.

[0086] Furthermore, in this embodiment, the data processing module uses a homography matrix transformation algorithm to spatially align images of multiple devices. Specifically, by simultaneously using four cameras to shoot the calibration board and selecting the same positioned markers from the same frame, a homography matrix for spatial alignment is calculated. Then, we apply these matrices to subsequent face images to achieve accurate spatial alignment of images between different devices.

[0087] Furthermore, in this embodiment, the data annotation module combines manual and machine learning techniques to perform data annotation, and the annotation information includes number, collection time, head posture, accessory type and device model.

[0088] The data storage module divides the image data into three storage types: real-time access data, near-line storage data and cold data based on the importance and access frequency of the data. The real-time access data is stored in a solid-state hard disk, the near-line storage data is stored in a mechanical hard disk, and the cold data is stored in a large-capacity tape library or cloud storage space.

[0089] Furthermore, in this embodiment, the data storage module compresses the image data using lossless compression technology.

[0090] Furthermore, in this embodiment, the data storage module uses an algorithm for merging adjacent data blocks to optimize the data blocks to be written during the data writing process.

[0091] Second embodiment

[0092] In order to verify the effect of the above embodiment, the following is an experimental part, which combines experiments to illustrate the role of this dataset: (1) studying the impact of different scene changes on infrared face recognition performance; (2) revealing the actual performance of the infrared face recognition model when considering the complexity of real life.

[0093] The recognition model architecture used in the experiment is FaceNet, which is based on the VGGFace2 pre-trained model and fine-tuned using the ComplexFace dataset and public datasets. In order to clarify the data used in the experiment, the data subset is represented by the format of "scene / device / time", where "device" can be A (AT61F), X (XT300) or U (T3S_A13), "time" can be S1 (first acquisition) or S2 (second acquisition), and "scene" can be Normal (natural state, that is, scenes without interference such as camouflage and movement), Disguise (camouflage occlusion), Sport (sports) or All (scenes containing all of the above). For example, the data subset Normal / A / S1 represents a face image in a natural state scene recorded by the AT61F device in the first acquisition.

[0094] In subsequent experiments, the dataset was divided into a training set of 49 people and a test set of 21 people. During the test, we randomly selected 10 face images from the normal scenes of the test set for each tester to form a registration set, and used the remaining images for testing. The average vector of the feature vectors of the registered images was used as the registration vector, and the cosine similarity between the registration vector and the test feature vector was used as the matching score. The system performance was evaluated using two indicators: Accuracy (Acc) and Equal Error Rate (EER).

[0095] Table 2 Cross-scenario results of the S1 subset of AT61F

[0096]

[0097] The results in Table 2 show that when the model is tested in camouflage and motion scenes, the performance is much lower than that in standard scenes, and the recognition performance after motion is also lower than before motion. These results show that scene changes such as camouflage, occlusion, and motion do pose significant challenges to infrared face recognition.

[0098] Table 3 Cross-time results of AT61F data

[0099]

[0100] Table 3 studies the effect of time variation on infrared face recognition. In order to eliminate the effect of device variation, only the data from the AT61F device is used. The results show that the model performs well when the registration set and the test set are from the same time period (i.e., both are from A / S1). However, when the registration images are from A / S1 and the test images are from A / S2, the model performance drops significantly. This shows that time variation has a great impact on model performance. Looking further at the A / [S1+S2] model, we can see the same trend as the A / S1 model, indicating that even if data from two time periods are used for training, there is no significant improvement in cross-time performance.

[0101] Table 4 Cross-device results of the S1 subset on three devices

[0102]

[0103]

[0104] Table 4 studies the impact of device changes on infrared face recognition. In order to eliminate the impact of time period changes, only the data of S1 is used. The results show that when focusing on the A / S1 model, it can be clearly seen that when the model is tested across devices, its performance drops significantly, and when tested on T3S_A13(U), the performance drops even more significantly. Further observation of the multi-device model [A+X+U] / S1 shows that it performs significantly better than the single-device model A / S1, and the performance of the model is significantly improved on all three test sets. This shows that the impact of cross-device can be effectively alleviated through cross-device training.

[0105] Table 5 EER (%) results across datasets

[0106]

[0107] Table 5 shows the results of using the ComplexFace dataset to test the models trained with three public datasets, SpeakingFaces (100 people for training and 42 people for testing), TUFTS (80 people for training and 32 people for testing), and IRIS (20 people for training and 10 people for testing). The training data of these three datasets were then merged to train a MIX model. Next, we tested the four models on the ComplexFace dataset, using the test set of three thermal imaging devices from S1. The registration used natural scenes (Normal / A / S1, Normal / X / S1, and Normal / U / S1), and the testing was performed in natural scenes and all scenes (ALL / A / S1, ALL / X / S1, and ALL / U / S1). Due to the mismatch in the size of the test set, the results are presented using the EER metric.

[0108] First, the performance of the corresponding models was evaluated on the corresponding SpeakingFaces, TUFTS, and IRIS test sets. The model performed best when the training set and test set came from the same data source, indicating that the model performs better on homogeneous data. The EER of the MIX model on different test sets is generally lower, indicating that collecting more diverse data can improve the generalization ability of the model.

[0109] Second, when the four models were tested on ComplexFace, their performance was quite good under "ideal" conditions (Normal / A / S1, Normal / X / S1, and Normal / U / S1), which are not affected by camouflage, motion, time, and device changes, and is comparable to the results obtained using the ComplexFace model (Table 2). This shows that the models can perform well on simple test data. However, when tested under more complex conditions (ALL / A / S1, ALL / X / S1, and ALL / U / S1), the performance dropped significantly and performed far worse than the results of the ComplexFace model. This shows that complex training data is needed to improve the quality of infrared face recognition models.

[0110] The EER results of the four models when scene changes are involved (ALL / A / S1, ALL / X / S1, and ALL / U / S1) are very poor, about 37%. These high EER values ​​indicate that although the models perform well on their respective test sets, they are almost useless in practical applications. This means that there is a significant gap between the performance reported in the literature and the actual test results, indicating that existing infrared face recognition research is still a long way from truly usable technology.

[0111] These high EER values ​​further highlight the importance of developing more diverse and complex datasets. Our ComplexFace dataset provides a more comprehensive testbed for infrared face recognition research by simulating various real-world conditions. This dataset can better evaluate and improve the performance of models in the face of real-world challenges, thereby promoting the advancement of technology in this field.

[0112] It should be noted that those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0113] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0114] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0115] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0116] It should be noted that in the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The present application may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0117] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0118] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A thermal infrared face recognition system combining scene diversity and complexity, characterized in that: The system comprises: A data collection module, including a plurality of thermal infrared cameras, which collects face videos of participants at a plurality of different head posture positions through the plurality of thermal infrared cameras to form a data set, wherein the data set includes features of facial occlusions, time variation features, and physiological state variation features; A data processing module, used to extract image frames, perform face detection and cropping, and optimize the quality of the data set, and synchronize the face videos collected by different thermal infrared cameras in the data set; A data annotation module, used for annotating the facial image data in the data set after data synchronization; A data storage module, used for classifying and compressing the face image data for storage based on the importance and access frequency of the data; Data application module, including software platforms for data management and processing, development of data visualization tools and standardized interfaces.

2. The system according to claim 1, characterized in that The data processing module corrects the inter-frame differences between the thermal infrared cameras based on a correction algorithm of the frame number, so as to achieve acquisition frame time alignment based on the inter-frame differences.

3. The system according to claim 2, characterized in that The steps of the frame number-based correction algorithm include: Select an image frame of a thermal infrared camera at a target time as a reference frame; Calculate the frame difference between the reference frame and the image frames captured by other thermal infrared cameras at the same target time; Based on the inter-frame difference, selecting an image frame that is closest to the reference frame in terms of acquisition time and the target time as a synchronization frame corresponding to the other thermal infrared camera; The synchronization frame is used as an image frame acquired at the same time as the reference frame, thereby achieving time alignment of the acquired frames.

4. The system according to claim 1, characterized in that The data processing module uses a homography matrix transformation algorithm to spatially align images of multiple devices.

5. The system according to claim 1, wherein: The data annotation module combines manual and machine learning techniques to perform data annotation, and the annotation information includes serial number, collection time, head posture, accessory type and device model.

6. The system according to claim 1, characterized in that The data storage module divides the image data into three storage types: real-time access data, near-line storage data and cold data based on the importance and access frequency of the data. The real-time access data is stored in a solid-state hard disk, the near-line storage data is stored in a mechanical hard disk, and the cold data is stored in a large-capacity tape library or cloud storage space.

7. The system according to claim 1, characterized in that The data storage module compresses the image data using lossless compression technology.

8. The system of claim 1, wherein: The data storage module optimizes the data blocks of the written data by using an algorithm for merging adjacent data blocks during the data writing process.

Images