Low-resolution temperature array face multi-modal perception method and device based on CNN

Abstract

The application discloses a low-resolution temperature array face multi-modal perception method and device based on a CNN, and the method comprises the following steps: acquiring an original temperature matrix output by a temperature sensor and performing normalization processing; inputting the normalized matrix into a multi-task convolutional neural network model to synchronously output a face confidence, a bounding box parameter and a depth estimation value; extracting a corresponding area temperature value from the original matrix according to the bounding box parameter to calculate an average temperature; and combining the bounding box parameter, the depth estimation value and a sensor field angle parameter to calculate a three-dimensional space coordinate of the face in a sensor coordinate system. The device comprises a memory, a processor and a stored computer program. The application realizes non-contact face detection, accurate temperature measurement and three-dimensional positioning, can intelligently drive automatic adjustment of a car cabin and a household air conditioner, and improves comfort and intelligent experience.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and device for low-resolution temperature array face multimodal perception based on CNN. Background Technology

[0002] In the fields of smart cockpits and smart homes, achieving personalized and comfortable experiences through seamless status perception and interaction with drivers, passengers, or family members has become a significant development trend. Among these, non-contact methods for facial presence detection, body temperature monitoring, and location tracking are key technological aspects of achieving intelligent temperature control. Current technologies typically rely on multiple sensor fusion schemes to achieve these functions, such as combining a color camera with an infrared temperature measurement module or a depth sensor. While such solutions can achieve some functions, they have significant drawbacks: multi-sensor systems are expensive, complex in structure, and require large spaces. Furthermore, the need to solve data synchronization and calibration issues between different sensors significantly increases the difficulty of system design and integration. Another approach is to use a single visible light camera to perform facial detection and tracking using computer vision algorithms, attempting to estimate distance or body temperature. However, visible light images are susceptible to changes in ambient lighting conditions, with reliability plummeting in low or strong light, and temperature information cannot be directly obtained, making it difficult to guarantee the accuracy of body temperature estimation.

[0003] While high-resolution infrared thermal imagers exist as a technological solution for facial recognition and temperature measurement through thermal image analysis, the high cost of these sensors and the computational demands of data processing limit their widespread adoption in cost-sensitive consumer applications. Furthermore, existing infrared data-based solutions are largely focused on high-temperature warnings or high-precision medical-grade temperature measurement, often requiring manual intervention or providing only single-point temperature data. They lack real-time integration with spatial location information, thus failing to support fully automated intelligent temperature control systems. Therefore, under current technological conditions, there is a lack of an integrated solution for real-time, concurrent facial detection, body temperature measurement, and 3D positioning based on a single, low-cost sensor in in-vehicle or home environments. This hinders the development of intelligent temperature control systems towards greater efficiency, economy, and automation. Summary of the Invention

[0004] To address the aforementioned shortcomings, this invention proposes a low-resolution temperature array-based multimodal face perception method and device based on CNN, which achieves non-contact face detection, accurate temperature measurement, and 3D positioning. It can intelligently drive the automatic adjustment of car cabins and home air conditioners, significantly improving comfort and intelligent experience.

[0005] This invention provides the following technical solution: a CNN-based low-resolution temperature array face multimodal perception method, comprising the following steps:

[0006] S1: Obtain the original temperature matrix SrcTemp output by the temperature sensor, with SrcTempW as the number of columns and SrcTempH as the number of rows;

[0007] S2: Normalize the original temperature matrix to obtain the normalized temperature matrix SrcTempNorm. i,j ;

[0008] S3: Input the normalized temperature matrix into the pre-trained multi-task convolutional neural network model. The multi-task convolutional neural network model simultaneously outputs three sets of prediction results: face confidence, face bounding box parameters, and face depth estimate.

[0009] S4: Based on the face bounding box parameters output by the convolutional neural network, extract the temperature values ​​of the corresponding regions from the original temperature matrix SrcTemp and calculate the average value as the final face temperature value FaceTemp;

[0010] S5: Calculate the three-dimensional spatial coordinates (X,Y,Z) of the face in the sensor coordinate system based on the face bounding box parameters output by the convolutional neural network, the face depth estimate, and the preset field of view parameters of the temperature sensor.

[0011] As an improvement, step S2 normalizes the original temperature matrix by linearly scaling the temperature values ​​to between 0 and 1. The specific method is as follows:

[0012]

[0013] Among them, SrcTemp i,j Here, Tmax is the temperature value in the i-th row and j-th column of the original temperature matrix, and SrcTemp Norm is the preset temperature range threshold. i,j This is the normalized value.

[0014] As an improvement, the convolutional neural network model in step S3 includes the following in sequence:

[0015] The first convolutional layer receives the normalized temperature matrix and outputs a 16-channel first feature map.

[0016] The first pooling layer, connected to the first convolutional layer, is used to downsample the first feature map;

[0017] The second convolutional layer is used to convolve the downsampled feature map and output a 32-channel second feature map.

[0018] The second pooling layer, connected to the second convolutional layer, is used to downsample the second feature map;

[0019] The third convolutional layer is used to convolve the downsampled feature map and output a 64-channel third feature map.

[0020] An adaptive pooling layer, connected to the third convolutional layer, is used to pool the third feature map to a fixed size;

[0021] The fully connected layer, connected to the adaptive pooling layer, is used to perform linear transformations on the pooled features.

[0022] The output layer consists of three independent sub-output layers, which are used to output face confidence, normalized coordinates and size of face bounding boxes, and face depth estimates, respectively.

[0023] As an improvement, the output layer includes:

[0024] The face confidence output layer uses the Sigmoid activation function to output the face confidence score.

[0025] The bounding box output layer uses the Sigmoid activation function to output the normalized center coordinates (x, y) and dimensions (w, h) of the face bounding box;

[0026] The depth estimation output layer uses a linear activation function to output the face depth estimate.

[0027] As an improvement, the multi-task convolutional neural network model is trained using a composite loss function, which is:

[0028] Loss=λ1·L cls +λ2·L box +λ3·L depth

[0029] Among them, L cls L is the binary cross-entropy loss for face confidence. box L is the mean squared error loss for the face bounding box parameters. depth Let λ1, λ2, and λ3 be the mean squared error loss of the face depth estimation, and let λ1, λ2, and λ3 be the weighting coefficients used to balance the losses of different tasks, where 0 ≤ λ1 ≤ 1, 0 ≤ λ2 ≤ 1, 0 ≤ λ3 ≤ 1, and λ1 + λ2 + λ3 = 1.

[0030] As an improvement, the final face temperature value FaceTemp in step S4 is calculated using the following formula:

[0031]

[0032] in,

[0033]

[0034] As an improvement, step S5 specifically includes the following steps:

[0035] S5.1: Take the face bounding box output by the convolutional neural network, and its coordinates (x, y) relative to the original temperature matrix and its coordinates (w, h) relative to the original temperature matrix, then the coordinates of the face corners in the original temperature matrix SrcTemp can be obtained, denoted as FaceSrcTemp, and represented in matrix form as follows:

[0036] S5.2: Obtain the depth of the face output by the convolutional neural network. This face depth is the Z coordinate in the sensor coordinate system. The horizontal field of view of the sensor is fixed and denoted as θ. w The sensor's vertical viewing angle field is also fixed, denoted as θ. h Thus, the X and Y coordinates in the sensor coordinate system are obtained, as expressed by the following formula:

[0037]

[0038] An apparatus for implementing the steps of any of the above-described CNN-based low-resolution temperature array face multimodal perception methods, the apparatus comprising a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0039] Compared with the prior art, the advantages of the present invention are as follows:

[0040] This approach integrates face detection, temperature measurement, and 3D spatial localization by combining a low-resolution temperature sensor with a multi-task convolutional neural network, offering comprehensive advantages in multimodal perception. The method utilizes normalization to improve data consistency and simultaneously outputs face confidence, bounding boxes, and depth information through the multi-task convolutional neural network, significantly improving detection accuracy and robustness under low-resolution thermal imaging conditions. The introduction of a composite loss function effectively balances the training objectives across different tasks, enhancing the overall model performance. Furthermore, face regions are extracted from the raw temperature data based on bounding box parameters, and the average temperature is calculated, avoiding the impact of normalization on actual temperature values ​​and ensuring the physical accuracy of the temperature measurement. By combining depth estimation with sensor field-of-view parameters, non-contact 3D spatial positioning is achieved. This technology can be widely applied in automotive smart cockpits and home environments. By detecting, measuring temperature, and locating the faces of passengers or family members, it provides a basis for automatic adjustment of the air conditioning system, such as automatically adjusting temperature, airflow intensity, and airflow direction, without requiring manual operation by the user. This significantly improves driving comfort and the smart home experience. The overall solution achieves highly integrated multimodal perception functions under the premise of low cost and low power consumption, and has good feasibility and promotion prospects. Attached Figure Description

[0041] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0042] Figure 1 This is the original low-resolution image of the face detection temperature measurement.

[0043] Figure 2 This is a normalized image of a face detected by temperature measurement.

[0044] Figure 3 For the final face detection temperature measurement image;

[0045] Figure 4 This is a schematic diagram of a multi-task convolutional neural network model. Detailed Implementation

[0046] like Figures 1 to 4 As shown, the low-resolution temperature array-based face multimodal perception method based on CNN includes the following steps:

[0047] S1: Obtain the original temperature matrix SrcTemp output by the temperature sensor, with SrcTempW as the number of columns and SrcTempH as the number of rows;

[0048] S2: Normalize the original temperature matrix by linearly scaling the temperature values ​​to between 0 and 1. The specific formula is as follows:

[0049]

[0050] Among them, SrcTemp i,j The temperature value in the i-th row and j-th column of the original temperature matrix, where Tmin and Tmax are preset temperature range thresholds, and SrcTemp Norm. i,j The value is the normalized value;

[0051] S3: Input the normalized temperature matrix into a pre-trained multi-task convolutional neural network model. The multi-task convolutional neural network model simultaneously outputs three sets of prediction results: face confidence, face bounding box parameters, and face depth estimate.

[0052] The convolutional neural network model includes, in order:

[0053] The first convolutional layer receives the normalized temperature matrix and outputs a 16-channel first feature map.

[0054] The first pooling layer, connected to the first convolutional layer, is used to downsample the first feature map;

[0055] Preferably, the first convolutional layer has 1 input channel, 16 output channels, a kernel size of 3×3, a stride of 1, and padding of 1. Then, it passes through the ReLU activation function and 2×2 max pooling operation to reduce the feature map size to 16×(SrcTempW / 2)×(SrcTempH / 2).

[0056] The second convolutional layer is used to convolve the downsampled feature map and output a 32-channel second feature map.

[0057] The second pooling layer, connected to the second convolutional layer, is used to downsample the second feature map;

[0058] Preferably, the second convolutional layer has 16 input channels, 32 output channels, a kernel size of 3×3, a stride of 1, padding of 1, ReLU activation, followed by 2×2 max pooling, reducing the size to 32×(SrcTempW / 4)×(SrcTempH / 4).

[0059] The third convolutional layer is used to convolve the downsampled feature map and output a 64-channel third feature map.

[0060] An adaptive pooling layer, connected to the third convolutional layer, is used to pool the third feature map to a fixed size;

[0061] The third convolutional layer has 32 input channels and 64 output channels. The kernel size is 3×3, stride is 1, padding is 1, ReLU activation is used, and then adaptive average pooling is performed to 64×(SrcTempW / 32)×(SrcTempH / 32).

[0062] The fully connected layer, connected to the adaptive pooling layer, is used to perform linear transformations on the pooled features.

[0063] Preferably, the pooled 64-dimensional feature vector is input into a fully connected layer, undergoes linear transformation and ReLU activation by 64 neurons to form a feature representation. This is then connected to three independent output layers.

[0064] The output layer consists of three independent sub-output layers, including:

[0065] The face confidence output layer uses the Sigmoid activation function to output the face confidence. Preferably, the face confidence output layer is a single neuron with an output range of [0,1].

[0066] The bounding box output layer uses the Sigmoid activation function to output the normalized center coordinates (x, y) and size (w, h) of the face bounding box. Preferably, the bounding box output layer has 4 neurons and the output range is [0, 1].

[0067] The depth estimation output layer uses a linear activation function to output the face depth estimate, which can be in centimeters. Preferably, the depth estimation output layer is a single neuron.

[0068] Supervised training was performed using temperature matrix data annotated with face location and depth. The multi-task convolutional neural network model was trained using a composite loss function, which is:

[0069] Loss=λ1·L cls +λ2·L box +λ3·L depth

[0070] Among them, L cls L is the binary cross-entropy loss for face confidence. box L is the mean squared error loss for the face bounding box parameters. depth λ1, λ2, and λ3 are the mean squared error loss of the face depth estimation, and λ1, λ2, and λ3 are the weighting coefficients used to balance the losses of different tasks. 0≤λ1≤1, 0≤λ2≤1, 0≤λ3≤1, λ1+λ2+λ3=1;

[0071] S4: Based on the face bounding box parameters output by the convolutional neural network, extract the temperature values ​​of the corresponding regions from the original temperature matrix SrcTemp and calculate the average value as the final face temperature value FaceTemp, calculated using the following formula:

[0072]

[0073] in,

[0074]

[0075] S5.1: Take the face bounding box output by the convolutional neural network, and its coordinates (x, y) relative to the original temperature matrix and its coordinates (w, h) relative to the original temperature matrix, then the coordinates of the face corners in the original temperature matrix SrcTemp can be obtained, denoted as FaceSrcTemp, and represented in matrix form as follows:

[0076] S5.2: Obtain the depth of the face output by the convolutional neural network. This face depth is the Z coordinate in the sensor coordinate system. The horizontal field of view of the sensor is fixed and denoted as θ. w The sensor's vertical viewing angle field is also fixed, denoted as θ. h Thus, the X and Y coordinates in the sensor coordinate system are obtained, as expressed by the following formula:

[0077]

[0078] An apparatus for implementing the steps of any of the CNN-based low-resolution temperature array face multimodal perception methods described above, the apparatus comprising a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0079] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0080] The units described in some embodiments of this disclosure can be implemented in software or in hardware. The described units can also be located in a processor, and the functions described above can be performed at least in part by one or more hardware logic components.

[0081] The above description only illustrates the preferred embodiments of the present invention and should not be construed as limiting the scope of the claims. The present invention is not limited to the above embodiments, and variations in its specific structure are permitted. All modifications made within the scope of the independent claims of this invention are also within the scope of protection of this invention.

Claims

1. A low-resolution temperature array-based multimodal face perception method based on CNN, characterized in that, Includes the following steps: S1: Obtain the raw temperature matrix output by the temperature sensor. Its column number is The number of rows is ; S2: Normalize the original temperature matrix to obtain the normalized temperature matrix. ; S3: Input the normalized temperature matrix into a pre-trained multi-task convolutional neural network model. The multi-task convolutional neural network model outputs three sets of prediction results simultaneously through three independent sub-output layers: face confidence, face bounding box parameters, and face depth estimate. S4: Based on the face bounding box parameters output by the convolutional neural network, from the original temperature matrix... Extract the temperature values ​​of the corresponding areas and calculate the average value as the final face temperature value. ; S5: Calculate the three-dimensional spatial coordinates (X,Y,Z) of the face in the sensor coordinate system based on the face bounding box parameters output by the convolutional neural network, the face depth estimate, and the preset field of view parameters of the temperature sensor. Step S5 specifically includes the following steps: S5.1: Take the face bounding box output by the convolutional neural network, and its coordinates (x, y) relative to the original temperature matrix and its coordinate dimensions (w, h) relative to the original temperature matrix, then the face bounding box in the original temperature matrix can be obtained. The coordinates of the corner points of the face in the image are denoted as . It is denoted in matrix form as follows: ； S5.2: Obtain the depth of the face output by the convolutional neural network. This face depth is the Z coordinate in the sensor coordinate system. The horizontal field of view of the sensor is fixed and denoted as... The sensor's vertical viewing angle field is also fixed, denoted as... Thus, the X and Y coordinates in the sensor coordinate system are obtained, as expressed by the following formula: ， ； Where X1 and X2 are the horizontal coordinates of the face bounding box, and Y1 and Y2 are the vertical coordinates of the face bounding box.

2. The CNN-based low-resolution temperature array face multimodal perception method according to claim 1, characterized in that, In step S2, the original temperature matrix is ​​normalized by linearly scaling the temperature values ​​to between 0 and 1. The specific formula is as follows: in, This represents the temperature value in the i-th row and j-th column of the original temperature matrix. and The preset temperature range threshold, This is the normalized value.

3. The CNN-based low-resolution temperature array face multimodal perception method according to claim 1, characterized in that, The convolutional neural network model in step S3 includes, in sequence: The first convolutional layer receives the normalized temperature matrix and outputs a 16-channel first feature map. The first pooling layer, connected to the first convolutional layer, is used to downsample the first feature map; The second convolutional layer is used to convolve the downsampled feature map and output a 32-channel second feature map. The second pooling layer, connected to the second convolutional layer, is used to downsample the second feature map; The third convolutional layer is used to convolve the downsampled feature map and output a 64-channel third feature map. An adaptive pooling layer, connected to the third convolutional layer, is used to pool the third feature map to a fixed size; A fully connected layer, connected to the adaptive pooling layer, is used to perform a linear transformation on the pooled features; The output layer outputs the face confidence score, the normalized coordinates and size of the face bounding box, and the estimated face depth, respectively.

4. The CNN-based low-resolution temperature array face multimodal perception method according to claim 3, characterized in that, The output layer includes: The face confidence output layer uses the Sigmoid activation function to output the face confidence score. The bounding box output layer uses the Sigmoid activation function to output the normalized center coordinates (x, y) and dimensions (w, h) of the face bounding box; The depth estimation output layer uses a linear activation function to output the face depth estimate.

5. The CNN-based low-resolution temperature array face multimodal perception method according to claim 3, characterized in that, The multi-task convolutional neural network model is trained using a composite loss function, which is: in, The binary cross-entropy loss for face confidence is used. The mean squared error loss is used for the face bounding box parameters. The mean squared error loss for face depth estimation , , The weighting coefficients are used to balance the losses of different tasks. , , , + + =1.

6. The CNN-based low-resolution temperature array face multimodal perception method according to claim 1, characterized in that, The final face temperature value in step S4 Calculated using the following formula: in, ， ， ， 。 7. An apparatus for implementing the steps of the CNN-based low-resolution temperature array face multimodal perception method as described in any one of claims 1 to 6, characterized in that: The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor.

Images