Body temperature monitoring system based on multi-mode sensor

Abstract

The invention discloses a body temperature monitoring system based on a multi-modal sensor, a visible light camera, a depth camera and a thermal imager are integrated into a whole, images, depth information and thermal imaging data of a human face can be obtained at the same time, the system can more accurately position a temperature measuring point on the human face through fusion of the multi-modal data, and the temperature measuring accuracy of the human face is improved. Measurement results of the thermal imager are dynamically adjusted through the correction module according to the temperature and humidity of the environment, the system can keep high temperature measurement accuracy under different environment conditions, the body temperature measurement results can be visually displayed to a user in real time through the design of the display module, the body temperature monitoring effect with quick response is achieved, and the user experience is improved. By integrating various sensors and an intelligent correction algorithm, intelligence and automation of body temperature monitoring are realized, operations such as face recognition, distance measurement, temperature measurement, environment perception, data correction and result display can be automatically completed, a body temperature monitoring task can be efficiently completed without manual intervention, and the intelligent level is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of body temperature monitoring, and in particular to a body temperature monitoring system based on a multimodal sensor. Background Art

[0002] Although significant progress has been made in the current field of temperature monitoring technology, there are still many challenges and shortcomings, especially under complex and changing environmental conditions. Traditional temperature monitoring methods, such as contact thermometers, are accurate in measurement, but the operation is cumbersome and time-consuming, and there is a risk of cross-infection. Although non-contact infrared thermometers can achieve rapid measurement, their measurement accuracy is often affected by multiple factors such as environmental temperature and humidity, human body movement status, etc., resulting in instability and errors in the measurement results.

[0003] In addition, most existing non-contact body temperature monitoring systems rely on a single sensor, such as using only infrared thermal imagers for temperature measurement. Although this single-modal sensor system can meet basic needs to a certain extent, due to the lack of multi-dimensional data support, its measurement results are often difficult to fully reflect the true body temperature of the human body. For example, infrared thermal imagers are easily affected by ambient temperature, resulting in measurement errors; at the same time, it is also unable to accurately distinguish the temperature differences between different parts of the human body, which may affect the accuracy of diagnosis.

[0004] In order to overcome these technical defects, people began to explore temperature monitoring technology based on multimodal sensors. However, the multimodal sensor systems on the market still face some challenges. First, the data fusion and calibration technology between different sensors is not yet mature, which limits the overall performance of the system. Second, the system's integration and intelligence level need to be improved to meet the needs of real-time, efficient and accurate temperature monitoring. Finally, the cost and complexity of the system are also important factors that restrict its widespread application. Summary of the invention

[0005] In view of this, the present invention proposes a body temperature monitoring system based on a multimodal sensor, which can solve the defects of the prior art that the data fusion and calibration technology between different sensors is still immature, the integration and intelligence level need to be improved, and the cost is high.

[0006] The technical solution of the present invention is achieved in this way:

[0007] A body temperature monitoring system based on a multimodal sensor includes a visible light camera, a depth camera, a thermal imager, an environmental temperature and humidity detection module, a correction module and a display module, wherein the visible light camera, the depth camera and the thermal imager are integrated into one, the visible light camera is used to capture a face image, the depth camera is used to measure the distance between the face and the camera, the thermal imager is used to align the distance information obtained by the depth camera with the image obtained by the visible light camera, and obtain a positioning temperature measurement point based on the registration result, and perform temperature measurement based on the positioning temperature measurement point, the environmental temperature and humidity detection module is used to obtain the temperature and humidity of the environment, the correction module is used to correct the temperature measured by the thermal imager based on the temperature and humidity of the environment, and the display module is used to display the body temperature measurement result in real time.

[0008] As a further optional solution of the body temperature monitoring system based on the multimodal sensor, the thermal imager aligns the distance information obtained by the depth camera with the image obtained by the visible light camera, specifically including:

[0009] Use dual-target positioning technology to obtain the internal and external parameters of the visible light camera and the depth camera as well as their relative position relationship;

[0010] Based on the results of binocular positioning, the depth image formed according to the distance information is registered with the visible light image.

[0011] As a further optional solution of the body temperature monitoring system based on the multimodal sensor, the result of the binocular positioning is to register the depth image formed according to the distance information with the visible light image, specifically including:

[0012] Preprocessing the depth image and the visible light image to obtain a preprocessed depth image and a visible light image;

[0013] Extracting feature points for registration from the preprocessed depth image and visible light image;

[0014] The feature points are described using feature descriptors, and the corresponding feature point pairs in the depth image and visible light image are found through feature matching algorithms;

[0015] Based on the matched feature point pairs, calculate the transformation matrix from the depth image to the visible light image;

[0016] The depth image is transformed according to the calculated transformation matrix so that it coincides with the visible light image in space, thereby obtaining a registered visible light image.

[0017] As a further optional solution of the body temperature monitoring system based on the multimodal sensor, the obtaining of the positioning temperature measurement point according to the registration result specifically includes:

[0018] Acquire infrared thermal images based on thermal imagers;

[0019] Detecting faces in the registered visible light images based on computer vision algorithms;

[0020] The camera parameters obtained by binocular calibration are used to calculate the transformation relationship between the registered visible light image and the infrared thermal image.

[0021] In the infrared thermal image, the temperature measurement point of the face is located according to the registered face area.

[0022] As a further optional solution of the body temperature monitoring system based on the multimodal sensor, the correction module corrects the temperature measured by the thermal imager according to the temperature and humidity of the environment, specifically including:

[0023] Constructing a multivariate regression model, wherein the input of the multivariate regression model is the temperature and humidity of the environment and the face temperature measured by the thermal imager, and the output is the compensated temperature;

[0024] The real-time detected temperature and humidity of the environment and the face temperature measured by the thermal imager are input into the multivariate regression model, and the multivariate regression model outputs the compensated temperature.

[0025] As a further optional solution of the body temperature monitoring system based on the multimodal sensor, the calculation formula of the multivariate regression model is as follows:

[0026] Tcom=β0+β1Tmeas+β2Tenv+β3Henv;

[0027] Wherein, Tcom is the compensated temperature;

[0028] Tmeas is the face temperature measured by a thermal imager;

[0029] The Tenv is the ambient temperature;

[0030] Henv is the ambient humidity;

[0031] The β0, β1, β2 and β3 are regression coefficients, which are obtained through regression analysis and are used to quantify the influence of different factors on temperature measurement.

[0032] As a further optional solution of the body temperature monitoring system based on the multimodal sensor, the system further includes an early warning module, which is used to perform early warning tracking when the body temperature is abnormal, specifically including:

[0033] The images and related information of people with abnormal body temperature are uploaded to the central server through the router. The central server distributes the images and information of people with abnormal body temperature to the surveillance cameras. The surveillance cameras use facial recognition technology to continuously track people with abnormal body temperature and record their whereabouts in real time.

[0034] A body temperature monitoring method based on a multimodal sensor, the method is based on any of the above-mentioned body temperature monitoring systems based on a multimodal sensor, and specifically comprises:

[0035] Capturing facial images using a visible light camera;

[0036] Using a depth camera integrated with a visible light camera to measure the distance between the face and the camera and generate distance information;

[0037] The distance information generated by the depth camera is received by the thermal imager, and the distance information is registered with the image obtained by the visible light camera to determine the location of the temperature measurement point;

[0038] Based on the located temperature measurement points, a thermal imager is used to measure the temperature to obtain preliminary body temperature data;

[0039] Use the environmental temperature and humidity detection module to obtain the environmental temperature and humidity information in real time;

[0040] The ambient temperature and humidity information obtained by the ambient temperature and humidity detection module is input into the correction module. The correction module corrects the preliminary body temperature data measured by the thermal imager according to the ambient temperature and humidity based on a preset correction model to obtain the final body temperature data;

[0041] The final body temperature data is transmitted to the display module in real time, and the body temperature measurement results are displayed through the display module.

[0042] A computing device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned body temperature monitoring method based on a multimodal sensor when executing the computer program.

[0043] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned body temperature monitoring method based on a multimodal sensor.

[0044] The beneficial effects of the present invention are as follows: by integrating the visible light camera, the depth camera and the thermal imager into one, the system can simultaneously obtain the image, depth information and thermal imaging data of the human face. The fusion of such multimodal data enables the system to more accurately locate the temperature measurement point on the human face, avoiding the temperature measurement error caused by factors such as the face posture and expression changes. The addition of the environmental temperature and humidity detection module enables the system to perceive and adapt to different environmental conditions in real time. The correction module dynamically adjusts the measurement results of the thermal imager according to the temperature and humidity of the environment. The system can maintain a high temperature measurement accuracy under different environmental conditions. The design of the display module enables the temperature measurement results to be displayed in real time and directly. The system can be displayed to users visually, which is especially important for temperature monitoring scenarios that require quick response. Due to the integrated design of visible light camera, depth camera and thermal imager, users can complete temperature measurement without complicated operations or adjustments. This one-click, non-contact measurement method not only improves the convenience of measurement, but also reduces the risk and cost of cross-infection. The system realizes the intelligence and automation of temperature monitoring by integrating multiple sensors and intelligent correction algorithms. The system can automatically complete a series of operations such as face recognition, distance measurement, temperature measurement, environmental perception, data correction and result display. It can efficiently complete temperature monitoring tasks without human intervention, thereby improving the level of intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 It is a schematic diagram of the composition of a body temperature monitoring system based on a multimodal sensor of the present invention;

[0047] Figure 2 It is a flow chart of a body temperature monitoring method based on a multimodal sensor of the present invention. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] refer to Figure 1-2A body temperature monitoring system based on a multimodal sensor includes a visible light camera, a depth camera, a thermal imager, an environmental temperature and humidity detection module, a correction module and a display module, wherein the visible light camera, the depth camera and the thermal imager are integrated into one, the visible light camera is used to capture a face image, the depth camera is used to measure the distance between the face and the camera, the thermal imager is used to align the distance information obtained by the depth camera with the image obtained by the visible light camera, and obtain a positioning temperature measurement point based on the alignment result, and perform temperature measurement based on the positioning temperature measurement point, the environmental temperature and humidity detection module is used to obtain the temperature and humidity of the environment, the correction module is used to correct the temperature measured by the thermal imager based on the temperature and humidity of the environment, and the display module is used to display the body temperature measurement result in real time.

[0050] In this embodiment, by integrating the visible light camera, depth camera and thermal imager into one, the system can simultaneously obtain the image, depth information and thermal imaging data of the face. The fusion of this multimodal data enables the system to more accurately locate the temperature measurement point on the face, avoiding temperature measurement errors caused by factors such as facial posture and expression changes. The addition of the environmental temperature and humidity detection module enables the system to perceive and adapt to different environmental conditions in real time. The correction module dynamically adjusts the measurement results of the thermal imager according to the temperature and humidity of the environment. The system can maintain a high temperature measurement accuracy under different environmental conditions. The design of the display module enables the temperature measurement results to be displayed in real time and intuitively. Show it to users, which is especially important for temperature monitoring scenarios that require quick response. Due to the integrated design of visible light cameras, depth cameras and thermal imagers, users can complete temperature measurement without complicated operations or adjustments. This one-click, non-contact measurement method not only improves the convenience of measurement, but also reduces the risk and cost of cross-infection. The system realizes the intelligence and automation of temperature monitoring by integrating multiple sensors and intelligent correction algorithms. The system can automatically complete a series of operations such as face recognition, distance measurement, temperature measurement, environmental perception, data correction and result display. It can efficiently complete temperature monitoring tasks without human intervention, thereby improving the level of intelligence.

[0051] Preferably, the thermal imager aligns the distance information obtained by the depth camera with the image obtained by the visible light camera, specifically including:

[0052] Use dual-target positioning technology to obtain the internal and external parameters of the visible light camera and the depth camera as well as their relative position relationship;

[0053] Based on the results of binocular positioning, the depth image formed according to the distance information is registered with the visible light image.

[0054] In this embodiment, the use of dual-target positioning technology can accurately obtain the internal and external parameters of the visible light camera and the depth camera and the relative position relationship between them. These parameters are the basis for subsequent image registration, which can ensure the spatial consistency of the depth image and the visible light image, thereby improving the accuracy of registration; through dual-target positioning, the system can automatically adapt to the slight differences and changes between different devices, such as lens distortion, installation errors, etc. This adaptive capability enables the system to maintain stable performance in different environments and enhances the robustness of the system; the registered depth image and visible light image can correspond more accurately, allowing the system to be based on the depth information in the face image The temperature measurement point can be accurately located on the image. This precise positioning capability helps to improve the accuracy of body temperature measurement and reduce errors caused by improper selection of temperature measurement points. The dual-target positioning and image registration process can be completed during system initialization and serve as the basis for subsequent measurement processes. Once the calibration and registration are completed, the system can quickly and accurately complete the body temperature measurement task, improving the efficiency of measurement. The registered depth image and visible light image provide the possibility for subsequent multimodal data fusion. The system can further combine the data from the thermal imager to perform more accurate temperature measurement and analysis of the temperature measurement points. This fusion of multimodal data helps to improve the overall performance and application scope of the system.

[0055] Preferably, the registering of the depth image formed according to the distance information with the visible light image based on the result of the binocular positioning specifically includes:

[0056] Preprocessing the depth image and the visible light image to obtain a preprocessed depth image and a visible light image;

[0057] Extracting feature points for registration from the preprocessed depth image and visible light image;

[0058] The feature points are described using feature descriptors, and the corresponding feature point pairs in the depth image and visible light image are found through feature matching algorithms;

[0059] Based on the matched feature point pairs, calculate the transformation matrix from the depth image to the visible light image;

[0060] The depth image is transformed according to the calculated transformation matrix so that it coincides with the visible light image in space, thereby obtaining a registered visible light image.

[0061] In this embodiment, preprocessing the depth image and the visible light image is an important step in the registration process. The preprocessing steps may include denoising, contrast enhancement, distortion correction, etc. These operations can improve image quality and reduce errors in subsequent processing, thereby improving the accuracy and reliability of registration; extracting feature points for registration is a key link in the registration process. These feature points are usually unique and stable points in the image, such as corner points, edge intersections, etc. By extracting these feature points, the system can more accurately find the correspondence between the two images and enhance the matching ability; using feature descriptors to describe feature points and finding corresponding feature point pairs in the depth image and the visible light image through feature matching algorithms are the core steps in the registration process. Feature descriptors can capture local information around feature points, allowing the system to distinguish different feature points and accurately match them. This matching The registration method has high accuracy and robustness, and can cope with challenges such as illumination changes and occlusions in images; calculating the transformation matrix from the depth image to the visible light image based on the matched feature point pairs is a key step in the registration process. This transformation matrix describes the spatial transformation relationship from the depth image to the visible light image, including translation, rotation, scaling, etc. By calculating the transformation matrix, the system can ensure the spatial consistency of the depth image and the visible light image, and provide accurate positioning information for subsequent temperature measurements; transforming the depth image according to the calculated transformation matrix so that it coincides with the visible light image in space is the ultimate goal of the registration process. Through image transformation, the system can eliminate the spatial difference between the depth image and the visible light image and achieve precise alignment of the two images. This precise registration provides a reliable foundation for subsequent temperature measurements and helps to improve the accuracy and reliability of measurements.

[0062] Preferably, the obtaining of the positioning temperature measurement point according to the registration result specifically includes:

[0063] Acquire infrared thermal images based on thermal imagers;

[0064] Detecting faces in the registered visible light images based on computer vision algorithms;

[0065] The camera parameters obtained by binocular calibration are used to calculate the transformation relationship between the registered visible light image and the infrared thermal image.

[0066] In the infrared thermal image, the temperature measurement point of the face is located according to the registered face area.

[0067] In this embodiment, the thermal imager can obtain infrared thermal images of the measured object, and these images contain the temperature information of the object. Through the infrared thermal images obtained by the thermal imager, the system can intuitively see the temperature distribution of the face area, providing basic data for the subsequent positioning of the temperature measurement points; the face is detected in the registered visible light image using a computer vision algorithm, and the face area can be accurately identified. This detection method based on image processing has high accuracy and robustness, and can cope with challenges such as different lighting conditions, facial postures and expressions. Through face detection, the system can determine the position and range of the face in the image, providing an important reference for the subsequent positioning of the temperature measurement points; the dual-target positioning technology is not only used for the registration of the visible light camera and the depth camera, but also for calculating the transformation relationship between the registered visible light image and the infrared thermal image. This transformation relationship describes the spatial correspondence between the two images, making the system It is able to map the face area in the visible light image to the infrared thermal image. Through the binocular positioning technology, the system can ensure the accuracy of this mapping, so as to accurately locate the face area in the infrared thermal image. In the infrared thermal image, according to the registered face area, the system can locate the temperature measurement points of the face. These temperature measurement points are usually areas where the temperature distribution in the face is relatively uniform and can represent the overall body temperature, such as the forehead and cheeks. By accurately locating the temperature measurement points, the system can reduce the errors caused by improper selection of temperature measurement points and improve the accuracy and reliability of body temperature measurement. It integrates a variety of technical means such as thermal imaging technology, computer vision algorithms and binocular positioning technology to achieve the goal of accurately locating the face temperature measurement points in infrared thermal images. The application of this comprehensive technology not only improves the measurement accuracy and reliability of the system, but also enhances the adaptability and flexibility of the system. The system can cope with body temperature monitoring tasks under different environments and conditions.

[0068] Preferably, the correction module corrects the temperature measured by the thermal imager according to the temperature and humidity of the environment, specifically including:

[0069] Constructing a multivariate regression model, wherein the input of the multivariate regression model is the temperature and humidity of the environment and the face temperature measured by the thermal imager, and the output is the compensated temperature;

[0070] The real-time detected temperature and humidity of the environment and the face temperature measured by the thermal imager are input into the multivariate regression model, and the multivariate regression model outputs the compensated temperature.

[0071] In this embodiment, the temperature and humidity of the environment are important factors that affect the measurement accuracy of the thermal imager. By constructing a multivariate regression model and taking the temperature and humidity of the environment as one of the input variables, the system can automatically consider the impact of these environmental factors on the measurement results, thereby outputting a more accurate compensated temperature. This correction method significantly improves the accuracy of body temperature measurement and reduces the error caused by environmental factors. The multivariate regression model can handle the complex relationship between multiple input variables, so it can adapt to the body temperature measurement needs under different environmental conditions. Whether in a high temperature and high humidity environment or a low temperature and low humidity environment, the system can calculate the corresponding compensation value through the model to ensure the accuracy of the measurement results. This adaptability enables the system to be used in a wider range of fields. Application in scene; The correction process is fully automated, without human intervention. The system can detect the temperature and humidity of the environment in real time, and immediately input the detection results and the face temperature measured by the thermal imager into the multivariate regression model to quickly calculate the compensated temperature. This real-time and automatic feature enables the system to respond quickly to environmental changes and provide instant body temperature measurement results; The multivariate regression model is trained based on a large amount of historical data, which includes the measurement results of the thermal imager and the actual body temperature values ​​under different environmental conditions. Through a data-driven approach, the model can learn the intrinsic relationship between environmental temperature and humidity and measurement errors, so as to more accurately predict and compensate for errors. This data-driven approach makes the system more intelligent and reliable.

[0072] Preferably, the calculation formula of the multivariate regression model is as follows:

[0073] Tcom=β0+β1Tmeas+β2Tenv+β3Henv;

[0074] Wherein, Tcom is the compensated temperature;

[0075] Tmeas is the face temperature measured by a thermal imager;

[0076] The Tenv is the ambient temperature;

[0077] Henv is the ambient humidity;

[0078] The β0, β1, β2 and β3 are regression coefficients, which are obtained through regression analysis and are used to quantify the influence of different factors on temperature measurement.

[0079] In this embodiment, the acquisition of regression coefficients β0, β1, β2, and β3 generally involves a series of statistical analysis and modeling steps, which are intended to determine the best linear relationship between independent variables (such as ambient temperature, directly measured temperature, etc.) and dependent variables (such as compensated temperature). The following are specific steps for obtaining these regression coefficients:

[0080] First, a series of observations including dependent variables (compensated temperature Tcom, which may be unknown before actual modeling, and usually a standard or known accurate temperature is used as a substitute) and independent variables (such as ambient temperature Tenv, directly measured temperature Tmeas, and other possible environmental parameters P3Henu) need to be collected. These data should cover different environmental conditions and measurement situations to ensure the generalization ability of the model.

[0081] Before modeling, data may need to be preprocessed, including but not limited to:

[0082] Cleaning data: removing outliers, missing values, etc.

[0083] Data transformation: Logarithmic transformation, standardization, or normalization of certain variables can be performed to improve the performance of the model.

[0084] Before performing regression analysis, it is necessary to check whether the data meets the assumptions of the linear regression model, including:

[0085] Linear relationship assumption: There should be a linear relationship between the independent and dependent variables;

[0086] Error term assumption: The error terms should be independent and identically distributed and follow a normal distribution;

[0087] No multicollinearity: There should not be a high correlation between the independent variables.

[0088] Using the collected data, the regression coefficients are estimated through regression analysis (such as the method of least squares). In multiple linear regression, the goal is to find a set of regression coefficients that minimizes the sum of squared errors between the predicted and actual values, which is usually achieved by solving normal equations or using optimization algorithms.

[0089] Coefficient interpretation: Each regression coefficient represents the average change in the dependent variable when the corresponding independent variable changes by one unit while keeping other variables unchanged.

[0090] Coefficient test: Use statistical tests (such as t-test) to evaluate whether each regression coefficient is significantly different from zero, that is, whether the corresponding independent variable has a significant impact on the dependent variable.

[0091] Preferably, the system further includes an early warning module, which is used to perform early warning tracking when the body temperature is abnormal, specifically including:

[0092] The images and related information of people with abnormal body temperature are uploaded to the central server through the router. The central server distributes the images and information of people with abnormal body temperature to the surveillance cameras. The surveillance cameras use facial recognition technology to continuously track people with abnormal body temperature and record their whereabouts in real time.

[0093] In this embodiment, the central server serves as an information hub and distributes images and information of people with abnormal body temperature to surveillance cameras. This cross-device collaborative working mode realizes real-time sharing and efficient use of information, so that surveillance cameras in different locations can jointly participate in the tracking of people with abnormal body temperature; surveillance cameras use advanced face recognition technology to continuously track people with abnormal body temperature. Face recognition technology has a high degree of accuracy and robustness, and can accurately identify target persons in complex environments to ensure the accuracy and reliability of tracking; the early warning module can not only track the whereabouts of people with abnormal body temperature in real time, but also record their activity trajectory. Through rapid early warning, precise tracking and real-time recording, the system can quickly lock on to the target population when an emergency occurs, and take effective measures to prevent, control and manage, to ensure public safety and social stability.

[0094] A body temperature monitoring method based on a multimodal sensor, the method is based on any of the above-mentioned body temperature monitoring systems based on a multimodal sensor, and specifically comprises:

[0095] Capturing facial images using a visible light camera;

[0096] Using a depth camera integrated with a visible light camera to measure the distance between the face and the camera and generate distance information;

[0097] The distance information generated by the depth camera is received by the thermal imager, and the distance information is registered with the image obtained by the visible light camera to determine the location of the temperature measurement point;

[0098] Based on the located temperature measurement points, a thermal imager is used to measure the temperature to obtain preliminary body temperature data;

[0099] Use the environmental temperature and humidity detection module to obtain the environmental temperature and humidity information in real time;

[0100] The ambient temperature and humidity information obtained by the ambient temperature and humidity detection module is input into the correction module. The correction module corrects the preliminary body temperature data measured by the thermal imager according to the ambient temperature and humidity based on a preset correction model to obtain the final body temperature data;

[0101] The final body temperature data is transmitted to the display module in real time, and the body temperature measurement results are displayed through the display module.

[0102] A computing device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned body temperature monitoring method based on a multimodal sensor when executing the computer program.

[0103] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned body temperature monitoring method based on a multimodal sensor.

[0104] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A body temperature monitoring system based on a multimodal sensor, characterized in that: It includes a visible light camera, a depth camera, a thermal imager, an environmental temperature and humidity detection module, a correction module and a display module. The visible light camera, the depth camera and the thermal imager are integrated into one. The visible light camera is used to capture facial images, the depth camera is used to measure the distance between the face and the camera, the thermal imager is used to align the distance information obtained by the depth camera with the image obtained by the visible light camera, and obtain the positioning temperature measurement point according to the alignment result, and measure the temperature according to the positioning temperature measurement point. The environmental temperature and humidity detection module is used to obtain the temperature and humidity of the environment, the correction module is used to correct the temperature measured by the thermal imager according to the temperature and humidity of the environment, and the display module is used to display the body temperature measurement result in real time.

2. A body temperature monitoring system based on a multimodal sensor according to claim 1, characterized in that: The thermal imager aligns the distance information obtained by the depth camera with the image obtained by the visible light camera, specifically including: Use dual-target positioning technology to obtain the internal and external parameters of the visible light camera and the depth camera as well as their relative position relationship; Based on the results of binocular positioning, the depth image formed according to the distance information is registered with the visible light image.

3. A body temperature monitoring system based on a multimodal sensor according to claim 2, characterized in that: Based on the result of the binocular positioning, the depth image formed according to the distance information is registered with the visible light image, specifically including: Preprocessing the depth image and the visible light image to obtain a preprocessed depth image and a visible light image; Extracting feature points for registration from the preprocessed depth image and visible light image; The feature points are described using feature descriptors, and the corresponding feature point pairs in the depth image and visible light image are found through feature matching algorithms; Based on the matched feature point pairs, calculate the transformation matrix from the depth image to the visible light image; The depth image is transformed according to the calculated transformation matrix so that it coincides with the visible light image in space, thereby obtaining a registered visible light image.

4. A body temperature monitoring system based on a multimodal sensor according to claim 3, characterized in that: The obtaining of the positioning temperature measurement point according to the registration result specifically includes: Acquire infrared thermal images based on thermal imagers; Detecting faces in the registered visible light images based on computer vision algorithms; The camera parameters obtained by binocular calibration are used to calculate the transformation relationship between the registered visible light image and the infrared thermal image. In the infrared thermal image, the temperature measurement point of the face is located according to the registered face area.

5. A body temperature monitoring system based on a multimodal sensor according to claim 4, characterized in that: The correction module corrects the temperature measured by the thermal imager according to the temperature and humidity of the environment, specifically including: Constructing a multivariate regression model, wherein the input of the multivariate regression model is the temperature and humidity of the environment and the face temperature measured by the thermal imager, and the output is the compensated temperature; The real-time detected temperature and humidity of the environment and the face temperature measured by the thermal imager are input into the multivariate regression model, and the multivariate regression model outputs the compensated temperature.

6. A body temperature monitoring system based on a multimodal sensor according to claim 5, characterized in that: The calculation formula of the multivariate regression model is as follows: Tcom=β0+β1Tmeas+β2Tenv+β3Henv; Wherein, Tcom is the compensated temperature; Tmeas is the face temperature measured by a thermal imager; The Tenv is the ambient temperature; Henv is the ambient humidity; The β0, β1, β2 and β3 are regression coefficients, which are obtained through regression analysis and are used to quantify the influence of different factors on temperature measurement.

7. A body temperature monitoring system based on a multimodal sensor according to claim 6, characterized in that: The system also includes an early warning module, which is used to perform early warning tracking when the body temperature is abnormal, specifically including: The images and related information of people with abnormal body temperature are uploaded to the central server through the router. The central server distributes the images and information of people with abnormal body temperature to the surveillance cameras. The surveillance cameras use facial recognition technology to continuously track people with abnormal body temperature and record their whereabouts in real time.

8. A body temperature monitoring method based on a multimodal sensor, characterized in that: The method is based on the body temperature monitoring system based on a multimodal sensor according to any one of claims 1 to 7, and specifically comprises: Capturing facial images using a visible light camera; Using a depth camera integrated with a visible light camera to measure the distance between the face and the camera and generate distance information; The distance information generated by the depth camera is received by the thermal imager, and the distance information is registered with the image obtained by the visible light camera to determine the location of the temperature measurement point; Based on the located temperature measurement points, a thermal imager is used to measure the temperature to obtain preliminary body temperature data; Use the environmental temperature and humidity detection module to obtain the environmental temperature and humidity information in real time; The ambient temperature and humidity information obtained by the ambient temperature and humidity detection module is input into the correction module. The correction module corrects the preliminary body temperature data measured by the thermal imager according to the ambient temperature and humidity based on a preset correction model to obtain the final body temperature data; The final body temperature data is transmitted to the display module in real time, and the body temperature measurement results are displayed through the display module.

9. A computing device, characterized in that The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the body temperature monitoring method based on a multimodal sensor as described in claim 8 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the body temperature monitoring method based on a multimodal sensor described in claim 8 are implemented.

Images