Millimeter wave radar privacy area real-time shielding system assisted by thermal imaging

Abstract

The invention discloses a thermal imaging-assisted millimeter wave radar privacy area real-time shielding system, and relates to the technical field of privacy protection and sensors, and the system comprises a privacy area detection module which is used for scanning a to-be-detected space through a millimeter wave radar, and detecting and marking a potential privacy area; the living body recognition module is used for collecting a temperature map in the privacy area through a thermal imaging sensor and extracting temperature change information; the data fusion module is used for fusing output results of the privacy area detection module and the living body recognition module; and the shielding control module is used for superposing a virtual shielding layer on the privacy area on the AR display terminal and triggering an alarm. According to the method, the problem of accurately protecting the privacy area in real time in a dynamic environment is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of privacy protection and sensor technology, more particularly, to a thermal imaging assisted millimeter wave radar privacy area real-time shielding system. BACKGROUND

[0002] With the rapid development of smart home, augmented reality (AR) and other technologies, sensor systems are increasingly widely used in daily life. Millimeter wave radar, due to its high penetration and sensitivity to object movement, has become an important tool for environmental perception; while thermal imaging technology, by capturing infrared radiation, can provide temperature distribution information, and has unique advantages in security monitoring and other fields. However, while these technologies provide convenience, privacy protection issues have gradually emerged. Especially in dynamic environments such as homes or public spaces, users may face the risk of accidental exposure of privacy areas. Currently, the problem to be solved is how to accurately identify and protect areas involving human bodies or other sensitive objects while maintaining the functionality of sensors, in order to cope with the challenges brought by environmental changes and individual differences. This not only relates to the practicality of technology, but also directly affects users' trust and acceptance of smart devices. SUMMARY

[0003] The technical problem to be solved by the present application is to provide a thermal imaging assisted millimeter wave radar privacy area real-time shielding system to solve the problems mentioned in the background.

[0004] In order to achieve the above-mentioned purpose, the present application adopts the following technical scheme:

[0005] A thermal imaging assisted millimeter wave radar privacy area real-time shielding system, comprising:

[0006] A privacy area detection module for scanning a to-be-measured space by a millimeter wave radar, detecting and marking potential privacy areas;

[0007] A living body recognition module for collecting temperature maps within the privacy areas by a thermal imaging sensor and extracting temperature change information;

[0008] A data fusion module for fusing the output results of the privacy area detection module and the living body recognition module;

[0009] A shielding control module for superimposing a virtual shielding layer on the privacy areas on an AR display terminal and triggering an alarm.

[0010] In some embodiments, the privacy area detection module performs the following steps: first, baseline scanning is performed in an empty environment to obtain a baseline reflection coefficient σ base ; the millimeter wave radar continuously collects reflection coefficient σ(t) at time t, and calculates Δσ = |σ(t) - σ baseThe return point cloud exceeding the preset threshold is subjected to DBSCAN clustering to generate a voxel cluster, and a space region corresponding to the voxel cluster is marked as a privacy region.

[0011] In some embodiments, the living body recognition module comprises a thermal imaging sensor configured to continuously acquire temperature data of the marked privacy region and calculate a temperature change rate per unit time according to ΔT / Δt, where ΔT represents a temperature difference and Δt represents a time interval.

[0012] In some embodiments, the data fusion module calculates a living body detection score S according to the following formula:

[0013]

[0014] where ΔT ref represents a reference temperature difference, Δt ref represents a reference time interval, w r and w h are preset weight coefficients.

[0015] In some embodiments, the data fusion module compares the score S with a preset threshold T, and if S>T, it is determined that a living body exists in the privacy region.

[0016] In some embodiments, the shielding control module superimposes a semi-transparent mosaic on the confirmed privacy region outline position in the field of view of the AR display terminal, and draws a red warning frame on the edge of the outline.

[0017] In some embodiments, the shielding control module is deployed in an edge computing unit, and the edge computing unit locally completes data fusion and image processing.

[0018] In some embodiments, the system further comprises a permission management module configured to receive a one-time dynamic code of a user and temporarily release or restore the shielding state of a designated region according to the dynamic code.

[0019] In some embodiments, after receiving the one-time dynamic code, the permission management module only releases the shielding of a designated private space that has passed authentication, and uploads logs such as time, region, and user information of the operation to a background system.

[0020] In some embodiments, the AR display terminal is intelligent glasses integrated with the privacy region detection module, the living body recognition module, and the data fusion module, or is a smart home display screen in communication with the edge computing unit.

[0021] The advantages of the present application over the prior art are that the present application realizes real-time detection and shielding of privacy areas by combining millimeter wave radar with thermal imaging technology, effectively solving the problem of insufficient precision in privacy protection in dynamic environments. The core advantage is that millimeter wave radar is used to scan the space and mark potential privacy areas, and then a thermal imaging sensor is used to analyze temperature changes to determine the presence of living beings, thereby significantly improving recognition accuracy and ensuring that sensitive areas are protected in a timely manner. In addition, the system is further optimized by superimposing a virtual shielding layer on the AR display terminal and triggering an alarm, allowing users to intuitively perceive the privacy protection status. At the same time, the application of edge computing units improves real-time performance and response speed, making it suitable for a variety of scenarios such as smart homes and smart glasses. The introduction of the permission management module also provides flexibility for users, allowing temporary adjustments to the shielding state through one-time dynamic codes, enhancing control while ensuring security. These improvements collectively enhance the practicality and user experience of the system. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is the overall schematic diagram of the system of the present application;

[0023] Figure 2 is a schematic diagram of the privacy area detection module of the system of the present application;

[0024] Figure 3 is a schematic diagram of the living body recognition module of the system of the present application;

[0025] Figure 4 is a schematic diagram of the shielding control and permission management module of the system of the present application. DETAILED DESCRIPTION

[0026] The specific embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0027] The present application discloses a thermal imaging assisted millimeter wave radar privacy area real-time shielding system, which integrates millimeter wave radar and thermal imaging technology to realize real-time detection and shielding of privacy areas in space, aiming to protect personal privacy while providing flexible permission management functions. The system is composed of multiple modules, including a privacy area detection module, a living body recognition module, a data fusion module, a shielding control module, and a permission management module, suitable for a variety of scenarios such as smart homes and smart glasses.

[0028] As Figure 1As shown, the system of this invention uses millimeter-wave radar to scan the space under test, detects and marks potential privacy areas, and then collects temperature information of these areas through a thermal imaging sensor to determine whether a living being (such as a human body) is present. Subsequently, the data fusion module comprehensively analyzes the detection results of radar and thermal imaging to conclude whether a living being exists. For privacy areas where a living being is confirmed to exist, the shielding control module overlays a virtual masking layer on the augmented reality (AR) display terminal and triggers an alert to protect privacy. In addition, the system also includes a permission management module that allows users to temporarily adjust the masking status of specific areas using a one-time dynamic code.

[0029] This invention emphasizes both real-time performance and localized processing, with some functions deployed on edge computing units to ensure low latency and high efficiency. The AR display terminal can be either portable smart glasses or a fixed smart home display that communicates with the edge computing unit.

[0030] like Figure 2 As shown, more specifically, in the system of this invention, the privacy region detection module uses millimeter-wave radar to scan space and detects and marks potential privacy regions by analyzing changes in radar reflection coefficients. Its operation consists of the following steps.

[0031] When the system is initialized or the space is unmanned, the millimeter-wave radar first performs a baseline scan and records the reflection coefficient of the space at this time, denoted as σ. base This baseline value reflects the electromagnetic wave reflection characteristics in an air environment and serves as a reference standard for subsequent testing. Baseline scans are typically performed after system installation or when significant environmental changes occur (such as furniture adjustments) to ensure data accuracy.

[0032] During system operation, the millimeter-wave radar continuously scans space, acquiring the reflection coefficient σ(t) at each time t. The difference between the current reflection coefficient and the baseline value is calculated as Δσ = |σ(t) - σ. base The system can identify areas in space where reflectivity changes. These changes may be caused by people entering, objects moving, etc. Δσ is calculated in absolute form to capture any significant changes, whether positive or negative.

[0033] To identify potential privacy zones, the system sets a preset threshold for Δσ. When Δσ exceeds this threshold, the system considers the reflectivity of the area to have changed significantly, potentially indicating the presence of privacy-sensitive objects. The specific value of the threshold needs to be calibrated based on the radar's sensitivity and the ambient noise level, and can be determined experimentally; for example, it could be set to 1.1σ. base up to 1.5σ base .

[0034] Subsequently, the system performs a DBSCAN clustering algorithm on the radar return point cloud data that exceeds the threshold. DBSCAN is a density-based clustering method that can identify dense clusters of points in space and exclude isolated noise points. In the system, DBSCAN groups the point cloud that exceeds the threshold into voxel clusters, and the spatial region corresponding to each voxel cluster is labeled as a potential privacy region.

[0035] The DBSCAN algorithm requires setting two parameters: neighborhood radius ε and minimum point number MinPts. ε defines the maximum distance between points, and MinPts defines the minimum number of points required to form a cluster. For example, in a home scenario, ε can be set to 0.5 meters, and MinPts can be set to 10 to identify areas the size of a human body. The adjustment of these parameters needs to be optimized according to the density of the radar point cloud and the expected size of the privacy region.

[0036] As shown in Figure 3 , the living body recognition module monitors the temperature of the marked privacy region through the thermal imaging sensor, extracts temperature change information, and assists in determining whether there is a living body in the region. The implementation process is as follows:

[0037] The thermal imaging sensor continuously collects temperature data for each marked privacy region and generates a temperature distribution map for the region. Thermal imaging technology detects the infrared radiation of an object's surface and calculates the temperature value, which can effectively distinguish between ambient temperature and living body temperature (such as human body temperature of 36-37°C).

[0038] To identify living bodies, the system focuses on the change in temperature over time. Unlike everyday objects, living bodies have small fluctuations in surface temperature due to physiological activities such as metabolism. The system calculates the temperature change rate ΔT / Δt per unit time, where ΔT represents the temperature difference in a certain period of time, and Δt represents the time interval. For example, if the temperature of a certain region rises from 36.5°C to 36.6°C in 1 second, it is likely that a person has entered the region, ΔT = 0.1°C, Δt = 1 second, and the change rate is 0.1°C / second.

[0039] The selection of Δt needs to balance sensitivity and stability. A too short Δt (such as 0.1 seconds) may cause noise interference, and a too long Δt (such as 10 seconds) may miss rapid changes. A typical value can be set between 1 second and 5 seconds, depending on the sampling rate of the sensor and the application scenario.

[0040] The data fusion module integrates the output results of the privacy region detection module and the living body recognition module to determine whether there is a living body in the privacy region. The core is to calculate the living body detection score and perform threshold comparison.

[0041] The system calculates the living body detection score S using the following formula:

[0042]

[0043] where ΔT ref represents the reference temperature difference, Δt ref represents the reference time interval, w r and w h are preset weight coefficients. ΔT ref / Δt ref is mainly used to eliminate dimensions, which can be selected according to specific conditions, such as 0.05-0.1℃ / s.

[0044] The design of the formula is based on the following logic: radar data is good at detecting physical changes in space, but cannot distinguish between living and non-living bodies; thermal imaging data can identify living body characteristics through temperature changes. Through weighted fusion, the system integrates the advantages of both to improve detection accuracy. The weight w r and w h usually ranges from 0 to 1, and (w r +w h =1). For example, if the thermal imaging data is more reliable, we can set (w r =0.4) and (w h =0.6). These values need to be optimized through experimental data to adapt to different environments and sensor performance.

[0045] The calculated score S is compared with the preset threshold T. If S>T, it is judged that there is a living body in the privacy area. The setting of the threshold T affects the sensitivity and false positive rate of the system. A lower T (such as 0.5) will improve the detection sensitivity, but may misjudge non-living bodies as living bodies; a higher T (such as 0.8) will reduce false positives, but may miss detection. The value of T needs to be determined through a large number of tests, for example, using ROC curve analysis to determine the optimal threshold.

[0046] As shown in Figure 4 , the shielding control module is responsible for virtually shielding and warning the privacy area confirmed to have a living body on the AR display terminal.

[0047] In the field of view of the AR display terminal, the system superimposes a semi-transparent mosaic on the outline position of the privacy area, blurring the image of the area to prevent privacy leakage. At the same time, a red warning box is drawn on the edge of the outline to remind the user that the area has been shielded. The transparency of the mosaic can be adjusted, for example, set to 50%, to balance privacy protection and scene recognizability. The red warning box is designed with high contrast to ensure eye-catching effect.

[0048] The shielding control module is deployed in an edge computing unit, which completes data fusion and image processing tasks locally. Edge computing avoids the delay of data transmission to the cloud, and is particularly suitable for scenarios with high real-time requirements. For example, in a smart home, an edge computing unit can be installed on a central control device in the living room, processing all sensor data and directly driving an AR display terminal.

[0049] The permission management module provides a flexible privacy control mechanism, allowing authorized users to temporarily adjust the shielding state of specific areas.

[0050] Users can apply for a one-time dynamic code through a preset method (such as a mobile app). The permission management module receives the dynamic code and performs authentication. If authentication is successful, the shielding state of the specified area is temporarily lifted or restored. For example, the dynamic code can be designed as a 6-digit number or letter combination, with a validity period of 5 minutes, and automatically expires after expiration.

[0051] To ensure security, detailed information such as the time, area, and user information of each operation is recorded in the log and uploaded to the background system. This facilitates administrators to track operation records and prevent unauthorized access. For example, the log may record "2023-10-01 14:30, User A lifts the shielding of the bedroom area, duration 5 minutes."

[0052] The AR display terminal is the user's interaction interface with the system, supporting two forms.

[0053] Smart glasses integrate privacy area detection, liveness detection, and data fusion modules, enabling real-time detection and shielding of privacy areas in the user's field of view while wearing them. This portable design is suitable for dynamic scenarios, such as privacy protection in public places.

[0054] The smart home display screen, as a fixed device, communicates with the edge computing unit and is suitable for home environments. Users can view and manage the status of privacy areas in their homes through the display screen, such as adjusting the shielding settings of the bedroom on the living room screen.

[0055] To illustrate the working principle of the system, the following describes a hotel.

[0056] Suppose the system is installed in a hotel bathroom, and the millimeter wave radar covers the entire area. When the hotel is unoccupied, the system performs baseline scanning and records σ base When the staff wearing smart glasses enter the hotel room to clean, and the customer enters the bathroom, the radar detects a change in the reflection coefficient of the bathroom area, Δσ exceeds the threshold. After clustering by DBSCAN, the bathroom area is marked as a potential privacy area.

[0057] The thermal imaging sensor then monitors the area. The data fusion module calculates the score S, and if S > T, it confirms the presence of a living body.

[0058] In the smart glasses worn by the staff, the toilet area is overlaid with a semi-transparent mosaic, and a red warning box is displayed. If the user needs to view the area, a dynamic code can be applied. After inputting the valid code, the shielding is temporarily removed, and the operation is automatically restored after completion.

[0059] The above merely describes the preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can make equivalent replacements or changes within the technical range disclosed by the present application according to the technical solutions and the inventive concept of the present application, which should be covered within the protection scope of the present application.

Claims

1. A thermal imaging assisted millimeter wave radar privacy zone real-time shielding system, characterized in that, The system comprises: a privacy area detection module for scanning a space to be measured by a millimeter wave radar, detecting and marking potential privacy areas; a living body recognition module for collecting temperature maps in the privacy areas by a thermal imaging sensor and extracting temperature change information; a data fusion module for fusing the output results of the privacy area detection module and the living body recognition module; a shielding control module for superimposing a virtual shielding layer on the privacy areas on an AR display terminal and triggering an alarm.

2. The system of claim 1, wherein, The privacy area detection module performs the following steps: first, baseline scanning in an empty environment to obtain baseline reflection coefficient σ base ; the millimeter wave radar continuously collects reflection coefficient σ (t) at time t, and calculates Δσ = |σ (t) - σ base |; for the return point cloud whose Δσ exceeds the preset threshold, DBSCAN clustering is performed to generate a voxel cluster, and the space area corresponding to the voxel cluster is marked as a privacy area.

3. The system of claim 2, wherein, The living body recognition module comprises a thermal imaging sensor for continuously collecting temperature data of the marked privacy areas and calculating a temperature change rate per unit time according to ΔT / Δt, wherein ΔT represents a temperature difference and Δt represents a time interval.

4. The system of claim 3, wherein, The data fusion module calculates a living body detection score S according to the following formula: where ΔT ref represents a reference temperature difference, Δt ref represents a reference time interval, w r and w h are preset weight coefficients.

5. The system of claim 4, wherein, The data fusion module compares the score S with a preset threshold T, and if S>T, it is determined that there is a living body in the privacy area.

6. The system of claim 1, wherein, The shielding control module superimposes a semi-transparent mosaic on the confirmed privacy area contour position in the field of view of the AR display terminal and draws a red warning frame on the contour edge.

7. The system of claim 1, wherein, The shielding control module is deployed in an edge computing unit, and the edge computing unit completes data fusion and image processing locally.

8. The system of claim 1, wherein, The system further comprises a permission management module for receiving a one-time dynamic code of a user and temporarily releasing or restoring the shielding state of a specified area according to the dynamic code.

9. The system of claim 8, wherein, After receiving the one-time dynamic code, the permission management module only releases the shielding of the specified private space that has passed authentication, and uploads logs such as the time, area and user information of the operation to a background system.

10. The system of claim 1, wherein, The AR display terminal is intelligent glasses integrated with the privacy area detection module, the living body recognition module and the data fusion module, or is a smart home display screen in communication with the edge computing unit.

Images