Night penetrating type scene reconstruction system based on millimeter wave radar and thermal imaging fusion

Abstract

The invention discloses a night penetration type scene reconstruction system based on millimeter wave radar and thermal imaging fusion, and relates to the technical field of multi-source data processing. A space point cloud collection module is arranged to scan indoor space through millimeter wave radar to generate depth point cloud data in real time; a temperature distribution collection module captures human body temperature distribution characteristics through a thermal imaging sensor, a data fusion module is set to fuse the depth point cloud data and the human body temperature distribution characteristics into three-dimensional scene data, and a risk point prediction module is set to input the fused three-dimensional scene data into a lightweight convolutional network. Risk points of the shielded area are predicted, an action detail reduction module is arranged, limb action details of the user are reduced through human body temperature distribution characteristics, and a posture early warning module triggers an early warning signal according to a risk point prediction result and the user action details and does not perform processing; the association analysis of the environmental risk and the user behavior is realized, and a more comprehensive security guarantee is provided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-source data processing, in particular to a night penetrating scene reconstruction system based on fusion of millimeter wave radar and thermal imaging. BACKGROUND

[0002] Night environment poses a serious challenge to the safe movement of the elderly and the disabled. With age, the vision, balance and reaction speed of the elderly generally decline, especially in low light environments. According to statistics, the incidence of night falls for people over 65 years old is more than three times that of daytime, and more than 40% of these falls result in serious injury or even life-threatening. For the visually impaired, the difficulty of night activities is multiplied, and they cannot perceive changes in the surrounding environment through vision, and their ability to identify potential dangers is extremely limited.

[0003] The risk factors in the night environment are diverse, including invisible obstacles, ground height changes, moving objects, and complex spatial structures. These risk factors are difficult to be discovered in time under insufficient light conditions, even for people with normal vision. For the elderly and the disabled, due to the decline in perception and reaction ability, these risk factors are more likely to cause falls, collisions and other safety accidents. Especially in unfamiliar environments, the lack of familiarity with the spatial layout further increases the risk of night activities.

[0004] Traditional visual aids such as flashlights, night vision devices, etc. rely on residual light or infrared reflection in the environment, and their effectiveness is limited in complete darkness or obstructed environments. Single-sensor-based assistance systems, such as pure vision systems or pure ultrasonic systems, have significantly reduced reliability in night environments. Vision systems are limited by light conditions and cannot penetrate obstacles; ultrasonic systems, although not affected by light, have low spatial resolution and are difficult to identify detailed features in complex environments.

[0005] Therefore, the present application proposes a night penetrating scene reconstruction system based on fusion of millimeter wave radar and thermal imaging. SUMMARY

[0006] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the present application proposes a night penetrating scene reconstruction system based on fusion of millimeter wave radar and thermal imaging, which realizes the correlation analysis of environmental risks and user behavior and provides more comprehensive safety protection.

[0007] To achieve the above-mentioned purpose, a night penetrating scene reconstruction system based on fusion of millimeter wave radar and thermal imaging is proposed, which includes a spatial point cloud collection module, a temperature distribution collection module, a data fusion module, a risk point prediction module, an action detail restoration module, and a navigation warning module; wherein each module is connected through electrical means.

[0008] a spatial point cloud collection module, which scans the indoor space through a millimeter wave radar to generate depth point cloud data in real time, the depth point cloud data being millimeter wave radar point cloud data, containing spatial coordinates and velocity and other motion characteristics of obstacles, and sending the depth point cloud data to a data fusion module;

[0009] a temperature distribution collection module, which captures human temperature distribution characteristics through a thermal imaging sensor; the human temperature distribution characteristics are output in the form of a thermal imaging temperature matrix, containing temperature distribution; and the human temperature distribution characteristics are sent to the data fusion module and a motion detail restoration module;

[0010] the data fusion module, which performs spatio-temporal alignment on the depth point cloud data and the human temperature distribution characteristics through an extended Kalman filtering algorithm to generate fused three-dimensional scene data; and sending the three-dimensional scene data to a risk point prediction module;

[0011] the risk point prediction module, which inputs the fused three-dimensional scene data into a lightweight convolutional network to predict risk points of an occluded area, and sends the risk points of the occluded area to a navigation warning module;

[0012] the motion detail restoration module, which restores user limb motion details through the human temperature distribution characteristics, and sends the user limb motion details to the navigation warning module;

[0013] the body posture warning module, which judges whether the user behavior is an abnormal body posture according to the risk point prediction result and the user motion details, and triggers a warning signal if the judgment is yes, otherwise, no processing is performed;

[0014] the real-time generation of the depth point cloud data includes the following steps:

[0015] Step 11: the millimeter wave radar collects three-dimensional point cloud data of the indoor space in real time by emitting millimeter wave signals of a predetermined frequency and receiving reflected signals;

[0016] Step 12: the millimeter wave radar outputs the collected three-dimensional point cloud data in a structured format, containing timestamp, spatial coordinates, reflection intensity, radial velocity and confidence information;

[0017] the capturing of the human temperature distribution characteristics through the thermal imaging sensor includes the following steps:

[0018] Step 21: the thermal imaging sensor acquires scene temperature distribution data in real time by an infrared detector array collecting thermal radiation information in the environment;

[0019] Step 22: pre-processing the collected temperature distribution data, including noise suppression, temperature calibration and spatial filtering, to improve the accuracy and spatial resolution of temperature measurement;

[0020] Step 23: Human target detection and segmentation on pre-processed temperature distribution data, extracting human region in foreground, generating human temperature distribution data;

[0021] Step 24: Organizing the human temperature distribution data into structured thermal imaging temperature matrix, including timestamp, temperature value distribution, human region mask and key point temperature information;

[0022] The depth point cloud data and the human temperature distribution feature are spatio-temporally aligned by an extended Kalman filtering algorithm to generate fused three-dimensional scene data, including the following steps:

[0023] Step 31: Time synchronization and spatial registration preprocessing of depth point cloud data of millimeter wave radar and temperature distribution features of thermal imaging sensor;

[0024] Step 32: Based on the depth point cloud data and the temperature distribution feature after time synchronization and spatial registration preprocessing, construct an extended Kalman filter state space model, define state vector, observation vector, state transition matrix and observation matrix;

[0025] Step 33: Perform prediction and update steps of extended Kalman filtering algorithm to realize probability fusion of depth point cloud data and human temperature distribution feature;

[0026] Step 34: Construct a fused three-dimensional scene model based on the fusion result after probability fusion, including spatial geometric structure, temperature distribution and motion feature;

[0027] The fused three-dimensional scene data is input into a lightweight convolutional network to predict risk points in the occluded area, including the following steps:

[0028] Step 41: Preprocess the fused three-dimensional scene data to convert it into an input format suitable for lightweight convolutional network processing;

[0029] Step 42: Construct a lightweight three-dimensional convolutional neural network model to extract features and predict risk points from pre-processed three-dimensional scene data;

[0030] The lightweight three-dimensional convolutional neural network adopts an encoder-decoder architecture, including three main modules: feature extraction, context understanding and risk point prediction.

[0031] In the feature extraction module, the network contains 4 consecutive three-dimensional convolution blocks. Each convolution block is composed of a 3x3x3 three-dimensional convolution layer, a batch normalization layer, a ReLU activation function and a 3x3x3 depth separable convolution. The depth separable convolution decomposes the standard convolution into two steps of depth convolution and pointwise convolution, significantly reducing the parameter quantity and computational complexity. The output channel number of the first convolution block is 32, and the channel number of each subsequent block is doubled, forming a feature pyramid structure.

[0032] The context understanding module adopts a design of a hollow convolution, and captures multi-scale context information through parallel convolution branches with different expansion rates (1, 2, 4, 8). Each branch contains a 3x3x3 hollow convolution, and the output channel number is 64. The outputs of the branches are weighted and fused through a channel attention mechanism to generate a feature map containing rich context information.

[0033] The risk point prediction module uses a transpose convolution layer to gradually restore the spatial resolution of the feature map. It contains 3 consecutive upsampling blocks, each composed of a 2x2x2 three-dimensional transpose convolution, a batch normalization layer and a ReLU activation function. The last layer is a 1x1x1 convolution layer, and the output channel number is 1, representing the probability value of each voxel containing a risk point.

[0034] Step 43: Post-processing the output of the lightweight convolutional network to identify and locate the risk points in the occluded area;

[0035] The method for restoring the details of the user's limb action through the human body temperature distribution feature comprises the following steps:

[0036] Step 51: Key point detection and pose estimation on the thermal imaging temperature matrix to identify human skeletal joints and construct a human skeleton model;

[0037] The skeletal joint detection adopts a multi-stage detection algorithm based on heat map regression. First, input the thermal imaging temperature matrix into a feature extraction network, which is composed of 5 convolution blocks, each containing two 3x3 convolution layers, a batch normalization layer and a ReLU activation function. The feature extraction network outputs multi-scale feature maps, capturing temperature distribution patterns at different scales.

[0038] On the basis of the multi-scale feature map, a heat map regression network is used to predict the position heat map of 17 human key points, including the head, neck, shoulder, elbow, wrist, hip, knee and ankle joints. The heat map regression network adopts a full convolution structure, and outputs a probability heat map for each key point, with the heat map peak corresponding to the possible position of the key point. Key point positioning is achieved by finding the local maximum value in the heat map, while calculating the detection confidence.

[0039] Based on the detected key points, a human skeleton model is constructed. The skeleton model adopts a tree structure, with the torso as the root node and the limbs as branches. Each joint node stores a three-dimensional spatial coordinate and a detection confidence. The connection relationship between the joint nodes is predefined according to human anatomy knowledge, forming a complete skeleton topology.

[0040] Step 52: Based on the constructed human skeleton model, combined with the human temperature distribution characteristics, the joint angles and limb postures are calculated through the inverse kinematics algorithm to generate the user limb action parameters;

[0041] Step 53: Time series analysis is performed on the limb action parameters under continuous time sequence to identify the user action mode and behavior intention, and generate the user limb action details;

[0042] The step of judging whether the user behavior is an abnormal body state according to the risk point prediction result and the user action details comprises the following steps:

[0043] Step 61: A user-risk point interaction model centered on the user skeleton is constructed to obtain the interaction state of each part of the user with the risk point;

[0044] Step 62: Based on the user-risk point interaction model and the interaction state, it is judged whether the user behavior is an abnormal body state;

[0045] The abnormal judgment is performed by using a multi-level abnormal body state detection algorithm; the multi-level abnormal body state detection algorithm contains three levels of abnormal detection mechanisms: rule-based static posture detection, statistics-based dynamic behavior analysis and learning-based comprehensive abnormal recognition.

[0046] Step 63: When it is judged that the user behavior is an abnormal body state, a warning signal is generated, otherwise no processing is performed.

[0047] Compared with the prior art, the beneficial effects of the present application are:

[0048] The spatial point cloud collection module takes advantage of the penetration of millimeter wave radar to obtain accurate depth point cloud data even in completely dark or obstructed environments. Millimeter wave radar works in a specific frequency band and is not limited by light conditions, and can penetrate obstacles such as lightweight walls and curtains to detect objects in the obstructed area. This module uses linear frequency continuous wave technology to collect real-time three-dimensional spatial coordinates and motion characteristics such as speed of obstacles in the environment, providing basic data for subsequent processing. The temperature distribution collection module captures human temperature distribution characteristics through thermal imaging sensors. The human body continuously releases heat, which appears as a clear temperature profile in thermal imaging sensors, and can be clearly identified even in dark environments. This module organizes the collected temperature data into a thermal imaging temperature matrix, including timestamps, temperature value distribution, human region mask, and key point temperature information, providing a basis for human posture analysis. The data fusion module is the core of the system, which uses an extended Kalman filter algorithm to realize the spatio-temporal alignment of millimeter wave radar point cloud data and thermal imaging temperature distribution. This algorithm constructs a state space model to define a state vector containing position, velocity, acceleration, and attitude angle, and realizes the probabilistic fusion of multi-source data. The fusion process takes into account the measurement error of the sensor and environmental noise to generate three-dimensional scene data with high reliability. The risk point prediction module inputs the fused three-dimensional scene data into a lightweight convolutional network to predict potential risk points in the obstructed area. This module can identify dangerous factors in the environment, such as sudden changes in ground height, narrow passages, unstable objects, etc., and evaluate their risk levels. The prediction results are output in the form of risk point coordinates and risk levels, providing decision-making basis for navigation warning. The action detail restoration module restores user limb action details by analyzing human temperature distribution characteristics. This module can identify key information such as walking posture, balance state, gait characteristics, etc., to determine whether the user's current motion state is abnormal. For example, if the system detects that the user's gait is unstable, the body inclination angle is too large, or the motion trajectory is abnormal, it will determine that it is a potential risk state. The body posture warning module analyzes the risk point prediction results and user action details to determine whether the user's behavior is in an abnormal body posture. When the system detects that the user is approaching a risk point in the environment and the user's posture is also unstable, it will trigger a warning signal to remind the user to adjust their behavior or provide auxiliary navigation suggestions, thereby avoiding potential safety accidents. It realizes penetration sensing in dark environments, can detect obstacles in obstructed areas; it realizes accurate analysis of human actions, can identify abnormal postures and potential risks; it realizes the correlation analysis of environmental risks and user behaviors, and provides more comprehensive safety protection. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 The module connection relationship diagram of the night penetration scene reconstruction system based on millimeter wave radar and thermal imaging fusion in embodiment 1 of the present application; DETAILED DESCRIPTION

[0050] The technical solutions of the present application will be described clearly and completely in combination with the embodiments below. Obviously, the described embodiments are only some of the embodiments of the present application, but not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0051] As shown in the embodiment of the present application, the night penetrating scene reconstruction system based on fusion of millimeter wave radar and thermal imaging includes a spatial point cloud collection module, a temperature distribution collection module, a data fusion module, a risk point prediction module, an action detail restoration module, and a navigation warning module. Each module is connected by electrical means. Figure 1 The spatial point cloud collection module scans the indoor space through a millimeter wave radar to generate depth point cloud data in real time. The depth point cloud data is millimeter wave radar point cloud data, which contains the spatial coordinates and velocity of obstacles and other motion characteristics. The depth point cloud data is sent to the data fusion module.

[0052] The temperature distribution collection module captures human temperature distribution characteristics through a thermal imaging sensor. The human temperature distribution characteristics are output in the form of a thermal imaging temperature matrix, which contains temperature distribution. The human temperature distribution characteristics are sent to the data fusion module and the action detail restoration module.

[0053] The data fusion module performs spatio-temporal alignment on the depth point cloud data and the human temperature distribution characteristics through an extended Kalman filter algorithm to generate fused three-dimensional scene data. The three-dimensional scene data is sent to the risk point prediction module.

[0054] The risk point prediction module inputs the fused three-dimensional scene data into a lightweight convolutional network to predict the risk points of the occluded area and sends the risk points of the occluded area to the navigation warning module.

[0055] The action detail restoration module restores user limb action details through human temperature distribution characteristics and sends the user limb action details to the navigation warning module.

[0056] The body posture warning module determines whether the user behavior is an abnormal body posture according to the risk point prediction result and the user action details. If it is determined to be yes, a warning signal is triggered, otherwise, no processing is performed.

[0057] Specifically, the real-time generation of depth point cloud data includes the following steps:

[0058] Step 11: The millimeter wave radar collects three-dimensional point cloud data of the indoor space in real time by emitting millimeter wave signals of a predetermined frequency and receiving reflected signals.

[0059] Step 12: The millimeter wave radar generates depth point cloud data in real time based on the collected three-dimensional point cloud data.

[0060] Specifically, the transmitting port of the millimeter wave radar generates continuous millimeter wave signals in a preset sweep frequency mode, the millimeter wave signals are reflected after encountering indoor objects, and are captured by the receiving port of the millimeter wave radar.

[0061] The receiving port mixes the reflected signals with the transmitted signals to generate intermediate frequency signals. By performing spectrum analysis on the intermediate frequency signals, the distance, azimuth angle and elevation angle information of the object are extracted. Specifically, the spectrum analysis converts the time domain signals into frequency domain signals through three-dimensional fast Fourier transform, and the peak values in the frequency domain correspond to the spatial position coordinates of the object. The signal processing unit of the millimeter wave radar outputs the extracted spatial information in the form of data packets to form three-dimensional point cloud data.

[0062] It should be noted that in the specific implementation of the present application, the sweep frequency mode of the millimeter wave radar can adopt a linear frequency modulation continuous wave technology, and the sweep frequency period and bandwidth are dynamically adjusted according to the indoor environment. The sweep frequency period is set to 10 ms to ensure that 100 frames of point cloud data can be obtained per second, meeting the real-time requirement; the bandwidth is set to 4 GHz to achieve a distance resolution of centimeters. The receiving port of the millimeter wave radar adopts a multi-channel MIMO (Multiple Input Multiple Output) array design, including 16 transmitting antennas and 16 receiving antennas, forming 256 virtual channels, each channel independently processing reflected signals in different directions to realize high-precision three-dimensional scanning of indoor space. The signal processing unit of the millimeter wave radar performs phase calibration and spatial synthesis on the multi-channel signals to eliminate multipath effects and environmental noise, and improve the spatial resolution and accuracy of the point cloud data.

[0063] Step 12: The millimeter wave radar outputs the collected three-dimensional point cloud data in a structured format, including timestamp, spatial coordinates, reflection intensity, radial velocity and confidence information;

[0064] In the embodiment of the present application, the depth point cloud data is output in a structured format, including timestamp, three-dimensional spatial coordinates (x, y, z), reflection intensity, radial velocity and point cloud confidence. Among them, the three-dimensional spatial coordinates are represented by a rectangular coordinate system with the millimeter wave radar as the origin; the reflection intensity reflects the reflection characteristics of the object surface; the radial velocity is measured by the Doppler effect and is used to identify moving objects; the point cloud confidence represents the reliability of the measurement result, which is calculated based on the signal-to-noise ratio.

[0065] Step 13: Preprocess the three-dimensional point cloud data collected by the millimeter wave radar, including noise filtering, clustering segmentation and feature extraction, and generate depth point cloud data after feature extraction;

[0066] Specifically, in the noise filtering stage of three-dimensional point cloud data, an adaptive threshold filtering algorithm is adopted, the filtering threshold is dynamically adjusted according to the environmental noise level, and the noise points with low signal-to-noise ratio are filtered out. The filtering algorithm combines the statistical outlier detection method to identify and remove the point cloud data with abnormal spatial distribution. The point cloud data after noise filtering has more uniform density and clearer spatial structure.

[0067] In the clustering segmentation stage of three-dimensional point cloud data, a density-based spatial clustering algorithm is adopted to segment the point cloud data into multiple independent object clusters. The clustering parameters are automatically adjusted according to the characteristics of indoor environment, including neighborhood radius and minimum point threshold. For each object cluster, its spatial bounding box, centroid position and volume information are calculated to form the spatial representation at the object level.

[0068] Finally, in the feature extraction stage of three-dimensional point cloud data, geometric features and motion features are extracted for each object cluster. The geometric features include principal direction vector, surface normal vector distribution and shape descriptor; the motion features include centroid velocity, angular velocity and motion trajectory. It can be understood that the object is an obstacle in space, and after the feature extraction stage, the spatial position and motion features such as velocity of each obstacle can be obtained.

[0069] Further, the capturing of human temperature distribution features by the thermal imaging sensor includes the following steps:

[0070] Step 21: The thermal imaging sensor acquires thermal radiation information in the environment through an infrared detector array, and obtains scene temperature distribution data in real time;

[0071] Specifically, in the preferred embodiment of the present application, the thermal imaging sensor uses a non-cooled microbolometer array as the core detection element, with an array resolution of 640x480 pixels and a thermal sensitivity better than 0.05℃. Each microbolometer unit can detect infrared radiation in the wavelength range of 8-14μm, which corresponds to the peak region of human body emitted thermal radiation.

[0072] During data acquisition, the thermal imaging sensor continuously acquires thermal radiation information in the scene at a frame rate of 30Hz. Each pixel records the radiation intensity of the corresponding spatial position, and the radiation intensity is converted to absolute temperature value through the thermal radiation formula, thereby generating temperature distribution data of the absolute temperature value of each pixel;

[0073] Step 22: Preprocess the collected temperature distribution data, including noise suppression, temperature calibration and spatial filtering, to improve the accuracy and spatial resolution of temperature measurement;

[0074] Specifically, in the noise suppression stage of temperature distribution data, a method combining time domain average filtering and spatial median filtering is adopted. Time domain average filtering performs weighted average on consecutive multiple thermal images to suppress random thermal noise; spatial median filtering effectively removes salt and pepper noise in thermal images while preserving temperature boundary information. The filtering parameters are adaptively adjusted according to the current environmental noise level, minimizing the loss of details while ensuring the filtering effect.

[0075] In the temperature calibration stage of temperature distribution data, a multi-point calibration strategy is adopted, using the built-in reference temperature source as the standard to establish the mapping relationship between radiation intensity and absolute temperature. The calibration process takes into account factors such as environmental temperature changes, lens attenuation, and atmospheric absorption, and achieves accurate calibration in the full temperature range through piecewise linear interpolation method. The temperature calibration process is automatically executed every 30 minutes to ensure the stability of temperature measurement over a long period of time.

[0076] In the spatial filtering stage of temperature distribution data, an adaptive Gaussian filtering algorithm is adopted to smooth the thermal images and improve the continuity of the temperature field. The size of the filter kernel is dynamically adjusted according to the target distance, with small size filter kernel used for close-range targets to preserve details and large size filter kernel used for long-range targets to enhance signal-to-noise ratio. At the same time, anisotropic diffusion filtering is applied in the edge region to maintain the sharpness of the temperature boundary and avoid loss of details caused by excessive smoothing.

[0077] Step 23: Perform human target detection and segmentation on the preprocessed temperature distribution data to extract the human region in the foreground and generate human temperature distribution data;

[0078] Specifically, the human target detection adopts a hybrid detection strategy based on temperature threshold and morphological features. First, taking advantage of the fact that the surface temperature of the human body is usually higher than the ambient temperature, an adaptive temperature threshold method is used, such as fluctuating between 35 degrees Celsius and 37 degrees Celsius, to preliminarily segment the possible human region.

[0079] Based on the preliminary segmentation, morphological operations including opening and closing operations are applied to optimize the segmentation results, fill internal cavities, and smooth the boundary contours. Subsequently, connected component analysis is used to identify independent human targets and calculate their spatial features, including area, perimeter, aspect ratio, and circularity. Based on these feature parameters, combined with prior knowledge of human morphology, non-human targets such as heat source equipment and pets are filtered out.

[0080] The distribution of absolute temperature values of each pixel point in the recognized human region can be used as human temperature distribution data;

[0081] Step 24: Organize the human body temperature distribution data into a structured thermal imaging temperature matrix, including timestamp, temperature value distribution, human body region mask, and key point temperature information;

[0082] Specifically, the thermal imaging temperature matrix adopts a multi-layer structure design, including a basic temperature layer, a segmentation mask layer, a key point layer, and a metadata layer. The basic temperature layer stores the original temperature values in the form of a floating-point matrix, with a resolution consistent with the thermal imaging sensor; the segmentation mask layer marks the human body region in the form of a binary matrix; the key point layer records the position coordinates and temperature values of specific parts of the human body, such as the head, torso, limbs, etc.; and the metadata layer contains auxiliary information such as timestamp, environmental reference temperature, and sensor parameters.

[0083] In further preferred embodiments of the present application, statistical analysis of human body temperature distribution can also be performed to calculate statistical characteristics such as mean, standard deviation, maximum, minimum, and quantile of temperature. This statistical characteristic reflects the overall characteristics of the human body temperature distribution. At the same time, temperature gradient information can be extracted to calculate the direction and amplitude of temperature change, forming a temperature gradient field, which is used to identify the temperature boundary and hot spot area on the surface of the human body.

[0084] Further, the human body temperature distribution data can also be standardized into a standard format. The temperature values are mapped to the integer range of 0-255 through linear transformation, facilitating storage and transmission; the spatial coordinates are mapped to the standard coordinate system through affine transformation, eliminating the effects of viewing angle and distance changes; and the timestamp uses a unified time reference, facilitating synchronization with other sensor data.

[0085] The final generated thermal imaging temperature matrix is output in the form of a structured data packet, containing the following key information: timestamp, accurate to the millisecond level; temperature distribution matrix, recording the temperature values of each pixel point in the human body region; human body segmentation mask, identifying the human body region in the image; human body key point coordinates and temperature, including the head, torso, limbs, etc. main parts; temperature statistical characteristics, describing the overall characteristics of the temperature distribution; and temperature gradient field, indicating the direction and amplitude of temperature change.

[0086] Further, the depth point cloud data and the human body temperature distribution features are spatio-temporally aligned through an extended Kalman filter algorithm to generate fused three-dimensional scene data, including the following steps:

[0087] Step 31: Time synchronization and spatial registration preprocessing of millimeter wave radar depth point cloud data and thermal imaging sensor temperature distribution features;

[0088] Specifically, in the time synchronization stage, a high-precision timestamp alignment strategy is adopted to establish a unified time reference for the millimeter wave radar and the thermal imaging sensor. The millimeter wave radar is 100 Hz, and the thermal imaging sensor is 30 Hz. The closest data frames are paired through a time window matching algorithm.

[0089] In the spatial registration preprocessing stage, a rigid transformation relationship between the millimeter wave radar coordinate system and the thermal imaging sensor coordinate system is first established. Through the rotation matrix R and the translation vector T obtained by prior calibration, the thermal imaging temperature matrix is mapped to the three-dimensional space.

[0090] Step 32: Based on the depth point cloud data and temperature distribution features after time synchronization and spatial registration preprocessing, an extended Kalman filter state space model is constructed, and a state vector, an observation vector, a state transition matrix, and an observation matrix are defined;

[0091] Specifically, the constructed extended Kalman filter adopts a 15-dimensional state vector, which includes the three-dimensional position coordinates (x, y, z) of the obstacle and the human body, the three-dimensional velocity (vx, vy, vz), the three-dimensional acceleration (ax, ay, az), the three-dimensional attitude angle (roll, pitch, yaw), and the three-dimensional angular velocity (ωx, ωy, ωz). The state vector completely represents the kinematic characteristics and attitude information of the target in the scene. Among them, roll, pitch, and yaw represent the roll angle, pitch angle, and yaw angle, respectively;

[0092] The state vector is defined as:

[0093] X=

[0094] [x, y, z, vx, vy, vz, ax, ay, az, roll, pitch, yaw, ωx, ωy, ωz] T The state transition model adopts a constant acceleration kinematic model, and the state transition equation is:

[0095] x(k+1)=x(k)+vx(k)·Δt+0.5·ax(k)·Δt 2 ;

[0096] y(k+1)=y(k)+vy(k)·Δt+0.5·ay(k)·Δt 2 ;

[0097] z(k+1)=z(k)+vz(k)·Δt+0.5·az(k)·Δt 2 ;

[0098] vx(k+1)=vx(k)+ax(k)·Δt;

[0099] vy(k+1) = vy(k) + ay(k) At;

[0100] vz(k+1) = vz(k) + az(k) At;

[0101] ax(k+1) = ax(k);

[0102] ay(k+1) = ay(k);

[0103] az(k+1) = az(k);

[0104] roll(k+1) = roll(k) + ox(k) At;

[0105] pitch(k+1) = pitch(k) + oy(k) At;

[0106] yaw(k+1) = yaw(k) + oz(k) At;

[0107] ox(k+1) = ox(k);

[0108] oy(k+1) = oy(k);

[0109] oz(k+1) = oz(k);

[0110] wherein At is the sampling time interval, k is each unit time, and the interval between every two unit times is At.

[0111] The observation vector is composed of the measurement data of the millimeter wave radar and the thermal imaging sensor:

[0112] Z = [xr, yr, zr, vr, u, v, T] T ;

[0113] wherein (xr, yr, zr) is the three-dimensional position measured by the millimeter wave radar, vr is the radial velocity, (u, v) is the thermal imaging image coordinate, and T is the temperature value.

[0114] Step 33: Perform the prediction and update steps of the extended Kalman filtering algorithm to realize the probability fusion of the depth point cloud data and the human body temperature distribution characteristics;

[0115] Specifically, the detailed implementation steps of the extended Kalman filtering algorithm include:

[0116] Step 321: Set the initial state vector X(0|0) to the initial observation value or a preset value

[0117] Set the initial state covariance matrix P(0|0) to a diagonal matrix, and the diagonal elements reflect the uncertainty of the initial state

[0118] Step 322: State prediction: X(k|k-1) = f(X(k-1|k-1)), where f is a nonlinear state transition function calculated according to the state transition equation above; calculate the state transition Jacobian matrix F_k:

[0119] For example, the partial derivative of position with respect to velocity is Δt, the partial derivative of velocity with respect to acceleration is Δt, and the rest are calculated according to the corresponding partial derivatives of the state transition equation

[0120] Predicted state covariance: P(k|k-1) = F_k·P(k-1|k-1)·F_k^T+Q_k; where Q_k is the process noise covariance matrix;

[0121] Step 323: Calculate observation prediction: Z(k|k-1) = h(X(k|k-1)), where h(·) is a nonlinear observation function that maps the state vector to the observation space; for millimeter wave radar observation: xr = x, yr = y, zr = z, vr = (x·vx + y·vy + z·vz) / sqrt(x 2 +y 2 +z 2 );

[0122] For thermal imaging observation: u = fx·x / z + cx, v = fy·y / z + cy, where fx, fy are focal lengths, and cx, cy are principal point coordinates;

[0123] Calculate the observation Jacobian matrix H_k:

[0124] For example, for millimeter wave radar position observation, the partial derivative of position with respect to position is 1; for thermal imaging observation, the partial derivative of u with respect to x is fx / z, and the partial derivative of u with respect to z is -fx·x / z 2 Calculate the observation residual: y_k = Z_k - Z(k|k-1), where Z_k is the actual observation value;

[0125] Calculate the residual covariance: S_k = H_k·P(k|k-1)·H_k^T+R_k; where R_k is the observation noise covariance matrix;

[0126] Calculate the Kalman gain: K_k = P(k|k-1)·H_k^T·S_k^(-1)

[0127] Update the state estimate: X(k|k) = X(k|k-1) + K_k·y_k

[0128] Update the state covariance: P(k|k) = (I - K_k·H_k)·P(k|k-1)

[0129] Step 314. Calculate Mahalanobis distance of observation residual: d 2 = y_k^T · S_k^(-1) · y_k;

[0130] If d 2 exceeds the preset distance threshold, which is usually taken as the 95% or 99% quantile of χ 2 distribution, the observation is considered abnormal and is not updated, otherwise steps 311 to 314 are repeated;

[0131] Step 34: Construct a fused three-dimensional scene model based on the fusion result after probability fusion, including spatial geometric structure, temperature distribution and motion characteristics.

[0132] Specifically, the three-dimensional scene model adopts a multi-layer structure design, including a geometric layer, a temperature layer and a motion layer.

[0133] In the geometric layer, the spatial structure of the scene is constructed using millimeter wave radar point cloud data. The point cloud data is processed by voxel grid to convert it into a regular three-dimensional grid representation.

[0134] In the temperature layer, the thermal imaging temperature matrix is mapped to the three-dimensional space to give the scene geometric structure temperature attributes. The mapping process uses the coordinate transformation relationship provided by the filter to project the two-dimensional temperature distribution onto the three-dimensional point cloud.

[0135] In the motion layer, the motion trajectory and velocity field of the obstacles in the scene are recorded. Based on the estimated velocity and acceleration information using the extended Kalman filter algorithm, the short-term motion trend of the obstacles is predicted and visualized in the form of a vector field.

[0136] Further, the step of inputting the fused three-dimensional scene data into a lightweight convolutional network to predict the risk points of the occluded area includes the following steps:

[0137] Step 41: Preprocess the fused three-dimensional scene data to convert it into an input format suitable for lightweight convolutional network processing.

[0138] Specifically, the fused three-dimensional scene data includes spatial geometric structure, temperature distribution and motion characteristics. The preprocessing stage of the three-dimensional scene data first voxelizes the three-dimensional scene data to discretize the continuous space into a regular three-dimensional grid. The voxel size is set to 10cm x 10cm x 10cm to ensure spatial resolution while controlling computational complexity.

[0139] For each voxel, the following voxel features are extracted and organized: spatial position coordinates (x, y, z), reflectance intensity value, temperature value, and motion velocity vector (vx, vy, vz). These voxel features are organized as a multi-channel three-dimensional tensor, forming the input data of the network. Specifically, the dimension of the input tensor is [B, C, D, H, W], where B is the batch size, C is the number of feature channels (including position, intensity, temperature, and velocity, a total of 6 channels), D, H, and W are the depth, height, and width dimensions, respectively, corresponding to the spatial dimensions after voxelization.

[0140] Step 42: Construct a lightweight three-dimensional convolutional neural network model to extract features and predict risk points from the preprocessed three-dimensional scene data;

[0141] Specifically, in the embodiments of the present application, the lightweight three-dimensional convolutional neural network adopts an encoder-decoder architecture, including three main modules: feature extraction, context understanding, and risk point prediction.

[0142] In the feature extraction module, the network contains 4 consecutive three-dimensional convolutional blocks. Each convolutional block is composed of a 3x3x3 three-dimensional convolutional layer, a batch normalization layer, a ReLU activation function, and a 3x3x3 depth separable convolution. The depth separable convolution decomposes the standard convolution into two steps: depth convolution and point-wise convolution, significantly reducing the number of parameters and computational complexity. The output channel number of the first convolutional block is 32, and the channel number of each subsequent block is doubled, forming a feature pyramid structure.

[0143] The context understanding module adopts a design of dilated convolution, capturing multi-scale context information through parallel convolution branches with different dilation rates (1, 2, 4, 8). Each branch contains a 3x3x3 dilated convolution, and the output channel number is 64. The outputs of each branch are weighted and fused through a channel attention mechanism to generate a feature map containing rich context information.

[0144] The risk point prediction module uses a transpose convolution layer to gradually restore the spatial resolution of the feature map. It contains 3 consecutive upsampling blocks, each composed of a 2x2x2 three-dimensional transpose convolution, a batch normalization layer, and a ReLU activation function. The last layer is a 1x1x1 convolution layer, with an output channel number of 1, representing the probability value of each voxel containing a risk point.

[0145] Step 43: Post-processing of the output of the lightweight convolutional network to identify and locate the risk points in the occluded area;

[0146] Specifically, the output of the lightweight convolutional network is a three-dimensional probability voxel map, and the value of each voxel represents the probability of the presence of a risk point at that location. In the post-processing stage, a threshold segmentation is first applied to mark voxels with a probability value greater than 0.7 as candidate risk points.

[0147] Subsequently, the candidate risk points are clustered using connected component analysis to identify a set of adjacent risk points in space. The centroid position, spatial range, and average probability value are calculated for each set of risk points as the feature description of the risk points.

[0148] Finally, based on the spatial position of the risk points and the scene structure, it is determined whether the risk points are located in the occluded area. The specific determination method is: from the observation point to the risk point centroid, detect whether the connecting line intersects with the surface of the obstacle in the scene. If there is an intersection, it is confirmed that the risk point is located in the occluded area.

[0149] The risk points in the occluded area are output in a structured data form, including the following information: the three-dimensional coordinates (x, y, z) of the risk points, the confidence value of the risk points, the identification of the occluded area to which the risk points belong, and the risk level evaluation of the risk points.

[0150] Further, the restoring user limb action details through the human body temperature distribution features comprises the following steps:

[0151] Step 51: key point detection and posture estimation are performed on the thermal imaging temperature matrix, human body skeleton joints are identified, and a human body skeleton model is constructed;

[0152] Specifically, the skeleton joint detection adopts a multi-stage detection algorithm based on heat map regression. First, the thermal imaging temperature matrix is input into a feature extraction network, which is composed of 5 convolutional blocks, each of which includes two 3x3 convolutional layers, a batch normalization layer and a ReLU activation function. The feature extraction network outputs multi-scale feature maps, which capture temperature distribution patterns at different scales.

[0153] Based on the multi-scale feature maps, a heat map regression network is used to predict the position heat map of 17 human body key points, including the head, neck, shoulder, elbow, wrist, hip, knee and ankle joints. The heat map regression network adopts a full convolutional structure, and outputs a probability heat map for each key point. The peak value of the heat map corresponds to the possible position of the key point. The key point positioning is realized by finding the local maximum value in the heat map, and the detection confidence is calculated.

[0154] Based on the detected key points, a human body skeleton model is constructed. The skeleton model adopts a tree structure, with the torso as the root node and the limbs as the branches. Each joint stores the three-dimensional spatial coordinates and the detection confidence. The connection relationship between the joints is predefined according to human anatomy knowledge, forming a complete skeleton topology structure.

[0155] Step 52: based on the constructed human body skeleton model, the human body temperature distribution features are combined, and the joint angles and limb postures are calculated through inverse kinematics algorithm to generate user limb action parameters;

[0156] Specifically, the inverse kinematics algorithm adopts a hierarchical optimization strategy, treating the human body as a multi-joint kinematic chain. First, the system establishes a human kinematics model, defining the degrees of freedom and movement range of each joint. The torso has 6 degrees of freedom, 3 positions and 3 rotations, and the joints of the limbs are set according to anatomical characteristics, such as 3 degrees of freedom for the shoulder joint and 1 degree of freedom for the elbow joint.

[0157] In the inverse kinematics solving process, a hybrid algorithm combining analytical method and numerical optimization is adopted. For simple joint chains with closed-form solutions, such as arms, the analytical method is directly used to calculate the joint angles; for complex multi-joint structures, the Jacobian transpose method is used for numerical solution. The Jacobian matrix represents the partial derivative of the end effector position with respect to the joint angle, and the error between the target position and the current position is minimized through iterative optimization.

[0158] The optimization objective function of inverse kinematics solving contains multiple constraint terms: first, the position constraint ensures that the key point position is consistent with the observation; second, the anatomical constraint limits the joint angle within the physiological feasible range; third, the temperature consistency constraint uses temperature distribution to verify the rationality of the pose.

[0159] It should be noted that the temperature distribution feature plays an important verification role in inverse kinematics solving. By establishing a temperature distribution model for each part of the human body, the overlapping and occlusion of the limbs under different poses will produce specific temperature distribution patterns. By comparing the theoretical temperature distribution under the predicted pose with the actual observed temperature distribution, the accuracy of the pose estimation is evaluated, and the optimization direction is guided.

[0160] The final generated user limb motion parameters include the following information: three-dimensional coordinates of each joint, joint angle values, position and pose of the torso, and confidence scores of each parameter, which completely describe the user's current limb pose and motion state.

[0161] Step 53: Perform time series analysis on the limb motion parameters in the continuous time sequence to identify user action patterns and behavior intentions, and generate user limb motion details;

[0162] Specifically, the time series analysis adopts a recurrent neural network architecture to process the time series of skeleton-related parameters in the user limb motion parameters. The skeleton flow network is input with the time series sequence composed of continuous frame limb motion parameters, and a bidirectional LSTM structure is used to capture the time series features of the action;

[0163] The recurrent neural network architecture action recognition module identifies pre-defined action categories based on the extracted time series features through a fully connected layer and a Softmax classifier, including basic actions such as walking, turning, squatting, and reaching out. The system uses a sliding window strategy to perform real-time action recognition on continuous data streams, with a window size of 2 seconds and a step size of 0.5 seconds, ensuring timely capture of action changes.

[0164] The behavior intention understanding module of the recurrent neural network architecture maps the identified basic action sequence to a high-level behavior intention. For example, an action-intention mapping model is established to capture the probabilistic relationship between action sequences and intentions by a conditional random field algorithm. For example, a continuous "reaching-grasping-withdrawing" action sequence can correspond to the behavior intention of "taking an object".

[0165] The finally generated user limb action details contain three levels of information: the bottom-level joint motion parameters, the middle-level basic action recognition results, and the high-level behavior intention understanding. These information are output in a structured data form, containing time stamp, action type, action parameters, intention label and confidence score.

[0166] Further, the step of judging whether the user behavior is an abnormal body state according to the risk point prediction result and the user action details comprises the following steps:

[0167] Step 61: constructing a user-risk point interaction model centered on the user skeleton to obtain the interaction state of each part of the user with the risk point;

[0168] The user-risk point interaction model calculates the spatial relationship of each joint with the surrounding risk points, including the shortest distance, relative orientation and approaching speed. The user-risk point interaction model adopts a hierarchical structure, divides the user's body into four main parts: head, torso, upper limbs and lower limbs, and respectively evaluates the interaction state of each main part with the risk point, such as contact, close distance or long distance, etc.

[0169] Step 62: judging whether the user behavior is an abnormal body state based on the user-risk point interaction model and the interaction state;

[0170] Specifically, in the preferred embodiment of the present application, the abnormality judgment is performed by using a multi-level abnormal body state detection algorithm; the multi-level abnormal body state detection algorithm contains three levels of abnormality detection mechanisms: rule-based static posture detection, statistics-based dynamic behavior analysis and learning-based comprehensive abnormality recognition.

[0171] In the static posture detection layer, based on the pre-defined ergonomics rules, the rationality of the current posture of the user is evaluated. In the specific implementation of the present application, the following key indicators can be detected: first, the torso inclination angle, when the torso is inclined forward or laterally by more than 30 degrees, it is marked as a potential abnormality; second, the joint angle constraint, such as when the knee joint bending angle is more than 120 degrees or less than 10 degrees, it is determined as an unnatural posture; third, the body balance state, the balance stability is evaluated by calculating the relationship between the center of gravity projection point and the support surface, when the center of gravity projection point is less than 10 centimeters away from the edge of the support surface, it is identified as an unstable state.

[0172] In the dynamic behavior analysis layer, the sliding time window method is adopted to analyze the time characteristics of the user's continuous action sequence. The window size is set to 2 seconds, and the step is 0.5 seconds, which ensures timely capture of action changes. In each time window, the system calculates the following dynamic features: first, the joint motion speed, when the speed of any joint node exceeds the preset threshold (such as head > 2 m / s, arm > 3 m / s), it is marked as abnormal; second, acceleration mutation, detect whether the joint acceleration appears sudden change, the change rate exceeds 5 m / s 2 abnormal; third, action continuity, by calculating the cosine similarity between consecutive poses to evaluate the smoothness of the action, for example, when the similarity is less than 0.7, it is identified as an incoherent action.

[0173] In the comprehensive abnormality recognition layer, combined with the interaction state of user-risk point, the final abnormality judgment is made through the SVM algorithm. The input feature vector of the SVM model contains 22 dimensions: 5 static posture features, 8 dynamic behavior features and 9 user-risk point interaction features. Static posture features include trunk inclination angle, main joint angle and body balance index; dynamic behavior features include joint speed, acceleration and posture continuity index; interaction features include the distance, relative speed and approach time of each part of the user to the nearest risk point.

[0174] In the actual implementation process of the present application, the SVM model adopts the radial basis function (RBF) kernel, the kernel parameter γ is set to 0.1, and the penalty parameter C is set to 10, which is optimized by the grid search method. The model output is a binary classification result: normal posture or abnormal posture. In order to improve the reliability of the judgment, the system also calculates the classification confidence, and only when the confidence is more than 0.85, the abnormal posture warning is triggered.

[0175] Step 63: When the user's behavior is judged to be an abnormal posture, a warning signal is generated, otherwise no processing is done.

[0176] Specifically, when it is determined that the user is in an abnormal posture state, a warning signal is immediately generated. The warning signal contains three parts of information: abnormal type, severity and suggested operation. The system generates a warning signal of the corresponding level according to the specific type of abnormal posture (such as unstable posture, dangerous approach or abnormal action) and the severity (low, medium and high levels).

[0177] The above embodiments are only used to illustrate the technical method of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical method of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical method of the present application.

Claims

1. A night penetrating scene reconstruction system based on fusion of millimeter wave radar and thermal imaging, characterized in that, The application comprises a spatial point cloud collection module, a temperature distribution collection module, a data fusion module, a risk point prediction module, an action detail restoration module, and a navigation warning module. The modules are connected through electrical means. The spatial point cloud collection module scans the indoor space through a millimeter wave radar to generate depth point cloud data in real time. The depth point cloud data is millimeter wave radar point cloud data, which contains the spatial coordinates and velocity of obstacles, and is sent to the data fusion module. The temperature distribution collection module captures human temperature distribution characteristics through a thermal imaging sensor. The human temperature distribution characteristics are output in the form of a thermal imaging temperature matrix, which contains temperature distribution. The human temperature distribution characteristics are sent to the data fusion module and the action detail restoration module. The data fusion module performs spatio-temporal alignment on the depth point cloud data and the human temperature distribution characteristics through an extended Kalman filter algorithm to generate fused three-dimensional scene data. The three-dimensional scene data is sent to the risk point prediction module. The risk point prediction module inputs the fused three-dimensional scene data into a lightweight convolutional network to predict the risk points of the occluded area and sends the risk points of the occluded area to the navigation warning module. The action detail restoration module restores user limb action details through human temperature distribution characteristics and sends the user limb action details to the navigation warning module. The body posture warning module determines whether the user behavior is an abnormal body posture according to the risk point prediction result and the user action details. If the determination is yes, a warning signal is triggered. Otherwise, no processing is performed.

2. The night time penetrating scene reconstruction system based on fusion of millimeter wave radar and thermal imaging according to claim 1, characterized in that, The real-time generation of depth point cloud data includes the following steps: Step 11: The millimeter wave radar collects three-dimensional point cloud data of the indoor space in real time by emitting millimeter wave signals of a predetermined frequency and receiving reflected signals. Step 12: The millimeter wave radar outputs the collected three-dimensional point cloud data in a structured format, which contains timestamp, spatial coordinates, reflection intensity, radial velocity, and confidence information.

3. The night time penetrating scene reconstruction system based on fusion of millimeter wave radar and thermal imaging according to claim 2, characterized in that, The human temperature distribution characteristics captured by the thermal imaging sensor include the following steps: Step 21: The thermal imaging sensor acquires scene temperature distribution data in real time by an infrared detector array to collect thermal radiation information in the environment. Step 22: The collected temperature distribution data is preprocessed, including noise suppression, temperature calibration, and spatial filtering, to improve the accuracy and spatial resolution of temperature measurement. Step 23: Human target detection and segmentation are performed on the preprocessed temperature distribution data to extract the human region in the foreground and generate human temperature distribution data. Step 24: The human temperature distribution data is organized into a structured thermal imaging temperature matrix, which contains timestamp, temperature value distribution, human region mask, and key point temperature information.

4. The night time penetrating scene reconstruction system based on fusion of millimeter wave radar and thermal imaging according to claim 3, characterized in that, The spatio-temporal alignment of the depth point cloud data and the human temperature distribution characteristics through the extended Kalman filter algorithm to generate fused three-dimensional scene data includes the following steps: Step 31: Time synchronization and spatial registration preprocessing are performed on the depth point cloud data of the millimeter wave radar and the temperature distribution characteristics of the thermal imaging sensor. Step 32: Based on the time synchronization and spatial registration of the preprocessed depth point cloud data and temperature distribution features, define the state vector, observation vector, state transition matrix and observation matrix to construct the extended Kalman filter state space model; Step 33: Perform the prediction and update steps of the extended Kalman filter algorithm to realize the probability fusion of the depth point cloud data and human body temperature distribution features; Step 34: Based on the fusion result after probability fusion, construct a fused three-dimensional scene model containing spatial geometric structure, temperature distribution and motion features.

5. The night time penetrating scene reconstruction system based on fusion of millimeter wave radar and thermal imaging according to claim 4, characterized in that, The method for predicting the risk points of the occluded area by inputting the fused three-dimensional scene data into a lightweight convolutional network comprises the following steps: Step 41: Preprocess the fused three-dimensional scene data to convert it into an input format suitable for processing by the lightweight convolutional network; Step 42: Construct a lightweight three-dimensional convolutional neural network model to extract features and predict risk points from the preprocessed three-dimensional scene data; Step 43: Post-process the output of the lightweight convolutional network to identify and locate the risk points in the occluded area.

6. The night time penetration scene reconstruction system based on fusion of millimeter wave radar and thermal imaging according to claim 5, characterized in that, The lightweight three-dimensional convolutional neural network adopts an encoder-decoder architecture and includes three main modules: feature extraction, context understanding and risk point prediction; In the feature extraction module, the network includes four consecutive three-dimensional convolutional blocks; each convolutional block consists of a 3x3x3 three-dimensional convolutional layer, a batch normalization layer, a ReLU activation function and a 3x3x3 depth separable convolution; the depth separable convolution decomposes the standard convolution into two steps: depth convolution and point-wise convolution, significantly reducing the number of parameters and computational complexity; the output channel number of the first convolutional block is 32, and the channel number doubles for each subsequent block, forming a feature pyramid structure; The context understanding module uses a design of dilated convolution to capture multi-scale context information through parallel convolution branches with different dilation rates (1, 2, 4, 8); each branch includes a 3x3x3 dilated convolution with 64 output channels; the outputs of each branch are weighted and fused through a channel attention mechanism to generate a feature map containing rich context information; The risk point prediction module uses a transpose convolution layer to gradually restore the spatial resolution of the feature map; it includes three consecutive upsampling blocks, each consisting of a 2x2x2 three-dimensional transpose convolution, a batch normalization layer and a ReLU activation function; the last layer is a 1x1x1 convolution layer with an output channel number of 1, representing the probability value of each voxel containing a risk point.

7. The night time penetration scene reconstruction system based on fusion of millimeter wave radar and thermal imaging according to claim 6, characterized in that, The method for restoring user limb action details through human body temperature distribution features comprises the following steps: Step 51: Perform key point detection and pose estimation on the thermal imaging temperature matrix to identify human skeletal joints and construct a human skeleton model; Step 52: Based on the constructed human skeleton model, combine the human body temperature distribution features and calculate the joint angles and limb poses through inverse kinematics algorithm to generate user limb action parameters; Step 53: Perform time series analysis on the limb action parameters in consecutive time sequences to identify user action patterns and behavior intentions and generate user limb action details.

8. The night time penetrating scene reconstruction system based on fusion of millimeter wave radar and thermal imaging according to claim 7, characterized in that, The skeleton joint detection adopts a multi-stage detection algorithm based on heat map regression; first, the thermal imaging temperature matrix is input into a feature extraction network composed of 5 convolutional blocks, each containing two 3x3 convolutional layers, batch normalization layers and ReLU activation functions; the feature extraction network outputs multi-scale feature maps that capture temperature distribution patterns at different scales; Based on the multi-scale feature maps, a heat map regression network is used to predict the position heat maps of 17 human body key points; the heat map regression network adopts a fully convolutional structure, and outputs a probability heat map for each key point, with the heat map peak corresponding to the key point; Key point positioning is achieved by finding the local maximum value in the heat map, while the detection confidence is calculated; Based on the detected key points, a human skeleton model is constructed; the skeleton model adopts a tree structure with the torso as the root node and the limbs as the branches; Each joint stores a three-dimensional spatial coordinate and a detection confidence; the connection relationship between the joints is predefined according to human anatomy knowledge, forming a complete skeleton topology structure.

9. The night penetrating scene reconstruction system based on fusion of millimeter wave radar and thermal imaging according to claim 8, wherein judging whether the user behavior is an abnormal posture based on the risk point prediction result and the user action details comprises the following steps: Step 61: constructing a user-risk point interaction model centered on the user skeleton to obtain the interaction state of each part of the user with the risk point; Step 62: judging whether the user behavior is an abnormal posture based on the user-risk point interaction model and the interaction state; Step 63: when the user behavior is judged to be an abnormal posture, generating an early warning signal, otherwise, no processing is performed.

10. The night penetrating scene reconstruction system based on fusion of millimeter wave radar and thermal imaging according to claim 9, wherein the abnormal posture judgment method is: Abnormal judgment is performed by using a multi-level abnormal posture detection algorithm; the multi-level abnormal posture detection algorithm contains three levels of abnormal detection mechanisms: rule-based static posture detection, statistics-based dynamic behavior analysis and learning-based comprehensive abnormal recognition.

Images