System and method for constructing a multi-modal human behavior perception dataset based on CSI

Abstract

This invention relates to a system and method for constructing a multimodal human behavior perception dataset based on CSI (Computer-Sensitive Interface). The system includes a wireless sensing subsystem for acquiring first CSI data packets of the human body, a visual capture subsystem for acquiring first RGB video, first depth video, second RGB video, and second depth video, a Linux-based central control unit, and a multimodal post-processing unit for cleaning and enhancing 3D skeletal joint data, denoising and extracting features from CSI data, and generating hierarchical natural language text descriptions. This invention constructs a large-scale, multidimensional, high-fidelity human behavior perception dataset acquisition system by deeply integrating high-frequency, fine-grained wireless signals with visual and semantic ground truth, ensuring the logical coupling of high-dimensional CSI signals and visual data.

Description

Technical Field

[0001] This invention relates to the field of data construction technology, specifically to a system and method for constructing a multimodal human behavior perception dataset based on CSI. Background Technology

[0002] Wireless sensing technology is an advanced means of achieving environmental monitoring and behavior capture using existing radio frequency communication infrastructure. Compared to traditional visual sensing or wearable sensors, this technology utilizes the multipath interference and diffraction laws of electromagnetic waves propagating in space, possessing the core advantages of non-contact, privacy protection, and non-line-of-sight (NLOS) sensing. With the advancement of the trend of sensor integration, using Wi-Fi signals to perform semantic modeling of human dynamics has become a key evolutionary direction in the fields of intelligent interaction and proactive health monitoring.

[0003] At the physical layer representation level, Channel State Information (CSI) exhibits extremely high spatial resolution by capturing the complex gain of each subcarrier in an orthogonal frequency division multiplexing (OFDM) system. CSI can meticulously depict the amplitude attenuation and phase drift of a signal caused by human movement interference during its propagation path, constructing a high-dimensional "action RF fingerprint." By increasing the sampling frequency to a certain level and combining it with the fine-grained features of hundreds of subcarriers under the Wi-Fi protocol, the system can capture even smaller amplitude limb disturbances, providing a robust physical layer foundation for high-precision human pose estimation.

[0004] A more serious problem is that current wireless sensing research is still limited by the breadth and accuracy of data sources. Most human behavior perception datasets are based on outdated hardware architectures (such as the Intel 5300 network card), with sparse subcarrier numbers and low sampling rates, resulting in severe spectral aliasing and information loss in the perceived features. In addition, the size of existing datasets is generally in the range of less than 10,000 cases, and the scenarios are mostly limited to single users, which easily leads to severe overfitting of deep learning models when dealing with heterogeneous environments. Moreover, due to the lack of a strict multimodal timestamp synchronization mechanism, the time asynchrony deviation between Wi-Fi signals and visual ground truth has also become a core bottleneck restricting further improvement in the accuracy of cross-modal feature fusion and pose reconstruction.

[0005] Therefore, providing a solution for constructing a large-scale, multi-dimensional, and high-fidelity multimodal human behavior perception dataset has become an urgent problem to be solved. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides a system and method for constructing a multimodal human behavior perception dataset based on CSI. The technical problem to be solved by this invention is achieved through the following technical solution: In a first aspect, the present invention provides a system for constructing a multimodal human behavior perception dataset based on CSI, comprising: A wireless sensing subsystem for collecting first CSI data packets of the human body, the wireless sensing subsystem comprising a first router located at the transmitting end and at least two second routers located at the receiving end; A visual capture subsystem is used to acquire a video set including a first RGB video, a first depth video, a second RGB video, and a second depth video. The visual capture subsystem includes a high-resolution sensing unit for acquiring high-definition first RGB video and first depth video, and a low-resolution annotation unit for acquiring low-definition second RGB video and second depth video. The Linux-based central control unit is used to control the wireless sensing subsystem and the visual capture subsystem to perform data synchronous acquisition, so as to obtain the first CSI data packet and the video set that are synchronously acquired, extract the first 3D skeletal joint points of the human body from the second RGB video and the second depth video, and realize cross-modal frame-level synchronization of the first CSI data packet, the video set and the first 3D skeletal joint points based on RTP timestamps, so that the first CSI data packet and the first 3D skeletal joint points are aligned with the videos in the video set. The multimodal post-processing unit performs bone data cleaning and enhancement on the aligned first 3D skeletal joints to obtain the second 3D skeletal joints, and performs denoising and feature extraction on the aligned first CSI data packet to obtain the second CSI data packet. The first RGB video and the second RGB video with the second 3D skeletal joints are input into the multimodal large language model to generate a hierarchical natural language text description containing action labels, limb coordination descriptions and micro-motion descriptions, thus completing the construction of the dataset.

[0007] Secondly, this invention provides a method for constructing a multimodal human behavior perception dataset based on CSI, including: Acquire the first CSI data packet acquired synchronously, including a video set comprising the first RGB video, the first depth video, the second RGB video, and the second depth video; Extract the first 3D skeletal joints of the human body from the second RGB video and the second depth video; Cross-modal frame-level synchronization of the first CSI data packet, the video set, and the first 3D skeletal joints is achieved based on RTP timestamps, so that the first CSI data packet and the first 3D skeletal joints are aligned with the video set. The first aligned 3D skeletal joints are cleaned and enhanced to obtain the second 3D skeletal joints. The first aligned CSI data packet is denoised and its features are extracted to obtain the second CSI data packet. The first RGB video and the second RGB video with the second 3D skeletal joints are input into a multimodal large language model to generate a hierarchical natural language text description containing action labels, limb coordination descriptions and micro-motion descriptions, thus completing the construction of the dataset.

[0008] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention provides a CSI-based multimodal human behavior perception dataset construction system. This system mainly consists of a wireless sensing subsystem, a visual capture subsystem, a Linux-based central control unit, and a multimodal post-processing unit. The system employs a single-input multiple-output (SIMO) wireless topology combined with multi-view visual capture technology. By deeply integrating high-frequency fine-grained wireless signals with visual and semantic ground truth, it constructs a large-scale, multi-dimensional, high-fidelity human behavior perception dataset acquisition system, ensuring the logical coupling of high-dimensional CSI signals and visual data.

[0009] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of a system for constructing a multimodal human behavior perception dataset based on channel state information, provided by the present invention. Figure 2 This is a logical architecture diagram of a multimodal data acquisition and automated storage system provided by the present invention; Figure 3 Equipment placement diagram in an office environment Figure 4 It is a location map of the subjects in an office environment, containing the accurate coordinates of five stations: A, B, C, D, and E. Figure 5 This is a flowchart illustrating a method for constructing a multimodal human behavior perception dataset based on channel state information, provided by the present invention. Detailed Implementation

[0011] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0012] In their research paper "MM-Fi: Multi-ModalNon-Intrusive 4D Human Dataset for Versatile Wireless Sensing" published at NeurIPS 2023, Jianfei Yang et al. proposed MM-Fi, a 4D human perception dataset integrating high-density CSI and multiple sensor modalities, such as... Figure 1 As shown, this research, through deep customization of commercial Wi-Fi devices, successfully increased the number of subcarriers from the traditional 30 to 114, achieving a nearly fourfold improvement in frequency domain resolution within a 40MHz bandwidth. This enables the capture of richer details of frequency-selective fading and subtle human movement features. The authors constructed a multimodal synchronization system comprising high-density Wi-Fi, millimeter-wave radar, lidar, and an RGB-D camera (Red Green Blue-Depth camera), encompassing 320,000 frames of data from 40 subjects across 27 categories of daily and rehabilitation movements. This provides core resources for continuous perception of wireless signals in the spatiotemporal dimensions. Although dense pose annotation still primarily relies on visual algorithms for generation, this dataset lays the foundation for cutting-edge research such as wireless human body mesh reconstruction.

[0013] In their research paper "WiMANS: A Benchmark Dataset for WiFi-based Multi-user Activity Sensing," published at ECCV 2024, Shuokang Huang et al. proposed WiMANS, the first large-scale benchmark dataset designed for multi-user wireless activity sensing. This dataset overcomes the limitations of previous Wi-Fi sensing methods, which were primarily confined to single-person scenarios, by providing over 9.4 hours of simultaneous dual-frequency (2.4 GHz and 5 GHz) channel state information (CSI) and corresponding ground truth video annotations. WiMANS encompasses 11,286 samples from 0 to 5 actors performing various action combinations (such as walking, sitting, waving, etc.), with fine-grained annotations for each sample's identity (ID), location, and action category. By simulating the challenges of multipath interference and signal interleaving in real-world environments, this dataset provides a core training resource and standard evaluation platform for multi-target feature decoupling, complex behavior recognition, and current multi-user pose estimation tasks.

[0014] Existing solutions also provide a method and system for establishing a cross-domain behavior recognition model based on CSI data. This method establishes a source domain behavior dataset and a target domain behavior dataset; segments and re-represents the data in both datasets; pairs the re-represented behavior data segments from the source domain dataset with those from the target domain dataset to obtain multiple data pairs; labels the data pairs formed by behaviors of the same category and those formed by behaviors of different categories differently; constructs a Siamese network model; inputs the data pairs and their corresponding labels into the Siamese network model to train it, obtaining the recognition model; collects behavior data of test subjects; performs segmentation and data pairing, inputs the paired data pairs into the recognition model, and outputs the recognition results; verification shows that the recognition method of this invention can achieve a high cross-domain recognition accuracy.

[0015] Existing solutions also provide a CSI action recognition method based on multi-channel fusion. This method involves a subject performing a predetermined action within the sensing area of ​​multiple WiFi signal transceiver pairs, collecting multi-view two-dimensional CSI data of the wireless signals, recording the corresponding action category labels, and obtaining a CSI dataset. The collected CSI dataset is then divided into training and test sets. An attention-based multi-channel feature fusion model is constructed to obtain training samples after time and subcarrier dimension weighting. Multi-view feature vectors for each training sample are obtained. Another attention-based multi-view feature fusion model is constructed to obtain the final feature representation of each training sample. Test samples from the test set are input into the trained action recognition network model to obtain the action recognition result. This method, based on the characteristics of wireless WiFi data, constructs an action recognition network model that can achieve high-precision passive behavior recognition tasks.

[0016] However, existing wireless human behavior perception datasets utilize outdated hardware platforms (such as the Intel 5300) with sparse subcarriers and low sampling rates. They fail to consider the need for high-resolution physical features in capturing small human movements, leading to significant loss of crucial information in the perceived features. Existing human behavior perception datasets are mostly single-modal channel state information data, lacking correlation with visual skeleton information and high-level natural language descriptions. They neglect the supporting role of multi-dimensional representations in understanding complex action semantics, resulting in insufficient spatial resolution when inferring fine-grained actions and difficulty in establishing a mapping from low-level physical features to high-level semantic concepts. Existing solutions often employ approximate synchronization or simple timing alignment methods when processing multimodal asynchronous signals, failing to consider the time deviation caused by the physical independence between high-sampling-rate wireless signals and fixed-frame-rate video streams. This results in inaccurate alignment between modalities, limiting the accuracy of cross-modal feature fusion and pose reconstruction. Existing human behavior perception datasets are generally small in scale (usually less than 10,000 cases) and limited to single-user and line-of-sight (LOS) environments, failing to fully cover complex indoor environments such as multi-user social interactions and non-line-of-sight (NLOS). This results in weak model generalization performance and a high risk of severe overfitting in heterogeneous environments.

[0017] Therefore, this invention provides a system for constructing a multimodal human behavior perception dataset based on channel state information. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of a multimodal human behavior perception dataset construction system based on channel state information provided by the present invention. The multimodal human behavior perception dataset construction system based on channel state information provided by the embodiments of the present invention includes: The wireless sensing subsystem is used to collect the first CSI data packet of the human body. The wireless sensing subsystem includes a first router located at the transmitting end and at least two second routers located at the receiving end. A visual capture subsystem is used to acquire a video set including a first RGB video, a first depth video, a second RGB video, and a second depth video. The visual capture subsystem includes a high-resolution sensing unit for acquiring high-definition first RGB video and first depth video, and a low-resolution annotation unit for acquiring low-definition second RGB video and second depth video. The Linux-based central control unit is used to control the wireless sensing subsystem and the visual capture subsystem to synchronously acquire data, obtain the first CSI data packet and video set, extract the first 3D skeletal joint points of the human body from the second RGB video and the second depth video, and realize cross-modal frame-level synchronization of the first CSI data packet, video set and the first 3D skeletal joint points based on RTP timestamps, so that the first CSI data packet and the first 3D skeletal joint points are aligned with the video in the video set. The multimodal post-processing unit performs bone data cleaning and enhancement on the aligned first 3D skeletal joints to obtain the second 3D skeletal joints, and performs denoising and feature extraction on the aligned first CSI data packet to obtain the second CSI data packet. The first RGB video and the second RGB video with the second 3D skeletal joints are input into the multimodal large language model to generate a hierarchical natural language text description containing action labels, limb coordination descriptions and micro-motion descriptions, thus completing the construction of the dataset.

[0018] In one specific embodiment, the wireless sensing subsystem operates in a 6GHz frequency band with a bandwidth of 80MHz, a sampling rate of 900Hz, and 598 effective subcarriers; the first router and the second router both support the Wi-Fi 6 protocol and are deployed in the sensing area.

[0019] Specifically, the topology deployment is as follows: At the transmitting end, a high-performance first router supporting the Wi-Fi 6 protocol and equipped with a built-in Intel AX210 network card is used. At the receiving end, at least two second routers with built-in Intel AX210 network cards and supporting the Wi-Fi 6 protocol, deployed at the edge of the sensing area, are used, thus forming a one-to-many sensing link. For example, one first router can be used at the transmitting end, and two second routers can be used at the receiving end, thus forming a one-to-two sensing link.

[0020] Frequency domain parameters of the wireless sensing subsystem: The system operates in the interference-free 6GHz band; it utilizes an 80MHz bandwidth to achieve real-time detection of 598 effective subcarriers. Compared to the traditional Intel 5300 solution, its frequency domain resolution is significantly improved, greatly enhancing its ability to capture small movements. Subcarrier and sampling rate settings: 598 effective subcarriers are set, and the system sampling rate is set to 900Hz to accurately capture the physical details of limb movements.

[0021] In one specific embodiment, the high-resolution sensing unit is a first camera configured in high-resolution mode, and the low-resolution annotation unit is a second camera configured in low-resolution mode. Both the first and second cameras are RealSense D435i cameras.

[0022] Specifically, the system in this embodiment deploys a first camera and a second camera. Both the first camera and the second camera simultaneously enable depth video streams and RGB video streams, but they are configured differently in terms of resolution settings and functional roles: The first camera is configured in high-resolution mode and is responsible for simultaneously acquiring high-definition RGB video streams and depth video streams. Its main function is to provide the dataset with detailed environmental backgrounds and human action details as a visual reference for generating fine-grained natural language action descriptions.

[0023] The second camera, configured in low-resolution mode, is responsible for simultaneously acquiring low-resolution RGB video streams and depth video streams. Its core function is to serve as the input source for the Nuictrack open-source toolkit (which provides skeleton tracking). Since the lower resolution is beneficial for improving the real-time performance and computational efficiency of skeleton extraction, this second camera focuses on real-time annotation of the coordinates of the 3D skeletal joints of the human body.

[0024] Spatial setup: The first and second cameras are fixedly placed directly above the first router at the transmitting end to ensure that the visual coordinate system is consistent with the horizontal displacement of the wireless signal transmission source.

[0025] like Figure 2 As shown, Figure 2 This is a logical architecture diagram of a multimodal data acquisition and automated storage system. It illustrates a single-transmit multiple-receive (SIMO) wireless sensing topology, including a transmitter and multiple receivers. Data 1, 2, and 3 correspond to wireless channel status information, high-resolution visual video streams (RGB video / depth video), and low-resolution visual video streams (RGB video skeleton points / depth video skeleton points), respectively. Ports 1, 2, and 3 correspond to the acquisition devices for the corresponding data: a Wi-Fi 6 acquisition device based on an AX210 network card, a high-resolution Realsense D435i video acquisition device, and a low-resolution Realsense D435i video acquisition device.

[0026] In one specific embodiment, the central control unit controls the wireless sensing subsystem and the visual capture subsystem to synchronously acquire data. The data acquisition terminal runs on Ubuntu 20.04 and uses the PicoScenes CSI tool. The specific process is as follows: data packets are injected through the AX210 network card, and the PicoScenes driver captures CSI information in kernel mode. Subsequently, the tool timestamps the acquired CSI signals, high / low resolution video streams, and skeletal coordinate files. Furthermore, each set of CSI measurements in the final exported data packet carries a precise timestamp index, facilitating direct processing in MATLAB or Python. The mapping.

[0027] In one specific embodiment, extracting the first 3D skeletal joints of the human body from the second RGB video and the second depth video includes: The second RGB video and the second depth video are received by the Nuictrack toolkit configured in the central control unit to extract the first 3D skeletal joints of the human body in real time.

[0028] Specifically, the data stream from the second camera is fed into the Nuicrack toolkit to extract the coordinates of standard human body keypoints in real time, i.e., the coordinates of the first 3D skeletal joints. This detailed pose data serves as the ground truth for training and is bound to the wireless signal.

[0029] Furthermore, since the wireless signal acquisition frequency (900Hz) and the visual sensor frame rate (30fps / 60fps) are driven independently by different clock sources at the physical level, and the Wi-Fi data packets have non-uniform delays in air interface transmission, there is a significant time asynchronous deviation between the two. Therefore, this invention designs a precise alignment and resampling algorithm based on the Real-time Transport Protocol (RTP).

[0030] In one specific embodiment, cross-modal frame-level synchronization of the first CSI data packet, the video set, and the first 3D skeletal joints is achieved based on RTP timestamps, so that the first CSI data packet and the first 3D skeletal joints are aligned with the video set, including: A1. The central control unit encapsulates a monotonically increasing RTP timestamp with a time sequence relationship for each first CSI data packet, each first 3D skeletal joint, and each frame of video in the video set. The RTP timestamp contains a sequence number and a timestamp reflecting the instant of data acquisition in the RTP header.

[0031] Here, the RTP timestamp is represented as:

[0032] in, For RTP timestamps For sampling sequence number, The unified reference clock is specifically a unified NTP / PTP high-precision network reference clock. Sampling frequency, The unit is timestamp. This is the amount of dynamic clock drift compensation for the system. A2. Select any one of the following as the alignment reference video: first RGB video, first depth video, second RGB video, and second depth video.

[0033] Specifically, since the first RGB video, the first depth video, the second RGB video, and the second depth video are acquired synchronously, any one of the first RGB video, the first depth video, the second RGB video, and the second depth video can be selected as the video used for alignment reference. This video used for alignment reference is the alignment reference video.

[0034] A3. Using the alignment reference video as a reference, translate all the first 3D bone joints so that each first 3D bone joint is aligned with the corresponding alignment reference video frame.

[0035] Specifically, because it takes time to parse the coordinates of the first 3D bone joints during acquisition, the timestamp of the first 3D bone joints will be offset relative to the timestamp of the video. Therefore, it is necessary to translate all the first 3D bone joints based on the alignment reference video so that they are aligned with the alignment reference video for each frame starting from the first first 3D bone joint.

[0036] A4. Based on the relationship between the RTP timestamp of the first CSI data packet in the CSI buffer pool and the RTP timestamp of the first frame alignment reference video, the CSI sequence in the CSI buffer pool is truncated or filled with null frames so that the first CSI data packet and the first frame alignment reference video are logically aligned in time.

[0037] Specifically, the central control unit temporarily stores all received first CSI data packets in the CSI buffer pool in chronological order. In this embodiment, to address the delay caused by device startup time differences, the system establishes a "starting frame anchor point" mechanism. At the beginning of the acquisition phase, the central control unit retrieves the first RTP timestamp of the video stream from the CSI buffer pool. Based on the chronological relationship between the first first CSI data packet in the CSI buffer pool and the first RTP timestamp of the alignment reference video, the central control unit adjusts the arrangement of the first CSI data packets to establish a stable starting alignment anchor point.

[0038] Furthermore, based on the relationship between the RTP timestamp of the first CSI data packet in the CSI buffer pool and the RTP timestamp of the first frame of the alignment reference video, the CSI sequence in the CSI buffer pool (the CSI sequence is a sequence composed of all first CSI data packets in the current CSI buffer pool in chronological order) is truncated or padded with null frames to ensure that the first CSI data packet and the starting frame of the first frame of the alignment reference video are logically aligned in time, including: The relationship between the RTP timestamp of the first CSI packet in the CSI buffer pool and the RTP timestamp of the first frame of the alignment reference video is determined. If the RTP timestamp of the first CSI packet is earlier than the RTP timestamp of the first frame of the alignment reference video, the header of the CSI sequence is truncated. The truncated portion is the part earlier than the RTP timestamp of the first frame of the alignment reference video, to ensure that the two are consistent. For logical alignment, if the RTP timestamp of the first CSI data packet is later than the RTP timestamp of the first alignment reference video, then a corresponding number of empty frames are added to the front of the CSI sequence to make the first CSI data packet and the starting frame of the first alignment reference video logically aligned in time, thus establishing a stable starting alignment anchor point.

[0039] A5. For each frame of the alignment reference video, use the RTP timestamp of the current frame's alignment reference video to define a corresponding time window in the CSI stream, and use a strategy of deleting more and adding less to adjust the number of the first CSI data packets within the time window. M Dynamically adjust to the theoretical number of CSI packages N Consistency is maintained until the alignment of all frames of the reference video with the first CSI packet is completed, theoretically involving the number of CSI packets. N The sampling frequency of the CSI stream is divided by the frame rate of the video stream.

[0040] Specifically, based on the system's parameter settings of a 900Hz sampling rate and a 30fps video frame rate, theoretically each frame of the alignment reference video should be associated with... The first CSI data packet. However, due to network jitter, the actual collected packet sequence is not uniformly distributed. Therefore, this system adopts a fixed ratio alignment algorithm of deletion and supplementation for frame-level alignment.

[0041] A5.1 For each frame of the alignment reference video, the span is defined in the CSI buffer pool using the RTP timestamp of the current frame's alignment reference video. The time window is ,in, For the first i The RTP timestamp of the frame alignment reference video. , For frame rate.

[0042] A5.2 Determine the number of the first CSI data packets within the time window. M Compared with the theoretical number of CSI packages N The relationship between them, if M > N Then, through the uniform sampling algorithm from M N first CSI data packets are uniformly selected from each first CSI data packet. These N first CSI data packets are used as CSI data packets aligned with the alignment reference video of the current frame to suppress redundant information caused by frequency-selective fading. M < N Then, the last first CSI data packet in the time window is sampled again to supplement the number of first CSI data packets in the time window. NTo ensure the logical integrity of the time domain dimension, follow the above method until the alignment of all frames of the reference video with the first CSI data packet is completed.

[0043] This embodiment transforms the original, discrete wireless CSI data packets into feature blocks that strictly correspond to the video frame sequence through the steps described above. This fixed-ratio alignment paradigm not only eliminates physical layer asynchronous biases but also provides logically consistent data support for the subsequent application of this dataset to tasks such as synchronous sensing and attitude reconstruction.

[0044] In this embodiment, in order to solve the problems of skeleton point jitter, abnormal jumps and data loss caused by the visual sensor during the acquisition process, this embodiment implements a complete skeleton data cleaning and enhancement scheme to solve these problems.

[0045] In one specific embodiment, the aligned first 3D skeletal joints are subjected to skeletal data cleaning and enhancement processing to obtain the second 3D skeletal joints, including: B1. If the coordinates of the aligned first 3D bone joint are (0, 0, 0) or outside the preset physical extreme value range, then the aligned first 3D bone joint is set to NaN (Not a Number). If the confidence field value of the aligned first 3D bone joint provided by the Nuictrack toolkit is less than the preset threshold, then the aligned first 3D bone joint is set to NaN and considered an invalid point to avoid introducing noise in subsequent calculations. The preset threshold is, for example, 0.2-0.3. Calculate the Euclidean displacement of the aligned first 3D bone joint between adjacent frames. If the Euclidean displacement exceeds the dynamic threshold, it is identified as abnormal jitter. The first aligned 3D skeleton joint is marked as an abnormal jump point and set to NaN. This mechanism effectively eliminates instantaneous "flying points" caused by sensor obstruction or light interference. The dynamic threshold is... , For absolute threshold, As a multiple factor, This represents the median of the historical frame shifts.

[0046] B2. For the first 3D skeletal joints set to NaN, perform missing frame detection to identify short-term missing points with a consecutive missing frame count less than or equal to t frames and medium-to-long-term missing points with a consecutive missing frame count greater than t frames. If it is a short-term missing point, use cubic spline interpolation or linear interpolation algorithms to calculate and fill the coordinates of the second 3D skeletal joints in the missing frame to maintain the continuity of limb movement speed. If it is a medium-to-long-term missing point and is not a critical point, use a Kalman filter to perform trajectory prediction and completion on the missing frame to generate the coordinates of the second 3D skeletal joints in the missing frame. If it is a medium-to-long-term missing point and is a critical point, use a spatiotemporal graph convolutional network (ST-GCN) or a temporal model to perform feature mapping and filling on the missing frame to generate the coordinates of the second 3D skeletal joints in the missing frame, ensuring the integrity of the skeleton structure. All remaining unmissing first 3D skeletal joints are then used as second 3D skeletal joints. For example, t is 5. For example, critical points are the torso, limbs, head, etc., and non-critical points are fingers, toes, etc.

[0047] In this embodiment, the propagation characteristics of Wi-Fi signals in indoor spaces are utilized to sense human behavior. When wireless signals propagate in complex indoor environments, they undergo reflection, diffraction, and scattering, resulting in multipath effects. When a subject moves within the sensing area, the length of the dynamic propagation path changes slightly, causing disturbances in the propagation delay and energy distribution of electromagnetic waves. In physical layer modeling, this impact of the indoor environment on the signal is typically described as the Channel Impulse Response (CIR).

[0048] Because this system uses the Wi-Fi 6 protocol based on Orthogonal Frequency Division Multiplexing (OFDM) technology, the wireless channel is divided into multiple mutually orthogonal subcarriers. According to signal processing principles, the CIR in the time domain and the Channel Frequency Response (CFR) in the frequency domain form a Fourier transform pair. In actual commercial Wi-Fi devices (such as the router with a built-in Intel AX210 network card used in this invention), the receiver directly captures the complex gain of each subcarrier in the frequency domain through physical layer link evaluation, thus forming the channel state information.

[0049] In one specific embodiment, the aligned first CSI data packet is denoised and its features are extracted to obtain the second CSI data packet, including: C1. Parse the first channel frequency response of each subcarrier at the corresponding time from the aligned first CSI data packet.

[0050] Specifically, CSI characterizes the attenuation and phase shift of a signal at a specific frequency from the transmitter to the receiver, for subcarriers. k The first channel frequency response at time t , can be represented as:

[0051] in, N For the total number of multiple paths, and Representing the first The amplitude gain and propagation delay of each path.

[0052] C2. For the first channel frequency response, Butterworth low-pass filtering is first performed to suppress high-frequency noise, and then the sampling phase offset is compensated by conjugate multiplication technique to obtain the second channel frequency response.

[0053] Specifically, RGB video, depth video, and video containing skeleton points can be directly stored. However, for CSI signals, due to the incomplete synchronization of the sampling clocks at the transmitting and receiving ends during actual sampling, the CSI signal will contain a large sampling phase offset, so some processing is required. A Butterworth low-pass filter is applied to the original first channel frequency response, and the sampling phase offset is eliminated using conjugate multiplication. The second channel frequency response is expressed as:

[0054] in, It is in time and frequency The first channel frequency response received by antenna 1; It is the conjugate complex number of the first channel frequency response received by antenna 2 at the same time and frequency. It is the preprocessed channel response result, i.e., the second channel frequency response.

[0055] This processing improves stability because the original CSI phase is usually chaotic and randomly changing. After conjugate multiplication, the phase becomes smooth and physically meaningful. Moreover, considering that this dataset will be used for diffusion models or semantic embedding learning, the relative phase obtained after this processing can reflect the changes in signal optical path difference caused by human activities (such as walking or gestures), which is an important input feature for subsequent diffusion models or semantic embedding learning.

[0056] C3. Obtain the second CSI data packet based on the second channel frequency response.

[0057] Specifically, the first CSI data packet with the second channel frequency response is used as the second CSI data packet.

[0058] In the system data acquisition process, the front-end sensing device detects physical layer packets in real time and encapsulates the parsed raw CSI data into... The format file is stored. To transform abstract radio frequency disturbances into features understandable by a computer vision model, this invention... The file further analyzes and extracts high-dimensional amplitude and phase matrices. Amplitude primarily reflects the energy attenuation of a signal during spatial propagation. When a human body blocks or absorbs electromagnetic waves, the amplitude fluctuates significantly, which is very effective for recognizing the amplitude of movements (such as squatting or walking). Phase is extremely sensitive to changes in path length. Even minute limb displacements can cause significant phase shifts, allowing for precise capture of subtle movement details. Amplitude and phase complement each other in terms of physical layer features, greatly improving frequency domain resolution and providing a more detailed underlying physical representation.

[0059] In this embodiment, a multimodal large language model (such as Qwen-VL) is also introduced, and a specific set of structured prompt words is designed. The model receives not only a first RGB video but also a second RGB video with second 3D skeletal joints as input. The generation of the multimodal large language model is divided into three levels: Action labeling layer: Identifies macro-level action categories (such as "hug" or "fall down").

[0060] Limb coordination description layer: Automatically analyzes the linkage logic of key joints such as shoulders, elbows, and wrists on both sides (such as "left shoulder synchronously adducts" and "right elbow slightly flexes to stabilize posture").

[0061] Micromotion description layer: Describes subtle anatomical changes (such as "shoulder adduction and flexion" and "wrist circumduction trajectory"), transforming abstract radiofrequency perturbations into interpretable anatomical language.

[0062] The final result is a layered natural language text description such as "The person is hugging, during which the right shoulder is retracted and bent forward, the right elbow is bent, the right wrist is wrapped around the other person's back, the left shoulder is retracted in sync, and the left elbow is slightly bent to stabilize the posture."

[0063] Finally, the corresponding hierarchical natural language text descriptions, second CSI data, second 3D skeleton joints, and video sets are combined and added to the dataset to complete the multidimensional cross-modal association annotation structure of "CSI signals, skeletons, and fine-grained text descriptions," thus completing the construction of the dataset.

[0064] Compared to the multimodal acquisition system used by MM-Fi, this invention achieves significant improvements in physical layer sensing accuracy and heterogeneous signal synchronization mechanisms. While the MM-Fi system achieves 114 subcarrier detection over commercial Wi-Fi through firmware customization, the multimodal human behavior perception dataset construction system built by this invention, based on the Wi-Fi 6 protocol and the 6GHz interference-free frequency band, significantly increases the number of effective subcarriers to 598, providing higher-dimensional frequency domain resolution. Regarding modal synchronization methods, compared to the approximate alignment method used by MM-Fi, this invention introduces timestamp alignment technology based on the Real-Time Transport Protocol (RTP), effectively solving the asynchronous deviation problem caused by independent hardware operation. Furthermore, in terms of environmental deployment, the system of this invention is specifically designed with a balanced distribution of line-of-sight (LOS) and non-line-of-sight (NLOS) acquisition schemes, improving the system's sensing robustness in complex indoor occlusion environments.

[0065] Compared to the construction method of WiMANS, this invention possesses core technological advantages in fine-grained feature capture and multi-dimensional semantic annotation systems. In terms of hardware perception performance, the WiMANS system provides only 270 subcarriers, while this invention achieves high-fidelity capture of limb movements by increasing the subcarrier scale to 598 and combining it with an ultra-high sampling rate of 900Hz. Regarding the annotation system's design methodology, WiMANS primarily focuses on discrete labels for identity, location, and action category, lacking a deep characterization of structured human posture. This invention innovatively enhances text description, specifying details such as head rotation, torso orientation, and limb changes when a person performs an action. This finer-grained text description facilitates a deep mapping from physical layer features to high-level complex behavioral semantics.

[0066] Existing wireless human behavior perception dataset acquisition platforms, such as the Intel 5300, are relatively outdated, suffering from sparse subcarrier numbers and low sampling rates. This leads to spectral aliasing and information loss when capturing small human movements. This invention utilizes the Wi-Fi 6 protocol and the interference-free 6GHz band to achieve fine-grained detection of 598 subcarriers and a high sampling rate of 900Hz. It also extracts high-dimensional amplitude and phase matrices from the original CSI signal, improving the resolution in both the frequency and time domains. This enables the capture of small limb perturbations and provides accurate underlying physical representations for human posture estimation.

[0067] Existing solutions often employ approximate alignment methods when synchronizing heterogeneous signals, failing to address the time asynchrony deviation caused by the physical independence between high-sampling-rate wireless signals and fixed-frame-rate video streams. This invention introduces a timestamp alignment technology based on the Real-Time Transport Protocol (RTP), which uses a unified high-precision clock source to affix standardized time tags to heterogeneous data. This solution establishes a traceable time alignment logic, aiming to control the time alignment deviation between modalities within a smaller range, providing a logically consistent data foundation for cross-modal feature fusion.

[0068] Existing human behavior perception datasets are mostly single-mode channel state information datasets and lack association with visual skeletons and high-level semantics, making it difficult for models to establish a mapping from physical features to complex semantic concepts. This invention not only includes CSI information but also RGB video, depth video, and skeleton points. Furthermore, by expanding text annotation information, this scheme provides more detailed supervision signals for models to learn the association between physical features and complex action concepts, supporting more complex cross-modal research tasks.

[0069] To address the limitations of existing human behavior perception datasets, which are generally small in scale (less than 10,000 cases) and confined to single-user, line-of-sight (LOS) environments, this invention constructs a dataset of over 44,000 CSI data samples, encompassing 22 categories of daily and post-rehabilitation actions performed by 15 subjects. By systematically collecting over 5,000 non-line-of-sight (NLOS) samples and multi-user social interaction samples in various indoor environments, this invention expands the diversity of the data, providing practical data support for improving the adaptability of perception models in complex and heterogeneous environments.

[0070] In summary, this invention provides a system for constructing a large-scale, multi-dimensional, and high-fidelity multimodal human behavior perception dataset. Utilizing the Wi-Fi 6 protocol and the interference-free 6GHz band, this invention achieves fine-grained detection of 598 subcarriers and a high sampling rate of 900Hz, enhancing the feature resolution capabilities in both the frequency and time domains and providing more detailed underlying physical features for human behavior perception. By introducing timestamp alignment technology based on real-time transmission protocols, this invention establishes a unified time reference for heterogeneous multimodal signals such as Wi-Fi signals and video streams, aiming to reduce time asynchrony deviations between modalities. Regarding dataset design, this invention constructs a large dataset (e.g., it can construct a dataset containing over 44,000 CSI data samples covering 15 subjects performing 22 categories of daily and post-rehabilitation actions), and balances the distribution of large-scale datasets in both line-of-sight and non-line-of-sight scenarios. By covering multiple users and multiple scenarios, the robustness and generalization performance of the perception model are improved. In addition to acquiring CSI signals, RGB video, and depth video, this invention also utilizes a multimodal large language model to transform basic action labels into complete action descriptions down to the details of the limbs, thereby establishing a multidimensional cross-modal association annotation structure of "signals, skeletons, and fine-grained text descriptions" and constructing a multimodal human behavior perception dataset based on CSI signals.

[0071] Example 2 This embodiment provides a specific large-scale, multi-scenario data construction process based on Embodiment 1.

[0072] like Figure 3 and Figure 4 The human behavior perception data acquisition scheme described in this invention is not a simple signal recording, but a rigorous physical environment setup and standardized action execution process, aiming to build a multi-dimensional deep association annotation dataset with high generalization ability.

[0073] (I) Data Collection Plan Scenario Setup: This embodiment simulates the multipath effect in a real-world indoor environment, specifically setting up three differentiated experimental scenarios that are frequently encountered in daily life. First, a small office with cramped space and many obstacles is selected as a typical environment, utilizing the intense electromagnetic wave reflection between walls to capture highly complex multipath signals. Second, a standard indoor workspace (such as a classroom) is selected as a balanced scenario to balance the visual distance (LOS) and perceptual features under partially occluded conditions. Finally, a spacious hall is selected as a weak multipath scenario, where electromagnetic propagation is mainly affected by reflections from the floor and ceiling, thus effectively extracting pure physical representations of human movements.

[0074] Device Placement: The topology and coordinate alignment of hardware devices are fundamental to ensuring high-fidelity data. The system employs a fixed transmitter and a movable receiver topology. During deployment, the visual sensor (RealSense D435i camera) is fixed directly above the transmitter host, ensuring that the visual coordinate system and the wireless signal transmitter are aligned horizontally. To enhance the model's adaptability to varying receiver sensitivities and signal attenuation, a "vertical swap" method is used to deploy the receiver in each environment, collecting data at two preset locations.

[0075] Character movement position settings: During the data collection process at each device placement position, for static movements, the character performs 4 actions at each of 5 different positions at random angles; for dynamic movements, the character performs 4 actions each in clockwise, counterclockwise, horizontal, vertical, and random diagonal directions.

[0076] Motion capture process setup: Six data points were collected for each motion under NLOS conditions (three horizontally and three vertically). Due to equipment relocation issues, a clockwork-style rotation of positions was used for daily motion capture. Specifically, under LOS conditions, data for person A was collected at positions a, b, and c; under NLOS conditions, data was collected at positions d and e. For person B, data was collected at positions b, c, and d under LOS conditions, and at positions a and e under NLOS conditions. During the capture process, 12 samples were first collected for all individuals under LOS conditions. Then, the equipment was moved outdoors to collect the remaining samples under NLOS conditions (NLOS data collection was not performed in the lobby due to limitations of the lobby setting).

[0077] Data collection sequence setup: Before the experiment, each student was given a data collection action sheet, which included grouping information for the three environments and the action execution position / direction information for each environment. Data collection was conducted by environment and group, with 15 participants divided into 3 groups. Data was collected from 5 participants daily. To avoid the influence of time and participants, three different groups of participants were used in the three environments. Data collection was completed in a total of 10-15 days.

[0078] In particular, strict settings are required for data collection from all three groups. The following example uses group one: Common actions: This category includes: drinking water, shaking head; waving hands, clapping hands, stretching; sitting down, squatting down, standing up, jumping; picking up objects, wiping, bowing; dressing; falling down, lying down, convulsing. The subject should refer to the following script and perform each action 4 times in the corresponding positions (A, B, C, D, E) according to the order. Specific details are as follows: Character 1: Stand at points A, B, and C and perform actions. After one of the routers is moved outside the wall (to collect non-line-of-sight NLOS data), the character performs actions again at points D and E.

[0079] Characters 2: B, C, and D, after the router's location changes, execute at E and A.

[0080] Characters 3: C, D, and E, after the router's location changes, execute at A and D.

[0081] Characters 4: D, E, and A, after the router's location changes, will execute at B and D.

[0082] Characters 5: E, A, and B, will execute at C and D after the router's location changes.

[0083] Walking-related actions: This category includes walking and running. The person performing the actions must refer to the following script and perform the actions four times each in the specified order: clockwise, counterclockwise, horizontal, vertical, and random diagonal.

[0084] Character 1: Executes clockwise, counterclockwise, and horizontally, and then vertically and randomly diagonally after the router's position changes.

[0085] Character 2: Execute counter-clockwise, horizontally, and vertically, and then randomly diagonally and clockwise after the router's position changes.

[0086] Character 3: Horizontal, vertical, and random diagonal movements, and then clockwise and counterclockwise movements after the router's position changes.

[0087] Character 4: Vertical, random diagonal, clockwise, and after the router position changes, execute in counterclockwise and horizontal directions.

[0088] Character 5: Randomly moves diagonally, clockwise, or counterclockwise, and executes horizontally or vertically after the router's position changes.

[0089] Jumping actions: This category includes standing jumps. The person performing the action must refer to the following script and perform the action 5 times each in the specified order: horizontal, vertical, random diagonal 1, and random diagonal 2.

[0090] Character 1: Executes horizontally, vertically, and randomly diagonally 1, and then randomly diagonally 2 after the router's position changes.

[0091] Character 2: Vertical, random diagonal 1, random diagonal 2, execute horizontally after the router position changes.

[0092] Character 3: Random diagonal 1, random diagonal 2, horizontal, and then vertical after the router position changes.

[0093] Character 4: Randomly diagonally 2, horizontally, vertically, and after the router's position changes, execute in random diagonally 1.

[0094] Character 5: Random diagonal 2, random diagonal 1, vertical, after the router position changes, execute horizontally.

[0095] Two-person actions: This category includes: handshake + hug. The person performing the action must refer to the following script and perform the action 5 times in different positions according to the specified combination of people and execution order.

[0096] Character 1 + Character 2: A, B, C, after the router's location changes, execute at D and E.

[0097] Characters 2 and 4 (B, C, and D) will execute at E and A after the router's location changes.

[0098] Characters 3 and 4: C, D, and E, will execute at A and D after the router's location changes.

[0099] Character 5 + Character 1: E, A, B, after the router's location changes, execute at C and D.

[0100] (II) CSI Data and Text Data Processing Noise Reduction and Denoising: RGB video, depth video, and video containing skeleton points can be directly stored. However, for CSI signals, due to the incomplete synchronization of the sampling clocks at the transmitting and receiving ends during actual sampling, the CSI signal will contain a large sampling phase offset, so some processing is required. A Butterworth low-pass filter is applied to the original channel frequency response, and the sampling phase offset is eliminated using conjugate multiplication.

[0101] This processing improves stability because the original CSI phase is usually chaotic and randomly changing. After conjugate multiplication, the phase becomes smooth and physically meaningful. Moreover, considering that this dataset will be used for diffusion models or semantic embedding learning, the relative phase obtained after this processing can reflect the changes in signal optical path difference caused by human activities (such as walking or gestures), which is an important input feature for subsequent diffusion models or semantic embedding learning.

[0102] Text information: A multimodal large language model (such as Qwen-VL) is introduced, and a specific set of structured prompt words is designed. The model receives not only video images but also extracted skeleton temporal features as input. The generation process consists of three levels: Action labeling layer: Identifies macro-level action categories (such as "hug" or "fall down").

[0103] Limb coordination layer: Automatically analyzes the linkage logic of key joints such as shoulders, elbows, and wrists on both sides (such as "left shoulder synchronously adducts" and "right elbow slightly flexes to stabilize posture").

[0104] Micromotion description layer: Describes subtle anatomical changes (such as "shoulder adduction and flexion" and "wrist circumduction trajectory"), transforming abstract radiofrequency perturbations into interpretable anatomical language.

[0105] The final result is a text description such as, "The person is hugging, during which the right shoulder is adducted and flexed forward, the right elbow is bent, the right wrist is wrapped around the other person's back, the left shoulder is simultaneously adducted, and the left elbow is slightly bent to stabilize the posture." Finally, this is combined with the corresponding CSI signal and skeleton point video to complete a multi-dimensional cross-modal association annotation structure of "CSI signal, skeleton, and fine-grained text description".

[0106] The multimodal human behavior perception dataset construction system provided by this invention has a more fine-grained physical layer feature analysis capability. Compared with datasets such as MM-Fi that use 114 subcarriers or outdated hardware such as Intel 5300, this invention utilizes the Wi-Fi 6 protocol to increase the number of subcarriers to 598, combined with a high sampling rate of 900Hz, and extracts high-dimensional amplitude and phase matrices from the original CSI signal, providing extremely high frequency and time domain resolution at the physical layer. This actual improvement in hardware parameters enables the system to capture more detailed motion details than previous sparse datasets, reducing the impact of spectral aliasing and information loss to a certain extent, and providing more solid raw physical layer data support for human behavior perception analysis.

[0107] The multimodal human behavior perception dataset construction system provided by this invention features a more standardized cross-modal synchronization method. Addressing the inherent physical asynchrony between high-sampling-rate wireless signals and fixed-frame-rate video streams, this invention abandons traditional coarse approximation synchronization methods and instead introduces timestamp encapsulation technology based on real-time transmission protocols. By unifying a high-precision clock source, this invention assigns a standardized time label to each data point, providing a more structured and logically consistent time calibration framework. Compared to existing technologies, this approach helps reduce time alignment deviations between modalities, providing a more reliable data alignment foundation for subsequent cross-modal feature fusion research.

[0108] The multimodal human behavior perception dataset construction system provided by this invention has a more detailed multi-dimensional semantic annotation system. Compared with datasets such as WiMANS, which are single-modality or only contain basic category labels, this invention supplements the semantic connotations lacking in wireless signals by introducing structured human skeleton coordinates and fine-grained natural language descriptions generated using a large language model. This provides multi-dimensional supervision signals for the model to learn the correlation between low-level physical features and high-level human behavior concepts, helping to improve the accuracy of representing complex actions.

[0109] The multimodal human behavior perception dataset construction system provided by this invention features a larger data scale and more diverse scene coverage. Existing human behavior perception datasets (such as MM-Fi and WiMANS) typically have small sample sizes and are mostly limited to laboratory or line-of-sight environments, which can easily lead to model overfitting. This invention constructs a dataset covering 15 subjects performing 22 categories of daily and post-rehabilitation actions, totaling over 44,000 CSI data samples, and achieves a balanced distribution of line-of-sight and non-line-of-sight scenes, fully covering complex real-world environments such as multi-user social interactions. By providing more diverse and challenging samples, this invention significantly enhances the adaptability of the perception model in complex and heterogeneous environments, meeting the generalization requirements of practical applications.

[0110] Example 3 Please see Figure 5 Based on Embodiments 1 and 2, this invention further provides a method for constructing a CSI-based multimodal human behavior perception dataset. This method utilizes the multimodal human behavior perception dataset construction system provided in the above embodiments. The method includes: Acquire the first CSI data packet acquired synchronously, including a video set comprising the first RGB video, the first depth video, the second RGB video, and the second depth video; Extract the first 3D skeletal joints of the human body from the second RGB video and the second depth video; Cross-modal frame-level synchronization of the first CSI data packet, video set, and first 3D skeletal joint point is achieved based on RTP timestamps, so that the first CSI data packet and the first 3D skeletal joint point are aligned with the video set. The first 3D skeletal joints after alignment are cleaned and enhanced to obtain the second 3D skeletal joints. The first CSI data packet after alignment is denoised and feature extracted to obtain the second CSI data packet. The first RGB video and the second RGB video with the second 3D skeletal joints are input into a multimodal large language model to generate a hierarchical natural language text description containing action labels, limb coordination descriptions and micro-motion descriptions, thus completing the construction of the dataset.

[0111] The implementation principle of the multimodal human behavior perception dataset construction method provided in this embodiment is similar to the implementation principle and technical effect of the multimodal human behavior perception dataset construction system provided in the above embodiments, and will not be repeated here.

[0112] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0113] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0114] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, any modifications made without departing from the inventive concept should be considered within the scope of protection of the present invention.

Claims

1. A system for constructing a multimodal human behavior perception dataset based on CSI, characterized in that, include: A wireless sensing subsystem for collecting first CSI data packets of the human body, the wireless sensing subsystem comprising a first router located at the transmitting end and at least two second routers located at the receiving end; A visual capture subsystem is used to acquire a video set including a first RGB video, a first depth video, a second RGB video, and a second depth video. The visual capture subsystem includes a high-resolution sensing unit for acquiring high-definition first RGB video and first depth video, and a low-resolution annotation unit for acquiring low-definition second RGB video and second depth video. The Linux-based central control unit is used to control the wireless sensing subsystem and the visual capture subsystem to perform data synchronous acquisition, so as to obtain the first CSI data packet and the video set that are synchronously acquired, extract the first 3D skeletal joint points of the human body from the second RGB video and the second depth video, and realize cross-modal frame-level synchronization of the first CSI data packet, the video set and the first 3D skeletal joint points based on RTP timestamps, so that the first CSI data packet and the first 3D skeletal joint points are aligned with the videos in the video set. The multimodal post-processing unit performs bone data cleaning and enhancement on the aligned first 3D skeletal joints to obtain the second 3D skeletal joints, and performs denoising and feature extraction on the aligned first CSI data packet to obtain the second CSI data packet. The first RGB video and the second RGB video with the second 3D skeletal joints are input into the multimodal large language model to generate a hierarchical natural language text description containing action labels, limb coordination descriptions and micro-motion descriptions, thus completing the construction of the dataset.

2. The multimodal human behavior perception dataset construction system according to claim 1, characterized in that, The wireless sensing subsystem operates in a 6GHz frequency band with a bandwidth of 80MHz, a sampling rate of 900Hz, and 598 effective subcarriers. Both the first router and the second router support the Wi-Fi 6 protocol and are deployed in the sensing area.

3. The multimodal human behavior perception dataset construction system according to claim 1, characterized in that, The high-resolution sensing unit is a first camera configured in high-resolution mode, and the low-resolution annotation unit is a second camera configured in low-resolution mode, with the first camera and the second camera located directly above the first router.

4. The multimodal human behavior perception dataset construction system according to claim 1, characterized in that, Extracting the first 3D skeletal joints of the human body from the second RGB video and the second depth video, including: The second RGB video and the second depth video are received by the Nuicrack toolkit configured in the central control unit to extract the first 3D skeletal joints of the human body in real time.

5. The multimodal human behavior perception dataset construction system according to claim 1, characterized in that, Cross-modal frame-level synchronization of the first CSI data packet, the video set, and the first 3D skeletal joints based on RTP timestamps, so that the first CSI data packet and the first 3D skeletal joints are aligned with the video set, including: Each of the first CSI data packets, each of the first 3D skeletal joints, and each video frame in the video set is encapsulated with a monotonically increasing RTP timestamp that has a time-series relationship. The RTP timestamp is represented as follows: in, For RTP timestamps For sampling sequence number, For a unified reference clock, Sampling frequency, The unit is timestamp. This is the amount of dynamic clock drift compensation for the system. Select any one of the first RGB video, the first depth video, the second RGB video, and the second depth video as the alignment reference video; Using the alignment reference video as a reference, translate all the first 3D bone joints so that each first 3D bone joint is aligned with the corresponding frame of the alignment reference video. Based on the relationship between the RTP timestamp of the first first CSI data packet in the CSI buffer pool and the RTP timestamp of the first frame of the alignment reference video, the CSI sequence in the CSI buffer pool is truncated or filled with null frames so that the first first CSI data packet and the first frame of the alignment reference video are logically aligned in time. For each frame of the alignment reference video, a corresponding time window is defined in the CSI stream using the RTP timestamp of the alignment reference video of the current frame, and a strategy of deleting more and adding less is adopted to determine the number of first CSI data packets within the time window. M Dynamically adjust to the theoretical number of CSI packages N The alignment continues until all frames of the alignment reference video are aligned with the first CSI data packet.

6. The multimodal human behavior perception dataset construction system according to claim 5, characterized in that, Based on the relationship between the RTP timestamp of the first CSI data packet in the CSI buffer pool and the RTP timestamp of the first frame of the alignment reference video, the CSI sequence in the CSI buffer pool is truncated or padded with null frames to ensure that the first CSI data packet and the starting frame of the first frame of the alignment reference video are logically aligned in time, including: The relationship between the RTP timestamp of the first first CSI data packet in the CSI buffer pool and the RTP timestamp of the first frame of the alignment reference video is determined. If the RTP timestamp of the first first CSI data packet is earlier than the RTP timestamp of the first frame of the alignment reference video, the header of the CSI sequence is truncated. If the RTP timestamp of the first first CSI data packet is later than the RTP timestamp of the first frame of the alignment reference video, a corresponding number of null frames are added to the front of the CSI sequence so that the first first CSI data packet and the starting frame of the first frame of the alignment reference video are logically aligned in time.

7. The multimodal human behavior perception dataset construction system according to claim 5, characterized in that, For each frame of the alignment reference video, a corresponding time window is defined in the CSI stream using the RTP timestamp of the alignment reference video of the current frame, and a strategy of deleting more and adding less is adopted to determine the number of first CSI data packets within the time window. M Dynamically adjust to the theoretical number of CSI packages N Consistent until the alignment of all frames of the alignment reference video with the first CSI data packet is completed, including: For each frame of the alignment reference video, the span in the CSI buffer is defined using the RTP timestamp of the alignment reference video of the current frame. The time window, the time window is ,in, For the first i The RTP timestamp of the frame alignment reference video. , For frame rate; Determine the number of the first CSI data packets within the time window. M With the theoretical number of CSI packages N The relationship between them, if M > N Then, through the uniform sampling algorithm from M N first CSI data packets are evenly selected from the first CSI data packets. M < N Then, the last first CSI data packet in the time window is sampled again to supplement the number of first CSI data packets in the time window. N The alignment of the alignment reference video with the first CSI data packet is completed for all frames until the alignment of all frames is completed.

8. The multimodal human behavior perception dataset construction system according to claim 1, characterized in that, The aligned first 3D bone joints are cleaned and enhanced to obtain the second 3D bone joints, including: If the coordinates of the first 3D bone joint after alignment are (0, 0, 0) or the coordinates of the first 3D bone joint after alignment are outside the preset physical extreme value range, then the first 3D bone joint after alignment will be set to NaN. If the confidence field value of the first aligned 3D bone joint provided by the Nuictrack toolkit is less than the preset threshold, then the first aligned 3D bone joint will be set to NaN. Calculate the Euclidean displacement of the first 3D bone joint after alignment between adjacent frames. If the Euclidean displacement is greater than a dynamic threshold, mark the aligned first 3D bone joint as an abnormal jump point and set it to NaN. The dynamic threshold is... , For absolute threshold, As a multiple factor, The median of the historical frame offsets; For the first 3D skeletal joints set to NaN, missing frame detection is performed to identify short-term missing points with a consecutive number of missing frames less than or equal to t frames and medium-to-long-term missing points with a consecutive number of missing frames greater than t frames. If it is a short-term missing point, cubic spline interpolation or linear interpolation algorithm is used to calculate and fill the coordinates of the second 3D skeletal joints in the missing frame. If it is a medium-to-long-term missing point and is not a critical point, a Kalman filter is used to perform trajectory prediction and completion on the missing frame to generate the coordinates of the second 3D skeletal joints in the missing frame. If it is a medium-to-long-term missing point and is a critical point, a spatiotemporal graph convolutional network or temporal model is used to perform feature mapping and filling on the missing frame to generate the coordinates of the second 3D skeletal joints in the missing frame. All other non-missing first 3D skeletal joints are used as second 3D skeletal joints.

9. The multimodal human behavior perception dataset construction system according to claim 1, characterized in that, The second CSI data packet is obtained by denoising and feature extraction of the aligned first CSI data packet, including: The first channel frequency response of each subcarrier at the corresponding time is parsed from the aligned first CSI data packet; For the first channel frequency response, Butterworth low-pass filtering is first performed to suppress high-frequency noise, and then the sampling phase offset is eliminated by conjugate multiplication technique to obtain the second channel frequency response; The second CSI data packet is obtained based on the second channel frequency response.

10. A method for constructing a multimodal human behavior perception dataset based on CSI, characterized in that, include: Acquire the first CSI data packet acquired synchronously, including a video set comprising the first RGB video, the first depth video, the second RGB video, and the second depth video; Extract the first 3D skeletal joints of the human body from the second RGB video and the second depth video; Cross-modal frame-level synchronization of the first CSI data packet, the video set, and the first 3D skeletal joints is achieved based on RTP timestamps, so that the first CSI data packet and the first 3D skeletal joints are aligned with the video set. The first aligned 3D skeletal joints are cleaned and enhanced to obtain the second 3D skeletal joints. The first aligned CSI data packet is denoised and its features are extracted to obtain the second CSI data packet. The first RGB video and the second RGB video with the second 3D skeletal joints are input into a multimodal large language model to generate a hierarchical natural language text description containing action labels, limb coordination descriptions and micro-motion descriptions, thus completing the construction of the dataset.

Images