Multi-person three-dimensional human body posture and spatial positioning combined reconstruction method and system

Abstract

The invention discloses a multi-person three-dimensional body posture and spatial positioning combined reconstruction method and system, and relates to the field of multi-modal three-dimensional reconstruction. The method comprises the steps that CSI signals are collected and preprocessed, and radio frequency feature representation is constructed; human body structured visual features are extracted by using a camera, and are aligned with CSI data through a timestamp and spatial calibration; a continuous three-dimensional radio frequency implicit field is constructed based on NeRFs, signal intensity is synthesized through a volume rendering technology, and deep fusion is carried out on the signal intensity and visual features under a cross-modal attention mechanism; a Transform encoder and double-decoder structure is adopted, three-dimensional joint point coordinates and spatial position coordinates of multiple persons are regressed, and joint optimization of postures and positioning is achieved through multitask attention. According to the method, the NeRF radio frequency field and the multi-head attention mechanism are fused to carry out multi-person three-dimensional human body posture and positioning combined reconstruction, and high-precision posture and spatial position reconstruction can be realized in a complex scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal three-dimensional reconstruction, in particular to a multi-person three-dimensional human posture and spatial positioning joint reconstruction method and system. BACKGROUND

[0002] With the rapid development of artificial intelligence, Internet of Things and intelligent sensing technology, the demand for accurate perception of human posture and three-dimensional spatial position in complex environments is increasing. Traditional human posture estimation technology has long relied on visual sensors such as RGB cameras and depth cameras, but its performance is significantly limited in actual scenarios such as weak light, severe occlusion, privacy sensitivity or multi-person dense interaction. Visual signals cannot penetrate obstacles, and continuous video acquisition also easily leads to privacy leakage risk, so a new perception paradigm is urgently needed as a supplement. In this context, WiFi signals, especially their channel state information (CSI), have gradually become a research hotspot in the field of human perception and positioning due to their strong penetration, convenient deployment and non-identity identification characteristics. In recent years, with the success of neural implicit representation (Neural Implicit Fields) in the visual field, researchers have begun to introduce it into radio frequency propagation modeling, thereby mapping wireless signals to continuous space functions, providing a new mathematical tool for understanding complex environments. The human posture estimation task is evolving from 2D to 3D and from single person to multiple people, and pure visual methods have obvious shortcomings in occlusion, weak light or privacy limited scenarios. Therefore, researchers propose to fuse WiFi CSI signals and visual information to build a complementary perception system. However, the generalization ability of WiFi-based perception models significantly decreases under different room layouts or hardware configurations, revealing the high sensitivity of existing methods to the environment and equipment. Although radio frequency perception and visual fusion show great potential, the field still faces multiple technical bottlenecks in actual deployment:

[0003] 1. Lack of continuous modeling of radio field: existing methods only process CSI as a time series signal, lacking explicit modeling of three-dimensional space propagation, making it difficult to establish a correspondence between the signal and the spatial geometry.

[0004] 2. Difficulty in separating multiple signals: In a multi-person environment, the reflection paths interfere with each other, and the signal is severely mixed, making it difficult to accurately distinguish the reflection components of different individuals.

[0005] 3. Separation of posture and positioning tasks: current methods independently process posture estimation and positioning tasks, failing to achieve joint optimization, resulting in a lack of spatial consistency in the outputs of the two tasks.

[0006] 4. High dependence on vision: performance significantly decreases under occlusion or light variation conditions, and most fusion methods still essentially rely on the supervision information provided by visual signals.

[0007] 5. Insufficient cross-scene generalization capability: changes in environmental layout or hardware devices can cause a sharp decline in model performance, lacking understanding and modeling of the physical nature of radio frequency signals. SUMMARY

[0008] The purpose of the present application is to provide a multi-person three-dimensional human pose and spatial positioning joint reconstruction method and system, to solve the spatial consistency and human body differentiation problems of existing WiFi-vision perception systems in multi-person and complex environments, and to achieve high-precision pose and spatial position reconstruction in complex scenes such as occlusion, multipath, and multi-person interaction.

[0009] To achieve the above purpose, the present application provides the following scheme: a multi-person three-dimensional human pose and spatial positioning joint reconstruction method based on a radio frequency field and a multi-head attention mechanism, comprising the following specific steps:

[0010] S1. Collecting multi-dimensional time series CSI signals and performing standardization and pseudo-image processing to form structured radio frequency features;

[0011] S2. Using a monocular or binocular camera to extract human structured visual features in real time, aligning with CSI data through timestamps and spatial calibration, and inputting the aligned visual feature data into a subsequent multi-modal fusion module;

[0012] S3. Inputting the preprocessed radio frequency features into a radio frequency field modeling module to construct a continuous implicit field describing the radio frequency propagation characteristics of a three-dimensional space, synthesizing signals through volume rendering technology, and outputting radio frequency volume rendering features;

[0013] S4. Using a multi-modal attention fusion method based on Transformer to deeply fuse the human structured visual features and the radio frequency volume rendering features under a cross-modal attention mechanism to generate final fusion features;

[0014] S5. On the basis of the fusion features, constructing a dual-branch joint optimization network for pose regression and spatial positioning regression, and realizing information interaction and joint optimization of the two tasks through a multi-head attention mechanism;

[0015] S6. According to the joint optimization results, outputting the 3D pose skeleton and 3D spatial position of each target through a dual-decoder, and performing end-to-end optimization using pose loss, positioning loss, and attention sparsity regularization terms.

[0016] Preferably, the standardization processing of the multi-dimensional time series CSI signals includes filtering and noise reduction, phase correction, and energy normalization; the specific steps are as follows:

[0017] S11. Filtering and Denoising: A combination of wavelet thresholding and Kalman filtering is used. First, wavelet packet decomposition is employed to decompose the continuous signal into wavelet coefficients of different scales and locations. The wavelet packet decomposition formula is as follows:

[0018] ;

[0019] in, This is the time-domain representation of the original CSI signal. It is the maximum number of levels in wavelet packet decomposition. It is the index of the current decomposition level. It is the index at the coefficient position of the current layer. These are the wavelet packet coefficients at scale j and position k. These are wavelet packet basis functions. For the decomposed wavelet coefficients, the Bayesian threshold is calculated:

[0020] ;

[0021] in, It is an adaptive threshold at scale j and position k. It is a noise standard deviation estimate. It is the signal length or the number of coefficients in the current sub-band. It is the median of the absolute values ​​of the coefficients of the current sub-band, and 0.6745 is the Gaussian distribution adjustment coefficient, used to estimate the standard deviation of the median;

[0022] Then apply the soft threshold function:

[0023] ;

[0024] in, These are the wavelet packet coefficients after thresholding. The original coefficient symbol, The absolute value of the original coefficients. To be a positive function, it equals x when x > 0, and 0 otherwise;

[0025] Finally, an extended Kalman filter model is established to further denoise and track signal dynamics:

[0026] ;

[0027] in, yes State vector at time step , It is the amplitude of the signal. It is the signal phase at time k. It is the Doppler frequency shift at time k. It is a state transition function. It is the observation function. is process noise, obeying zero-mean Gaussian distribution, is observation noise, obeying zero-mean Gaussian distribution;

[0028] S12, phase correction: linear phase deviation elimination is adopted, and the calculation formula is:

[0029]

[0030] wherein, is the phase after eliminating linear deviation, is the phase of the original measured CSI, is the subcarrier index, is the total number of subcarriers, is the timing offset data amount, is the phase offset;

[0031] S13, energy normalization:

[0032]

[0033] wherein, is the normalized signal amplitude, is the original signal amplitude, is the local mean estimation at time t, is the local standard deviation estimation at time t; and are estimated by a sliding window.

[0034] Preferably, the manner of pseudo-imaging processing of the multi-dimensional timing CSI signal is that the multi-dimensional timing CSI signal is reconstructed into a two-dimensional feature map through time-space-frequency three-dimensional mapping, forming a structured radio frequency feature representation.

[0035] Preferably, step S2 includes the following contents:

[0036] S21, according to the application scene, the camera type is selected, the lens is calibrated, and a unified world coordinate system is established, the camera collects the original image, real-time lens distortion correction is performed, the corrected image is sent into the attitude estimation model, and the human body structured visual feature is output;

[0037] S22, the video frame and the CSI data are aligned by using hardware triggering and software time stamp, and a unified space-time mapping relationship is established through joint calibration;

[0038] S23, the human body structured visual feature is packaged with the time stamp and the camera pose, and is paired with the CSI data according to the space-time mapping relationship.

[0039] Preferably, the following is performed: ​​The specific steps for radio frequency field modeling and constructing a continuous implicit field describing the radio frequency propagation characteristics in three-dimensional space are as follows:

[0040] S31. Arrange WiFi transmitters and several CSI receivers in the scene to establish an implicit radio frequency (RF) spatial field. The implicit function of the RF field is:

[0041] ;

[0042] in, Therefore A neural network mapping function with parameters. It is the coordinates of any query point in three-dimensional space. For the direction of signal propagation, The pitch angle, It is the azimuth angle. It is the amplitude decay function. It is the phase response function. Description in Point along direction The intensity of the scattered signal; the implicit function of the radio frequency field is specifically implemented through an MLP network:

[0043] ;

[0044] in, For position encoding functions;

[0045] S32. Using volumetric rendering technology, the signal intensity observed at a specific receiver is synthesized from a continuous radio frequency field. The volumetric rendering integral is performed along the ray path from the transmitter to the receiver to calculate the total signal intensity at the receiver:

[0046] ;

[0047] in, It is along the direction The total received signal strength, the integration path is ray Definition, from the transmitting end Depart, along the direction spread, For ray parameters, and Let these represent the near-field and far-field boundaries of the integral, respectively. Transmittance represents the signal from... spread to And the probability of no decay occurring;

[0048] ;

[0049] S33. Optimize the network parameters by minimizing the difference between the predicted signal and the actual observation. The loss function is:

[0050] ;

[0051] in, The actual measured CSI signal strength. The CSI signal strength of the synthesized signal. It is a regularization term;

[0052] S34. Determine whether the loss function L converges. If it converges, fix the MLP parameters and output the optimized continuous implicit field.

[0053] Preferably, the specific steps to obtain the final fusion features are as follows:

[0054] S41. The structured visual features of the human body are used as query Q, and the radio frequency volume rendering features are used as key K and value V. They are fused through a multi-head attention mechanism to obtain enhanced fused features.

[0055] S42. By linear projection, the features of different modalities are projected onto a unified dimension to obtain new Q, K, V;

[0056] S43. Divide the projected Q, K, V into h heads and compute the attention of each head in parallel.

[0057] S44. After stitching together all the heads, obtain the multi-head attention output through linear projection;

[0058] S45. After multi-head attention, residual connections and layer normalization are performed to output the final fused features after multimodal attention fusion and feedforward network enhancement.

[0059] Preferably, the specific steps for constructing the dual-branch joint optimization network are as follows:

[0060] S51. In order to preserve spatial information, sinusoidal position coding is added to the final fused features to obtain coded fused features;

[0061] S52. Construct a pose regression branch to decode the 3D joint coordinates of the human body from the encoded fusion features, and construct a positioning regression branch to decode the spatial position coordinates from the encoded fusion features.

[0062] S53. Introduce a cross-task attention mechanism between the attitude regression branch and the localization regression branch to facilitate information exchange.

[0063] Preferably, in S6, the pose and position are adjusted by optimizing the total loss function composed of pose loss, localization loss and attention sparse regularization term, and finally the 3D pose skeleton and 3D position of multiple people are output.

[0064] On the other hand, providing a basis A multi-person 3D human pose and spatial positioning joint reconstruction system based on radio frequency field and multi-head attention mechanism, including CSI acquisition and preprocessing module, video assistance module, The module includes an RF field modeling module, a multimodal fusion module, a dual-branch joint optimization module, and an inference module; among them,

[0065] The CSI acquisition and preprocessing module is used to acquire multi-dimensional time-series CSI signals and perform standardization and pseudo-image processing to form structured radio frequency features.

[0066] The video assistance module is used to extract human structured visual features in real time using a monocular or binocular camera, and to align them with CSI data through timestamps and spatial calibration. The aligned visual feature data will be input into the subsequent multimodal fusion module.

[0067] The The radio frequency field modeling module is used to initialize the radio frequency field using the radio frequency features. Radio frequency field modeling is used to construct a continuous implicit field that describes the radio frequency propagation characteristics in three-dimensional space. Signal synthesis is performed through volume rendering technology to output radio frequency volume rendering features.

[0068] The multimodal fusion module is used to employ a Transformer-based multimodal attention fusion method to deeply fuse the human structured visual features and the radio frequency volume rendering features under a cross-modal attention mechanism, generating the final fused features.

[0069] The dual-branch joint optimization module constructs a dual-branch joint optimization network based on fused features, performs pose regression and spatial localization regression respectively, and realizes information interaction and joint optimization between the two tasks through a multi-head attention mechanism;

[0070] The inference module, based on the joint optimization results, outputs the 3D pose skeleton and 3D spatial position of each target through dual decoders, and performs end-to-end optimization using pose loss, localization loss and attention sparse regularization term.

[0071] According to the above-described technical solution provided by the present invention, the present invention discloses the following technical effects: through fusion The RF field and multi-head attention mechanism are used to jointly reconstruct the pose and positioning of multiple three-dimensional human bodies, which can achieve high-precision pose and spatial position reconstruction in complex scenarios with occlusion, multipath, and multi-person interaction. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 This is a flowchart of the method of the present invention;

[0074] Figure 2 This is a flowchart of the CSI acquisition and preprocessing method of the present invention;

[0075] Figure 3 This is a flowchart of the radio frequency field rendering structure of the present invention;

[0076] Figure 4 This is a flowchart of the multi-head attention fusion method of the present invention;

[0077] Figure 5 This is a schematic diagram of the dual-branch joint optimization network used in this invention. Detailed Implementation

[0078] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0079] The purpose of this invention is to provide a method based on A method for joint reconstruction of multi-person 3D human pose and spatial localization using radio frequency field and multi-head attention mechanism, such as Figure 1 As shown, the specific steps include the following:

[0080] S1. Acquire multi-dimensional time-series CSI signals and perform standardization and pseudo-image processing to form structured radio frequency features;

[0081] S2. Use a monocular or binocular camera to extract the structured visual features of the human body in real time, and align them with CSI data through timestamps and spatial calibration. The aligned visual feature data will be input into the subsequent multimodal fusion module.

[0082] S3. Input the preprocessed radio frequency characteristics The radio frequency field modeling module constructs a continuous implicit field describing the radio frequency propagation characteristics in three-dimensional space, performs signal synthesis through volume rendering technology, and outputs radio frequency volume rendering features.

[0083] S4: A Transformer-based multimodal attention fusion method is adopted to deeply fuse human structured visual features and radio frequency volume rendering features under a cross-modal attention mechanism to generate the final fused features.

[0084] S5: Based on the fused features, a dual-branch joint optimization network is constructed to perform pose regression and spatial localization regression respectively. The information interaction and joint optimization of the two tasks are realized through a multi-head attention mechanism.

[0085] S6: Based on the joint optimization results, the 3D pose skeleton and 3D spatial position of each target are output through dual decoders, and end-to-end optimization is performed using pose loss, localization loss and attention sparse regularization term.

[0086] In this embodiment, an indoor environment with complex scenarios including multi-person interaction, occlusion, and low light is selected for verification. Furthermore, in S1, the raw channel state information acquired by the distributed Wi-Fi device undergoes processing including filtering and noise reduction, phase correction, energy normalization, and pseudo-imageization. Multi-scale time-domain features of the signal are extracted through time-series modeling to form a stable feature representation sensitive to multi-person actions, providing a high-quality input foundation for subsequent RF field construction and attention analysis.

[0087] like Figure 2 As shown, the specific operation process in step S1 is as follows:

[0088] Hardware and Data Acquisition Specifications: A distributed architecture with one transmitting node and six receiving nodes is adopted. All nodes are connected to a central synchronization server via Gigabit Ethernet to ensure a sampling time synchronization error of <1ms. Receiving nodes are evenly deployed around the perimeter of the room, with a spacing of 3±0.5 meters and a fixed installation height of 1.2 meters. The transmitting end (TX) can be a commercial AP (802.11ac), configured with a 4×4 MIMO antenna array, with a stable transmit power of 20dBm. Each receiving node (RX) is equipped with an Intel 5300 wireless network card and four omnidirectional antennas.

[0089] Data acquisition process: First, baseline data is collected in an unmanned environment to record static reflection characteristics. Data acquisition scenarios are then set up according to experimental requirements, including single / multi-person and different activity types (walking, sitting, interaction, etc.). The sampling frequency is f=1000Hz, bandwidth B=40MHz, and number of subcarriers K=56 (802.11ac). Data timestamps from all receiving nodes are aligned to the master clock and stored in a unified HDF5 data format, containing complete metadata. Visual reference data (RGB-D camera) is simultaneously acquired for ground truth annotation to ensure accurate timestamp matching.

[0090] Raw CSI signals are acquired using a distributed hardware architecture. The standardization process for multidimensional time-series CSI signals includes filtering and noise reduction, phase correction, and energy normalization; the specific steps are as follows:

[0091] S11. Filtering and Denoising: To address noise and phase shift issues in the original CSI data, a multi-step approach is used to improve data stability. A combination of wavelet thresholding and Kalman filtering is employed for noise reduction. First, wavelet packet decomposition is used to decompose the continuous signal into wavelet coefficients at different scales and locations. The wavelet packet decomposition formula is as follows:

[0092] ;

[0093] in, This is the time-domain representation of the original CSI signal. It is the maximum number of levels in wavelet packet decomposition. It is the index of the current decomposition level. It is the index at the coefficient position of the current layer. These are the wavelet packet coefficients at scale j and position k. These are wavelet packet basis functions. For the decomposed wavelet coefficients, the Bayesian threshold is calculated:

[0094] ;

[0095] in, It is an adaptive threshold at scale j and position k. It is a noise standard deviation estimate. It is the signal length or the number of coefficients in the current sub-band. It is the median of the absolute values ​​of the coefficients of the current sub-band, and 0.6745 is the Gaussian distribution adjustment coefficient, used to estimate the standard deviation of the median;

[0096] Then apply the soft threshold function:

[0097] ;

[0098] in, These are the wavelet packet coefficients after thresholding. The original coefficient symbol, This represents the absolute value of the original coefficients. To be a positive function, it equals x when x > 0, and 0 otherwise;

[0099] Finally, an extended Kalman filter model is established to further denoise and track signal dynamics:

[0100] ;

[0101] in, yes State vector at time step , It is the amplitude of the signal. It is the signal phase at time k. It is the Doppler frequency shift at time k. It is a state transition function. It is the observation function. It is process noise, and follows a zero-mean Gaussian distribution. It is observation noise, which follows a zero-mean Gaussian distribution;

[0102] S12. Phase Correction: Due to hardware deviations and environmental factors, the original CSI phase... Distortion exists, which is eliminated using linear phase deviation. The calculation formula is as follows:

[0103] ;

[0104] in, It is the phase after eliminating linearity deviation. It is the phase of the original CSI measurement. It is a subcarrier index. It is the total number of subcarriers. It is the amount of time-series offset data. It is a phase shift;

[0105] S13. To eliminate signal strength fluctuations caused by environmental changes, adaptive energy normalization is performed:

[0106] ;

[0107] in, It is the normalized signal amplitude. It is the original signal amplitude. It is a local mean estimate at time t. It is a local standard deviation estimate at time t; and Estimation via sliding window:

[0108] ;

[0109] ;

[0110] The method for pseudo-image processing of multidimensional time-series CSI signals is as follows: the multidimensional time-series CSI signals are reconstructed into two-dimensional feature maps through time-space-frequency three-dimensional mapping, forming a structured radio frequency feature representation. These pseudo-images serve as... Inputs to modules and multi-head attention mechanisms.

[0111] Furthermore, in S2, a monocular camera scheme is adopted. After lens calibration and coordinate system establishment are completed, structured visual features such as human skeleton heatmaps are extracted in real time. Then, using hardware triggering and software timestamps, video frames and CSI subcarrier data are aligned at the millisecond to microsecond level, and a unified spatiotemporal mapping relationship is established through joint calibration. Specifically, step S2 includes the following:

[0112] Camera Configuration and Calibration: Select a monocular RGB camera based on the application scenario, with a resolution of at least 1280×720, a frame rate ≥30fps, an installation height of 2.0-3.0 meters, and a tilt angle of 15°-30°. Camera intrinsic parameter calibration employs the Zhang Zhengyou calibration method, solving for the intrinsic parameter matrix and lens distortion coefficients using checkerboard images from different viewpoints. Camera extrinsic parameter calibration is performed by solving the PnP problem, calculating the rotation and translation relationship between the camera coordinate system and the world coordinate system, and establishing a unified world coordinate system.

[0113] Alignment is achieved using timestamps and CSI subcarrier indices: High-precision hardware timestamps are recorded for each image frame and each CSI data packet at both the camera driver and CSI driver layers. All device times are synchronized to a unified master clock based on NTP or PTP protocols. In post-processing, the image frames and CSI data packets with the closest timestamps are associated with data from the same time period.

[0114] Visual feature extraction: The camera captures raw images, performs real-time lens distortion correction, and feeds the corrected images into the pose estimation model, outputting human skeleton heatmaps, joint coordinates, or visual feature tensors. The visual features are packaged with timestamps and camera pose, paired with CSI data according to spatiotemporal mapping, and output as a query vector Q to the multi-head attention fusion module.

[0115] like Figure 3 As shown, in S3 Radio frequency (RF) field modeling involves learning the three-dimensional continuous distribution of RF signals using an MLP (Multi-Level Processing) algorithm to achieve the mapping from "spatial location to RF attributes." The specific operations are as follows:

[0116] S31. Deploy WiFi transmitters and several CSI receivers in the scene, establish an implicit radio frequency spatial field, and establish a continuous mapping relationship between spatial location, propagation direction, and radio frequency physical quantities:

[0117] ;

[0118] in, Therefore A neural network mapping function with parameters. It is the coordinates of any query point in three-dimensional space. For the direction of signal propagation, The pitch angle, It is the azimuth angle. It is the amplitude decay function. It is the phase response function. Description in Point along direction The intensity of the scattered signal; the implicit function of the radio frequency field is specifically implemented through an MLP network:

[0119] ;

[0120] in, For location encoding functions, low-dimensional inputs are mapped to high-frequency spaces, enhancing the network's ability to model high-frequency details;

[0121] S32. Using volumetric rendering technology, the signal intensity observed at a specific receiver is synthesized from a continuous radio frequency field. The volumetric rendering integral is performed along the ray path from the transmitter to the receiver to calculate the total signal intensity at the receiver:

[0122] ;

[0123] in, It is along the direction The total received signal strength, the integration path is ray Definition, from the transmitting end Depart, along the direction spread, For ray parameters, and Let these represent the near-field and far-field boundaries of the integral, respectively. Transmittance represents the signal from... spread to And the probability of no decay occurring;

[0124] ;

[0125] In practical calculations, the integral is discretized through hierarchical sampling:

[0126] ;

[0127] S33. Optimize the network parameters by minimizing the difference between the predicted signal and the actual observation. The loss function is:

[0128] ;

[0129] in, The actual measured CSI signal strength. The CSI signal strength of the synthesized signal. It is a regularization term that prevents overfitting, promotes smooth solutions, and enhances the physical rationality of the radio frequency field.

[0130] S34. Determine whether the loss function L converges. If it converges, fix the MLP parameters and output the optimized continuous implicit field.

[0131] The radio frequency field modeling module learns a continuous three-dimensional radio frequency field through the above process. This field implies the geometry of the environment (because obstacles will cause signal attenuation and reflective surfaces will affect directional intensity). This radio frequency field provides rich spatial features for the subsequent multi-head attention mechanism, thereby supporting the joint reconstruction of multi-person three-dimensional human pose and localization.

[0132] Furthermore, in S4, a Transformer-based multimodal attention fusion method is adopted. Video skeleton heatmap features are used as query vectors, and radio frequency volume rendering features are used as key-value pairs. A cross-modal attention mechanism is utilized to guide the separation of multiple targets in the radio frequency features and to complete the feature filling of occluded areas using visual semantic information, generating a fused representation that combines spatial accuracy and semantic integrity. For example... Figure 4 As shown, the specific operation process of step 4 is as follows:

[0133] This step uses visual skeleton heatmaps as a guide to separate multi-person target information from radio frequency features, achieving cross-modal feature enhancement. The specific operations are as follows:

[0134] First, the input features are preprocessed, using visual features as the query and radio frequency features as the key and value. These are then fused using a multi-head attention mechanism to obtain enhanced fused features. Next, linear projection is used to project the features from different modalities onto a unified dimension, resulting in new (Q, K, V) values. Cross-modal attention is then calculated. The projected Q, K, V values ​​are divided into h heads, and for each attention head, the following calculation is performed:

[0135] ;

[0136] After stitching together all the heads, the multi-head attention output is obtained through linear projection:

[0137] ;

[0138] ;

[0139] From the above description, we know that the query (Q) comes from visual features, and the key (K) comes from radio frequency (RF) features. The attention weight is obtained by calculating the similarity between Q and K. This weight indicates which RF features are related to each visual key. The value (V) is also an RF feature. The attention weight is used to perform a weighted summation of V, which is equivalent to extracting useful information from the RF features based on visual guidance.

[0140] After multi-head attention, residual connections and layer normalization are performed to enhance the output:

[0141] ;

[0142] ;

[0143] Dropout is a regularization technique used for... Adding the original visual features enhances the information, and then applying LayerNorm stabilizes the distribution of activation values, accelerating training; for First The input is a feedforward network (FNN), which undergoes a first-layer linear transformation and ReLU activation function, followed by a second-layer linear transformation and Dropout, then residual connections. Finally, the output is the final fused feature after multimodal attention fusion and feedforward network enhancement. .

[0144] Furthermore, such as Figure 5 As shown, the specific steps of step S5 include the following:

[0145] S51. In order to preserve spatial information, the fused features Add sine position encoding:

[0146] ;

[0147] ;

[0148] Where pos is the position index. It's a dimensional index. The input after adding position encoding is: .

[0149] S52. Next, we design two independent decoders for pose estimation and localization regression, respectively:

[0150] S521, Pose Regression Branch: From Output Features The coordinates of the 3D key points of the human body are decoded.

[0151] First, initialize a set of learnable parameters as a query, called attitude query:

[0152] ;

[0153] Where K represents the number of people and J represents the number of key points.

[0154] Then, key information is extracted from the encoded features through multi-layer cross-attention:

[0155] ;

[0156] ;

[0157] Finally, after multi-layer decoding, the 3D coordinates of each joint will be regressed using MLP:

[0158] ;

[0159] ;

[0160] in, Indicates the first The first of the individuals 3D coordinates of each joint It returns to the network and ultimately obtains the attitude output. .

[0161] S522, Localization Regression Branch: The goal of the localization decoder is to extract features... The 3D position of the human body is decoded.

[0162] First, a set of learnable parameters is initialized as the query, called the location query. ;

[0163] ;

[0164] Where K represents the number of people, each You can query the spatial location of a specific person;

[0165] Then, spatial attention is decoded:

[0166] ;

[0167] ;

[0168] Finally, after multi-layer decoding, each query will be mapped to 3D coordinates via MLP:

[0169] ;

[0170] in, Indicates the first The individual's 3D position is ultimately used to obtain the positioning output. .

[0171] S53. Introduce a cross-task attention mechanism for information exchange between the pose regression branch and the localization regression branch:

[0172] ;

[0173] ;

[0174] This interaction allows pose estimation to be aware of the global location context, and allows location estimation to benefit from human structural information.

[0175] Furthermore, in S6, the pose and position are adjusted by optimizing the total loss function, which consists of pose loss, localization loss and attention sparse regularization term, and finally outputs the 3D pose skeleton and 3D position of multiple people.

[0176] like Figure 5 As shown, the specific operation process of step S6 is as follows:

[0177] Attention sparsity regularization:

[0178] ;

[0179] ;

[0180] ;

[0181] in, Sparsity measures based on entropy It is an L1 regularization term. It is attention weight. It is the sequence length of the attention weights. This represents the loss function of the attention module, which implements regularization of the attention mechanism. It is the number of attention layers. It refers to the number of heads receiving multi-head attention;

[0182] The total loss function consists of pose loss, localization loss, and attention sparse regularization term:

[0183] ;

[0184] ;

[0185] ;

[0186] Among them, attitude loss It is the mean square error between the predicted key coordinates and the actual coordinates, and the positioning loss. It is the mean square error between the predicted position and the actual position. The pose and position are adjusted by optimizing the loss, and finally the 3D pose skeleton and 3D position of multiple people are output.

[0187] On the other hand, providing a basis A multi-person 3D human pose and spatial positioning joint reconstruction system based on radio frequency field and multi-head attention mechanism, including CSI acquisition and preprocessing module, video assistance module, The module includes an RF field modeling module, a multimodal fusion module, a dual-branch joint optimization module, and an inference module; among them,

[0188] The CSI acquisition and preprocessing module is used to acquire multi-dimensional time-series CSI signals and perform standardization and pseudo-image processing to form structured radio frequency features.

[0189] The video assistance module is used to extract the structured visual features of the human body in real time using a monocular or binocular camera. It aligns the visual feature data with CSI data through timestamps and spatial calibration. The aligned visual feature data will be input into the subsequent multimodal fusion module.

[0190] The radio frequency field modeling module is used to initialize the radio frequency field using the radio frequency features. Radio frequency field modeling is used to construct a continuous implicit field that describes the radio frequency propagation characteristics in three-dimensional space. Signal synthesis is performed through volume rendering technology to output radio frequency volume rendering features.

[0191] The multimodal fusion module is used to deeply fuse the human structured visual features and the radio frequency volume rendering features under a cross-modal attention mechanism using a Transformer-based multimodal attention fusion method to generate the final fused features.

[0192] The dual-branch joint optimization module constructs a dual-branch joint optimization network based on fused features, performing pose regression and spatial localization regression respectively, and realizing information interaction and joint optimization between the two tasks through a multi-head attention mechanism;

[0193] The inference module, based on the joint optimization results, outputs the 3D pose skeleton and 3D spatial position of each target through dual decoders, and performs end-to-end optimization using pose loss, localization loss and attention sparse regularization term.

[0194] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multi-person three-dimensional human pose and spatial positioning joint reconstruction method based on NeRF2 radio frequency field and multi-head attention mechanism, characterized in that, The specific steps include the following: S1, collect multi-dimensional time sequence CSI signals and perform standardization and pseudo-image processing to form structured radio frequency features; S2, use a monocular or binocular camera to extract human structured visual features in real time, align with the CSI data through timestamp and spatial calibration, and input the aligned visual feature data to a multi-modal fusion module; S3, input the radio frequency features into a NeRF2 radio field modeling module to construct a continuous implicit field describing the radio propagation characteristics of a three-dimensional space, synthesize signals through volume rendering technology, and output radio volume rendering features; S4, use a multi-modal attention fusion method based on Transformer to deeply fuse the human structured visual features and the radio volume rendering features under a cross-modal attention mechanism to generate final fusion features; S5, based on the fusion features, construct a dual-branch joint optimization network to perform pose regression and spatial positioning regression respectively, and realize information interaction and joint optimization of the two tasks through a multi-head attention mechanism; S6, according to the joint optimization result, output the 3D pose skeleton and 3D spatial position of each target through a double decoder, and perform end-to-end optimization using pose loss, positioning loss, and attention sparsity regularization terms.

2. The multi-person 3D human pose and spatial localization joint reconstruction method based on NeRF2 radio frequency field and multi-head attention mechanism according to claim 1, wherein, The standardization processing of the multi-dimensional time sequence CSI signals includes filtering and noise reduction, phase correction, and energy normalization. Specifically, the filtering and noise reduction is performed using a combination of wavelet threshold denoising and Kalman filtering, and the phase correction is performed using a linear phase bias elimination method.

3. The multi-person 3D human pose and spatial localization joint reconstruction method based on NeRF2 radio frequency field and multi-head attention mechanism according to claim 1, characterized in that, The pseudo-image processing method for the multi-dimensional time sequence CSI signals is to reconstruct the multi-dimensional time sequence CSI signals into a two-dimensional feature map through time-space-frequency three-dimensional mapping to form a structured radio frequency feature representation.

4. The multi-person 3D human pose and spatial localization joint reconstruction method based on NeRF2 radio frequency field and multi-head attention mechanism according to claim 1, wherein, Step S2 includes the following: S21, select a camera type according to the application scenario, perform lens calibration, establish a unified world coordinate system, collect original images by the camera, perform real-time lens distortion correction, input the corrected images into a pose estimation model, and output human structured visual features; S22, use hardware triggering and software timestamps to align video frames and CSI data, and establish a unified space-time mapping relationship through joint calibration; S23, package the human structured visual features with timestamps and camera poses, and pair them with CSI data according to the space-time mapping relationship.

5. The multi-person 3D human pose and spatial localization joint reconstruction method based on NeRF2 radio frequency field and multi-head attention mechanism according to claim 1, wherein, The specific steps for NeRF2 radio field modeling to construct a continuous implicit field describing the radio propagation characteristics of a three-dimensional space are as follows: S31, arrange WiFi transmitting ends and several CSI receiving ends in the scene to establish a radio space implicit field, and implement the radio field implicit function through an MLP network: S32, use volume rendering technology to synthesize the signal strength observed at a specific receiving end from the continuous radio field, and perform integration along the ray path from the transmitting end to the receiving end to calculate the total signal strength at the receiving end: S33, optimize the network parameters by minimizing the difference between the predicted signal and the actual observed signal through a loss function; S34, judge whether the loss function converges, if converges, fix the MLP parameters, output the optimized continuous implicit field.

6. The multi-person 3D human pose and spatial localization joint reconstruction method based on NeRF2 radio frequency field and multi-head attention mechanism according to claim 1, wherein, The specific steps for obtaining the final fusion feature are: S41, taking the human structured visual feature as a query Q, the radio body rendering feature as a key K and a value V, and performing fusion through a multi-head attention mechanism to obtain an enhanced fusion feature; S42, projecting the features of different modalities to a unified dimension through linear projection to obtain new Q, K, and V; S43, dividing the projected Q, K, and V into h heads, and calculating the attention of each head in parallel; S44, concatenating all the heads and obtaining the multi-head attention output through linear projection; S45, after the multi-head attention, performing residual connection and layer normalization to output the final fusion feature after multi-modal attention fusion and feedforward network enhancement.

7. The multi-person 3D human pose and spatial localization joint reconstruction method based on NeRF2 radio frequency field and multi-head attention mechanism according to claim 1, wherein, The specific steps for constructing the dual-branch joint optimization network are: S51, in order to retain spatial information, adding sinusoidal position encoding to the final fusion feature to obtain an encoded fusion feature; S52, constructing a pose regression branch to decode the 3D joint coordinates of the human body from the encoded fusion feature, and constructing a positioning regression branch to decode the spatial position coordinates from the encoded fusion feature; S53, introducing a cross-task attention mechanism between the pose regression branch and the positioning regression branch for information interaction.

8. The multi-person 3D human pose and spatial positioning joint reconstruction method based on NeRF2 radio frequency field and multi-head attention mechanism according to claim 1, wherein, In S6, the pose and position are adjusted by optimizing the loss through a total loss function composed of a pose loss, a positioning loss, and an attention sparsity regularization term, and finally the 3D pose skeleton and 3D position of multiple persons are output.

9. A multi-person 3D human pose and spatial positioning joint reconstruction system based on NeRF2 radio frequency field and multi-head attention mechanism, characterized in that, It comprises a CSI acquisition and preprocessing module, a video auxiliary module, a NeRF2 radio field modeling module, a multi-modal fusion module, a dual-branch joint optimization module, and an inference module. The CSI acquisition and preprocessing module is used to acquire multi-dimensional time-series CSI signals and perform preprocessing to form structured radio features. The video auxiliary module is used to extract human structured visual features in real time and align them with CSI data to output aligned visual features. The NeRF2 radio field modeling module is used to construct a continuous three-dimensional radio implicit field based on NeRF2 and output radio body rendering features. The multi-modal fusion module is used to perform deep fusion of the aligned visual features and the radio body rendering features under a cross-modal attention mechanism to generate a final fusion feature. The dual-branch joint optimization module is used to construct a dual-branch joint optimization network, perform pose regression and spatial positioning regression respectively, and realize information interaction and joint optimization of the two tasks through a multi-head attention mechanism. The inference module is used to output the 3D pose skeleton and 3D spatial position of each target through a double decoder according to the joint optimization result and perform end-to-end optimization.

Images