A three-dimensional sleeping posture detection method based on RFID equipment
Through RFID tag arrays and convolutional neural networks, combined with Kinect V2 devices, the problem of inaccurate identification of static three-dimensional sleeping positions in the existing technology is solved, high-precision and stable three-dimensional sleeping positions detection is achieved, and more abundant sleep information is provided.
Patent Information
- Application Number
- CN202211018800.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-24
- Publication Date
- 2025-08-22
- Estimated Expiration
- 2042-08-24
AI Technical Summary
The prior art cannot accurately identify three-dimensional sleeping positions, especially during static sleeping, which cannot provide fine-grained three-dimensional human joint position information, and wireless signals are susceptible to external environment interference, resulting in instability.
Using RFID tag array combined with Kinect V2 devices, RSSI, phase and binary information are extracted through data segmentation, preprocessing and convolutional neural network models, a three-dimensional sleeping posture detection method is constructed, and the data augmentation and attention mechanisms are used to improve recognition accuracy and stability.
It realizes high-precision and stable three-dimensional sleeping posture detection, overcomes the problems of wireless signal resolution limitation and environmental interference, and provides richer sleep information for health monitoring and disease diagnosis.
Smart Images
Figure CN115660042B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of wireless signal three-dimensional posture perception, and relates to a three-dimensional sleeping posture detection method based on RFID equipment. Background Art
[0002] In sleep health monitoring, sleeping posture has important significance and impact on sleep quality, physical health, disease detection, and auxiliary treatment. Existing sleep posture monitoring methods mostly focus on sleeping posture classification, such as supine, prone, and side sleeping. However, a single sleeping posture classification lacks richer body posture information during sleep. In fact, three-dimensional position information of key body parts such as the torso, legs, arms, and head is not only important for achieving comfortable sleep, but also for disease diagnosis. 3D sleeping posture displays a sleeping posture hologram from multiple angles, which can identify more natural and realistic sleeping postures. At the same time, it provides rich information for long-term sleep monitoring and disease diagnosis, and has strong practical significance and value.
[0003] Existing wireless sensing sleep posture detection can only provide coarse-grained posture classification. Due to the resolution limitations of low-frequency wireless devices, it is unable to provide fine-grained three-dimensional positioning of human joints. For example, wireless devices such as Wi-Fi, FMCW, and RFID are deployed around the bed to monitor sleep posture by analyzing the changes in wireless signal characteristics generated by different sleeping postures, including signal reception strength (RSS), channel state information (CSI), and phase. However, these methods focus on identifying sleeping posture classification or determining body orientation and are unable to accurately identify sleeping postures in three dimensions. Furthermore, wireless signals are easily interfered with by the external environment, making stability and robustness difficult to achieve.
[0004] In addition, some work has explored the feasibility of inferring the human skeleton using the reflection principle of wireless signals. For example, devices such as Wi-Fi, RF signals, and millimeter waves are deployed around a person and the 3D posture is extracted by capturing the reflection of the signals under human movement. However, such methods rely on the person walking and moving to achieve good accuracy. In contrast, the posture of a person during sleep remains static for most of the time, so these methods are not suitable for static 3D sleeping postures. Summary of the Invention
[0005] The purpose of the present invention is to provide a three-dimensional sleeping posture detection method based on RFID equipment to solve the technical problems existing in the prior art that three-dimensional sleeping posture cannot be accurately identified and is not applicable to static three-dimensional sleeping posture recognition.
[0006] In order to achieve the above object, the technical solution adopted by the present invention is:
[0007] A three-dimensional sleeping posture detection method based on RFID equipment specifically comprises the following steps:
[0008] Step 1: Deploy RFID tag arrays for data collection and simultaneously collect point cloud data;
[0009] Step 2: Data segmentation and elimination: The RFID data collected in step 1 is segmented using multiple time windows. The fluctuation of the tag array signal in each time window is counted to eliminate the data caused by turning over during sleep.
[0010] Step 3: Data preprocessing: perform RF image conversion and preprocessing on the data in each time window obtained in step 2; divide the obtained data into training set and test set;
[0011] Step 4: Perform data enhancement on the RSSI matrix, phase matrix, and binarization matrix in the training set obtained in step 3 to obtain an enhanced training set;
[0012] Step 5: Build a convolutional neural network model and extract data features from the data obtained in steps 3 and 4. After training, a trained convolutional neural network model is obtained. The data to be tested is input into the model to obtain a three-dimensional sleeping posture.
[0013] Furthermore, the operation of step 1 is as follows: deploy a 28*21 RFID tag array, place the RFID tag array on the mattress, use an antenna to place it above the bed, and the distance between the antenna and the tag array is 1.6m to 2.3m; at the same time, use Kinect V2 to collect synchronized point cloud information and mark the three-dimensional sleeping posture.
[0014] Furthermore, step 2 includes the following sub-steps:
[0015] Step 2.1: Data segmentation: The RFID data collected in step 1 is segmented into 3-second time windows. The data in each time window is represented as follows:
[0016] rssi m,n (t) = {rssi 1 m,n ,rssi 2 m,n ,rssi 3 m,n ,...,rssi i m,n}#(1)
[0017] phase m,n (t) = {phase 1 m,n ,phase 2 m,n ,phase 3 m,n ,...,phasei m,n}#(2)
[0018] Among them, rssi m,n (t) represents the RSSI sequence value of the tag at position m,n in the tth time window, phase m,n (t) represents the sequence value of the phase of the tag at position m,n in the t-th time window, i represents the number of times the tag at position m,n is read in t time windows;
[0019] Step 2.2: Data removal: Calculate the sum of the variances of the RSSI sequences and phase sequences of all unblocked tags in each time window, remove the data in the time window greater than the threshold, and retain the data in the time window within the threshold.
[0020] Furthermore, in step 2.2, the threshold of the RSSI sum of variance is set to 5, and the threshold of the phase sum of variance is set to 8.
[0021] Furthermore, step 3 includes the following sub-steps:
[0022] Step 3.1: Use Formula 3 and Formula 4 to calculate the mean of the RSSI sequence and phase sequence of the unblocked tag in each time window obtained in Step 2. At the same time, assign the blocked tag's RSSI mean value to -100 and the phase mean value to 8. Then, combine the tag array information to obtain the RSSI matrix and phase matrix:
[0023]
[0024]
[0025] in, Indicates the mean of the rssi sequence with label position m,n in a certain time window. Indicates the mean of the phase sequence with label position m,n in a certain time window;
[0026] Step 3.2: Binarize the RSSI matrix obtained in step 3.1, set the threshold to -100, and set the values of the elements in the RSSI matrix that are greater than the threshold to zero, and those that are equal to the threshold to 1, to obtain the binary matrix:
[0027]
[0028] Among them, b m,n Indicates the binary data with label position m,n in a certain time window;
[0029] Step 3.3: De-noise the phase matrix and RSSI matrix obtained in step 3.1 and the binarized matrix obtained in step 3.2 to obtain the denoised RSSI matrix, phase matrix and binarized matrix;
[0030] Step 3.4: Use Formula 6 and Formula 7 to normalize the denoised RSSI matrix and phase matrix obtained in Step 3.3, respectively, to obtain the normalized RSSI matrix, phase matrix, and binarization matrix:
[0031]
[0032]
[0033] Among them, rssi m,n is the rssi value of the label position m,n after denoising, rssi min is the minimum value of the rssi matrix after denoising, rssi max is the maximum value of the rssi matrix after denoising, phase m,n is the phase value of the label position m,n after denoising, phase min is the minimum value of the phase matrix after denoising, phase max is the maximum value of the phase matrix after denoising;
[0034] Step 3.5: Use the bilinear interpolation method to double upsample the normalized RSSI matrix, phase matrix, and binarization matrix obtained in step 3.4, and expand the matrix size to twice the original size to obtain the processed RSSI matrix, phase matrix, and binarization matrix; divide the obtained data into a training set and a test set.
[0035] Furthermore, in step 3.3, a Gaussian distribution function is used for denoising.
[0036] Furthermore, step 4 includes the following sub-steps:
[0037] Step 4.1: Mirror the RSSI matrix, phase matrix, and binarization matrix obtained in step 3 horizontally.
[0038] Step 4.2: Perform size transformation on the data obtained in step 3;
[0039] Step 4.3: Repeat the processing of step 3.5 on the RSSI matrix, phase matrix and binarization matrix obtained in steps 4.1 and 4.2 to obtain the enhanced RSSI matrix, phase matrix and binarization matrix.
[0040] Furthermore, the specific operation of step 4.3 is as follows: the two-dimensional mapping space of the sleeping posture is determined by the area where the value 1 is located in the binary matrix obtained in step 3.4; then, the left and right boundary ranges of the sleeping posture are found according to the two-dimensional mapping space of the sleeping posture, and the data outside the boundary are cut to change the binary matrix from 28*21 to 28*n, where n<21; at the same time, the rssi matrix and phase matrix obtained in step 3.4 are cut synchronously to obtain rssi matrix and phase matrix of size 28*n; then, the rssi matrix, phase matrix and binary matrix of size 28*n are spliced on both sides respectively The matrix of 28*n size RSSI matrix, phase matrix and binarization matrix is changed to 28*21 size.
[0041] Furthermore, step 5 includes the following sub-steps:
[0042] Step 5.1: Build a convolutional neural network in a PyTorch-based environment: First, build a 2D convolutional layer and introduce an attention mechanism. Then, connect two 2D convolutional layers to combine the 2D convolutional layer and the attention mechanism. Then, expand the output of the 2D convolutional layer to 3D and connect it to the 6-layer 3D convolutional layer. Finally, connect the 3D convolutional layer to the fully connected layer to build the convolutional neural network model.
[0043] Step 5.2: Set the loss function in the training neural network and use the Adam optimizer with adaptive learning rate to train the network model;
[0044] Step 5.3: Use the enhanced training set obtained in step 4 to train the constructed convolutional neural network model to obtain a trained convolutional neural network model.
[0045] Step 5.4: Use the test set to input the trained model obtained in step 5.3 and output the detected 3D sleeping posture results.
[0046] Furthermore, the step 5.1 includes the following operations:
[0047] Step 5.1.1, construct a two-dimensional convolution layer with a convolution kernel size of 3×3, a padding of 1×1, a stride of 1, an output channel number of 64, an activation function of LeakyReLU, and use BatchNorm2d to normalize the data; then connect the channel attention model, and then connect six two-dimensional convolution layers; the structure of the six two-dimensional convolution layers: the convolution kernel size of the first convolution layer is 3×3, the padding is 1×1, the stride is 1, the output channel number is 128, the activation function is LeakyReLU, use BatchNorm2d to normalize the data, and use a maximum pooling layer with a size of 2 and a stride of 2; the convolution kernel size of the second convolution layer is 3×3, the padding is 1×1, the stride is 1, the output channel number is 256, the activation function is LeakyReLU, and use BatchNorm2d to normalize the data; the convolution kernel size of the third convolution layer is 3×3, the padding is 1×1, and the stride is The length is 1, the number of output channels is 256, the activation function is LeakyReLU, the data is normalized using BatchNorm2d, and a maximum pooling layer with a size of 2 and a stride of 2 is used; the convolution kernel size of the fourth convolution layer is 3×3, the padding is 1×1, the stride is 1, the number of output channels is 512, the activation function is LeakyReLU, and the data is normalized using BatchNorm2d; the convolution kernel size of the fifth convolution layer is 3×3, the padding is 1×1, the stride is 1, the number of output channels is 1024, the activation function is LeakyReLU, and the data is normalized using BatchNorm2d; the convolution kernel size of the sixth convolution layer is 3×3, the padding is 1×1, the stride is 1, the number of output channels is 1024, the activation function is LeakyReLU, the data is normalized using BatchNorm2d, and a maximum pooling layer with a size of 2 and a stride of 2 is used;
[0048] In step 5.1.2, the output of the two-dimensional convolutional layer is expanded to three dimensions and connected to six three-dimensional convolutional layers; the structure of the six three-dimensional convolutional layers is as follows: the convolution kernel size of the first three-dimensional convolutional layer is 3×3×3, the padding is 1×1×1, the step size is 1, the number of output channels is 1025, the activation function is LeakyReLU, and the data is normalized using BatchNorm2d; the convolution kernel size of the second three-dimensional convolutional layer is 3×3×3, the padding is 1×1, the step size is 1, the number of output channels is 512, the activation function is LeakyReLU, and the data is normalized using BatchNorm2d; the third three-dimensional convolutional layer uses deconvolution, the convolution kernel size is 2×2×2, the padding is 1×1×1, the step size is 1, and the output channels are 1025. The number of channels is 256, the activation function is LeakyReLU, and BatchNorm2d is used to normalize the data; the convolution kernel size of the fourth convolution layer is 3×3×3, the padding is 1×1×1, the stride is 1, the number of output channels is 128, the activation function is LeakyReLU, and BatchNorm2d is used to normalize the data; the convolution kernel size of the fifth convolution layer is 3×3×3, the padding is 1×1×1, the stride is 1, the number of output channels is 64, the activation function is LeakyReLU, and BatchNorm2d is used to normalize the data; the convolution kernel size of the sixth convolution layer is 3×3×3, the padding is 1×1×1, the stride is 1, the number of output channels is 14, and the activation function is Sigmoid;
[0049] In step 5.1.3, the three-dimensional convolutional layer is connected to the fully connected layer to construct a convolutional neural network model; the fully connected layer is constructed using three layers, the first fully connected layer has 4096 nodes, the activation function is ReLU, and the Dropout is 0.5; the second fully connected layer has 4096 nodes, the activation function is ReLU, and the Dropout is 0.5; the third fully connected layer has 42 nodes, and the final result is obtained by output.
[0050] Furthermore, the loss function constructed in step 5.2 is:
[0051]
[0052] Among them, L m Represents the loss value in training the convolutional neural network, N represents the joint number, P i Represents the three-dimensional coordinates of the detected i-th joint, G i Represents the 3D coordinates of the i-th joint in the groundtruth.
[0053] Compared with the prior art, the present invention has the following beneficial effects:
[0054] The present invention extracts RSSI, phase, and binarization information from blocked and unblocked RFID tags in a three-dimensional sleeping posture. It then uses a convolutional neural network model to extract 3D human joint points from these RSSI, phase, and binarization information to obtain 3D sleeping posture detection results. Simultaneously, data enhancement methods are used to improve the accuracy, stability, and generalization of 3D sleeping posture skeleton detection. This overcomes the limitations of existing wireless signals, which cannot achieve fine-grained 3D sleeping posture detection, and the instability caused by wireless signal susceptibility to interference. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 is a flow chart of the method of the present invention;
[0056] Figure 2 are the rssi matrix, phase matrix and binarized matrix grayscale image;
[0057] Figure 3 is the image after data augmentation horizontal mirroring;
[0058] Figure 4 is the size of the data augmentation transform bed;
[0059] Figure 5 This is the framework diagram of the convolutional neural network;
[0060] Figure 6 It is the detection result of the method of the present invention. DETAILED DESCRIPTION
[0061] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0062] like Figure 1 As shown, the three-dimensional sleeping posture detection method based on RFID equipment of the present invention specifically includes the following steps:
[0063] Step 1: Deploy the RFID tag array and collect data. Based on the experimental requirements, deploy the RFID tag array, set up the data collection scenario, and collect RFID data to construct a dataset. Simultaneously, use a Kinect V2 to simultaneously collect point cloud data for 3D sleep position labeling.
[0064] Specifically, we deployed a 28x21 array of H47 RFID tags, each with its own unique location and name. We then placed the RFID tag array on the mattress and placed a circularly polarized antenna with a 915 MHz frequency and 8 dBi gain above the bed, with the antenna and tag array positioned between 1.6 and 2.3 meters apart. We also used a Kinect V2 to collect synchronized point cloud data and annotate 3D sleeping postures. This was used to train a supervised convolutional neural network model and compare 3D sleeping posture detection results.
[0065] In this embodiment, data collection was performed on 36 subjects, and 25,000 data were collected, of which 20,000 data were used as a training set for training the convolutional neural network, and 5,000 data were used as a test set for testing the convolutional neural network.
[0066] Advantages of this step: This step generates a dataset suitable for 3D sleeping posture recognition using RFID wireless signals, meeting the data requirements of the method of the present invention. Furthermore, during the data collection process, external interference factors can be identified, allowing us to fundamentally improve the robustness of the method.
[0067] Step 2: Data segmentation and elimination: Segment the RFID data collected in step 1 using multiple time windows, and count the fluctuations in the tag array signal in each time window to eliminate data from sleep tossing. This includes the following sub-steps:
[0068] Step 2.1: Data Segmentation. Based on the effect of sleeping posture on RFID tags, RFID tags are divided into blocking tags and non-blocking tags. Blocking tags are tags that fail to receive data due to the influence of sleeping posture. Non-blocking tags are tags that can still receive signals through the antenna despite the influence of sleeping posture. Tags collected in Step 1 are non-blocking tags. The RFID data collected in Step 1 is segmented into 3-second time windows. The data in each time window is represented as:
[0069] rssi m,n (t) = {rssi 1 m,n ,rssi 2 m,n ,rssi 3 m,n ,...,rssi i m,n}#(1)
[0070] phase m,n (t) = {phase 1 m,n ,phase 2 m,n,phase 3 m,n ,...,phase i m,n}#(2)
[0071] Among them, rssi m,n (t) represents the RSSI sequence value of the tag at position m,n in the tth time window, phase m,n (t) represents the sequence value of the phase of the tag at position m,n in the t-th time window, and i represents the number of times the tag at position m,n is read in t time windows.
[0072] Step 2.2: Data Removal. Calculate the sum of the variances of the RSSI sequences and phase sequences of all unblocked tags in each time window, remove the data in time windows greater than the threshold, and retain the data in time windows within the threshold.
[0073] Specifically, through step 2.1, we obtain the information of all non-blocking tags read in each time window (including the tag name, RSSI information, and phase information). The RSSI sequence and phase sequence of all tags in each time window can be retrieved through the tag name.
[0074] For step 2.2, considering that most sleepers are in a relatively static state, it is necessary to eliminate data from turning over and retain the static posture data. Because wireless signals are interfered with by the environment, when a person turns over, the values of the RSSI sequence and phase sequence fluctuate greatly. Variance is an evaluation indicator that reflects the magnitude of the value fluctuation. In this case, the sum of the variances of the RSSI sequences and the phase sequences of all tags in the time window are statistically calculated, and then the data of the corresponding time windows where the RSSI or phase variances are greater than the threshold are eliminated. In this embodiment, based on experimental data, the threshold for the RSSI variance is set to 5, and the threshold for the phase variance is set to 8.
[0075] The advantages of this step are: by dividing the data into time windows, the collected data can be further refined, which is convenient for data processing. At the same time, the redundant data brought by the turning over process is eliminated, ensuring that the data in the data set are all static sleeping posture data, thus ensuring the accuracy of the experiment.
[0076] Step 3: Data preprocessing: Perform RF image conversion and preprocessing on the data in each time window obtained in step 2. This includes the following sub-steps:
[0077] Step 3.1: Using Formula 3 and Formula 4, calculate the mean of the RSSI sequence and phase sequence of the unblocked tag in each time window obtained in Step 2. At the same time, assign the RSSI mean of the blocked tag to -100 and the phase mean to 8. Then, combine the tag array information to obtain the RSSI matrix and phase matrix:
[0078]
[0079]
[0080] in, Indicates the mean of the rssi sequence with label position m,n in a certain time window. It represents the mean of the phase sequence with label position m,n in a certain time window.
[0081] The advantage of this step is that it can extract three-dimensional information about sleeping posture from the tag array. According to experimental data, the impact of the human body on the tag can be divided into blocked tags and unblocked tags. Blocked tags indicate that the human body completely blocks the communication between the tag and the antenna, while the signal of the unblocked tag can reflect the three-dimensional spatial information of the human body. The RSSI value of the unblocked tag is between -70 and -80, and the phase value of the unblocked tag is between 0 and 2π. By assigning the blocked tag a value outside the RSSI value range and phase value range of the blocked tag, we can more clearly distinguish the difference between the blocked tag and the unblocked tag. It also provides us with more accurate information for extracting three-dimensional posture information from the RSSI matrix and phase matrix. Combining the tag value with the tag's position information is more conducive to combining three-dimensional posture information with two-dimensional plane information, thereby improving the accuracy of three-dimensional sleeping posture.
[0082] Step 3.2: Binarize the RSSI matrix obtained in step 3.1, set the threshold to -100, and set the values of the elements in the RSSI matrix that are greater than the threshold to zero, and those that are equal to the threshold to 1, to obtain the binary matrix:
[0083]
[0084] Among them, b m,n Represents the binary data with label position m,n in a certain time window.
[0085] See also Figure 2 , is the grayscale image of the RSSI matrix, phase matrix and binarization matrix.
[0086] The advantage of this step is that the binarized matrix contains the 2D mapping information of the 3D posture. We can distinguish blocked and unblocked labels from the RSSI matrix values. The blocked label has a value of -100, while the unblocked label has a value between -80 and -70. Therefore, we set a threshold of -100. RSSI values greater than -100 are assigned to unblocked labels and a value of 0 in the binarized matrix, while RSSI values equal to -100 are assigned to blocked labels and a value of 1 in the binarized matrix. The binarized matrix reflects the area of the sleeping posture on the 2D plane and the 2D mapping information of the limbs on the plane.
[0087] Step 3.3: Use Gaussian distribution function The phase matrix and RSSI matrix obtained in step 3.1 and the binarized matrix obtained in step 3.2 are denoised respectively to obtain the denoised RSSI matrix, phase matrix and binarized matrix.
[0088] The advantage of this step is that it effectively removes noise from the RSSI matrix, phase matrix, and binarization matrix. Due to inherent communication issues with the tags, there are some blocked tags that are not affected by sleeping posture. We use Gaussian denoising to reduce the impact of this phenomenon.
[0089] Step 3.4: Use Formula 6 and Formula 7 to normalize the denoised RSSI matrix and phase matrix obtained in Step 3.3, respectively, to obtain the normalized RSSI matrix, phase matrix, and binarization matrix:
[0090]
[0091]
[0092] Among them, rssi m,n is the rssi value of the label position m,n after denoising, rssi min is the minimum value of the rssi matrix after denoising, rssi max is the maximum value of the rssi matrix after denoising, phase m,n is the phase value of the label position m,n after denoising, phase min is the minimum value of the phase matrix after denoising, phase max is the maximum value of the phase matrix after denoising.
[0093] The advantage of this step is that it reduces the differences in RSSI matrix values and phase matrix data under different time windows, which is conducive to better feature extraction in the input convolutional neural network.
[0094] Step 3.5: Use the bilinear interpolation method to double-upsample the normalized RSSI matrix, phase matrix, and binarization matrix obtained in step 3.4, increasing the matrix size to twice its original size. This yields the processed RSSI matrix, phase matrix, and binarization matrix. The resulting data is divided into a training set and a test set.
[0095] The advantage of this step is that it doubles the matrix size through bilinear interpolation, which helps improve the matrix resolution. Due to the size and deployment of the labels themselves, the resulting RSSI matrix, phase matrix, and binarized matrix have low resolution, which hinders feature extraction when input into a convolutional neural network. Bilinear interpolation upsampling the RSSI matrix, phase matrix, and binarized matrix by a factor of 2 improves resolution, facilitating input into a convolutional neural network to extract 3D pose information.
[0096] Step 4: Perform data augmentation on the RSSI matrix, phase matrix, and binarized matrix in the training set obtained in Step 3 to obtain the enhanced training set. Data augmentation specifically includes flipping and resizing to enhance the dataset and improve the stability and generalization ability of 3D sleeping skeleton detection. This includes the following sub-steps:
[0097] Step 4.1: Mirror the RSSI matrix, phase matrix, and binarization matrix obtained in step 3 horizontally. According to the horizontal mirroring formula:
[0098] a′ m,n =a m,n-k #(8)
[0099] where a′ m,n The label position after mirroring is m,n, and the RSSI matrix, phase matrix and binarization matrix after horizontal mirroring are obtained.
[0100] The advantages of this step are: expanding the data set and saving the time consumption of data collection. Figure 3 The first row of images is the data obtained in step 3.4 (source data), and the second row of images is the mirror-flipped data. As can be seen, by synchronously mirroring the RSSI matrix, phase matrix, and binarization matrix, and also mirror-flipping the posture, we expand the diversity of sleeping postures in our dataset and improve the detection results of the convolutional neural network.
[0101] Step 4.2: Label Array Transformation. Depending on the size of the bed, we can deploy label arrays of different sizes (28*n) to accommodate beds of varying sizes. In this method, the experimental data we collected was collected with a label matrix size of 28*21. We simulated the generation of different label matrix sizes by varying the size of the 28*21 label matrix, making this method applicable to beds of varying sizes.
[0102] Specifically, for a smaller label array, the size is 28*n, where n<21, we use the area where the value 1 is located in the binary matrix obtained in step 3.4 to determine the two-dimensional mapping space of the sleeping posture. Then, based on the two-dimensional mapping space of the sleeping posture, we find the left and right boundary ranges of the sleeping posture, cut the data outside the boundary, and change the binary matrix from 28*21 to 28*n. At the same time, the rssi matrix and phase matrix obtained in step 3.4 are cut synchronously to obtain a rssi matrix and phase matrix of size 28*n. Then, the rssi matrix, phase matrix and binary matrix of size 28*n are spliced on both sides. The 28*n size RSSI matrix, phase matrix and binarization matrix are changed to 28*21 size to ensure the same size of the input convolutional neural network.
[0103] The advantages of this step are: expanding the data set, saving the time consumption of data collection, and obtaining sleeping posture RFID data of different bed sizes for model training. Figure 5 , is the image after size transformation. Through matrix transformation, the fixed-size matrix data is converted into matrix data of different sizes, and then transformed into matrix data of 28*21 size. This is conducive to making the method of the present invention applicable to beds of different sizes and maintaining high-precision three-dimensional sleeping posture detection results. See Figure 3 The first row of images is the data obtained in step 3.4 (source data), and the second row of images is the RSSI matrix, phase matrix and binarization matrix data obtained after label size transformation.
[0104] Step 4.3: Repeat the process of step 3.5 (i.e., double upsampling) with the RSSI matrix, phase matrix, and binarization matrix obtained in steps 4.1 and 4.2 to obtain the final enhanced RSSI matrix, phase matrix, and binarization matrix. Ensure that the input size of the convolutional neural network is consistent.
[0105] The advantage of this step is that it improves the resolution of the enhanced data while ensuring that the input data size of the convolutional neural network is consistent.
[0106] Step 5: Build a convolutional neural network model and extract data features from the data obtained in steps 3 and 4 to obtain a three-dimensional sleeping posture. After training, a trained convolutional neural network model is obtained. The data to be tested is input into the model to obtain a three-dimensional sleeping posture. This includes the following sub-steps:
[0107] Step 5.1: Build a convolutional neural network in a PyTorch-based environment. First, build a 2D convolutional neural network and introduce the attention mechanism model. Then, build a 3D convolutional neural network and connect it to the 2D convolutional neural network. Finally, connect it to the fully connected layer. Set the output of the convolutional neural network.
[0108] like Figure 5 Specifically, it includes the following contents:
[0109] In step 5.1.1, first construct a 2D convolutional layer with a kernel size of 3×3, 1×1 padding, a stride of 1, 64 output channels, and a LeakyReLU activation function. Use BatchNorm2d to normalize the data. Then connect the channel attention model and six 2D convolutional layers to combine 2D convolution with the attention mechanism. The structure of the six two-dimensional convolutional layers: the convolution kernel size of the first convolutional layer is 3×3, the padding is 1×1, the stride is 1, the number of output channels is 128, the activation function is LeakyReLU, the data is normalized using BatchNorm2d, and a maximum pooling layer with a size of 2 and a stride of 2 is used; the convolution kernel size of the second convolutional layer is 3×3, the padding is 1×1, the stride is 1, the number of output channels is 256, the activation function is LeakyReLU, the data is normalized using BatchNorm2d; the convolution kernel size of the third convolutional layer is 3×3, the padding is 1×1, the stride is 1, the number of output channels is 256, the activation function is LeakyReLU, the data is normalized using BatchNorm2d, and a large The convolution kernel size of the fourth convolution layer is 3×3, the padding is 1×1, the stride is 1, the number of output channels is 512, the activation function is LeakyReLU, and BatchNorm2d is used to normalize the data; the convolution kernel size of the fifth convolution layer is 3×3, the padding is 1×1, the stride is 1, the number of output channels is 1024, the activation function is LeakyReLU, and BatchNorm2d is used to normalize the data; the convolution kernel size of the sixth convolution layer is 3×3, the padding is 1×1, the stride is 1, the number of output channels is 1024, the activation function is LeakyReLU, and BatchNorm2d is used to normalize the data, and a maximum pooling layer with a size of 2 and a stride of 2 is used.
[0110] In step 5.1.2, the output of the two-dimensional convolutional layer is expanded to three dimensions and connected to six three-dimensional convolutional layers. The structure of these six three-dimensional convolutional layers is as follows: the convolution kernel size of the first three-dimensional convolutional layer is 3×3×3, the padding is 1×1×1, the stride is 1, the number of output channels is 1025, the activation function is LeakyReLU, and BatchNorm2d is used to normalize the data; the convolution kernel size of the second three-dimensional convolutional layer is 3×3×3, the padding is 1×1, the stride is 1, the number of output channels is 512, the activation function is LeakyReLU, and BatchNorm2d is used to normalize the data; the third three-dimensional convolutional layer uses deconvolution, the convolution kernel size is 2×2×2, the padding is 1×1×1, the stride is 1, the number of output channels is 256, and the activation function is LeakyReLU. U, use BatchNorm2d to normalize the data; the convolution kernel size of the fourth convolution layer is 3×3×3, the padding is 1×1×1, the stride is 1, the number of output channels is 128, the activation function is LeakyReLU, and BatchNorm2d is used to normalize the data; the convolution kernel size of the fifth convolution layer is 3×3×3, the padding is 1×1×1, the stride is 1, the number of output channels is 64, the activation function is LeakyReLU, and BatchNorm2d is used to normalize the data; the convolution kernel size of the sixth convolution layer is 3×3×3, the padding is 1×1×1, the stride is 1, the number of output channels is 14, and the activation function is Sigmoid.
[0111] In step 5.1.3, the three-dimensional convolutional layer is connected to the fully connected layer to construct a convolutional neural network model suitable for the method of the present invention. The fully connected layer is constructed using three layers: the first fully connected layer has 4096 nodes, the activation function is ReLU, and the dropout is 0.5; the second fully connected layer has 4096 nodes, the activation function is ReLU, and the dropout is 0.5; the first fully connected layer has 42 nodes, and the final result is output.
[0112] The advantages of this step are: introducing a channel attention mechanism into the two-dimensional convolutional layer, combining the two-dimensional convolutional neural network with the attention mechanism, and extracting two-dimensional features from the RSSI matrix, phase matrix, and binarized matrix. The two-dimensional feature data is then converted into three-dimensional features, and the three-dimensional skeleton features are further extracted through the three-dimensional convolutional layer. The model introduces an attention mechanism, which can better extract more useful information from the blocked labels and unblocked labels in the RSSI matrix, phase matrix, and binarized matrix; then the two-dimensional information is converted into three-dimensional information in a hierarchical and progressive manner, which is more conducive to our extraction of high-precision three-dimensional skeletons. This step determines the structure and specific parameters of the convolutional neural network, ensuring the feasibility and reliability of the implementation of this method, and achieving high precision and stability in three-dimensional sleeping posture.
[0113] Step 5.2: Set the loss function in the training neural network and use the Adam optimizer with adaptive learning rate to train the network model. The loss function is expressed as:
[0114]
[0115] Among them, L m Represents the loss value in training the convolutional neural network. N represents the joint number, P i Represents the three-dimensional coordinates of the detected i-th joint, G i Represents the 3D coordinates of the i-th joint in the groundtruth.
[0116] The advantage of this step is that our goal is to optimize the neural network to obtain high-precision 3D sleeping posture detection results. Using the loss function and Adam to constrain the neural network training process can update the neural network parameters and optimize the neural network training process.
[0117] Step 5.3: Use the enhanced training set obtained in step 4 to train the constructed convolutional neural network model to obtain a trained convolutional neural network model.
[0118] Specifically, the convolutional neural network constructed takes three input channels: the RSSI matrix, the phase matrix, and the binarization matrix. All data in the training set is iterated 80 times to obtain a trained convolutional neural network model, which is used to detect 3D sleeping postures.
[0119] The advantage of this step is that the enhanced dataset is used to train a convolutional neural network for 3D sleeping posture detection, resulting in highly accurate and stable results. Furthermore, the trained convolutional neural network is applicable to 3D sleeping posture detection for beds of varying sizes.
[0120] Step 5.4: Use the test set to input the trained model obtained in step 5.3 and output the detected 3D sleeping posture results.
[0121] Specifically, to test the stability and applicability of the convolutional neural network trained by this method for 3D sleeping posture detection, our test set not only includes the test data divided in step 3.5 but also includes data from sleeping postures in different rooms and on beds of different sizes. This data is used to test the trained convolutional neural network, assessing its high accuracy, stability, and applicability to different rooms and beds.
[0122] The advantages of this step are: the availability, high accuracy, stability and applicability of the method of the present invention are evaluated through the test set, and the feasibility, reliability and stability of the method of the present invention are verified.
[0123] The following is the result of applying the method of the present invention to a data set to demonstrate the accuracy of its three-dimensional sleeping posture detection.
[0124] (1) The table shows the basic information of the dataset.
[0125] Table 1
[0126] Number of dataset samples Number of training samples Number of test samples 25000 20000 50000
[0127] We collected data from 36 people as a dataset, of which 29 people's data were used to train the model and 7 people's data were used to test the model.
[0128] (2) Evaluation criteria:
[0129] Following our specific implementation steps, we complete the 3D sleeping posture detection task. We use the mean joint position error (MPJPE) as a specific indicator of the 3D skeleton. The MPJPE formula is as follows:
[0130] E MPJPE (i)=||Pi-Gi||2
[0131] Among them, i represents the joint number, P i Represents the three-dimensional coordinates of the detected i-th joint, G i Represents the 3D coordinates of the i-th joint in the groundtruth.
[0132] MPJPE is an evaluation index for three-dimensional sleeping posture. The smaller the index, the higher the accuracy of the detected three-dimensional sleeping posture.
[0133] (3) Result analysis:
[0134] First, this method is to use RFID equipment to detect the three-dimensional sleeping skeleton and see if it can obtain a fine-grained three-dimensional skeleton. Through the detection of convolutional neural networks, we can obtain the results shown in the following table:
[0135] Table 2
[0136] joint Error (cm) joint Error (cm) head 6.88 neck 5.05 left shoulder 7.05 left hip 6.56 left elbow 11.08 left knee 7.63 left wrist 16.44 left ankle 10.06 right shoulder 7.00 right hip 6.60 right elbow 10.48 right knee 7.82 right wrist 14.67 right ankle 9.70
[0137] The final accuracy of the skeleton detection method of the present invention is 9.07cm. As shown in Table 2, the average error of each joint of the three-dimensional sleeping posture detected by this method can be concluded from the data in the table that the errors of most joints can be kept within 10cm, while the errors of the elbows and wrists are slightly larger. This is because the degrees of freedom of the elbows and wrists are relatively high compared to other joints. At the same time, because the arms are thinner, the impact on the signal is smaller, so the error is larger than that of other joints. Figure 6 The figure shows the final result of this method. We can see that this method can generate a detailed three-dimensional sleeping posture.
[0138] Secondly, to investigate the robustness and generalization of our method, we applied data augmentation to adapt the method to beds of different sizes and in different rooms. We compared the results for different rooms and beds of different sizes with and without data augmentation.
[0139] The following table shows the accuracy of sleeping postures in different rooms with and without data augmentation:
[0140] Table 3
[0141]
[0142]
[0143] As shown in Table 3, when no data augmentation was used, the error was large and the accuracy was low in different rooms. However, after using data augmentation, the error was small and the accuracy was significantly improved in both rooms. This shows that data augmentation improves the stability and generalization ability of this method, allowing it to be applied to different rooms while still maintaining high-precision detection results.
[0144] We collected sleeping posture data for beds of different sizes in the same room to evaluate the stability and applicability of this method. The size of the label array varies with bed size, so sleeping postures on beds of different sizes have a significant impact on the results of this method. We used data augmentation to reduce the impact of bed size on the accuracy of this method's 3D sleeping posture detection. The following table shows the accuracy of the results for different bed sizes with and without data augmentation:
[0145] Table 4
[0146] Error without data enhancement (cm) Error after data enhancement (cm) 180cm 13.05 10.07 150cm 9.98 9.07 80cm 29.26 9.34
[0147] Table 4 shows the detection results of the three-dimensional sleeping posture under beds of different sizes by the method of the present invention, and also shows the detection results of the three-dimensional sleeping posture under beds of different sizes after data enhancement. In the table, we can clearly see that without using data enhancement, our accuracy for a bed of 150 cm is within 10 cm, while the three-dimensional sleeping posture results for beds of 180 cm and 80 cm are poor, indicating that the method cannot accurately detect the three-dimensional sleeping posture under beds of different sizes without using data enhancement. After using data enhancement, it can be seen that the three-dimensional sleeping posture results for beds of 180 cm and 80 cm are greatly improved, and are basically stable at about 10 cm, indicating that data enhancement enables the method of the present invention to have stable detection results for three-dimensional sleeping postures on beds of different sizes.
Claims
1. A three-dimensional sleeping posture detection method based on RFID equipment, characterized in that: The specific steps include: Step 1: Deploy RFID tag arrays for data collection and simultaneously collect point cloud data; Step 2: Data segmentation and elimination: The RFID data collected in step 1 is segmented using multiple time windows. The fluctuation of the tag array signal in each time window is counted to eliminate the data caused by turning over during sleep. Step 3: Data preprocessing: perform RF image conversion and preprocessing on the data in each time window obtained in step 2; divide the obtained data into training set and test set; Step 4: Perform data enhancement on the RSSI matrix, phase matrix, and binarization matrix in the training set obtained in step 3 to obtain an enhanced training set; It includes the following sub-steps: Step 4.1: Mirror the RSSI matrix, phase matrix, and binarization matrix obtained in step 3 horizontally. Step 4.2: Perform size transformation on the data obtained in step 3; Step 4.3: Repeat the process of step 3.5 for the RSSI matrix, phase matrix and binarization matrix to obtain the enhanced RSSI matrix, phase matrix and binarization matrix; the specific operations are as follows: The two-dimensional mapping space of the sleeping posture is determined by the area where the value 1 is located in the obtained binary matrix; then, the left and right boundary ranges of the sleeping posture are found according to the two-dimensional mapping space of the sleeping posture, and the data outside the boundary are cut to change the binary matrix from 28*21 to 28*n, where n<21; at the same time, the RSSI matrix and phase matrix obtained in step 3.4 are cut synchronously to obtain a 28*n RSSI matrix and phase matrix; then, the RSSI matrix, phase matrix and binary matrix of size 28*n are spliced on both sides respectively. Matrix, change the 28*n size RSSI matrix, phase matrix and binarization matrix into 28*21 size; Step 3.4: Use the following two formulas to normalize the denoised RSSI matrix and phase matrix respectively to obtain the normalized RSSI matrix, phase matrix and binarization matrix: in, is the RSSI value of the tag position m,n after denoising, is the minimum value of the rssi matrix after denoising, is the maximum value of the rssi matrix after denoising, is the phase value of the label position m,n after denoising, is the minimum value of the phase matrix after denoising, is the maximum value of the phase matrix after denoising; Step 3.5: using a bilinear interpolation method, performing two-fold upsampling on the normalized RSSI matrix, phase matrix, and binarization matrix obtained in step 3.4, respectively, to expand the size of the matrix to twice the original size, thereby obtaining processed RSSI matrix, phase matrix, and binarization matrix; and dividing the obtained data into a training set and a test set; Step 5: Build a convolutional neural network model and extract data features from the data obtained in steps 3 and 4. After training, a trained convolutional neural network model is obtained. The data to be tested is input into the model to obtain a three-dimensional sleeping posture.
2. The RFID-based three-dimensional sleeping posture detection method according to claim 1, wherein: The operation of step 1 is as follows: deploy a 28*21 RFID tag array, place the RFID tag array on the mattress, and use an antenna placed above the bed. The distance between the antenna and the tag array is 1.6m~2.3m; at the same time, use Kinect V2 to collect synchronized point cloud information and mark the three-dimensional sleeping posture.
3. The RFID-based three-dimensional sleeping posture detection method according to claim 2, wherein: The step 2 includes the following sub-steps: Step 2.1: Data segmentation: The RFID data collected in step 1 is segmented into 3-second time windows. The data in each time window is represented as follows: in, Indicates the RSSI sequence value of the tag at position m,n in the t-th time window, It represents the sequence value of the phase of the tag at position m,n in the t-th time window, and i represents the number of times the tag at position m,n is read in t time windows; Step 2.2: Data removal: Calculate the sum of the variances of the RSSI sequences and phase sequences of all unblocked tags in each time window, remove the data in the time window greater than the threshold, and retain the data in the time window within the threshold.
4. The RFID-based three-dimensional sleeping posture detection method according to claim 3, wherein: In step 2.2, the threshold of the RSSI variance sum is set to 5, and the threshold of the phase variance sum is set to 8.
5. The RFID-based three-dimensional sleeping posture detection method according to claim 4, wherein: Step 3 includes the following sub-steps: Step 3.1: Use Formula 3 and Formula 4 to calculate the mean of the RSSI sequence and phase sequence of the unblocked tag in each time window obtained in Step 2. At the same time, assign the blocked tag's RSSI mean value to -100 and the phase mean value to 8. Then, combine the tag array information to obtain the RSSI matrix and phase matrix: in, Indicates the mean of the rssi sequence with label position m,n in a certain time window. Indicates the mean of the phase sequence with label position m,n in a certain time window; Step 3.2: Binarize the RSSI matrix obtained in step 3.1, set the threshold to -100, and set the values of the elements in the RSSI matrix that are greater than the threshold to zero, and those that are equal to the threshold to 1, to obtain the binary matrix: in, Indicates the binary data with label position m,n in a certain time window; Step 3.3: De-noise the phase matrix and RSSI matrix obtained in step 3.1 and the binarized matrix obtained in step 3.2 to obtain the denoised RSSI matrix, phase matrix and binarized matrix; Step 3.4: Use Formula 6 and Formula 7 to normalize the denoised RSSI matrix and phase matrix obtained in Step 3.3, respectively, to obtain the normalized RSSI matrix, phase matrix, and binarization matrix: in, is the RSSI value of the tag position m,n after denoising, is the minimum value of the rssi matrix after denoising, is the maximum value of the rssi matrix after denoising, is the phase value of the label position m,n after denoising, is the minimum value of the phase matrix after denoising, is the maximum value of the phase matrix after denoising; Step 3.5: Use the bilinear interpolation method to double upsample the normalized RSSI matrix, phase matrix, and binarization matrix obtained in step 3.4, and expand the matrix size to twice the original size to obtain the processed RSSI matrix, phase matrix, and binarization matrix; divide the obtained data into a training set and a test set.
6. The RFID-based three-dimensional sleeping posture detection method according to claim 5, wherein: In step 3.3, Gaussian distribution function is used for denoising.
7. The RFID-based three-dimensional sleeping posture detection method according to claim 6, wherein: The step 5 includes the following sub-steps: Step 5.1: Build a convolutional neural network in a PyTorch-based environment: First, build a 2D convolutional layer and introduce an attention mechanism. Then, connect two 2D convolutional layers to combine the 2D convolutional layer and the attention mechanism. Then, expand the output of the 2D convolutional layer to 3D and connect it to the 6-layer 3D convolutional layer. Finally, connect the 3D convolutional layer to the fully connected layer to build the convolutional neural network model. Step 5.2: Set the loss function in the training neural network and use the Adam optimizer with adaptive learning rate to train the network model; Step 5.3: Use the enhanced training set obtained in step 4 to train the constructed convolutional neural network model to obtain a trained convolutional neural network model; Step 5.4: Use the test set to input the trained model obtained in step 5.3 and output the detected 3D sleeping posture results.
8. The RFID-based three-dimensional sleeping posture detection method according to claim 7, wherein: The step 5.1 includes the following operations: Step 5.1.1, construct a two-dimensional convolution layer with a convolution kernel size of 3×3, a padding of 1×1, a stride of 1, an output channel number of 64, an activation function of LeakyReLU, and use BatchNorm2d to normalize the data; then connect the channel attention model, and then connect six two-dimensional convolution layers; the structure of the six two-dimensional convolution layers: the convolution kernel size of the first convolution layer is 3×3, the padding is 1×1, the stride is 1, the output channel number is 128, the activation function is LeakyReLU, use BatchNorm2d to normalize the data, and use a maximum pooling layer with a size of 2 and a stride of 2; the convolution kernel size of the second convolution layer is 3×3, the padding is 1×1, the stride is 1, the output channel number is 256, the activation function is LeakyReLU, and use BatchNorm2d to normalize the data; the convolution kernel size of the third convolution layer is 3×3, the padding is 1×1, and the stride is The length is 1, the number of output channels is 256, the activation function is LeakyReLU, the data is normalized using BatchNorm2d, and a maximum pooling layer with a size of 2 and a stride of 2 is used; the convolution kernel size of the fourth convolution layer is 3×3, the padding is 1×1, the stride is 1, the number of output channels is 512, the activation function is LeakyReLU, and the data is normalized using BatchNorm2d; the convolution kernel size of the fifth convolution layer is 3×3, the padding is 1×1, the stride is 1, the number of output channels is 1024, the activation function is LeakyReLU, and the data is normalized using BatchNorm2d; the convolution kernel size of the sixth convolution layer is 3×3, the padding is 1×1, the stride is 1, the number of output channels is 1024, the activation function is LeakyReLU, the data is normalized using BatchNorm2d, and a maximum pooling layer with a size of 2 and a stride of 2 is used; In step 5.1.2, the output of the two-dimensional convolutional layer is expanded to three dimensions and connected to six three-dimensional convolutional layers; the structure of the six three-dimensional convolutional layers is as follows: the convolution kernel size of the first three-dimensional convolutional layer is 3×3×3, the padding is 1×1×1, the step size is 1, the number of output channels is 1025, the activation function is LeakyReLU, and the data is normalized using BatchNorm2d; the convolution kernel size of the second three-dimensional convolutional layer is 3×3×3, the padding is 1×1, the step size is 1, the number of output channels is 512, the activation function is LeakyReLU, and the data is normalized using BatchNorm2d; the third three-dimensional convolutional layer uses deconvolution, the convolution kernel size is 2×2×2, the padding is 1×1×1, the step size is 1, and the output channels are 1025. The number of channels is 256, the activation function is LeakyReLU, and BatchNorm2d is used to normalize the data; the convolution kernel size of the fourth convolution layer is 3×3×3, the padding is 1×1×1, the stride is 1, the number of output channels is 128, the activation function is LeakyReLU, and BatchNorm2d is used to normalize the data; the convolution kernel size of the fifth convolution layer is 3×3×3, the padding is 1×1×1, the stride is 1, the number of output channels is 64, the activation function is LeakyReLU, and BatchNorm2d is used to normalize the data; the convolution kernel size of the sixth convolution layer is 3×3×3, the padding is 1×1×1, the stride is 1, the number of output channels is 14, and the activation function is Sigmoid; In step 5.1.3, the three-dimensional convolutional layer is connected to the fully connected layer to construct a convolutional neural network model; the fully connected layer is constructed using three layers, the first fully connected layer has 4096 nodes, the activation function is ReLU, and the Dropout is 0.5; the second fully connected layer has 4096 nodes, the activation function is ReLU, and the Dropout is 0.5; the third fully connected layer has 42 nodes, and the final result is obtained by output.
9. The RFID-based three-dimensional sleeping posture detection method according to claim 8, wherein: The loss function constructed in step 5.2 is: in, Represents the loss value in training the convolutional neural network, N represents the joint number, Represents the three-dimensional coordinates of the detected i-th joint, Represents the 3D coordinates of the i-th joint in the groundtruth.