3D human body grid construction system Wi-Mesh based on WiFi vision

Abstract

The invention discloses a 3D human body grid construction system Wi-Mesh based on WiFi vision, which relates to the technical field of 3D imaging and comprises a signal transmitting module, a signal receiving module, a signal processing module, a deep learning model and a user interaction module. According to the invention, the shape and action of the human body in the space are identified by adopting the wifi signal, the wifi signal is transmitted and received by adopting the existing wifi communication equipment, the equipment deployment is simple, the cost benefit is relatively high, the wifi signal is not influenced by a wall body or other obstacles, the propagation of the wifi signal does not depend on light, and the wifi signal can be transmitted and received conveniently. The application effect of the system in various environments is greatly improved, the system interacts through reflection of a wifi signal after encountering a human body, and the use convenience and comfort of the system are greatly improved without wearing any sensor or mark.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional imaging, and in particular to a 3D human body mesh construction system Wi-Mesh based on WiFi vision. Background Art

[0002] 3D mesh is a representation of 3D objects composed of different vertices and polygons, and is widely used in virtual reality (VR), augmented reality (AR) systems, etc. Traditional 3D human mesh construction techniques rely on cameras or dedicated hardware, which perform poorly in non-line-of-sight (NLoS) and poorly illuminated environments, and often require wearable sensors or markers, limiting their application scope and user experience.

[0003] Estimating and constructing the 3D network of the human body has many uses in reality. For example, by capturing different postures of users, more realistic AR services can be obtained in the fields of entertainment, healthcare, etc.; by obtaining the body shape or body features of the human body, an identity-based personnel management system can be used to authorize and authenticate users; by identifying the continuous movement of the human body, factory managers and supervisors can easily determine whether the operation of workers on a production line meets the standards through the system.

[0004] For example, a method and system for constructing a human body 3D mesh model by fusing millimeter waves and images, published as CN116188726A, uses commercial millimeter-wave radar AWR1443BOOST and DCA1000EVM as millimeter-wave signal transceivers to acquire images, extracts the positions of key points of the human body and estimates the contour of the human body by fusing the two signals, and finally regresses a three-dimensional mesh model of the human body. The present invention has no requirements for lighting conditions and has a certain robustness to extreme weather and other conditions that cause the image method to fail.

[0005] The above technical solutions are highly dependent on hardware devices in practical applications, and their application scope and user experience are highly restricted, and are not applicable to practical applications.

[0006] Therefore, it is necessary to invent a 3D human body mesh construction system Wi-Mesh based on WiFi vision to solve the above problems. Summary of the invention

[0007] The purpose of the present invention is to provide a 3D human body mesh construction system Wi-Mesh based on WiFi vision to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solutions: A 3D human body mesh construction system Wi-Mesh based on WiFi vision, comprising a signal transmitting module, a signal receiving module, a signal processing module, a deep learning model and a user interaction module;

[0009] A signal transmitting module, the signal transmitting module is used to transmit a WiFi signal. The signal transmitting module can transmit the WiFi signal in different directions in the space, and make the WiFi signal propagate and reflect in the space in the form of radio waves;

[0010] A signal receiving module, wherein a plurality of signal receiving modules are provided, and the plurality of signal receiving modules are used to receive the WiFi signal reflected back by the human body, and the plurality of signal receiving modules can receive signals from different angles to provide multi-angle data;

[0011] A signal processing module, the signal processing module is used to process the wifi signal received by the signal receiving module, and calculate the two-dimensional angle of arrival (2D AoA) of the wifi signal according to the characteristics of the wifi signal received at different angles, so as to determine the direction from which the wifi signal arrives at the signal receiving module, and a communication module is provided between the signal processing module and the signal receiving module, and the communication module is provided as a wireless signal transmitter or a wired signal transmitter;

[0012] A deep learning model, which is used to convert the two-dimensional angle of arrival (2DAoA) data obtained by the signal processing module into a 3D mesh representation, and perform deep learning in the process of converting multiple sets of data to further understand the shape and movement of the human body and improve the accuracy of 3D human body mesh construction;

[0013] The user interaction module is used for the user to interact with the system. It can display the visualization results of the 3D grid constructed by the deep learning model and provide related interactive functions.

[0014] Preferably, the signal transmitting module is configured as a wifi signal transmitter, and the wifi signal transmitter is installed in the space.

[0015] Preferably, the signal receiving module is configured as a receiving antenna, and a plurality of the receiving antennas are dispersedly installed in the space, and the plurality of the receiving antennas can receive wifi signals from different angles.

[0016] Preferably, the signal processing module is configured as a signal processor, which can analyze and store the wifi signal data received by the signal receiving module, and analyze the characteristics of the wifi signal data to calculate the two-dimensional angle of arrival (2D AoA) of the wifi signal after encountering human body reflection.

[0017] Preferably, the characteristics of the wifi signal data are signal strength and signal propagation time. By analyzing the signal strength characteristics of the wifi signal data, the number of reflections of the wifi signal can be obtained. By analyzing the signal propagation time characteristics of the wifi signal data, the time it takes for the wifi signal to reach the human body and be reflected to the signal receiving module after being emitted by the signal transmitting module can be obtained.

[0018] Preferably, the deep learning of the deep learning model comprises the following steps:

[0019] S1. Obtain the initial deep learning model and sample data;

[0020] S2, inputting the extracted and analyzed multiple sets of sample data into the deep learning model, and inferring based on the relevant data before and after the data to realize the concatenation of multiple sets of two-dimensional angle of arrival (2D AoA) data, converting the multiple sets of two-dimensional angle of arrival (2D AoA) data into 3D grid data, and storing the 3D grid data;

[0021] S3. Analyze and compare the multiple groups of result data stored in step S2 with the real data, eliminate the data with large model loss, and retain the data with higher accuracy to achieve the effect of deep learning.

[0022] Preferably, in step S1, the sample data is two-dimensional angle of arrival (2D AoA) data, and the two-dimensional angle of arrival (2D AoA) data is converted into spatial and dynamic information data, redundant information is removed, and the most relevant features are focused on to derive vectors representing the shape and movement of the human body so that the deep learning model can estimate the 3D mesh data.

[0023] Preferably, in step S2, the distributed training includes the following steps:

[0024] S21, classifying multiple groups of two-dimensional angle of arrival (2D AoA) data according to acquisition time and spatial features, and sending the classified two-dimensional angle of arrival (2D AoA) data to the deep learning model;

[0025] S22. Analyze and process the two-dimensional angle of arrival (2D AoA) data through an initial deep learning model, convert the two-dimensional angle of arrival (2D AoA) data into 3D grid data, and store multiple groups of data in groups.

[0026] Preferably, in step S3, the analysis and comparison of the training result data can be performed by manual comparison and machine comparison, and the 3D mesh data calculated by the deep learning model is compared with the real human body shape and movement data in the space to screen out a data group with higher accuracy.

[0027] Preferably, the user interaction module is configured as a user interaction interface, and the user interaction interface can provide users with interactive functions such as editing, saving and sharing of 3D mesh data.

[0028] Technical effects and advantages of the present invention:

[0029] 1. The present invention uses wifi signals to identify the shape and movement of the human body in the space. The wifi signals are sent and received using existing wifi communication equipment. The equipment deployment is simple and cost-effective. The wifi signals will not be affected by walls or other obstacles, and the propagation of wifi signals does not rely on light, which greatly improves the application effect of the system in various environments. The system interacts through the reflection of the wifi signal after encountering the human body, and does not need to wear any sensors or markers, which greatly improves the convenience and comfort of the system.

[0030] 2. The present invention processes data by constructing a deep learning model. During the application of the system, the deep learning model can not only ensure the stable operation of the system, but also continuously improve the analysis accuracy according to the accumulation of data processing volume, thereby realizing autonomous learning of the system, thereby effectively improving the accuracy of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] The present invention provides Figure 1 A 3D human mesh construction system Wi-Mesh based on WiFi vision is shown, including a signal transmission module, a signal receiving module, a signal processing module, a deep learning model and a user interaction module;

[0034] A signal transmitting module is used to transmit Wi-Fi signals. The signal transmitting module can transmit Wi-Fi signals in different directions in the space, and make the Wi-Fi signals propagate and reflect in the space in the form of radio waves;

[0035] The signal transmitting module is set as a wifi signal transmitter, and the wifi signal transmitter is installed in the space;

[0036] A signal receiving module, wherein a plurality of signal receiving modules are provided, and the plurality of signal receiving modules are used to receive the WiFi signal reflected back by the human body, and the plurality of signal receiving modules can receive signals from different angles to provide multi-angle data;

[0037] The signal receiving module is configured as a receiving antenna, and multiple receiving antennas are dispersedly installed in the space. The multiple receiving antennas can receive Wi-Fi signals from different angles.

[0038] When the Wifi signal detects a certain part of the human body, the human body can reflect the signal to the receiving antenna. However, some parts of the human body where the human body and the receiving antenna overlap will be blocked, resulting in incomplete reflection of the Wifi signal. Multiple sets of receiving antennas can receive reflected Wifi signals from multiple directions of the human body to ensure data integrity.

[0039] A signal processing module, the signal processing module is used to process the wifi signal received by the signal receiving module, and calculate the two-dimensional angle of arrival (2D AoA) of the wifi signal according to the characteristics of the wifi signal received at different angles, so as to determine the direction from which the wifi signal reaches the signal receiving module. A communication module is provided between the signal processing module and the signal receiving module, and the communication module is provided as a wireless signal transmitter or a wired signal transmitter;

[0040] The signal processing module is configured as a signal processor, which can analyze and store the wifi signal data received by the signal receiving module, and analyze the characteristics of the wifi signal data to calculate the two-dimensional angle of arrival (2D AoA) of the wifi signal after encountering human body reflection;

[0041] The characteristics of wifi signal data are signal strength and signal propagation time. By analyzing the signal strength characteristics of wifi signal data, the number of reflections of wifi signal can be obtained. By analyzing the signal propagation time characteristics of wifi signal data, the time it takes for wifi signal to reach human body and be reflected to signal receiving module after being sent by signal transmitting module can be obtained.

[0042] Deep learning model: The deep learning model is used to convert the two-dimensional angle of arrival (2D AoA) data obtained by the signal processing module into a 3D mesh representation, and perform deep learning in the process of converting multiple sets of data to further understand the shape and movement of the human body and improve the accuracy of 3D human body mesh construction;

[0043] Deep learning of a deep learning model consists of the following steps:

[0044] S1. Obtain the initial deep learning model and sample data;

[0045] The sample data is 2D AoA data. The 2D AoA data is converted into spatial and dynamic information data, redundant information is removed, and the most relevant features are focused on to derive vectors representing the shape and movement of the human body so that the deep learning model can estimate the 3D mesh data.

[0046] S2, inputting the extracted and analyzed multiple sets of sample data into the deep learning model, and inferring based on the relevant data before and after the data to realize the concatenation of multiple sets of two-dimensional angle of arrival (2D AoA) data, converting the multiple sets of two-dimensional angle of arrival (2D AoA) data into 3D grid data, and storing the 3D grid data;

[0047] Since the movement of the human body is continuous and two adjacent movements are closely related, when processing the two-dimensional angle of arrival (2D AoA) data, the data before and after the individual data are correlated to correct each other, thereby improving the accuracy of 3D mesh data construction.

[0048] Distributed training consists of the following steps:

[0049] S21, classifying multiple groups of two-dimensional angle of arrival (2D AoA) data according to acquisition time and spatial features, and sending the classified two-dimensional angle of arrival (2D AoA) data to the deep learning model;

[0050] S22, analyzing and processing the two-dimensional angle of arrival (2D AoA) data through the initial deep learning model, converting the two-dimensional angle of arrival (2D AoA) data into 3D grid data, and storing the multiple groups of data in groups;

[0051] S3, analyzing and comparing the multiple groups of result data stored in step S2 with the real data, eliminating the data with large model loss, and retaining the data with higher accuracy to achieve the effect of deep learning;

[0052] The analysis and comparison of training result data can be carried out by manual comparison and machine comparison. The 3D mesh data calculated by the deep learning model is compared with the real human body shape and movement data in space to screen out the data group with higher accuracy.

[0053] User interaction module: The user interaction module is used for interaction between users and the system. It can display the visualization results of the 3D mesh constructed by the deep learning model and provide related interactive functions;

[0054] The user interaction module is configured as a user interaction interface, which can provide users with interactive functions such as editing, saving and sharing of 3D mesh data.

[0055] In order to better illustrate the advantages of the technical solution of the present invention, this embodiment discloses the following experiments:

[0056] Experiment 1

[0057] This experiment was conducted in an indoor environment. A wifi transmitter and two wifi receivers were deployed at the experimental site. The distance between the wifi transmitter and the wifi receiver was 2 meters. The scene size of the experimental site was 6m*6m. 20 people of different heights and genders were selected to participate in the experiment. Each person performed daily activities in the experimental scene, including circular walking, straight walking, arm movement, leg movement, torso rotation, etc. During the experiment, the wifi data of the wifi receiver mobile phone was obtained, and the wifi data was processed by Wi-Mesh, a 3D human mesh construction system based on WiFi vision, to construct 3D mesh data;

[0058] The system results are compared with the results of three existing systems based on CSI, Doppler and 1D AoA, and the error data of the average vertex position and joint position of the four systems are obtained as shown in the following table:

[0059]

[0060] In summary, the use of this system can obtain higher-precision 3D human body mesh data in a conventional environment.

[0061] Experiment 2

[0062] This experiment was conducted in a dark indoor environment. A wifi transmitter and two wifi receivers were deployed at the experimental site. The distance between the wifi transmitter and the wifi receiver was 2 meters. The scene size of the experimental site was 6m*6m. 20 people of different heights and genders were selected to participate in the experiment. Each person performed daily activities in the experimental scene, including circular walking, straight walking, arm movement, leg movement, torso rotation, etc. During the experiment, the wifi data of the wifi receiver mobile phone was obtained, and the wifi data was processed by Wi-Mesh, a 3D human mesh construction system based on WiFi vision, to construct 3D mesh data;

[0063] The system results are compared with the results of three existing systems based on CSI, Doppler and 1D AoA, and the error data of the average vertex position and joint position of the four systems are obtained as shown in the following table:

[0064]

[0065] In summary, the use of this system can obtain higher-precision 3D human body mesh data in dark and invisible environments.

[0066] Experiment 3

[0067] This experiment was conducted in an indoor environment. A wifi transmitter and two wifi receivers were deployed at the experimental site. The distance between the wifi transmitter and the wifi receiver was 2 meters. There were four experimental sites. The scene sizes of the four experimental sites were 3m*3m, 4m*4m, 5m*5m, and 6m*6m respectively. 20 people of different heights and genders were selected to participate in the experiment. Each person performed daily activities in the experimental scene, including circular walking, straight walking, arm movement, leg movement, torso rotation, etc. During the experiment, the wifi data of the wifi receiver mobile phone was obtained, and the wifi data was processed by Wi-Mesh, a 3D human mesh construction system based on WiFi vision, to construct 3D mesh data;

[0068] The system results are compared with the results of three existing systems based on CSI, Doppler and 1D AoA, and the error data of the average vertex position and joint position of the four systems are obtained. The average positioning position error data of the four systems at four experimental locations are shown in the following table:

[0069] Location 1 Location 2 Location 3 Location 4 Based on CSI 13.26 14.24 11.08 12.12 Doppler-based 7.92 8.01 7.98 7.12 Based on 1D AoA 4.32 4.21 4.67 3.98 Wi-Mesh 3.21 3.18 2.96 2.88

[0070] The average joint position error data of the four systems at four experimental locations are shown in the following table:

[0071]

[0072] In summary, the present system can be used to obtain higher-precision 3D human body mesh data in environments of different sizes.

[0073] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A 3D human body mesh construction system based on WiFi vision, Wi-Mesh, characterized in that: It includes a signal transmission module, a signal receiving module, a signal processing module, a deep learning model and a user interaction module; A signal transmitting module, the signal transmitting module is used to transmit a WiFi signal. The signal transmitting module can transmit the WiFi signal in different directions in the space, and make the WiFi signal propagate and reflect in the space in the form of radio waves; A signal receiving module, wherein a plurality of signal receiving modules are provided, and the plurality of signal receiving modules are used to receive the WiFi signal reflected back by the human body, and the plurality of signal receiving modules can receive signals from different angles to provide multi-angle data; A signal processing module, the signal processing module is used to process the wifi signal received by the signal receiving module, and calculate the two-dimensional angle of arrival (2D AoA) of the wifi signal according to the characteristics of the wifi signal received at different angles, so as to determine the direction from which the wifi signal arrives at the signal receiving module, and a communication module is provided between the signal processing module and the signal receiving module, and the communication module is provided as a wireless signal transmitter or a wired signal transmitter; A deep learning model, which is used to convert the two-dimensional angle of arrival (2D AoA) data obtained by the signal processing module into a 3D mesh representation, and perform deep learning in the process of converting multiple sets of data to further understand the shape and movement of the human body and improve the accuracy of 3D human body mesh construction; The user interaction module is used for the user to interact with the system. It can display the visualization results of the 3D grid constructed by the deep learning model and provide related interactive functions.

2. According to claim 1, a 3D human body mesh construction system based on WiFi vision Wi-Mesh is characterized in that: The signal transmitting module is configured as a wifi signal transmitter, and the wifi signal transmitter is installed in the space.

3. According to claim 1, a 3D human body mesh construction system based on WiFi vision Wi-Mesh is characterized in that: The signal receiving module is configured as a receiving antenna, and a plurality of the receiving antennas are dispersedly installed in the space. The plurality of receiving antennas can receive Wi-Fi signals from different angles to obtain Wi-Fi signal data reflected by the human body at different angles.

4. The 3D human body mesh construction system Wi-Mesh based on WiFi vision according to claim 1, characterized in that: The signal processing module is configured as a signal processor, which can analyze and store the wifi signal data received by the signal receiving module, and analyze the characteristics of the wifi signal data to obtain the two-dimensional angle of arrival (2D AoA) data of the wifi signal after encountering human body reflection.

5. The 3D human body mesh construction system Wi-Mesh based on WiFi vision according to claim 4, characterized in that: The characteristics of the wifi signal data are signal strength and signal propagation time. By analyzing the signal strength characteristics of the wifi signal data, the number of reflection times of the wifi signal can be obtained. By analyzing the signal propagation time characteristics of the wifi signal data, the time it takes for the wifi signal to reach the human body and be reflected to the signal receiving module after being emitted by the signal transmitting module can be obtained.

6. The WiFi vision-based 3D human body mesh construction system Wi-Mesh according to claim 1, characterized in that: The deep learning of the deep learning model comprises the following steps: S1. Obtain an initial deep learning model and multiple sets of sample data; S2, inputting the extracted and analyzed multiple sets of sample data into the deep learning model, and inferring based on the relevant data before and after the data to realize the concatenation of multiple sets of two-dimensional angle of arrival (2D AoA) data, converting the multiple sets of two-dimensional angle of arrival (2D AoA) data into 3D grid data, and storing the 3D grid data; S3. Analyze and compare the multiple groups of result data stored in step S2 with the real data, eliminate the data with large model loss, and retain the data with higher accuracy to achieve the effect of deep learning.

7. The WiFi vision-based 3D human body mesh construction system Wi-Mesh according to claim 6, characterized in that: In step S1, the sample data is two-dimensional angle of arrival (2D AoA) data, and the two-dimensional angle of arrival (2D AoA) data is converted into spatial and dynamic information data, redundant information is removed, and the most relevant features are focused on to derive vectors representing the shape and movement of the human body so that the deep learning model can estimate the 3D mesh data.

8. The WiFi vision-based 3D human body mesh construction system Wi-Mesh according to claim 6, characterized in that: In step S2, the distributed training includes the following steps: S21, classifying multiple groups of two-dimensional angle of arrival (2D AoA) data according to acquisition time and spatial features, and sending the classified two-dimensional angle of arrival (2D AoA) data to the deep learning model; S22. Analyze and process the two-dimensional angle of arrival (2D AoA) data through an initial deep learning model, convert the two-dimensional angle of arrival (2D AoA) data into 3D grid data, and store multiple groups of data in groups.

9. The WiFi vision-based 3D human body mesh construction system Wi-Mesh according to claim 6, characterized in that: In step S3, the analysis and comparison of the training result data can be performed by manual comparison and machine comparison, and the 3D mesh data calculated by the deep learning model is compared with the real human body shape and movement data in the space to screen out a data group with higher accuracy.

10. The WiFi vision-based 3D human body mesh construction system Wi-Mesh according to claim 1, characterized in that: The user interaction module is configured as a user interaction interface, and the user interaction interface can provide users with interactive functions such as editing, saving and sharing of 3D mesh data.

Images