Gesture tracking method and system based on acoustic signal

Abstract

According to the gesture tracking method and system based on the acoustic signals, distance, angle and speed information of signals reflected by multiple targets is fused based on a signal model of triangular chirp signals, the signals are analyzed through a joint estimation algorithm, signal separation is achieved through signal elimination and reconstruction, and multi-dimensional parameters of each target are obtained. According to the invention, the limit of motion tracking by using acoustic signals is broken through, and continuous real-time tracking is carried out on a plurality of targets at the same time with centimeter-level precision.

Description

Technical Field

[0001] The present invention relates to the field of wireless sensing technology, and in particular to a gesture tracking method and system based on acoustic signals. Background Art

[0002] As the Internet of Things continues to penetrate into various industries, the level of industry informatization and networking continues to improve, and smart devices and their applications have become an indispensable part of people's lives. In the current era of the Internet of Everything, smart devices such as smart watches, smart phones, smart cars, and smart homes have become indispensable elements in people's daily lives. The vigorous development of these devices and applications has completely changed people's lives, which has also greatly stimulated the research on human-computer interaction technology. People hope to better understand and apply these smart devices through various technologies to achieve more efficient and convenient human-computer interaction.

[0003] As the intersection of the Internet of Things and artificial intelligence, wireless sensing and recognition technology is an important part of human-computer interaction technology. Compared with traditional sensor sensing and recognition, the biggest feature of wireless sensing and recognition technology is that it is equipment-free and contactless. When sensing and identifying people and the environment, it is no longer necessary to deploy special sensors and wired lines, and users do not need to wear any equipment. They use wireless signals for perception, such as sound, light, and radio frequency signals. Taking radio frequency signals as an example, after the signal is transmitted by the signal transmitter, it will propagate in space through a variety of paths, including direct, reflected, and scattered. After these signals propagate through different paths and reach the receiver, they will be superimposed to form a composite signal carrying spatial information. Wireless sensing and recognition technology can analyze the changes in the signal during the propagation process, obtain distance, speed, and angle information, and can provide high-precision positioning, gesture capture, motion recognition, passive object detection and tracking, thereby realizing a wide range of new services such as scene perception and environmental reconstruction. After years of development, various basic and verification work on wireless sensing and recognition technology has been completed, and has achieved application results in many fields. For example, in the medical field, wireless sensing technology can detect the vital signs of patients and remotely monitor and diagnose them; in smart home scenarios, it can realize home security monitoring and control and management of smart home devices such as smart TVs and smart curtains. The development of wireless sensing and recognition technology has pushed human perception of the physical world to a new stage of comprehensive intelligence, and has also created more possibilities for the integration of human-computer interaction and intelligence.

[0004] With the increasing popularity of smart terminals such as mobile phones, speakers and wearable devices, intelligent sensing technologies based on various sensing media have attracted widespread attention from researchers. For contactless tracking and identification, among different types of sensing media, acoustic signals have stronger sensitivity and adaptability to environmental changes, and compared with other wireless signals, such as WiFi and RFID, acoustic signals have lower propagation speed in the air, so acoustic sensing has inherent advantages in sensing granularity and tracking accuracy. Acoustic sensing can utilize ubiquitous audio devices without competing for radio resources with other home devices. Recent efforts have improved acoustic sensing accuracy to the millimeter level. By transmitting and receiving signals through speakers and microphones, intelligent recognition based on acoustic signals has been extended to a wider range of applications, such as acoustic positioning, contactless motion tracking, gesture recognition and action posture recognition, as well as monitoring of important human physiological parameters (for example, driver fatigue detection and breathing activity).

[0005] Nowadays, using gestures to control devices has become a research hotspot in the field of wireless perception and interaction. This device-free technology allows users to control devices using movements or gestures. It is considered a natural, convenient and potential way of human-computer interaction and has been widely used in many application scenarios. For example, in the field of smart homes, gesture tracking and recognition technology allows users to control the switches, adjust the volume and brightness of smart home devices through gestures without touching the devices. This is particularly useful when the user's hands are dirty, wet or it is inconvenient to touch the device; in the field of virtual reality, gesture tracking and recognition technology allows users to manipulate objects in the virtual environment through gestures, providing a more natural way of interaction for virtual reality games and training.

[0006] Existing gesture tracking and recognition can be divided into two categories: device-based and device-free (contactless tracking and recognition). Device-based tracking uses wearable sensors. However, wearable sensor-based methods are usually cumbersome due to the inconvenience of wearing sensors. Device-free gesture tracking and recognition can be roughly divided into three categories: vision-based, radio frequency signal-based, and acoustic signal-based. Vision-based methods suffer from poor lighting conditions or occlusion. Radio frequency-based methods require the use of Wi-Fi or specialized equipment. Secondly, Wi-Fi measurements are too coarse-grained and cannot provide fine accuracy or recognize small gestures. In recent years, some gesture recognition systems based on acoustic signals have been proposed. Compared with sensor-based, vision-based, and radio frequency-based methods, the advantage of acoustic signal-based methods is that they do not require good lighting conditions or any equipment to be worn on the user, and provide fine-grained gesture recognition by analyzing the reflected signals of the hand. Summary of the invention

[0007] The technical problem to be solved by the present invention is how.

[0008] The present invention solves the above technical problems through the following technical means:

[0009] The hand gesture tracking method based on acoustic signals includes the following steps:

[0010] Mixed signal calculation steps: the denoised triangular chirp signal is divided into an up-chirp signal and a down-chirp signal, the separated up-chirp and down-chirp signals are respectively orthogonally demodulated to obtain a preliminary mixed signal, the high-frequency part of the obtained preliminary mixed signal is filtered out to obtain the in-phase part and the orthogonal part of the signal respectively, the same part and the orthogonal part of the up-chirp and down-chirp signals are respectively mixed to obtain an up-chirp mixed signal and a down-chirp mixed signal; the mixed signal of multiple targets includes multiple up-chirp mixed signals and multiple down-chirp mixed signals; the flight time of the reflected signal can be obtained from the up-chirp and down-chirp mixed signals;

[0011] Signal separation steps: first eliminate the noise of the upper and lower mixed signals, then search for the optimal parameters of the distance information r, angle information θ and speed information v within the set parameter range, and then calculate the signal attenuation factor based on the optimal parameters of the distance information r and speed information v; when there are multiple targets, search for the strongest reflected signal in the current mixed signal of multiple targets each time through an iterative method to obtain the distance information, angle information and speed information parameters that match the signal, and then calculate the signal attenuation factor of the reflected signal. After the iteration, the parameters of different targets can be obtained;

[0012] Steps in object matching tracking: Using weighted l1-norm

[0013]

[0014] where p t It represents the vector composed of the estimated values ​​of the four dimensional parameters at time t: signal attenuation factor, distance information, angle information and speed information, expressed as β is a scale vector that normalizes the difference values ​​of the four dimensions to the same scale; the parameter vector obtained by jointly estimating the two consecutive signal periods with the smallest l1 distance is considered to belong to the same target.

[0015] Furthermore, the calculation process of the mixed signal in the mixed signal calculation step is:

[0016] First, the chirp signal S T _uq is Hilbert transformed with its own signal and the up-chirp signal to obtain the 90° phase shift version S′ of the transmission signal T _up are multiplied to obtain a first preliminary mixed signal and a second preliminary mixed signal, respectively. The first preliminary mixed signal and the second preliminary mixed signal are respectively passed through a low-pass filter to obtain the same part S I _up(t) and the orthogonal part SQ _up(t), the same part and the orthogonal part are combined by a complex function to finally obtain the up-chirp mixed signal; the down-chirp mixed signal is obtained in the same way; the expression is:

[0017]

[0018] In the formula, S M _up(t) is the up-chirp mixed signal, S M _down(t) is the down-chirp mixed signal.

[0019] Furthermore, the multi-target mixed signal is expressed as follows: Assuming there are L targets, the signal received by each microphone should be a composite signal formed by the superposition of all target reflection signals. At the i-th signal sampling point, the mixed signal of all target reflections received by the k-th microphone is expressed as follows:

[0020]

[0021] Among them, α l , r l , v l They respectively represent the amplitude attenuation factor, distance information and speed information of the lth target reflection signal.

[0022] Furthermore, the signal attenuation factor calculation process of a single target in the signal separation step is:

[0023] Assume that a triangular chirp signal contains 2N sample points, where the up-chirp part and the down-chirp part each contain N samples, and there are K microphones; first consider the up-chirp part. For each microphone, there are N signal sample points. For each possible set of (r, v), the phase change is reconstructed based on the signal flight time, so each microphone can reconstruct a phase change estimation vector The size of the vector is N. The signal measurement value of each sample is multiplied by the conjugate of the phase change of the reconstructed signal based on the parameters, eliminating the phase shift of each sample in the distance and velocity dimensions, and all the results are added together to obtain the joint estimate of the upper chirp part, and the same is true for the lower chirp part;

[0024]

[0025] The absolute values ​​of the joint estimates of the upper and lower chirp parts are added together to obtain the result of the joint estimator, which can be expressed as

[0026] E(r,v)=||E_up(r,v)|| 2 +||E_down(r,v)|| 2

[0027] The result of the joint estimator E(r,v) will be a set of optimal parameters is maximized when , because all signals are in phase at this time; therefore, the problem of finding the optimal parameter can be expressed as

[0028]

[0029] Finally, we get two optimal parameters After the estimated values ​​of

[0030]

[0031] Further, the method for separating multiple target signals is:

[0032] First, based on the joint estimator, the path and parameters corresponding to the strongest signal in the mixed signal are obtained, and the parameters obtained according to the up-and-down chirp mixed signal formula are used to reconstruct the strongest path signal, and the reconstructed signal is subtracted from the original mixed signal to obtain the remaining mixed signal; then, the joint estimator is used again to estimate the strongest path signal from the remaining signals, and the joint estimation of the next target is performed to obtain the parameter information of the n+1th target; until the power of the remaining signal is less than a predefined threshold, the joint estimation is terminated.

[0033] The present invention also provides a gesture tracking system based on acoustic signals, comprising:

[0034] Mixed signal calculation module: the denoised triangular chirp signal is divided into an up-chirp signal and a down-chirp signal, and the separated up-chirp and down-chirp signals are respectively orthogonally demodulated to obtain a preliminary mixed signal, and the high-frequency part of the obtained preliminary mixed signal is filtered out to obtain the in-phase part and the orthogonal part of the signal respectively, and the same part and the orthogonal part of the up-chirp and down-chirp signals are respectively mixed to obtain an up-chirp mixed signal and a down-chirp mixed signal; the mixed signal of multiple targets includes multiple up-chirp mixed signals and multiple down-chirp mixed signals; the flight time of the reflected signal can be obtained from the up-chirp and down-chirp mixed signals;

[0035] Signal separation module: first eliminate the noise of the upper and lower mixed signals, then search for the optimal parameters of the distance information r, angle information θ and speed information v within the set parameter range, and then calculate the signal attenuation factor based on the optimal parameters of the distance information r and speed information v; when there are multiple targets, the strongest reflected signal in the current mixed signal of multiple targets is searched each time through an iterative method to obtain the distance information, angle information and speed information parameters matching the signal, and then calculate the signal attenuation factor of the reflected signal. After the iteration, the parameters of different targets can be obtained;

[0036] Target matching tracking module: using weighted l1-norm

[0037]

[0038] where p t It represents the vector composed of the estimated values ​​of the four dimensional parameters at time t: signal attenuation factor, distance information, angle information and speed information, expressed as β is a scale vector that normalizes the difference values ​​of the four dimensions to the same scale; the parameter vector obtained by jointly estimating the two consecutive signal periods with the smallest l1 distance is considered to belong to the same target.

[0039] Furthermore, the calculation process of the mixed signal in the mixed signal calculation module is:

[0040] First, the chirp signal S T _uq is Hilbert transformed with its own signal and the up-chirp signal to obtain the 90° phase shift version S′ of the transmission signal T _up are multiplied to obtain a first preliminary mixed signal and a second preliminary mixed signal, respectively. The first preliminary mixed signal and the second preliminary mixed signal are respectively passed through a low-pass filter to obtain the same part S I _up(t) and the orthogonal part S Q _up(t), the same part and the orthogonal part are combined by a complex function to finally obtain the up-chirp mixed signal; the down-chirp mixed signal is obtained in the same way; the expression is:

[0041]

[0042] In the formula, S M _up(t) is the up-chirp mixed signal, S M _down(t) is the down-chirp mixed signal.

[0043] Furthermore, the multi-target mixed signal is expressed as follows: Assuming there are L targets, the signal received by each microphone should be a composite signal formed by the superposition of all target reflection signals. At the i-th signal sampling point, the mixed signal of all target reflections received by the k-th microphone is expressed as follows:

[0044]

[0045] Among them, α l , r l , v l They respectively represent the amplitude attenuation factor, distance information and speed information of the lth target reflection signal.

[0046] Furthermore, the signal attenuation factor calculation process of a single target in the signal separation module is:

[0047] Assume that a triangular chirp signal contains 2N sample points, where the up-chirp part and the down-chirp part each contain N samples, and there are K microphones; first consider the up-chirp part. For each microphone, there are N signal sample points. For each possible set of (r, v), the phase change is reconstructed based on the signal flight time, so each microphone can reconstruct a phase change estimation vector The size of the vector is N. The signal measurement value of each sample is multiplied by the conjugate of the phase change of the reconstructed signal based on the parameters, eliminating the phase shift of each sample in the distance and velocity dimensions, and all the results are added together to obtain the joint estimate of the upper chirp part, and the same is true for the lower chirp part;

[0048]

[0049] The absolute values ​​of the joint estimates of the upper and lower chirp parts are added together to obtain the result of the joint estimator, which can be expressed as

[0050] E(r,v)=||E_up(r,v)|| 2 +||E_down(r,v)|| 2

[0051] The result of the joint estimator E(r,v) will be a set of optimal parameters is maximized when , because all signals are in phase at this time; therefore, the problem of finding the optimal parameter can be expressed as

[0052]

[0053] Finally, we get two optimal parameters After the estimated values ​​of

[0054]

[0055] Further, the method for separating multiple target signals is:

[0056] First, based on the joint estimator, the path and parameters corresponding to the strongest signal in the mixed signal are obtained, and the parameters obtained according to the up-and-down chirp mixed signal formula are used to reconstruct the strongest path signal, and the reconstructed signal is subtracted from the original mixed signal to obtain the remaining mixed signal; then, the joint estimator is used again to estimate the strongest path signal from the remaining signals, and the joint estimation of the next target is performed to obtain the parameter information of the n+1th target; until the power of the remaining signal is less than a predefined threshold, the joint estimation is terminated.

[0057] The advantages of the present invention are:

[0058] The present invention realizes a new type of non-contact tracking based on acoustic triangular chirp signals, which breaks through the limit of motion tracking using acoustic signals and continuously tracks multiple targets in real time with centimeter-level accuracy. The present invention verifies its tracking performance through simulation and real-life scenarios, demonstrates its feasibility in real-life applications, and confirms its great potential for deployment on smart devices, providing a technical solution for a more natural and friendly human-computer interaction method from the perspective of acoustic wireless perception. Specifically, the present invention proposes a signal model based on triangular chirp signals to fuse the distance, angle, and speed information of signals reflected from multiple targets, analyzes the signal through a joint estimation algorithm, achieves signal separation through signal elimination and reconstruction, and obtains multi-dimensional parameters of each target.

[0059] The present invention proposes solutions to some interference system tracking accuracy problems, such as alignment algorithm can effectively solve the start time error caused by the inherent limitations of hardware equipment and software coding, and background multipath subtraction can subtract the direct path contained in the mixed reflection signal as well as most of the noise and other interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 An algorithm flow chart of a gesture tracking method based on acoustic signals in an embodiment of the present invention;

[0061] Figure 2 1 is a calculation process of a mixed signal in an embodiment of the present invention (taking the above chirped mixed signal as an example, the calculation process of the lower chirped mixed signal is the same);

[0062] Figure 3 It is a multi-objective parameter joint estimation iterative process in an embodiment of the present invention;

[0063] Figure 4 The figure is a comparison of the signal reconstruction process in the embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] Glossary:

[0066] Acoustic wireless sensing is a technology that uses sound waves to detect, locate and identify objects. It obtains information about target objects by analyzing the way sound waves propagate in space. Acoustic wireless sensing can be used in many fields, including wireless communications, security monitoring, medical diagnosis, etc.

[0067] FMCW wave is a commonly used wireless communication technology, mainly used for radar ranging and speed measurement. In FMCW wave, the frequency of the signal is modulated in a continuous manner and changes linearly with time at a certain rate. Since the single-chord signal is highly periodic, one flight time will correspond to the same phase in multiple cycles, resulting in ambiguity in the calculation of distance. Compared with the single-chord signal, the FMCW signal has unique advantages. Since the signal frequency of the FMCW wave changes continuously, the distance and speed of the target can be calculated by measuring the frequency difference and phase difference of the signal, and higher measurement accuracy and resolution can be achieved. This feature will also reduce the interference caused by the signal arriving through multiple paths to the measurement result. The arrival time of each path is different, and the signal frequency difference generated is also different, so the signal receiver can distinguish multiple paths to improve the reliability of the signal.

[0068] Signal flight time refers to the time it takes for a wireless signal to reach the receiver from the transmitter. In radar, wireless communication and positioning systems, accurately measuring the signal flight time is crucial to determining the target position and distance. By measuring the signal flight time, the propagation distance of the signal in space can be calculated, which can then be used for applications such as positioning, ranging and speed measurement.

[0069] The signal arrival angle refers to the angle of the direction from which the signal arrives at the receiving antenna. In wireless communication systems, understanding the signal arrival angle can help optimize antenna layout and signal coverage, improve communication quality and system performance. The signal arrival angle is also an important parameter in radar and positioning systems, used to determine the azimuth angle of the target.

[0070] The Doppler effect refers to the phenomenon that the frequency of a signal changes when the source or receiver moves relative to the receiver or source. When the source or receiver moves toward the receiver or source, the frequency of the signal increases, which is called the positive Doppler effect; when the source or receiver moves away from the receiver or source, the frequency of the signal decreases, which is called the negative Doppler effect. The Doppler effect has important applications in radar, wireless communication and mobile communication systems, such as speed measurement, frequency offset compensation and speed estimation in mobile communications.

[0071] This embodiment provides a gesture tracking method based on acoustic signals. Figure 1 As shown, the steps are as follows:

[0072] Mixed signal calculation steps (i.e. Figure 1Establishing the mathematical signal model part): This implementation uses a continuous triangular chirp signal, and each transmitted triangular chirp signal consists of an upper chirp part with a linearly increasing signal frequency and a lower chirp part with a linearly decreasing signal frequency. First, the signal received by the microphone is passed through a high-pass filter to remove some low-frequency noise. According to the upper and lower chirp parts of the transmission signal, the received signal is divided into an upper chirp received signal and a lower chirp received signal by intercepting the sampling points. The separated upper and lower chirp signals are respectively orthogonally demodulated, and the obtained preliminary mixed signal is passed through a low-pass filter to filter out the high-frequency part of the preliminary mixed signal, and the in-phase part and the orthogonal part of the signal are respectively obtained, and they are combined to obtain a mixed signal.

[0073] Steps for jointly estimating separated signals: The joint estimation algorithm used in this implementation is designed based on the characteristics of triangular chirp signals. It can gradually separate the reflected signals from the mixed signals of multiple targets and estimate the target parameters from the reflected signals. Before jointly estimating the mixed signals, the system first eliminates the background multipath interference. These interferences include the direct path from the speaker to the microphone and the reflected interference generated from the static environment. Then a joint estimator is constructed to search for the optimal parameters of distance, angle, and speed within the parameter range, and finally the signal attenuation is calculated. When there are multiple targets, the strongest reflected signal of the current signal is searched each time through an iterative method to obtain the parameters that match the signal, and the steps of joint estimation are repeated with the remaining signals to obtain the parameters of different targets.

[0074] Steps of target matching tracking: In order to continuously track the moving target and obtain its trajectory, the system needs to continuously perform joint estimation to obtain the position and motion parameters of the target at different time points. When multiple targets need to be tracked, the system's joint estimation algorithm outputs a set of parameters for multiple targets. The key challenge is how to associate the position and motion parameter estimates of the same target at different timestamps, especially when the trajectories of multiple targets overlap. This implementation solves this problem by using parameters in more dimensions, because within a given timestamp, it is rare for multiple targets to have similar parameters in all dimensions. For example, by resolving the ambiguity caused by the similarity in distance in two-dimensional range space and angle of incidence (AoA).

[0075] The above steps are described in detail below.

[0076] Without loss of generality, a dedicated Seeed Respeaker 6-Mic circular microphone array and a Lecoo speaker Raspberry Pi 4B were built in an everyday open space, and Matlab software was run on a computer equipped with an AMD Ryzen 53550H processor and 16GB to process and analyze the data. In this implementation, the triangular chirp signal used by the system has a starting frequency of 18kHz, an ending frequency of 22kHz, and a bandwidth of 4kHz. The frequency of the upper chirp part of the signal increases linearly from the starting frequency to the ending frequency, with a duration of 0.04s, and the lower chirp part decreases linearly from the ending frequency to the starting frequency, with a duration of 0.04s, so a complete signal cycle is 0.08s. By processing the collected sound signals, this implementation can track moving targets, verifying the effectiveness and feasibility of this implementation.

[0077] Mixed signal calculation steps: Figure 1 The mathematical signal model is established in the part.

[0078] This implementation uses the unique properties of FMCW waves to measure the signal flight time. The signal needs to be preprocessed before establishing a mathematical signal model. First, the signal received by the microphone is passed through a high-pass filter to remove some low-frequency noise. The transmitted and received signals are continuous triangular chirp signals, and in the subsequent processing, the up-chirp part and the down-chirp part are processed and analyzed respectively, so it is necessary to separate the up-chirp and down-chirp of the received signal. According to the up-chirp and down-chirp parts of the transmitted signal, the received signal is divided into an up-chirp received signal and a down-chirp received signal by intercepting the sampling points.

[0079] Next, the signal mixing process is performed to obtain the signal flight time of a single target. Figure 2 The mixed signal portion can be obtained by multiplying the up-chirp and down-chirp portions of the transmitted signal with the corresponding portions of the received signal. Specifically, using trigonometric equations, for example: The mixed signals of up-chirp signal and down-chirp signal can be expressed as

[0080]

[0081] Among them, f0, f1, B, T, α, and τ represent the starting frequency, ending frequency, frequency bandwidth, signal duration, amplitude attenuation caused by signal reflection, and signal flight time of the triangular chirp signal, respectively, and f1 is the sum of f0 and B.

[0082] like Figure 2 As shown, the chirp signal S T _up as an example, first the chirp signal S T_up is Hilbert transformed with its own signal and the up-chirp signal to obtain a 90° phase-shifted version S′ of the transmitted signal T _up are multiplied to obtain a first preliminary mixed signal and a second preliminary mixed signal, respectively. The first preliminary mixed signal and the second preliminary mixed signal are respectively passed through a low-pass filter to obtain the same part S I _up(t) and the orthogonal part S Q _up(t), the same part and the orthogonal part are combined by a complex function, and finally the up-chirp mixed signal is obtained, which is expressed as

[0083]

[0084] In the formula, S M _up(t) is the up-chirp mixed signal, S M _down(t) is the down-chirp mixed signal.

[0085] The flight time of the reflected signal can be analyzed from the above mixed signal, and then the distance, angle and speed information of the target can be extracted from the flight time by jointly estimating the separated signal. Figure 2 The calculation process of the chirp mixed signal is introduced by taking the chirp signal as an example. Similarly, Figure 2 The down-chirped mixed signal is calculated by the method.

[0086] The signal flight time calculation method based on the above single target can be extended to multiple targets. Assuming there are L targets, the signal received by each microphone should be a composite signal formed by the superposition of all target reflection signals. At the i-th signal sampling point, the mixed signal of all target reflections received by the k-th microphone can be expressed as

[0087]

[0088] where α l , r l , v l They represent the signal attenuation factor, distance information and speed information of the lth target reflection signal respectively. The ultimate goal of the system tracking algorithm is to separate the reflection signal of each individual target from the mixed signal reflected by all targets one by one, and analyze the corresponding multi-dimensional parameters of each target from the signal, so as to obtain the spatial position and motion state information of the target.

[0089] Steps to jointly estimate the separated signals:

[0090] Before jointly estimating the mixed signal, the system first eliminates background multipath interference. These interferences include the direct path from the speaker to the microphone and the reflection interference generated from the static environment. Since the strength of the direct path signal is much higher than the strength of the other various signals that arrive after reflection, the direct path is used as an anchor point to achieve alignment of the received signals.

[0091] For each signal sample sampled by the kth microphone at the i-th sampling timestamp, the signal can be represented by the attenuation factor α and the phase change Model it, where the phase change of the signal is caused by the distance r of the target relative to the microphone and the speed v of the target. in represents the total phase change of the signal sampled by the kth microphone at the ith sampling timestamp, and Represent the signal phase change caused by distance and speed respectively. When the distance and speed parameters are correctly estimated, the phase change calculated based on the estimated parameters is The measured value of the actual phase change Should be approximately equal.

[0092]

[0093] in and represent the phase variables calculated based on the estimated distance parameter and speed parameter, respectively.

[0094] If the phase change based on these accurate estimates is removed from the measured value at each sampling point As the equation

[0095]

[0096] in(·) * For the conjugate operation, a phase value of 0 (i.e. αe j0 / 2), which means that the remaining signal after removing the signal reconstructed based on the estimated value is in phase. At this time, the signal strength after superimposing the remaining signals of all sampling points reaches the maximum value. According to this principle, the values ​​of distance and speed parameters (r, v) can be searched within the preset parameter search range. When the superimposed signal is the largest, the parameters at this time are the optimal parameters.

[0097] Assume that a triangular chirp signal contains 2N sample points, of which the up-chirp part and the down-chirp part each contain N samples, and the experiment contains K microphones. First consider the up-chirp part. For each microphone, there are N signal sample points. For each possible set of (r, v), the system can reconstruct the phase change based on the signal flight time, so each microphone can reconstruct a phase change estimation vector The size of the vector is N. The signal measurement value of each sample is multiplied by the conjugate of the phase change of the reconstructed signal based on the parameters to eliminate the phase shift caused by each sample in the distance and velocity dimensions, and all the results are added together to obtain the joint estimate of the upper chirp part, and the same is true for the lower chirp part.

[0098]

[0099] It should be noted here that the result of the joint estimator is composed of the absolute value of the joint estimate of the upper chirp part and the joint estimate of the lower chirp part, so a complete triangular chirp signal is required to obtain a set of optimal parameters. The absolute value of the joint estimate of the upper and lower chirp parts is added to obtain the result of the joint estimator, which can be expressed as

[0100] E(r,v)=||E_up(r,v)|| 2 +||E_down(r,v)|| 2

[0101] The result of the joint estimator E(r,v) will be a set of optimal parameters , because all signals are in phase at this time. Therefore, the problem of finding the optimal parameters can be expressed as

[0102]

[0103] Finally, we get two optimal parameters After the estimated values ​​are obtained, the system calculates the signal attenuation by combining these estimated values

[0104]

[0105] The single-target joint estimation algorithm obtained above can be extended to multiple targets. Figure 3 As shown, the joint estimator first obtains the path and parameters corresponding to the strongest signal in the mixed signal, and then calculates the chirp mixed signal S M _up(t), S M_down(t) is used to reconstruct the strongest path signal, and the reconstructed signal is subtracted from the original mixed signal to obtain the remaining mixed signal. Then, the joint estimation algorithm is used again to estimate the strongest path signal from the remaining signals, and the joint estimation of the next target is performed to obtain the parameter information of the n+1th target. The joint estimation ends when the power of the remaining signal is less than a predefined threshold.

[0106] In the above formula, α and They represent theoretical value and calculated value respectively, which is convenient for description and distinction. The same is true for other parameters.

[0107] Steps of target matching tracking:

[0108] Since the target is moving, the position parameters and motion parameters of the target will change in different signal times. In order to continuously track the target and obtain the target's motion trajectory, the system needs to continuously perform joint estimation to obtain the position and motion parameters of the target at different times. In each time window, when there are multiple targets to be tracked, the joint estimation algorithm of the system outputs a parameter set for multiple targets. How to associate the position, i.e., the estimated values ​​of motion parameters, of the same target at different timestamps has become a major challenge for the system, especially when multiple targets have overlapping trajectories. The key idea of ​​the system's solution to match target parameters is that the movement of the target is continuous (especially human movement). Since the signal period is extremely short (only 0.08s), the difference between the estimated parameters obtained in two consecutive periods from the same target should be small. However, when the trajectories of two targets overlap, the parameters of the two targets become similar, resulting in tracking confusion. This implementation uses parameters from more dimensions to solve this problem, because within a given timestamp, it is rare for multiple targets to have similar parameters in all dimensions. For example, the ambiguity caused by similar distances can be resolved in high-dimensional space (i.e., two-dimensional range space and AoA). To quantify the overlap of different target trajectories, the system uses the weighted l1-norm

[0109]

[0110] where p t Represents a vector composed of the estimated values ​​of the four dimensional parameters (signal attenuation factor, distance information, angle information and speed information) at time t, expressed as β is a scale vector that normalizes the difference values ​​of the four dimensions to the same scale. The parameter vector obtained by jointly estimating two consecutive signal cycles with the smallest l1 distance is considered to belong to the same target.

[0111] This implementation implements a new type of non-contact tracking based on acoustic triangular chirp signals, which breaks the limit of motion tracking using acoustic signals and continuously tracks multiple targets in real time with centimeter-level accuracy. This implementation verifies its tracking performance through simulation and real-world scenarios, demonstrates its feasibility in real-world applications, and confirms its great potential for deployment on smart devices, providing a technical solution for a more natural and friendly human-computer interaction method from the perspective of acoustic wireless perception.

[0112] In order to realize this system, this paper proposes a signal model based on triangular chirp signal to fuse the distance, angle and speed information of the signals reflected from multiple targets, analyzes the signals through a joint estimation algorithm, realizes signal separation through signal elimination and reconstruction, and obtains the multi-dimensional parameters of each target.

[0113] This implementation proposes solutions to some interference system tracking accuracy issues. For example, the alignment algorithm can effectively solve the start time errors caused by the inherent limitations of hardware devices and software encoding, and the background multipath subtraction can subtract the direct path contained in the mixed reflection signal as well as most of the noise and other interference.

[0114] This implementation implemented this system on a Raspberry Pi equipped with a speaker and microphone array, and conducted a comprehensive test and evaluation of the system in terms of tracking accuracy, resolution, and factors affecting system performance.

[0115] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A gesture tracking method based on acoustic signals, characterized in that: The following steps are involved: Mixed signal calculation steps: the denoised triangular chirp signal is divided into an up-chirp signal and a down-chirp signal, the separated up-chirp and down-chirp signals are respectively orthogonally demodulated to obtain a preliminary mixed signal, the high-frequency part of the obtained preliminary mixed signal is filtered out to obtain the in-phase part and the orthogonal part of the signal respectively, the in-phase part and the orthogonal part of the up-chirp and down-chirp signals are respectively mixed to obtain an up-chirp mixed signal and a down-chirp mixed signal; The mixed signal of multiple targets includes multiple up-chirp mixed signals and multiple down-chirp mixed signals; the flight time of the reflected signal can be obtained from the up-chirp mixed signals; Signal separation steps: first eliminate the noise of the upper and lower mixed signals, then search for the optimal parameters of the distance information r, angle information θ and speed information v within the set parameter range, and then calculate the signal attenuation factor based on the optimal parameters of the distance information r and speed information v; when there are multiple targets, search for the strongest reflected signal in the current mixed signal of multiple targets each time through an iterative method to obtain the distance information, angle information and speed information parameters that match the signal, and then calculate the signal attenuation factor of the reflected signal. After the iteration, the parameters of different targets can be obtained; Steps in object matching tracking: Using weighted l1-norm where p t It represents the vector composed of the estimated values ​​of the four dimensional parameters at time t: signal attenuation factor, distance information, angle information and speed information, expressed as β is a scale vector that normalizes the difference values ​​of the four dimensions to the same scale; the parameter vector obtained by jointly estimating the two consecutive signal periods with the smallest l1 distance is considered to belong to the same target.

2. The method for tracking gestures based on acoustic signals according to claim 1, characterized in that: The calculation process of the mixed signal in the mixed signal calculation step is: First, the chirp signal S T _up is Hilbert transformed with its own signal and the up-chirp signal to obtain a 90° phase-shifted version S′ of the transmitted signal T _up are multiplied to obtain a first preliminary mixed signal and a second preliminary mixed signal, respectively. The first preliminary mixed signal and the second preliminary mixed signal are respectively passed through a low-pass filter to obtain the same part S I _up(t) and the orthogonal part S Q _up(t), combine the same part and the orthogonal part through a complex function, and finally obtain the up-chirp mixed signal. The down-chirp mixed signal is obtained in the same way; the expression is: In the formula, S M _up(t) is the up-chirp mixed signal, S M _down(t) is the down-chirp mixed signal.

3. The method for tracking gestures based on acoustic signals according to claim 2, characterized in that: The multi-target mixed signal is expressed as follows: Assuming there are L targets, the signal received by each microphone should be a composite signal formed by the superposition of all target reflection signals. At the i-th signal sampling point, the mixed signal of all target reflections received by the k-th microphone is expressed as follows: Among them, α l , r l , v l They respectively represent the amplitude attenuation factor, distance information and speed information of the lth target reflection signal.

4. The method for tracking gestures based on acoustic signals according to claim 3, characterized in that: The calculation process of the signal attenuation factor of a single target in the signal separation step is: Assume that a triangular chirp signal contains 2N sample points, where the up-chirp part and the down-chirp part each contain N samples, and there are K microphones; first consider the up-chirp part. For each microphone, there are N signal sample points. For each possible set of (r, v), the phase change is reconstructed based on the signal flight time, so each microphone can reconstruct a phase change estimation vector The size of the vector is N. The signal measurement value of each sample is multiplied by the conjugate of the phase change of the reconstructed signal based on the parameters, eliminating the phase shift of each sample in the distance and velocity dimensions, and all the results are added together to obtain the joint estimate of the upper chirp part, and the same is true for the lower chirp part; The absolute values ​​of the joint estimation values ​​of the upper and lower chirp parts are added to obtain the result of the joint estimator, which can be expressed as E(r,v)=||E_up(r,v)|| 2 +||E_down(r,v)|| 2 The result of the joint estimator E(r,v) will be a set of optimal parameters is maximized when , because all signals are in phase at this time; therefore, the problem of finding the optimal parameter can be expressed as Finally, we get two optimal parameters After the estimated values ​​of 5. The method for tracking gestures based on acoustic signals according to claim 4, characterized in that: The method of multi-target signal separation is: First, based on the joint estimator, the path and parameters corresponding to the strongest signal in the mixed signal are obtained, and the parameters obtained according to the up-and-down chirp mixed signal formula are used to reconstruct the strongest path signal, and the reconstructed signal is subtracted from the original mixed signal to obtain the remaining mixed signal; then, the joint estimator is used again to estimate the strongest path signal from the remaining signals, and the joint estimation of the next target is performed to obtain the parameter information of the n+1th target; until the power of the remaining signal is less than a predefined threshold, the joint estimation is terminated.

6. A gesture tracking system based on acoustic signals, characterized in that: include: Mixed signal calculation module: the denoised triangular chirp signal is divided into an up-chirp signal and a down-chirp signal, the separated up-chirp and down-chirp signals are respectively orthogonally demodulated to obtain a preliminary mixed signal, the high-frequency part of the obtained preliminary mixed signal is filtered out to obtain the in-phase part and the orthogonal part of the signal respectively, the in-phase part and the orthogonal part of the up-chirp and down-chirp signals are respectively mixed to obtain an up-chirp mixed signal and a down-chirp mixed signal; The mixed signal of multiple targets includes multiple up-chirp mixed signals and multiple down-chirp mixed signals; the flight time of the reflected signal can be obtained from the up-chirp mixed signals; Signal separation module: first eliminate the noise of the upper and lower mixed signals, then search for the optimal parameters of the distance information r, angle information θ and speed information v within the set parameter range, and then calculate the signal attenuation factor based on the optimal parameters of the distance information r and speed information v; when there are multiple targets, the strongest reflected signal in the current mixed signal of multiple targets is searched each time through an iterative method to obtain the distance information, angle information and speed information parameters matching the signal, and then calculate the signal attenuation factor of the reflected signal. After the iteration, the parameters of different targets can be obtained; Target matching tracking module: using weighted l1-norm where p t It represents the vector composed of the estimated values ​​of the four dimensional parameters at time t: signal attenuation factor, distance information, angle information and speed information, expressed as β is a scale vector that normalizes the difference values ​​of the four dimensions to the same scale; the parameter vector obtained by jointly estimating the two consecutive signal periods with the smallest l1 distance is considered to belong to the same target.

7. The gesture tracking system based on acoustic signals according to claim 6, characterized in that: The calculation process of the mixed signal in the mixed signal calculation module is: First, the chirp signal S T _up is Hilbert transformed with its own signal and the up-chirp signal to obtain a 90° phase-shifted version S′ of the transmitted signal T _up are multiplied to obtain a first preliminary mixed signal and a second preliminary mixed signal, respectively. The first preliminary mixed signal and the second preliminary mixed signal are respectively passed through a low-pass filter to obtain the same part S I _up(t) and the orthogonal part S Q _up(t), combine the same part and the orthogonal part through a complex function, and finally obtain the up-chirp mixed signal; the down-chirp mixed signal is obtained in the same way; the expression is: In the formula, S M _up(t) is the up-chirp mixed signal, S M _down(t) is the down-chirp mixed signal.

8. The gesture tracking system based on acoustic signals according to claim 7, characterized in that: The multi-target mixed signal is expressed as follows: Assuming there are L targets, the signal received by each microphone should be a composite signal formed by the superposition of all target reflection signals. At the i-th signal sampling point, the mixed signal of all target reflections received by the k-th microphone is expressed as follows: Among them, α l , r l , v l They respectively represent the amplitude attenuation factor, distance information and speed information of the lth target reflection signal.

9. The gesture tracking system based on acoustic signals according to claim 8, characterized in that: The calculation process of the signal attenuation factor of a single target in the signal separation module is: Assume that a triangular chirp signal contains 2N sample points, where the up-chirp part and the down-chirp part each contain N samples, and there are K microphones; first consider the up-chirp part. For each microphone, there are N signal sample points. For each possible set of (r, v), the phase change is reconstructed based on the signal flight time, so each microphone can reconstruct a phase change estimation vector The size of the vector is N. The signal measurement value of each sample is multiplied by the conjugate of the phase change of the reconstructed signal based on the parameters, eliminating the phase shift of each sample in the distance and velocity dimensions, and all the results are added together to obtain the joint estimate of the upper chirp part, and the same is true for the lower chirp part; The absolute values ​​of the joint estimates of the upper and lower chirp parts are added together to obtain the result of the joint estimator, which can be expressed as E(r,v)=||E_up(r,v)|| 2 +||E_down(r,v)|| 2 The result of the joint estimator E(r,v) will be a set of optimal parameters is maximized when , because all signals are in phase at this time; therefore, the problem of finding the optimal parameter can be expressed as Finally, we get two optimal parameters After the estimated values ​​of 10. The gesture tracking system based on acoustic signals according to claim 9, characterized in that: The method of multi-target signal separation is: First, based on the joint estimator, the path and parameters corresponding to the strongest signal in the mixed signal are obtained, and the parameters obtained according to the up-and-down chirp mixed signal formula are used to reconstruct the strongest path signal, and the reconstructed signal is subtracted from the original mixed signal to obtain the remaining mixed signal; then, the joint estimator is used again to estimate the strongest path signal from the remaining signals, and the joint estimation of the next target is performed to obtain the parameter information of the n+1th target; until the power of the remaining signal is less than a predefined threshold, the joint estimation is terminated.

Images