Daily scene millimeter wave sensing non-human target filtering method and system

Abstract

The invention provides a daily scene millimeter wave sensing non-human target filtering method and system, and the method comprises the steps: transmitting a millimeter wave to a target scene based on a millimeter wave radar, collecting an echo signal, mapping a point cloud sequence of the echo signal to a high-dimensional frequency domain space, and obtaining a first point cloud sequence; inputting the first point cloud sequence into a preset uniform motion filtering module, and mapping the output of the uniform motion filtering module to a time domain space to obtain a second point cloud sequence; the second point cloud sequence is input into a preset periodic motion filtering module, the periodic motion filtering module is provided with a multi-head attention layer, a linear layer and a multi-layer perceptron layer, and a point cloud feature sequence is output through the multi-layer perceptron layer; and inputting the point cloud feature sequence into a preset classifier to complete point cloud classification.

Description

TECHNICAL FIELD

[0001] The present solution relates to the technical field of millimeter wave, and particularly relates to a daily scene millimeter wave perception non-human target filtering method and system. BACKGROUND

[0002] In recent years, millimeter wave (mmWave) has shown great potential in human perception. With its fine-grained perception ability and privacy protection characteristics (i.e. without visual imaging), mmWave technology is expected to play an important role in home privacy scenarios. Especially with the increasing popularity of smart home systems, indoor intelligent applications based on mmWave perception (such as human presence detection and health monitoring, etc.) have attracted widespread attention.

[0003] However, the real indoor environment is not ideal (i.e. neither spacious and transparent, nor only human targets exist). Various non-human targets exist widely in real home scenarios, including swinging curtains, running fans, and false targets (i.e. "Ghost artifacts") generated by signal multipath effects (i.e. signals reflected on the surface of metal, glass, etc. and interact with human bodies). These non-human targets coexist with human targets, and existing algorithms are difficult to effectively distinguish between the two, resulting in frequent false alarms in downstream tasks, such as false detection of human presence due to curtain swinging. SUMMARY

[0004] In view of this, the embodiments of the present solution provide a daily scene millimeter wave perception non-human target filtering method and system to eliminate or improve one or more defects in the prior art.

[0005] One aspect of the present solution provides a daily scene millimeter wave perception non-human target filtering method, the steps of which include: Based on the millimeter wave radar, millimeter waves are emitted to the target scene, and echo signals are collected, and the point cloud sequence of the echo signals is mapped to a high-dimensional frequency domain space to obtain a first point cloud sequence; The first point cloud sequence is input into a preset uniform motion filtering module, and the output of the uniform motion filtering module is mapped to a time domain space to obtain a second point cloud sequence; The second point cloud sequence is input into a preset periodic motion filtering module, the periodic motion filtering module is provided with a multi-head attention layer, a linear layer and a multi-layer perception layer, and a point cloud feature sequence is output through the multi-layer perception layer; The point cloud feature sequence is input into a preset classifier to complete point cloud classification.

[0006] By adopting the scheme, on one hand, the scheme converts time domain features into frequency domain representation, filters out non-deterministic spectral peaks in a low frequency band by using dynamically learnable weights, so as to suppress uniform motion interference; for periodic motion interference (such as fan rotation and swing), the scheme divides a motion space into a plurality of sub-regions, and applies global autocorrelation analysis in each sub-region to separate different periodic components (such as rotation component and swing component) and suppress them respectively; on the other hand, the scheme uses a contrastive learning framework to align human kinematics high-dimensional features from an optical sensor with spatiotemporal features extracted by a millimeter wave, so as to filter out ghost artifact components that do not meet human kinematics constraints, and introduces a physics-informed neural network (PINN) to ensure the rationality of human kinematics and ensure accurate identification of human or non-human.

[0007] In some embodiments of the present scheme, in the step of mapping the point cloud sequence of the echo signal to a high-dimensional frequency domain space to obtain the first point cloud sequence, the dimension of the point cloud sequence is raised by a linear layer, and then the data in the time domain is mapped to the frequency domain by a fast Fourier transform to obtain the first point cloud sequence.

[0008] In some embodiments of the present scheme, in the step of inputting the first point cloud sequence into the preset uniform motion filtering module, the uniform motion filtering module extracts information in the frequency domain of the first point cloud sequence by using a multi-layer perception.

[0009] In some embodiments of the present scheme, in the step of inputting the second point cloud sequence into the preset periodic motion filtering module, linear transformation is performed on the corresponding data of each point cloud in the second point cloud sequence to obtain a query vector, a key vector and a value vector, and a periodic interference component is filtered by a linear layer and a multi-layer perception layer.

[0010] In some embodiments of the present scheme, in the step of inputting the point cloud feature sequence into the preset classifier to complete point cloud classification, the point cloud feature sequence and the point cloud data in the original point cloud sequence are dimensionally unified, so that the data dimension of each point cloud in the point cloud feature sequence is the same as the data dimension of the point cloud in the original point cloud sequence, and the point cloud feature sequence after dimensional unification is input into the preset classifier to complete point cloud classification.

[0011] In some embodiments of the present scheme, the method further includes pre-training the model, and the pre-training step includes calculating a point cloud classification loss, pre-training the model including the uniform motion filtering module, the periodic motion filtering module and the classifier based on the value of the point cloud classification loss, and calculating the point cloud classification loss by using the following formula: wherein, represents a value of the point cloud classification loss, T represents a frame number of the point cloud sequence of the echo signal, and N represents a number of point clouds in each frame of the point cloud sequence of the echo signal, represents a value of the point cloud classification loss of each point cloud, represents a real point cloud class label, represents a predicted point cloud class output by the classifier.

[0012] In some embodiments of the present solution, the step of pre-training further comprises: synchronously collecting data by using a depth camera and a millimeter wave radar to obtain joint coordinates, inputting the joint coordinates into a Ghost artifact filtering module, sequentially processing the joint coordinates by an implicit neural representation layer and a joint encoder in the Ghost artifact filtering module to obtain a joint feature sequence, and calculating a human body limb motion constraint loss based on the joint feature sequence; aligning features of the joint feature sequence and a point cloud feature sequence, and calculating a human body motion sequence feature alignment loss based on the joint feature sequence and the point cloud feature sequence after feature alignment.

[0013] In some embodiments of the present solution, in the step of calculating the human body motion sequence feature alignment loss based on the joint feature sequence and the point cloud feature sequence after feature alignment, the human body motion sequence feature alignment loss is calculated by using the following formula: wherein, represents a value of the human body motion sequence feature alignment loss, is a feature dimension after feature alignment of the joint feature sequence and the point cloud feature sequence, and respectively represent a covariance matrix of the point cloud feature sequence and the joint feature sequence, represents a Frobenius norm.

[0014] In some embodiments of the present solution, in the step of calculating the human body limb motion constraint loss based on the joint feature sequence, the human body limb motion constraint loss is calculated by using the following formula: wherein, represents a value of the human body limb motion constraint loss, represents a value of the joint coordinate constraint loss, a value representing mean square error, A is the number of frames of millimeter wave radar collected data, a value representing any one frame of millimeter wave radar collected data, is any one joint node, is a preset hyperparameter, a value representing a value representing coordinates of the joint node in the joint feature sequence, a value representing a value representing joint coordinates of the joint node in the millimeter wave radar collected data; a value representing a value representing a value representing the preset moment imbalance term of the joint node; a value representing an inertia matrix of the joint node, a value representing joint angular acceleration of the joint node, a value representing a preset a coupling value of centrifugal force and Coriolis force of the joint node, a value representing joint angular velocity of the joint node in a value representing a value representing joint gravity of the joint node.

[0015] In some embodiments of the present scheme, the step of pre-training further comprises calculating a total loss based on the human limb motion constraint loss, the human motion sequence feature alignment loss, and the point cloud classification loss, pre-training the model comprising the uniform motion filtering module, the periodic motion filtering module, and the classifier based on the total loss, and calculating the total loss using the following formula: wherein, a value representing the total loss, a value representing the point cloud classification loss, a value representing the human motion sequence feature alignment loss, a value representing the human limb motion constraint loss, and are both preset hyperparameters.

[0016] The second aspect of the present scheme also provides a daily scene millimeter wave perception non-human target filtering system, which comprises a computer device, the computer device comprising a processor and a memory, the memory storing computer instructions, and the processor being configured to execute the computer instructions stored in the memory, and the system implements the steps implemented by the method as described above.

[0017] The third aspect of the present solution also provides a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the steps of the aforementioned daily scene millimeter wave perception non-human target filtering method.

[0018] Additional advantages, objects, and features of the present solution will be apparent from the following detailed description and drawings, and from the claims.

[0019] Those skilled in the art will understand that the objects and advantages of the present solution are not limited to the above specifically described, and the above and other objects capable of being achieved by the present solution will be more clearly understood according to the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings, which are included to provide a further understanding of the present solution and constitute a part of this application, illustrate embodiments of the present solution and together with the description serve to explain the principle of the present solution.

[0021] Figure 1 A schematic diagram of an embodiment of the daily scene millimeter wave perception non-human target filtering method of the present solution; Figure 2 A schematic diagram of the overall architecture of the daily scene millimeter wave perception non-human target filtering method of the present solution; Figure 3 A schematic diagram of the processing architecture of the uniformity motion filtering module of the present solution; Figure 4 A schematic diagram of the processing architecture of the periodic motion filtering module of the present solution; Figure 5 A schematic diagram of the processing architecture of the Ghost artifact filtering module of the present solution. DETAILED DESCRIPTION

[0022] In order to make the objects, technical solutions and advantages of the present solution clearer, the present solution will be further described in detail below in combination with embodiments and drawings. Herein, the schematic embodiments of the present solution and their descriptions are used to explain the present solution, but not as a limitation to the present solution.

[0023] It should also be noted herein that, in order to avoid the present solution being obscured by unnecessary details, only structures and / or processing steps closely related to the present solution are shown in the drawings, and other details not closely related to the present solution are omitted.

[0024] In millimeter-wave target detection, traditional algorithms (e.g., Constant False Alarm Rate detection, CFAR) usually detect targets by extracting regions with signal energy exceeding an adaptive threshold. Recent research can be broadly categorized into two classes: one class adopts holistic modeling methods to distinguish human and non-human targets through machine learning-driven spatio-temporal feature analysis; the other class adopts semantic localization methods to extract human signals by annotating human-related regions in radar observations. However, existing methods fail to fully consider the key kinematic differences between human and non-human interferences (i.e., a human body can be regarded as a system composed of multiple rigid bodies, and the inter-joint motion of the human body is coordinated but not deterministic), resulting in poor performance in home scenarios.

[0025] To solve the above problems, the present scheme adopts a deep learning architecture to enable it to perceive the kinematic characteristics of heterogeneous targets, which utilizes two types of discriminative interference features: (1) statistical characteristics of regular motion interference (such as uniform motion of curtains, periodic motion of fans); (2) ghost artifacts caused by non-ideal reflection patterns of multipath effects. The framework includes three complementary modules: a frequency domain dynamic filtering module for suppressing interference caused by motion uniformity, a cross-subspace spatio-temporal autocorrelation analysis module for eliminating periodic motion interference, and a kinematic feature alignment module for suppressing ghost targets caused by multipath effects.

[0026] As shown in Figure 1 The present scheme proposes a method for filtering non-human targets in daily scene millimeter-wave perception, which includes the following steps: Step S100, based on the millimeter-wave radar, millimeter-wave is emitted to the target scene, and the echo signal is collected, the point cloud sequence of the echo signal is mapped to a high-dimensional frequency domain space, and a first point cloud sequence is obtained; In the specific implementation process, first, a conventional signal processing algorithm is used to generate a millimeter-wave point cloud: ① perform Range-FFT operation on the received millimeter-wave signal to obtain the Range spectrum of the signal; ② use a static filtering algorithm to remove static information in the environment to retain the motion characteristics of dynamic targets; ③ apply Capon Beamforming method to generate Range-Azimuth matrix for subsequent target spatial positioning; ④ use CFAR (Constant False Alarm Rate) algorithm to process the above matrix to remove noise signals and retain target echoes; ⑤ further perform Elevation estimation and Doppler-FFT operation in each Range-Azimuth dimension to obtain the Elevation information and radial velocity information of the target.

[0027] Step S200, input the first point cloud sequence into a preset uniform motion filtering module, and map the output of the uniform motion filtering module to a time domain space to obtain a second point cloud sequence; Step S300, input the second point cloud sequence into a preset periodic motion filtering module, the periodic motion filtering module is provided with a multi-head attention layer, a linear layer and a multi-layer perception layer, and a point cloud feature sequence is output through the multi-layer perception layer; Step S400, input the point cloud feature sequence into a preset classifier to complete point cloud classification.

[0028] The scheme is aimed at the two common non-human interferences in the above-mentioned daily scene, and needs to solve two key technical difficulties: 1. Uncertainty of regular motion interference; In a home scene, a moving object usually produces a regular motion trajectory, for example, swinging of a curtain and rotation of a fan. The curtain motion shows uniformity and can be regarded as a quasi-rigid body, the velocity of each part of which remains relatively consistent; the fan motion shows periodicity, the motion trajectory of which changes periodically with time and can be predicted through a linear model. However, such regular motion also has uncertainty: the uniformity of the curtain will be disturbed by wind or human body, resulting in inconstant velocity; the multi-degree-of-freedom periodic motion (such as rotation and swinging) of the fan will be coupled into non-periodic motion and the periodic timing of each motion component can also change. To solve the above problems, the scheme converts time domain features into frequency domain representation, filters out non-deterministic spectral peaks in the low frequency band by using dynamically learned weights, to suppress uniform motion interference; for periodic motion interference (such as fan rotation and swinging), the scheme divides the motion space into multiple sub-regions, and applies global autocorrelation analysis in each sub-region to separate different periodic components (such as rotation component and swinging component) and suppress them respectively.

[0029] 2. Mirror image of ghost artifact; in a home environment, due to multipath propagation effect, a spatio-temporal mirror image phenomenon will occur between human body movement and ghost artifact, and the dynamic symmetry of reflected signal will simulate the original movement trajectory. To solve this problem, the present scheme first finds that although the ghost artifact is coupled with the human body in kinematics, multipath attenuation will cause incomplete or distortion of its movement representation, for example, the ghost artifact only retains part of the limbs (such as arms), and the movement trajectory is distorted. This phenomenon violates the kinematic law of mirror reflection, and presents a nonlinear deformation similar to the "funhouse mirror" effect. Therefore, the present scheme uses a contrast learning framework to align the high-dimensional kinematic features of the human body from the optical sensor with the spatio-temporal features extracted by the millimeter wave, thereby filtering out the ghost artifact components that do not meet the kinematic constraints of the human body, and introducing a physics-informed neural network (PINNs) to ensure the reasonableness of human kinematics.

[0030] With the above scheme, on the one hand, by converting the time domain features into frequency domain representation, the dynamically learnable weights are used to filter out the non-deterministic spectral peaks in the low frequency band to suppress uniform motion interference; for periodic motion interference (such as fan rotation and swinging), the present scheme divides the motion space into multiple sub-regions, and applies global autocorrelation analysis in each sub-region to separate different periodic components (such as rotation component and swinging component) and suppress them respectively; on the other hand, by using a contrast learning framework, the high-dimensional kinematic features of the human body from the optical sensor are aligned with the spatio-temporal features extracted by the millimeter wave, thereby filtering out the ghost artifact components that do not meet the kinematic constraints of the human body, and introducing a physics-informed neural network (PINNs) to ensure the reasonableness of human kinematics and ensure accurate identification of human or non-human.

[0031] In some embodiments of the present scheme, in the step of mapping the point cloud sequence of the echo signal to a high-dimensional frequency domain space to obtain a first point cloud sequence, the dimension of the point cloud sequence is raised by a linear layer, and then the data in the time domain is mapped to the frequency domain by a fast Fourier transform to obtain the first point cloud sequence.

[0032] In some embodiments of the present scheme, in the step of inputting the first point cloud sequence into a preset uniform motion filtering module, the uniform motion filtering module extracts information in the frequency domain of the first point cloud sequence using a multi-layer perception.

[0033] In specific implementation, the uniform motion filtering module of the present scheme is used to effectively extract human point cloud features from the environment of non-human objects (such as curtains, plants) with uniform motion. To achieve this purpose, the present scheme designs a frequency domain filtering process based on neural network, which dynamically suppresses non-human interference by using the frequency spectrum characteristics of the motion signal.

[0034] Specifically, as shown in Figure 2 , the point cloud sequence defining the echo signal is , the present module first maps the input point cloud to a high-dimensional frequency domain space through a linear layer. Since the uniform motion interference (such as curtains, plants) shows obvious low-frequency components in the frequency domain, traditional filtering algorithms can suppress signals with significant isolated peaks in the frequency domain. However, in actual environment, the change of environmental conditions and time fluctuations, combined with external non-stationary interference, make these feature peaks unstable. As a result, the energy of low-frequency components is no longer concentrated at a fixed value, but fluctuates within a certain range, showing a diffuse distribution, and traditional filtering algorithms are difficult to completely suppress such interference.

[0035] Therefore, the model described in the present scheme uses learnable weights to weight each feature element in the frequency domain to suppress uniform motion interference. Then, the multi-layer perceptron (MLP) is used to extract the key information of the human motion signal in the frequency domain, and the feature is converted back to the time domain through inverse Fourier transform, providing input for the subsequent processing of the periodic motion filtering module. In summary, the uniform motion filtering module separates the motion signal using spectral information, where uniform motion objects (such as curtains, plants) usually have high energy concentration in a few low-frequency regions in the frequency domain, while human motion shows a more dispersed frequency spectrum distribution. Through training, the learnable weights dynamically adjust the contribution of each frequency component, thereby effectively attenuating the interference signal after inverse transformation and retaining only the key information reflecting human motion.

[0036] In some embodiments of the present scheme, in the step of inputting the second point cloud sequence into the preset periodic motion filtering module, linear transformation is performed on the corresponding data of each point cloud in the second point cloud sequence to obtain a query vector, a key vector and a value vector, and the periodic interference components are filtered through a linear layer and a multi-layer perceptron layer.

[0037] In specific implementation, the periodic motion filtering module of the present scheme is used to effectively extract human motion features from non-human objects (such as fans, air conditioners) with periodic motion. To effectively filter periodic motion interference features, the present module needs to solve two main technical challenges: (1) some objects are composed of multiple independent parts, if the motion periods of each part are not commensurate, considering the object as a whole will lead to merging into non-periodic motion; (2) the uncertainty of the duration of the interference object cycle makes it difficult for self-correlation analysis with fixed lag coefficients to effectively suppress interference.

[0038] The specific process is shown in Figure 4 To alleviate the interference caused by the independent motion of each part of the object, the present scheme first performs linear transformation on the spatial motion time series data of each object to obtain Queries (Q), Keys (K) and Values (V), and then uses the multi-head attention mechanism to project the different motion parts of the object into a high-dimensional subspace, i.e. (Q, K, V are divided into h heads, and each head has a dimension of , where is the total projection dimension). To solve the uncertainty of the motion period of the interfering object, the present scheme uses the self-attention mechanism to capture the time correlation of different time steps in the input sequence, i.e. Since the periodic subspace self-attention matrix exhibits a significant horizontal or vertical interval high-weight band (as shown in Figure 3 periodic characteristics), the model then concatenates each subspace feature and filters the periodic interference components through a linear layer and a multi-layer perception (MLP) layer, retaining only the key features of human motion.

[0039] In some embodiments of the present scheme, in the step of inputting the point cloud feature sequence into a preset classifier to complete point cloud classification, the dimension of the point cloud data in the original point cloud sequence is unified with the point cloud feature sequence, so that the data dimension of each point cloud in the point cloud feature sequence is the same as that of the point cloud in the original point cloud sequence. The point cloud feature sequence after dimension unification is input into a preset classifier to complete point cloud classification. The classifier can be a softmax classifier.

[0040] In some embodiments of the present scheme, the method further includes pre-training the model, and the pre-training step includes calculating a point cloud classification loss, and pre-training the model including the uniformity motion filtering module, the periodic motion filtering module and the classifier based on the value of the point cloud classification loss. The point cloud classification loss is calculated using the following formula: wherein, represents the value of the point cloud classification loss, T is the number of frames of the point cloud sequence of the echo signal, N is the number of point clouds in each frame of the point cloud sequence of the echo signal, represents the value of the point cloud classification loss of each point cloud, represents the true point cloud class label, represents the predicted point cloud class output by the classifier.

[0041] As shown in Figure 1 and 5 in some embodiments of the present scheme, the pre-training step further includes: The depth camera and the millimeter wave radar are used to synchronously collect data to obtain joint coordinates, the joint coordinates are input into a ghost artifact filtering module, and the joint coordinates are sequentially processed through an implicit neural representation layer and a joint encoder in the ghost artifact filtering module to obtain a joint feature sequence, and a human body limb motion constraint loss is calculated based on the joint feature sequence. The joint feature sequence and the point cloud feature sequence are aligned in features, and a human body motion sequence feature alignment loss is calculated based on the joint feature sequence and the point cloud feature sequence after feature alignment.

[0042] In the specific implementation process, the ghost artifact filtering module of the present scheme is used to extract human body point cloud features from an environment with ghost artifacts. In a home environment, when a signal propagates to a receiver via a human body in the environment and along multiple paths, ghost artifacts that are confused with real human body targets are easily generated. Since the overall motion trend of such artifacts is consistent with that of the real human body, it is difficult to directly filter them by relying only on position and trajectory information. On the other hand, ghost signals are attenuated after multiple reflections in the environment, resulting in a decrease in the number of point cloud points. However, in a quasi-stationary state (such as when a person is in a sitting position reading or typing), the point cloud density also presents a low level, so it is not possible to distinguish between real human bodies and ghosts by relying only on the number of point cloud points.

[0043] Inspired by the idea of contrastive learning, the specific process is as shown in Figure 5 The upper half of the figure is the main network, which processes the millimeter wave perception point cloud data collected in the real environment through a point cloud encoder (including a uniform motion filtering module and a periodic motion filtering module) to obtain a point cloud motion feature sequence containing only human bodies and ghosts. The lower half is the auxiliary network, which is a supplementary source of human body limb motion features. Specifically, the ghost artifact filtering module uses a depth camera to collect joint coordinates of human body motion in the real environment, and processes them through an implicit neural representation (INR) and a joint encoder to generate a feature sequence containing only human body motion features. Unlike the sparse and localized distribution characteristics of millimeter wave radar point clouds, depth camera data only focuses on human joint information and can provide more comprehensive representation of human body limb motion. Based on the principle of contrastive learning, the millimeter wave perception feature sequence and the human body joint feature sequence generated by the depth camera are aligned. Since the motion characteristics of ghosts do not conform to the motion rules of human joints, the millimeter wave perception features will tend to retain only the attributes related to human motion, thereby effectively filtering the ghost interference.

[0044] Due to the feature alignment in high-dimensional space, the point cloud does not need to be assigned to a specific body part, thus effectively avoiding the problem of inaccurate matching of point cloud and body part in previous work. In addition, to alleviate the joint coordinate fluctuation caused by the difference in time synchronization of the two modalities (the frame rate of the depth camera is higher than that of the millimeter wave radar), the INR module is used to enhance the continuity of the feature sequence between frames. Further, after the auxiliary network completes the feature alignment, it maps the features to joint coordinates to complete the network training. The scheme participates in pre-training through the Ghost artifact filtering module, so that the backbone network learns effective ghost feature filtering capability in the point cloud encoding process.

[0045] In some embodiments of the present scheme, in the step of calculating the human motion sequence feature alignment loss based on the joint feature sequence and the point cloud feature sequence after feature alignment, the human motion sequence feature alignment loss is calculated using the following formula: wherein, represents the value of the human motion sequence feature alignment loss, is the feature dimension after feature alignment of the joint feature sequence and the point cloud feature sequence, and respectively represent the covariance matrix of the point cloud feature sequence and the joint feature sequence, represents the Frobenius norm.

[0046] In some embodiments of the present scheme, in the step of calculating the human body limb motion constraint loss based on the joint feature sequence, the human body limb motion constraint loss is calculated using the following formula: wherein, represents the value of the human body limb motion constraint loss, represents the value of the joint coordinate constraint loss, represents the value of the mean square error, A is the number of frames of millimeter wave radar collected data, represents any frame of millimeter wave radar collected data, is any joint node, is a preset hyperparameter, represents the coordinate of the joint node in the frame in the joint feature sequence, represents the joint coordinate of the joint node in the frame in the millimeter wave radar collected data; denotes frame of the value of the preset moment imbalance term of the joint; denotes the inertia matrix of the joint, denotes the joint angular acceleration of the joint, denotes the preset the coupling value of the centrifugal force and the Coriolis force of the joint, denotes the joint angular velocity of the joint in frame of denotes the joint gravity of the joint.

[0047] In some embodiments of the present scheme, the step of pre-training further comprises calculating a total loss based on the human limb motion constraint loss, the human motion sequence feature alignment loss and the point cloud classification loss, pre-training the model comprising the uniform motion filtering module, the periodic motion filtering module and the classifier based on the total loss, and calculating the total loss using the following formula: wherein, denotes the value of the total loss, denotes the value of the point cloud classification loss, denotes the value of the human motion sequence feature alignment loss, denotes the value of the human limb motion constraint loss, and are preset hyperparameters.

[0048] In summary, the core of the present scheme is to combine physical information constraints and neural network-based methods to filter out interfering objects that do not conform to human motion characteristics. The overall architecture is shown in Figure 1 , which has three main modules for filtering the above three types of non-human interference: (1) Uniform motion interference. In a home scene, wind blowing or object collision may generate non-human noise such as curtains, plants, etc., which interfere with the human recognition task. Therefore, the present scheme proposes a uniform filtering module to filter out such noise. Since the distribution, speed, etc. of such objects in the point cloud space have uniformity, this module is based on Fourier transform (FT) and filtering ideas to filter out objects with significant feature peaks. However, the motion caused by wind blowing and collision in real scenes is not constant and varies with the environment and the material of the object itself. Therefore, the present scheme combines neural network algorithms to design a dynamically learnable filter to adapt to the uniform motion object noise in space.

[0049] (2) Periodic motion interference. For the high periodic non-human motion interference such as fans and air conditioners in real scenes, the periodic filtering module is proposed in the scheme. Since such objects move periodically along a specific trajectory, the module is based on autocorrelation analysis for filtering. However, the motion trajectory and periodic time window of different objects in different environments differ, and selecting a specified lag coefficient for autocorrelation analysis of the sequence can easily lead to false filtering. In addition, the independent motion of multiple parts of such objects may interfere with the periodicity judgment, for example, the rotation and oscillation of the fan. Therefore, the module uses a self-attention mechanism to establish the correlation of the point cloud features of each frame in the sequence from a global perspective, and combines a multi-head mechanism to separate different motion frequency components of each part of the object to analyze the periodic motion of each part, thereby achieving more accurate filtering.

[0050] (3) Ghost shadows generated by signal multipath through the human body show complexity and randomness in motion, and their trajectories are highly similar to human activities, which are the main interference that causes confusion in existing human recognition. However, non-ideal plane reflection and environmental complexity in daily scenes result in significant signal energy attenuation, making the Ghost shadow relatively incomplete or distorted compared to the human body. Based on this finding, the Ghost shadow filtering module is proposed in the scheme, which first extracts the human joint coordinates generated by the depth camera, and then uses a neural network to perform feature transformation and modal alignment on the point cloud of each object to eliminate the Ghost shadow that is inconsistent with the human motion features. In addition, the module is designed to facilitate the backbone network to gradually learn this function, and no auxiliary network is required in the inference stage.

[0051] After the point cloud sequence in the spatial domain is processed by the above three filtering modules, only the features related to human activities are retained. By connecting the features after the original sequence is mapped by a multi-layer perceptron (MLP) with the filtered features, this operation is equivalent to marking the human body in the target in a high-dimensional feature space. The finally trained model can distinguish between human and non-human entities in the spatial domain. In addition, for the inevitable loss and distortion of joint coordinates when only a single or a small number of auxiliary optical sensors can be deployed in a home environment, the scheme proposes a constraint based on a physical information neural network (PINN).

[0052] The scheme can accurately filter out non-human targets (including regular motion and ghost shadows) in complex daily scenes, achieving a classification accuracy of over 97% in distinguishing between human and non-human targets. In addition, the invention has strong generalization ability and maintains high classification accuracy for individuals and new motion interferences that have not been seen in different environments. The invention can be used as a base to avoid the decline in the effectiveness of subsequent downstream tasks (such as activity recognition, presence perception, etc.) caused by the presence of non-human targets.

[0053] The beneficial effects of the scheme include: 1. By analyzing the macro-kinematic differences between human and non-human targets, effective differentiation between the two can be achieved in complex home environments.

[0054] 2. Based on the motion difference prior, a corresponding neural network algorithm is designed, and the neural network is divided into three non-human interference filtering modules to dynamically remove the above interference.

[0055] 3. The uniform motion filtering module filters the dynamic peaks of uniform motion with learnable weights in the frequency domain; the periodic motion filtering module decomposes the periodic motion with multiple degrees of freedom into each subspace using multi-head self-attention, and uses the high-low interval weight of the self-attention matrix specific to periodic motion to filter the interference; the ghost artifact filtering module uses the distortion and absence of ghost relative to the human body to align the millimeter wave perception features with the human joint features of the depth camera using the idea of contrastive learning, to filter the ghost that does not conform to normal limb motion.

[0056] 4. The scheme designs a corresponding loss function for the requirement of the neural network structure for noise filtering: point cloud classification loss, human motion sequence feature alignment loss, and human limb motion constraint. The tasks are respectively to define the neural network human / non-human classification task, filter the ghost artifact features, and ensure the reasonableness of the depth camera human motion features.

[0057] 5. The scheme can accurately filter out non-human targets (including regular motion and ghost artifacts) in complex daily scenarios, achieving a classification accuracy of more than 97% in distinguishing between human and non-human targets. In addition, the invention has strong generalization ability and maintains high classification accuracy for individuals and new motion interference that have not been seen in different environments.

[0058] The embodiment of the scheme also provides a daily scene millimeter wave perception non-human target filtering system, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory, when the computer instructions are executed by the processor, the system realizes the steps implemented by the method as described above.

[0059] The embodiment of the scheme also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the steps implemented by the aforementioned daily scene millimeter wave perception non-human target filtering method. The computer readable storage medium can be a tangible storage medium, such as random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the technical field.

[0060] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this solution. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this solution are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave.

[0061] It should be clarified that this solution is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this solution is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this solution.

[0062] In this solution, features described and / or illustrated for one implementation may be used in the same or similar manner in one or more other implementations, and / or combined with or replaced by features of other implementations.

[0063] The above description is merely a preferred embodiment of this solution and is not intended to limit the solution. For those skilled in the art, various modifications and variations can be made to this solution. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this solution should be included within the protection scope of this solution.

Claims

1. A method for filtering non-human targets using millimeter-wave sensing in everyday scenarios, characterized in that, The steps of this method include: Based on the millimeter-wave radar transmitting millimeter waves to the target scene and collecting the echo signals, the point cloud sequence of the echo signals is mapped to a high-dimensional frequency domain space to obtain the first point cloud sequence; The first point cloud sequence is input into a preset uniformity motion filtering module, and the output of the uniformity motion filtering module is mapped to the time domain space to obtain the second point cloud sequence. The second point cloud sequence is input into a preset periodic motion filtering module, which is equipped with a multi-head attention layer, a linear layer and a multi-layer perceptron layer, and outputs a point cloud feature sequence through the multi-layer perceptron layer. The point cloud feature sequence is input into a preset classifier to complete the point cloud classification.

2. The method for filtering non-human targets in everyday scenarios using millimeter-wave sensing according to claim 1, characterized in that, In the step of mapping the point cloud sequence of the echo signal to a high-dimensional frequency domain space to obtain the first point cloud sequence, the dimension of the point cloud sequence is increased by a linear layer, and then the time domain data is mapped to the frequency domain by a fast Fourier transform to obtain the first point cloud sequence.

3. The method for filtering non-human targets in everyday scenarios using millimeter-wave sensing according to claim 1, characterized in that, In the step of inputting the first point cloud sequence into the preset uniformity motion filtering module, the uniformity motion filtering module uses a multilayer perceptron to extract information in the frequency domain of the first point cloud sequence.

4. The method for filtering non-human targets in everyday scenarios using millimeter-wave sensing according to claim 1, characterized in that, In the step of inputting the second point cloud sequence into the preset periodic motion filtering module, the corresponding data of each point cloud in the second point cloud sequence is linearly transformed to obtain the query vector, key vector and value vector, and the periodic interference components are filtered through the linear layer and the multi-layer perceptron layer.

5. The method for filtering non-human targets in everyday scenarios using millimeter-wave sensing according to claim 1, characterized in that, In the step of inputting the point cloud feature sequence into a preset classifier to complete point cloud classification, the point cloud feature sequence and the point cloud data in the original point cloud sequence are dimension-unified so that the data dimension of each point cloud in the point cloud feature sequence is the same as the data dimension of the point cloud in the original point cloud sequence. The point cloud feature sequence after dimension unification is then input into the preset classifier to complete point cloud classification.

6. The method for filtering non-human targets in everyday scenarios using millimeter-wave sensing according to any one of claims 1 to 5, characterized in that, The method further includes pre-training the model, which involves calculating a point cloud classification loss and pre-training the model, including a uniformity motion filtering module, a periodic motion filtering module, and a classifier, based on the value of the point cloud classification loss. The point cloud classification loss is calculated using the following formula: in, The value represents the point cloud classification loss, where T is the number of frames in the point cloud sequence of the echo signal, and N is the number of points in each frame of the point cloud sequence of the echo signal. This represents the point cloud classification loss value for each point cloud instance. Indicates the category label of the real point cloud. This indicates the predicted point cloud category output by the classifier.

7. The method for filtering non-human targets in everyday scenarios using millimeter-wave sensing according to claim 6, characterized in that, The pre-training steps also include: Data is collected synchronously using a depth camera and millimeter-wave radar to obtain joint coordinates. The joint coordinates are then input into the Ghost artifact filtering module. The Ghost artifact filtering module performs sequential processing through the implicit neural representation layer and joint encoder to obtain a joint feature sequence. The human limb motion constraint loss is calculated based on the joint feature sequence. The joint feature sequence and the point cloud feature sequence are aligned, and the human motion sequence feature alignment loss is calculated based on the aligned joint feature sequence and the point cloud feature sequence.

8. The method for filtering non-human targets in everyday scenarios using millimeter-wave sensing according to claim 7, characterized in that, In the step of calculating the feature alignment loss of the human motion sequence based on the joint feature sequence and the point cloud feature sequence after feature alignment, the following formula is used to calculate the feature alignment loss of the human motion sequence: in, This represents the value of the human motion sequence feature alignment loss. The feature dimension is the result of aligning the joint feature sequence with the point cloud feature sequence. and Let represent the covariance matrices of the point cloud feature sequence and the joint feature sequence, respectively. This represents the Frobenius norm.

9. The method for filtering non-human targets in everyday scenarios using millimeter-wave sensing according to claim 7, characterized in that, In the step of calculating the human limb motion constraint loss based on joint feature sequences, the following formula is used to calculate the human limb motion constraint loss: in, This represents the value indicating the loss of motor restraint in human limbs. This represents the value of the joint coordinate constraint loss. This represents the mean square error, where A is the number of frames in the millimeter-wave radar data acquisition. This represents any frame of data acquired by the millimeter-wave radar. For any key point, These are preset hyperparameters. express Frames The coordinates of the joints in the joint feature sequence express Frames Joint coordinates of key points in millimeter-wave radar data; express Frames The value of the preset torque imbalance term at the joint; express The inertia matrix of the joints, express Joint angular acceleration at the joint points Indicates preset The coupling value of centrifugal force and Coriolis force at the joint point express The key point is The joint angular velocity of the frame, express Joint gravity at the joint points.

10. A millimeter-wave sensing non-human target filtering system for everyday scenarios, characterized in that, The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method as described in any one of claims 1 to 9.

Images