Millimeter-wave-based method for non-carrying device positioning and home activity perception

Abstract

The present invention discloses a method for carrying-free device positioning and home activity perception based on millimeter waves, relates to the technical field of indoor positioning and activity perception, and aims to solve the lack of a technology for carrying-free device positioning and home activity perception that can achieve full-area perception, safety, reliability, and privacy protection; starting from radar point cloud data, a complete radar sensor spatial layout and parameter calibration scheme is designed, which can realize multi-radar collaborative continuous tracking and positioning, eliminate environmental occlusion and long-distance signal attenuation problems, and meet the needs of a large range of complex indoor environments; according to the sensor ranging and angle measurement range parameter characteristics and the physical characteristics of radar point clouds, a point cloud preprocessing method is designed to effectively eliminate the point cloud multipath problem in indoor environments and improve the robustness of subsequent positioning technology.

Description

Technical Field

[0001] The present invention relates to the field of indoor positioning and activity sensing technology, and specifically to a method for portable device-free positioning and home activity sensing based on millimeter waves. Background Art

[0002] Population aging is one of the serious problems that need to be solved in the rapid development of cities. Indoor elderly assisted living health monitoring technology has become an important research field. Location estimation is a key requirement for realizing indoor LBS (location-based services), such as medical care, security management, etc. For the health care and home activity perception of the elderly, it is not only necessary to estimate the user's location in real time, but also not to require users to carry professional positioning equipment at all times.

[0003] There are many methods for indoor location estimation, such as Bluetooth, UWB, audio, etc. These methods can provide high-precision tracking results, but they must meet the premise that the user and his signal receiving device are inseparable, which is a challenge for the elderly because they cannot always remember to wear or carry the device, and the device is always likely to be misplaced. In addition, vision-based technology can achieve device-free positioning, however, considering privacy issues and the invasiveness of cameras, user acceptance of vision-based technology in home and commercial environments is low.

[0004] There are also studies on device-free positioning based on millimeter-wave radar, but the existing solutions focus on sensing information about slow-moving personnel, including breathing, turning heads, etc., from the level of raw electronic signals. Moreover, the premise for acquiring the above-mentioned tiny movements is that the sensing distance needs to be close enough.

[0005] In summary, there is currently a lack of a technology that can achieve full-area perception, security, reliability, and privacy protection for device-free positioning and home activity perception. Summary of the invention

[0006] In view of the problems existing in the prior art, the present invention discloses a method for portable device positioning and home activity perception based on millimeter waves. The technical solution adopted is, including the spatial layout and parameter calibration of millimeter wave radar sensors, millimeter wave radar point cloud data preprocessing, pedestrian target positioning and tracking based on millimeter radar point cloud data, and user home activity perception based on millimeter wave radar.

[0007] As a preferred technical solution of the present invention, the millimeter-wave radar sensor spatial layout and parameter calibration include a millimeter-wave radar spatial layout solution for locating scene size and layout, thereby achieving full coverage of the scene while maximizing the reduction of sensor layout costs; a millimeter-wave radar sensor parameter calibration system, which calibrates the internal and external parameters of the sensor.

[0008] As a preferred technical solution of the present invention, the interference point cloud elimination method designed based on the millimeter-wave radar sensor's own parameters and the geometric constraints of the space in which it is located includes a multipath false point elimination method based on parameter constraints such as the millimeter-wave radar sensor's ranging range and field of view angle and the spatial geometric constraints of the positioning scene, and an environmental background point cloud elimination method based on the millimeter-wave radar point cloud features in an unmanned static environment.

[0009] As a preferred technical solution of the present invention, the pedestrian target positioning and tracking based on millimeter radar point cloud data includes designing a user positioning algorithm based on the optics clustering algorithm according to the continuity and aggregation of the dynamic target reflection point cloud in the spatial distribution; on the basis of the initial clustering of the point cloud, according to the characteristics such as the radial velocity of the point cloud, dynamic multipath false point elimination is realized; and further designing a target tracking and trajectory estimation algorithm based on the Hungarian algorithm and the Kalman filter algorithm.

[0010] As a preferred technical solution of the present invention, the user home activity perception based on millimeter-wave radar includes designing a user home activity perception method based on the combination of millimeter-wave radar positioning and tracking results and spatial map semantic information.

[0011] As an optimal technical solution of the present invention, the installation coordinates of the millimeter-wave radar sensor are designed according to the effective coverage range of the millimeter-wave radar sensor signal and the size of the scene area. The installation coordinates include X, Y, Z, posture and installation quantity M. The specific layout criteria are that the posture is pitch angle pitch, roll angle roll, and heading angle yaw. The installation height is between 2 meters and 3.5 meters. The installation position is close to the corner or close to the wall. The pitch angle of the installation posture is between -10° and -20°, the roll angle is 0±0.5°, and the heading angle is manually adjusted according to the shape of the positioning space and calibrated by a subsequent calibration method; the internal parameters of the sensor refer to the millimeter-wave radar ranging error scaling factor, and the external parameters of the sensor include the posture and position of the millimeter-wave radar sensor relative to the local navigation coordinate system; the millimeter-wave radar sensor is deployed in the scene, and the internal parameters and external parameters of each millimeter-wave radar sensor are calibrated. The calibration method includes the following steps:

[0012] Step 101, external parameter calibration: The conversion formula of point cloud data between the radar sensor coordinate system and the positioning coordinate system can be expressed as follows:

[0013]

[0014] Where P n =[x n ,y n ,z n ] T Represents the coordinates of the point cloud in the positioning coordinate system, P r =[xr ,y r ,z r ] T Represents the coordinates of the point cloud in the sensor coordinate system, Represents the transformation matrix from the sensor coordinate system to the positioning coordinate system, including three-dimensional rotation R = [pitch, roll, yaw] T and three-dimensional translation t=[t x ,t y ,t z ] T ; Among them, the pitch angle pitch and roll angle roll in the three-dimensional rotation are calibrated by directly measuring their true values ​​with the goniometer; the heading angle yaw and the three-dimensional translation t are calibrated using the least squares method:

[0015] Where P n =[x n ,y n ,z n ] T Represents the coordinates of the point cloud in the positioning coordinate system, P r =[x r ,y r ,z r ] T Represents the coordinates of the point cloud in the sensor coordinate system, Represents the transformation matrix from the sensor coordinate system to the positioning coordinate system, including three-dimensional rotation R = [pitch, roll, yaw] T and three-dimensional translation t=[t x ,t y ,t z ] T ; Among them, the pitch angle pitch and roll angle roll in the three-dimensional rotation are calibrated by directly measuring their true values ​​with the goniometer; the heading angle yaw and the three-dimensional translation t are calibrated using the least squares method:

[0016] Place corner reflectors for point cloud calibration at N known calibration points in the positioning space, and record the true value of each calibration point in the positioning coordinate system. And the measured values ​​in the sensor coordinate system Where i = 1, 2, ..., N; according to the known measured pitch angle and roll angle, the measured values ​​in the sensor coordinate system are converted to the intermediate transition coordinate system:

[0017]

[0018] The transformation relationship from the intermediate transition coordinate system to the positioning coordinate system can be expressed as:

[0019]

[0020] in

[0021]

[0022] The point cloud coordinate transformation relationship from the intermediate transition coordinate system to the positioning coordinate system can be rearranged as:

[0023]

[0024] make Re-describe the above formula as N (≥2) sets of measurement-true values ​​can form N (≥2) observation equations as above:

[0025]

[0026] Further order H=[H1 H2 … H N ] T ; So we can use the least squares algorithm to solve X:

[0027] X=(H T H) -1 H T Y

[0028] The heading parameter can be obtained by Calculated;

[0029] Step 102, internal parameter calibration:

[0030] Transformation matrix from known sensor coordinate system to positioning coordinate system Under the premise of , the three-axis error scaling factor of the radar point cloud measured by millimeter-wave radar sensing is: S = [s x s y s z ] T , then there is the following conversion relationship:

[0031]

[0032] Using N (≥2) sets of measurement-true values ​​and the least squares algorithm, the millimeter-wave radar three-axis error scaling factor S can be solved.

[0033] As a preferred technical solution of the present invention, the multipath false point elimination method based on parameter constraints such as the ranging range and field of view angle of the millimeter wave radar sensor and the spatial geometric constraints of the positioning scene includes the following steps:

[0034] Step 201, using the set ranging range r of the millimeter wave radar sensor, directly remove the point cloud whose ranging value is greater than the ranging range;

[0035] Step 202: using the horizontal and vertical field of view angle parameters θ of the millimeter wave radar sensor itself h ,θ v , directly eliminate the point clouds whose horizontal and vertical incident angles in the measured point cloud are larger than the corresponding field of view angle;

[0036] Step 203 , using the geometric size parameters [length, width, height] of the positioning space, directly remove the point clouds in the measured point cloud that exceed any parameters.

[0037] As a preferred technical solution of the present invention, the method for removing environmental background point cloud based on millimeter wave radar point cloud features in an unmanned static environment comprises the following steps:

[0038] Step 204, collecting millimeter wave radar point cloud data in an unmanned static environment for a period of time to establish a background point cloud data set E;

[0039] Step 205 , in each frame of point cloud data measured in real time, point clouds appearing in the background point cloud data set are eliminated to achieve background point cloud elimination.

[0040] As a preferred technical solution of the present invention, the user positioning algorithm based on the optics clustering algorithm is designed according to the continuity and aggregation of the dynamic target reflection point cloud in spatial distribution, including the following steps:

[0041] Consider the point cloud data set X = {x 1 ,x 2 ,…,x N}, given the clustering radius parameter ∈ and the minimum number of point clouds within the radius M, the core distance cd of each point cloud is calculated by the following steps = {cd 1 ,…,cd i ,…,cd N} and reachable distance rd = {rd 1 ,…,rd i ,…,rd N}, and the point cloud output list P:

[0042] Step 301: for any point cloud data set X={x 1 ,x 2 ,…,x N}, given clustering parameter ∈,M;

[0043] Step 302, calculate the distance matrix D between all point clouds, initialize the elements of the access list V that marks whether the point cloud has been processed to zero, indicating that all point clouds have not been visited; initialize the seed list S to be empty; initialize the point cloud output list P to be empty;

[0044] Step 303, taking out any unvisited point cloud from the point cloud data set X and adding it to the seed list S;

[0045] Step 304: Take the first point cloud x from the seed list S i , where 1≤i≤N represents the sequence number of the point cloud in the set X; in V, the point is set as visited and x i Add to the end of the output list P; get x i The point cloud set H within the neighborhood radius ∈; if the number of set H is less than M, then the point cloud x i The reachable distance and core distance are set to infinity, x i is judged as a non-core point; otherwise, x i Determine it as a core point and calculate the core distance cd of the point cloud by the following formula i and the reachable distance rd of each point in H j :

[0046]

[0047]

[0048] Among them, d(·) represents the Euclidean distance, H j (x i ) represents the distance x in the set H i The jth closest point, 1≤j≤num(H); finally, add the point cloud in set H to the seed list S and sort them according to the reachable distance;

[0049] Step 305: for the same point clouds in the list S, retain the point cloud with the smallest reachable distance, and update the reachable distance of the point cloud to the reachable distance of this point cloud;

[0050] Step 306: if the point cloud data set X is not empty, then repeat steps 3, 4 and 5; otherwise, output the output list P of the point cloud, as well as the core distance and reachable distance of each point;

[0051] Step 307, set a new neighborhood radius According to the output point cloud core distance cd and reachable distance rd, as well as the point cloud output list P, the cluster label of each point cloud is determined by the following algorithm:

[0052]

[0053] Step 308, according to the cluster labels L to which each point cloud outputted in step 7 belongs, extract the point clouds with the same cluster labels and combine them into a set, and take the average position of the point clouds in the set as the positioning result of the cluster;

[0054] In the above steps, if the seed list S is not empty, repeat steps 4 and 5; otherwise, execute step 6; based on the initial clustering of the point cloud, according to the characteristics of the point cloud radial velocity, dynamic multipath false point removal is realized. For each cluster, if the mean value of the radial velocity of the point cloud in the cluster is less than the set threshold μ, and its variance is less than the set threshold σ, all the point clouds in the cluster are determined to be multipath false points and are removed;

[0055] According to the above, a target tracking and trajectory estimation algorithm based on the Hungarian algorithm and the Kalman filter algorithm is further designed. A target tracking algorithm based on the Hungarian algorithm is designed. First, a new tracking task is created for the point cloud cluster that has newly entered the scene, and the tracking tasks that cannot be associated for a certain period of time are deleted; then the Hungarian algorithm is used to associate the output result of the point cloud cluster at the current moment with the current tracking task, which mainly includes the following steps:

[0056] Assume that the tracking task set at the current moment is U = {u 1 ,u 2 ,…,u n}, where u i ={p i ,v i}, p i and v i They represent the tracking task u respectively. i The position and velocity in the positioning coordinate system at the current moment; the positioning result set of each point cloud cluster at the current moment is W = {w 1 ,w 2 ,…,w m},in and Respectively represent the point cloud cluster positioning results w i The current position in the positioning coordinate system and the position relative to the millimeter wave radar sensor P L Radial velocity; construct the distance matrix between U and W elements The element in row i and column j Indicates position u i To position w j Euclidean distance; if m≠n, then for the matrix Add dummy rows or columns of all zeros to make it a square matrix; then, find the global best association by the following steps:

[0057] Step 401, let the matrix Subtract the smallest number in each row from each row, and then subtract the smallest number in each column from each column to get the matrix

[0058] Step 402: Cover the matrix with the least combination of rows and columns If all zero elements in , an optimal association can be found in If found, go to step 4; otherwise, go to step 3 and continue updating

[0059] Step 403, in step 402, use the minimum combination of rows and columns to cover the matrix After all the zero elements are found, find the minimum value of the remaining uncovered elements, the minimum value is defined as s, and all Subtract s from the uncovered elements in Add s to the elements in s that are covered by both the row and column in step 402, and get a new Then execute step 402;

[0060] Step 404, in The row i and column j of the zero element in represents the corresponding u i and w j There may be an association. If the row and column where the element is located are removed, the remaining submatrix We can still find an optimal association in i and w j Perform the associated output and continue in the remaining sub-matrices Find the best association in i and w j Cannot be associated, examine the remaining zero elements;

[0061] Design a trajectory estimation algorithm based on the Kalman filter algorithm:

[0062] Tracking task and point cloud cluster positioning results for a pair of associations at time k+1 First, the system equations are created using the uniform linear motion assumption. Estimate the state at the next moment:

[0063]

[0064] where ε k represents the zero-mean Gaussian white noise in the state estimation process, Δt represents the time interval between adjacent moments; k+1 point cloud cluster positioning result And the predicted estimation results The relationship between can be expressed by the following measurement equation:

[0065]

[0066] in Represents the velocity vector From the position To the millimeter wave radar sensor position P L Direction vector The angle between k Represents the measured zero-mean Gaussian white noise; using the Kalman filter method, the main steps of estimating the current tracking trajectory state include:

[0067] Step 405, predict the current state: in represents the state transfer matrix, represents the state estimation error covariance matrix, Q k represents the state noise error covariance;

[0068] Step 406 updates the predicted state using the measured value: first calculate the Kalman gain:

[0069] Then update the status:

[0070]

[0071] in, represents the measurement matrix; R k represents the measurement noise error covariance matrix.

[0072] As a preferred technical solution of the present invention, the user home activity perception method based on the combination of millimeter wave radar positioning tracking results and spatial map semantic information includes the following steps:

[0073] Step 501 utilizes the aforementioned target positioning and tracking results, combined with the semantic information of the space map, to realize the user's home activity perception;

[0074] Step 502: According to the home activity perception results, the user's daily routine is counted. According to the high correlation between spatial position and spatial activity in the home environment, when the positioning target is detected to be in the same location area within a certain period of time, it is determined that the positioning target is engaged in home activities related to the area. Suppose the set of activity types that need to be perceived is A = {a1, a2, ..., a n}, where a i ={name i ,region i ,duration i} indicates the i-th activity, name i Indicates the name of the activity, region i Indicates the area of ​​activity, duration i Indicates the minimum duration threshold for determining the activity type;

[0075] Step 503: When the positioning target appears in a certain area region within a period of time Γ i The cumulative time Γ Greater than or equal to duration i , then it is determined that the main activities during this period include activity name i .

[0076] The beneficial effects of the present invention are as follows: 1. Starting from radar point cloud data, a complete radar sensor spatial layout and parameter calibration scheme is designed, which can realize multi-radar collaborative continuous tracking and positioning, eliminate environmental occlusion and long-distance signal attenuation problems, and can meet a wide range of complex indoor environments.

[0077] 2. Based on the sensor ranging and angle measurement range parameter characteristics and the physical characteristics of the radar point cloud, a point cloud preprocessing method was designed to effectively eliminate the point cloud multipath problem in indoor environments and improve the robustness of subsequent positioning technology.

[0078] 3. The present invention uses a Kalman filter algorithm to perform a trajectory estimation method based on millimeter radar point cloud data. In the event that the target is lost, the Kalman filter can be used to predict it based on historical data, thereby ensuring the smoothness of target tracking.

[0079] 4. The present invention relies on the continuous position perception of the target user in complex indoor scenes, designs a user home activity perception method, and provides useful data for the analysis of the target user's home life status. At the same time, it can provide basic data for application services such as home health care for the elderly without any privacy leakage issues. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the following will briefly introduce the drawings required for the specific implementation or the prior art description. In all the drawings, each element or part is not necessarily drawn according to the actual scale.

[0081] Figure 1 It is a schematic diagram of the process of the millimeter wave-based non-carrying device positioning and home activity sensing technology of the present invention;

[0082] Figure 2 It is a schematic diagram of the overall algorithm flow of the present invention;

[0083] Figure 3 It is a schematic diagram of the sensor layout and sensor coordinate and positioning coordinate framework of the present invention;

[0084] Figure 4 This is a schematic diagram of the final layout of the millimeter wave radar of the present invention;

[0085] Figure 5This is a schematic diagram of the millimeter wave radar point cloud data preprocessing result of the present invention. Figure 1 ;

[0086] Figure 6 This is a schematic diagram of the millimeter wave radar point cloud data preprocessing result of the present invention. Figure 2 ;

[0087] Figure 7 A schematic diagram of clustering positioning results based on point cloud of the present invention;

[0088] Figure 8 It is a schematic diagram of the target continuous tracking trajectory result of the present invention. DETAILED DESCRIPTION

[0089] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. In the description of the present invention, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0090] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0091] Example 1

[0092] like Figures 1 to 8 As shown, the present invention discloses a method for portable device positioning and home activity perception based on millimeter waves, and the technical solution adopted includes sensor deployment, parameter calibration, data preprocessing, target positioning and tracking, and home activity perception.

[0093] The specific steps of this solution technology are as follows:

[0094] Step 1: Spatial layout and parameter calibration of millimeter wave radar sensors

[0095] According to the effective coverage range of the millimeter-wave radar sensor signal and the size of the scene area, the installation coordinates of the millimeter-wave radar sensor are designed. The installation coordinates include X, Y, and Z axes, posture, and installation quantity M. The posture also includes pitch angle pitch, roll angle roll, and heading angle yaw. The specific layout criteria are: the installation height is 2 meters, the installation position is close to the corner or close to the wall, the pitch angle of the installation posture is between -10°, the roll angle is 0°, and the heading angle is manually adjusted according to the shape of the positioning space and calibrated by the subsequent calibration method, such as Figure 3 As shown;

[0096] The parameter calibration of the sensor is divided into internal parameters and external parameters. The internal parameters of the sensor refer to the radar ranging error scaling factor, and the external parameters include the attitude and position of the millimeter-wave radar sensor relative to the local navigation coordinate system. The millimeter-wave radar sensor is deployed in the scene, and the internal parameters and external parameters of each millimeter-wave radar sensor are calibrated. The calibration method includes the following steps:

[0097] Step 101, external parameter calibration: The conversion formula of point cloud data between the radar sensor coordinate system and the positioning coordinate system can be expressed as follows:

[0098]

[0099] Where P n =[x n ,y n ,z n ] T Represents the coordinates of the point cloud in the positioning coordinate system, P r =[x r ,y r ,z r ] T Represents the coordinates of the point cloud in the sensor coordinate system, Represents the transformation matrix from the sensor coordinate system to the positioning coordinate system, including three-dimensional rotation R = [pitch, roll, yaw] T and three-dimensional translation t=[t x ,t y ,t z ] T ; Among them, the pitch angle pitch and roll angle roll in the three-dimensional rotation are calibrated by directly measuring their true values ​​with the goniometer; the heading angle yaw and the three-dimensional translation t are calibrated using the least squares method:

[0100] Place corner reflectors for point cloud calibration at N known calibration points in the positioning space, and record the true value of each calibration point in the positioning coordinate system. And the measured values ​​in the sensor coordinate system Where i = 1, 2, ..., N; according to the known measured pitch angle and roll angle, the measured values ​​in the sensor coordinate system are converted to the intermediate transition coordinate system:

[0101]

[0102] The transformation relationship from the intermediate transition coordinate system to the positioning coordinate system can be expressed as:

[0103]

[0104] in

[0105]

[0106] The point cloud coordinate transformation relationship from the intermediate transition coordinate system to the positioning coordinate system can be rearranged as:

[0107]

[0108] make Re-describe the above formula as N (≥2) sets of measurement-true values ​​can form N (≥2) observation equations as above:

[0109]

[0110] Further order H=[H1 H2 … H N ] T ; So we can use the least squares algorithm to solve X:

[0111] X=(H T H) -1 H T Y

[0112] The heading parameter can be obtained by Calculated;

[0113] Step 102, internal parameter calibration:

[0114] Transformation matrix from known sensor coordinate system to positioning coordinate system Under the premise of , the three-axis error scaling factor of the radar point cloud measured by millimeter-wave radar sensing is: S = [s x s y s z ] T , then there is the following conversion relationship:

[0115]

[0116] Using N (≥2) sets of measurement-true values ​​and the least squares algorithm, the millimeter-wave radar three-axis error scaling factor S can be solved;

[0117] Step 2: A multipath false point elimination method based on the millimeter-wave radar sensor ranging range, field of view and other parameter constraints and the positioning scene space geometric constraints, and an environmental background point cloud elimination method based on the millimeter-wave radar point cloud features in an unmanned static environment.

[0118] Step 201: Parameter constraints

[0119] Aiming at the influence of indoor multipath, a parameter constraint method such as the ranging range and field of view of the millimeter-wave radar sensor is first proposed. The point cloud with a ranging value greater than the ranging range is directly eliminated by using the set ranging range r of the millimeter-wave radar sensor. Secondly, the horizontal and vertical field of view parameters θ of the millimeter-wave radar sensor are used to calculate the distance between the points and the target points. h ,θ v , directly eliminate the point clouds whose horizontal and vertical incident angles in the measured point cloud are larger than the corresponding field of view angle;

[0120] Step 202: spatial geometric constraints

[0121] Following the above steps, we use the geometric size parameters [length, width, height] of the positioning space to directly remove the point clouds that exceed any parameters in the measured point cloud. The results before and after step 2 are compared. Figure 5 Figure 6 As shown;

[0122] Step 203: Eliminate the environmental background point cloud

[0123] First, the millimeter-wave radar point cloud data in an unmanned static environment for a period of time is collected to establish a background point cloud data set E; secondly, in each frame of point cloud data F measured in real time, the point cloud that appears in the background point cloud data set is removed to achieve background point cloud removal. The formula for removing background point clouds is as follows:

[0124] D=d(F i (x,y),E j (x,y))+δd(F i (v),E j (v))

[0125] Where d(·) represents the Euclidean distance, δ represents the coefficient, and if D is less than a given threshold, the i point in the current frame F is considered to be the same as the j point in the background point cloud dataset E, and then it is removed;

[0126] Step 3: According to the continuity and aggregation of the dynamic target reflection point cloud in spatial distribution, a user positioning algorithm based on optics clustering algorithm is designed.

[0127] Step 301: User positioning based on optics clustering algorithm

[0128] Consider the point cloud data set X = {x 1 ,x 2 ,…,x N}, given the clustering radius parameter ∈ and the minimum number of point clouds within the radius M, the core distance cd of each point cloud is calculated by the following steps = {cd 1 ,…,cd i ,…,cd N} and reachable distance rd = {rd 1 ,…,rd i ,…,rd N}, and the point cloud output list P. The specific process is as follows:

[0129] Step 301: for any point cloud data set X={x 1 ,x 2 ,…,x N}, given clustering parameter ∈,M;

[0130] Step 302, calculate the distance matrix D between all point clouds, initialize the elements of the access list V that marks whether the point cloud has been processed to zero, indicating that all point clouds have not been visited; initialize the seed list S to be empty; initialize the point cloud output list P to be empty;

[0131] Step 303, taking out any unvisited point cloud from the point cloud data set X and adding it to the seed list S;

[0132] Step 304: Take the first point cloud x from the seed list S i , where 1≤i≤N represents the sequence number of the point cloud in the set X; in V, the point is set as visited and x i Add to the end of the output list P; get x i The point cloud set H within the neighborhood radius ∈; if the number of set H is less than M, then the point cloud x i The reachable distance and core distance are set to infinity, x i is judged as a non-core point; otherwise, x i Determine it as a core point and calculate the core distance cd of the point cloud by the following formula i and the reachable distance rd of each point in H j :

[0133]

[0134]

[0135] Among them, d(·) represents the Euclidean distance, H j (xi ) represents the distance x in the set H i The jth closest point, 1≤j≤num(H); finally, add the point cloud in set H to the seed list S and sort them according to the reachable distance;

[0136] Step 305: for the same point clouds in the list S, retain the point cloud with the smallest reachable distance, and update the reachable distance of the point cloud to the reachable distance of this point cloud; if the seed list S is not empty, repeat steps 304 and 305; otherwise, execute step 306;

[0137] Step 306, if the point cloud data set X is not empty, then repeat steps (303), (304) and (305); otherwise, output the output list P of the point cloud, as well as the core distance and reachable distance of each point;

[0138] Step 307, set a new neighborhood radius According to the output point cloud core distance cd and reachable distance rd, as well as the point cloud output list P, the cluster label of each point cloud is determined by the following algorithm:

[0139]

[0140] Step 308: According to the cluster labels L to which each point cloud outputted in step 307 belongs, point clouds with the same cluster labels are extracted and combined into a set, and the average position of the point clouds in the set is taken as the positioning result of the cluster;

[0141] Step 4: Based on the initial clustering of the point cloud, the dynamic multipath false point removal is realized according to the characteristics of the point cloud such as radial velocity; the target tracking and trajectory estimation algorithm based on the Hungarian algorithm and Kalman filter algorithm is further designed;

[0142] Dynamic multipath false point elimination: According to the characteristics of point cloud radial velocity, dynamic multipath false point elimination is realized. For each cluster, if the mean value of the radial velocity of the point cloud in the cluster is less than the set threshold μ, and its variance is less than the set threshold σ, then all the point clouds in the cluster are judged as multipath false points and eliminated;

[0143] Target tracking and trajectory estimation algorithms based on the Hungarian algorithm and Kalman filter algorithm, such as Figure 8 The specific method is as follows:

[0144] Design a target tracking algorithm based on the Hungarian algorithm: First, create a new tracking task for the point cloud cluster that has newly entered the scene, and delete the tracking tasks that cannot be associated for a certain period of time;

[0145] Then, the Hungarian algorithm is used to associate the point cloud cluster output result at the current moment with the current tracking task. The main steps include:

[0146] Assume that the tracking task set at the current moment is U = {u 1 ,u 2 ,…,u n}, where u i ={p i ,v i}, p i and v i They represent the tracking task u respectively. i The position and velocity in the positioning coordinate system at the current moment; the positioning result set of each point cloud cluster at the current moment is W = {w 1 ,w 2 ,…,w m},in and Respectively represent the point cloud cluster positioning results w i The current position in the positioning coordinate system and the position relative to the millimeter wave radar sensor P L Radial velocity; construct the distance matrix between U and W elements The element in row i and column j Indicates position u i To position w j Euclidean distance; if m≠n, then for the matrix Add dummy rows or columns of all zeros to make it a square matrix; then, find the global best association by the following steps:

[0147] Step 401, let the matrix Subtract the smallest number in each row from each row, and then subtract the smallest number in each column from each column to get the matrix

[0148] Step 402: Cover the matrix with the least combination of rows and columns If all zero elements in , an optimal association can be found in If found, execute step 404; otherwise, execute step 403 and continue updating

[0149] Step 403, in step 402, use the minimum combination of rows and columns to cover the matrix After all the zero elements are found, find the minimum value (defined as s) among the remaining uncovered elements and set all Subtract s from the uncovered elements in Add s to the elements in s that are covered by both the row and column in step 402, and get a new Then execute step 402;

[0150] Step 404, in The row i and column j of the zero element in represents the corresponding u i and w j There may be an association. If the row and column where the element is located are removed, the remaining submatrix We can still find an optimal association in i and w j Perform the associated output and continue in the remaining sub-matrices Find the best association in i and w j Cannot be associated, examine the remaining zero elements;

[0151] Design a trajectory estimation algorithm based on the Kalman filter algorithm:

[0152] Tracking task and point cloud cluster positioning results for a pair of associations at time k+1 First, the system equations are created using the uniform linear motion assumption. Estimate the state at the next moment:

[0153]

[0154] where ε k represents the zero-mean Gaussian white noise in the state estimation process, and Δt represents the time interval between adjacent moments;

[0155] Point cloud cluster positioning results at time k+1 And the predicted estimation results The relationship between can be expressed by the following measurement equation:

[0156]

[0157] in Represents the velocity vector From the position To the millimeter wave radar sensor position P L Direction vector The angle between k Represents the measured zero-mean Gaussian white noise; using the Kalman filter method, the main steps of estimating the current tracking trajectory state include:

[0158] Step 405, predict the current state: in represents the state transfer matrix, represents the state estimation error covariance matrix, Q k represents the state noise error covariance;

[0159] Step 406, update the predicted state using the measured value: first calculate the Kalman gain:

[0160] Then update the status:

[0161]

[0162] in, represents the measurement matrix; R k represents the measurement noise error covariance matrix;

[0163] Step 5, home activity perception uses the above-mentioned target positioning and tracking results, combined with the semantic information of the spatial map, such as kitchen, living room, bedroom, bathroom, door and other information, to realize the user's home activity perception, including sleeping, cooking, going to the toilet, going out, going home, watching TV, etc.; then according to the home activity perception results, the user's daily routine is counted; according to the high correlation between spatial position and spatial activity in the home environment, when the positioning target is detected to be in the same location area within a certain period of time, it is determined that the positioning target is engaged in home activities related to the area; for example, when the position of the positioning target is continuously monitored in the kitchen, it is determined that the target user's home activity is "cooking"; let the set of activity types that need to be perceived be A = {a1, a2, ..., a n}, where a i ={name i ,region i ,duration i} indicates the i-th activity, name i Indicates the name of the activity, region i Indicates the area of ​​activity, duration i Indicates the minimum duration threshold for determining the activity type; when the positioning target appears in a certain area region within a period of time Γ i The cumulative time Γ′ is greater than or equal to duration i , then it is determined that the main activities during this period include activity name i .

[0164] Although the specific embodiments of the present invention are described in detail above, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the purpose of the present invention, and modifications or deformations without creative labor are still within the protection scope of the present invention.

Claims

1. A method for positioning and sensing home activities without portable devices based on millimeter waves, characterized by: The following steps are involved: Step 1: spatial layout and parameter calibration of millimeter wave radar sensors; Step 2: millimeter wave radar point cloud data processing; Steps 3 to 4: Pedestrian target positioning and tracking based on millimeter radar point cloud data; Step 5: User home activity perception based on millimeter-wave radar; The millimeter wave radar sensor spatial layout and parameter calibration include a millimeter wave radar spatial layout scheme for locating scene size and layout; a millimeter wave radar sensor parameter calibration system for calibrating internal and external parameters of the sensor; According to the effective coverage range of the millimeter-wave radar sensor signal and the size of the scene area, the installation coordinates of the millimeter-wave radar sensor are designed. The installation coordinates include X, Y, Z, posture and installation quantity M. The specific layout criteria are that the posture is pitch angle pitch, roll angle roll, and heading angle yaw. The installation height is between 2 meters and 3.5 meters. The installation position is close to the corner or close to the wall. The pitch angle of the installation posture is between -10° and -20°, the roll angle is 0±0.5°, and the heading angle is manually adjusted according to the shape of the positioning space and calibrated by the subsequent calibration method. The internal parameters of the sensor refer to the millimeter-wave radar ranging error scaling factor, and the external parameters of the sensor include the attitude and position of the millimeter-wave radar sensor relative to the local navigation coordinate system. The millimeter-wave radar sensor is deployed in the scene, and the internal parameters and external parameters of each millimeter-wave radar sensor are calibrated. The calibration method includes the following steps: Step 101, external parameter calibration: The conversion formula of point cloud data between the radar sensor coordinate system and the positioning coordinate system is expressed as follows: Where P n =[x n ,y n ,z n ] T Represents the coordinates of the point cloud in the positioning coordinate system, P r =[x r ,y r ,z r ] T Represents the coordinates of the point cloud in the sensor coordinate system, Represents the transformation matrix from the sensor coordinate system to the positioning coordinate system, including three-dimensional rotation R = [pitch, roll, yaw] T and three-dimensional translation t=[t x ,t y ,t z ] T ; Among them, the pitch angle pitch and roll angle roll in the three-dimensional rotation are calibrated by directly measuring their true values ​​with the goniometer; the heading angle yaw and the three-dimensional translation t are calibrated using the least squares method: Place corner reflectors for point cloud calibration at N known calibration points in the positioning space, and record the true value of each calibration point in the positioning coordinate system. And the measured values ​​in the sensor coordinate system Where i = 1, 2, ..., N; according to the known measured pitch angle and roll angle, the measured values ​​in the sensor coordinate system are converted to the intermediate transition coordinate system: The transformation relationship from the intermediate transition coordinate system to the positioning coordinate system is expressed as: in The point cloud coordinate transformation relationship from the intermediate transition coordinate system to the positioning coordinate system is rearranged as follows: make Re-describe the above formula as N≥2 sets of measurement-true values ​​form N≥2 observation equations: Further order H=[H1 H2 … H N ] T ; Then, we can use the least squares algorithm to solve X: X=(H T H) -1 H T Y The heading parameter is Calculated; Step 102, internal parameter calibration: Transformation matrix from known sensor coordinate system to positioning coordinate system Under the premise of , the three-axis error scaling factor of the radar point cloud measured by millimeter-wave radar sensing is: S = [s x s y s z ] T , then there is the following conversion relationship: Using N≥2 sets of measurement-true values ​​and the least squares algorithm, the millimeter-wave radar three-axis error scaling factor S can be solved; The pedestrian target positioning and tracking based on millimeter wave radar point cloud data includes designing a user positioning algorithm based on the optics clustering algorithm according to the continuity and aggregation of the dynamic target reflection point cloud in spatial distribution; based on the initial clustering of the point cloud, the dynamic multipath false point removal is realized according to the radial velocity characteristics of the point cloud; and further designing a target tracking and trajectory estimation algorithm based on the Hungarian algorithm and the Kalman filter algorithm; According to the continuity and aggregation of the dynamic target reflection point cloud in spatial distribution, designing a user positioning algorithm based on the optics clustering algorithm includes the following steps: Consider the point cloud data set X = {x 1 ,x 2 ,…,x N }, given the clustering radius parameter ∈ and the minimum number of point clouds within the radius M, the core distance cd of each point cloud is calculated by the following steps = {cd 1 ,…,cd i ,…,cd N } and reachable distance rd = {rd 1 ,…,rd i ,…,rd N }, and the point cloud output list P: Step 301: for any point cloud data set X={x 1 ,x 2 ,…,x N }, given clustering parameter ∈,M; Step 302, calculate the distance matrix D between all point clouds, initialize the elements of the access list V that marks whether the point cloud has been processed to zero, indicating that all point clouds have not been visited; initialize the seed list S to be empty; initialize the point cloud output list P to be empty; Step 303, taking out any unvisited point cloud from the point cloud data set X and adding it to the seed list S; Step 304: Take the first point cloud x from the seed list S i , where 1≤i≤N represents the sequence number of the point cloud in the set X; in V, the point is set as visited and x i Add to the end of the output list P; get x i The point cloud set H within the neighborhood radius ∈; if the number of point clouds in the set H is less than M, then the point cloud x i The reachable distance and core distance are set to infinity, x i is judged as a non-core point; otherwise, x i Determine it as a core point and calculate the core distance cd of the point cloud by the following formula i and the reachable distance rd of each point in H j : Among them, d(·) represents the Euclidean distance, H j (x i ) represents the distance x in the set H i The jth closest point, 1≤j≤num(H); finally, add the point cloud in set H to the seed list S and sort them according to the reachable distance; Step 305: for the same point clouds in the list S, retain the point cloud with the smallest reachable distance, and update the reachable distance of the point cloud to the reachable distance of this point cloud; Step 306: if the point cloud data set X is not empty, then repeat steps 303, 304 and 305; otherwise, output the output list P of the point cloud, as well as the core distance and reachable distance of each point; Step 307, set a new neighborhood radius According to the output point cloud core distance cd and reachable distance rd, as well as the point cloud output list P, the cluster label of each point cloud is determined by the following algorithm: Algorithm 1: Optics millimeter wave radar point cloud clustering extraction Input: point cloud core distance cd, reachable distance rd, output list P, clustering parameters Initialization: Cluster label: clusterID=-1, k=1 Output: Cluster labels of each point cloud L = {l 1 ,l 2 ,…,l N } For i = 1, 2, ..., N: j = P(i); if or j =+∞, then: If cd j ≠+∞ and but: clusterID=k;k=k+1;l j =clusterID; no but: l j ＝-1； no but: l j =clusterID; Step 308: According to the cluster labels L to which each point cloud outputted in step 307 belongs, point clouds with the same cluster labels are extracted and combined into a set, and the average position of the point clouds in the set is taken as the positioning result of the cluster; Based on the initial clustering of point clouds, dynamic multipath false point elimination is achieved according to the radial velocity characteristics of point clouds. For each cluster, if the mean of the radial velocity of the point cloud in the cluster is less than the set threshold μ, and its variance is less than the set threshold σ, all the point clouds in the cluster are determined to be multipath false points and eliminated. The target tracking and trajectory estimation algorithm based on the Hungarian algorithm and the Kalman filter algorithm is further designed. The target tracking algorithm based on the Hungarian algorithm is designed. First, a new tracking task is created for the point cloud cluster that has newly entered the scene, and the tracking tasks that cannot be associated for a certain period of time are deleted. Then, the output result of the point cloud cluster at the current moment is associated with the current tracking task using the Hungarian algorithm, including the following steps: Assume that the tracking task set at the current moment is U = {u 1 ,u 2 ,…,u n }, where u i ={p i ,v i }, p i and v i They represent the tracking task u respectively. i The position and velocity in the positioning coordinate system at the current moment; the positioning result set of each point cloud cluster at the current moment is W = {w 1 ,w 2 ,…,w m },in and Respectively represent the point cloud cluster positioning results w i The current position in the positioning coordinate system and the position relative to the millimeter wave radar sensor P L Radial velocity; construct the distance matrix between U and W elements The element in row i and column j Indicates position u i To position w j Euclidean distance; if m≠n, then for the matrix Add dummy rows or columns of all zeros to make it a square matrix; then, find the global best association by the following steps: Step 401, let the matrix Subtract the smallest number in each row from each row, and then subtract the smallest number in each column from each column to get the matrix Step 402: Cover the matrix with the least combination of rows and columns If all zero elements in , an optimal association can be found in If found, execute step 404; otherwise, execute step 403 and continue updating Step 403, in step 402, use the minimum combination of rows and columns to cover the matrix After all the zero elements are found, find the minimum value of the remaining uncovered elements, the minimum value is defined as s, and all Subtract s from the uncovered elements in Add s to the elements in s that are covered by both the row and column in step 402, and get a new Then execute step 402; Step 404, in The row i and column j of the zero element in represents the corresponding u i and w j There may be an association. If the row and column where the element is located are removed, the remaining submatrix We can still find an optimal association in i and w j Perform the associated output and continue in the remaining sub-matrices Find the best association in i and w j Cannot be associated, examine the remaining zero elements; Design a trajectory estimation algorithm based on the Kalman filter algorithm: Tracking task and point cloud cluster positioning results for a pair of associations at time k+1 First, the system equations are created using the uniform linear motion assumption. Estimate the state at the next moment: where ε k represents the zero-mean Gaussian white noise in the state estimation process, and Δt represents the time interval between adjacent moments; Point cloud cluster positioning results at time k+1 And the predicted estimation results The relationship between them can be expressed by the following measurement equation: in Represents the velocity vector From the position To the millimeter wave radar sensor position P L Direction vector The angle between k Represents the measured zero-mean Gaussian white noise; using the Kalman filter method, the steps of estimating the current tracking trajectory state include: Step 405, predict the current state: in represents the state transfer matrix, represents the state estimation error covariance matrix, Q k represents the state noise error covariance; Step 406 updates the predicted state using the measured value: first calculate the Kalman gain: Then update the status: and in, represents the measurement matrix; R k represents the measurement noise error covariance matrix.

2. The method for non-carrying device positioning and home activity sensing based on millimeter waves according to claim 1 is characterized in that: An interference point cloud elimination method based on the millimeter-wave radar sensor's own parameters and the geometric constraints of the space in which it is located is designed, including a multipath false point elimination method based on the millimeter-wave radar sensor's ranging range, field of view angle parameter constraints and positioning scene space geometric constraints, and an environmental background point cloud elimination method based on the millimeter-wave radar point cloud characteristics in an unmanned static environment.

3. The method for non-carrying device positioning and home activity sensing based on millimeter waves according to claim 1, characterized in that: The user home activity perception based on millimeter wave radar includes designing a user home activity perception method based on the combination of millimeter wave radar positioning and tracking results and spatial map semantic information.

4. The method for non-carrying device positioning and home activity sensing based on millimeter waves according to claim 2, characterized in that: The multipath false point elimination method based on the ranging range of the millimeter wave radar sensor, the field of view angle parameter constraints and the positioning scene space geometric constraints comprises the following steps: Step 201, using the set ranging range r of the millimeter wave radar sensor, directly remove the point cloud whose ranging value is greater than the ranging range; Step 202: using the horizontal and vertical field of view angle parameters θ of the millimeter wave radar sensor itself h ,θ v , directly eliminate the point clouds whose horizontal and vertical incident angles in the measured point cloud are larger than the corresponding field of view angle; Step 203 , using the geometric size parameters [length, width, height] of the positioning space, directly remove the point clouds in the measured point cloud that exceed any parameters.

5. The method for non-carrying device positioning and home activity sensing based on millimeter waves according to claim 2, characterized in that: The method for removing environmental background point cloud based on millimeter wave radar point cloud features in an unmanned static environment comprises the following steps: Step 204, collecting millimeter wave radar point cloud data in an unmanned static environment for a period of time to establish a background point cloud data set E; Step 205 , in each frame of point cloud data measured in real time, point clouds appearing in the background point cloud data set are eliminated to achieve background point cloud elimination.

6. The method for non-carrying device positioning and home activity sensing based on millimeter waves according to claim 3, characterized in that: The method for perceiving user home activities based on the combination of millimeter wave radar positioning and tracking results and spatial map semantic information comprises the following steps: Step 501 utilizes the aforementioned target positioning and tracking results, combined with the semantic information of the space map, to realize the user's home activity perception; Step 502: According to the home activity perception results, the user's daily routine is counted. According to the high correlation between spatial position and spatial activity in the home environment, when the positioning target is detected to be in the same location area within a certain period of time, it is determined that the positioning target is engaged in home activities related to the area. Suppose the set of activity types that need to be perceived is A = {a1, a2, ..., a n }, where a i ={name i ,region i ,duration i } indicates the i-th activity, name i Indicates the name of the activity, region i Indicates the area of ​​activity, duration i Indicates the minimum duration threshold for determining the activity type; Step 503: When the positioning target appears in a certain area region within a period of time Γ i The cumulative time Γ' is greater than or equal to duration i , then it is determined that the activities during this period include activity name i .

Images