Tracking target state recognition method and device based on millimeter wave sensing

Abstract

The present invention provides a method and device for identifying the state of a tracking target based on millimeter wave perception, the method comprising: acquiring point cloud data in real time based on a millimeter wave signal received by a millimeter wave radar; tracking a target based on the acquired point cloud data, and determining spatial coordinate data of the tracking target; splitting the point cloud data related to the tracking target within a time window of a predetermined length by frame, and generating a multidimensional data matrix corresponding to each frame of data based on the split data; and inputting the multidimensional data matrix into a pre-trained neural network model, the neural network model comprising a multi-layer convolutional layer, an LSTM network layer and a fully connected classifier, the fully connected classifier outputting a prediction result of the state of the tracking target, the state of the tracking target comprising a plurality of action states; and determining a final state of the tracking target based on the prediction result of the tracking target state and a pre-established finite state machine.

Description

Technical Field

[0001] The present invention relates to the field of millimeter wave sensing technology, and in particular to a method and device for identifying the state of a tracking target based on millimeter wave sensing. Background Art

[0002] Millimeter wave technology can provide sub-millimeter accuracy and can penetrate certain specific materials, such as plastics and clothing, and is not easily affected by environmental conditions such as rain, fog, dust and snow. Millimeter waves can also be used as a very valuable sensing technology to detect targets and provide the distance, speed and angle of these targets. Millimeter wave sensors have been widely used in fields such as autonomous driving, industry, drones and medical applications. Various millimeter wave sensing technologies, such as millimeter wave-based gesture recognition, gait recognition, and heartbeat and breathing recognition, can provide users with a more intelligent, convenient and interesting product experience. With the rapid development and maturity of 5G technology, millimeter wave wireless modules will be widely installed in mobile phones, wearable devices and more IoT devices.

[0003] Millimeter wave technology can realize non-contact personnel status recognition without cameras, which has the advantages of privacy protection, good comfort, passive monitoring, etc. Existing status recognition also includes video-based status recognition technology, wearable sensor-based status recognition technology and RF signal-based status recognition technology.

[0004] Among them, the state recognition technology based on video requires the installation of a camera device within the sensing range, and through continuous video shooting, the continuous action changes of the target are captured, and then the position of the target in each frame is detected by algorithms such as target detection. The state recognition algorithm is used to extract the actions completed by the target in multiple frames. Machine learning or deep learning methods are usually used for state recognition. Since the state recognition technology based on video uses camera equipment, there is a risk of privacy leakage in some sensitive environments, such as bedrooms and bathrooms, and user acceptance is low, while the cost is high. In addition, the video-based method has high requirements for light and cannot work in adverse environments such as smoke, rain and snow.

[0005] State recognition technology based on wearable devices requires users to actively wear relevant sensor devices, and use instruments such as gyroscopes and accelerometers to capture changes in the user's limbs or torso. Through data denoising, processing and analysis, the sensor signal change curve of the user in different states can be obtained, and the user state can be identified by matching with the state curve template or other identification methods. However, wearing sensors will affect the user experience, the comfort level is low, and the continuity of data collection depends on the user's cooperation. In addition, there are additional burdens such as device charging and regular maintenance. Moreover, since the sensors are worn on different parts of the user, such as the wrist or ankle, the data obtained mainly focuses on the changes in the part when completing a certain action, rather than the user's whole body movement information, so there are fewer recognizable actions and the accuracy is not very high. For example, the recognition rate for slow falls is low.

[0006] The state recognition technology based on RF (radio frequency) signals uses radar to emit RF waves and use the reflected signals to obtain the position and state of the target. However, the resolution of distance measured by traditional RF signals is far worse than that of millimeter wave signals, and positioning and tracking are not as precise as millimeter waves.

[0007] There is currently a technology that uses frequency modulated continuous wave (FMCW) as a millimeter wave signal to achieve trajectory tracking and fall detection, but this technology is currently limited to identifying limited states, such as falls, that is, the identifiable states are relatively single. In addition, it uses thresholds for logical judgment based on point cloud data information, and the accuracy is not high enough.

[0008] Although millimeter wave recognition technology is beginning to be applied to personnel status recognition, how to more accurately recognize more types of personnel status remains a problem to be solved. Summary of the invention

[0009] In view of this, an embodiment of the present invention provides a method and device for identifying a tracking target state based on millimeter wave sensing to eliminate or improve one or more defects existing in the prior art.

[0010] One aspect of the present invention provides a method for identifying a tracking target state based on millimeter wave sensing, the method comprising the following steps:

[0011] Based on the millimeter wave signal received by the millimeter wave radar, point cloud data is acquired in real time, wherein the point cloud data includes spatial coordinate data, Doppler velocity information and signal-to-noise ratio of each detection point;

[0012] Perform target tracking based on the obtained point cloud data to determine the spatial coordinate data of the tracked target;

[0013] Splitting the point cloud data related to the tracking target within a time window of a predetermined length by frame, generating a multidimensional data matrix corresponding to each frame of data based on the split data, wherein the multidimensional data matrix includes at least data of multiple attributes, and the data of the multiple attributes includes a spatial coordinate attribute and at least one of the following attributes: a Doppler velocity attribute and a signal-to-noise ratio attribute;

[0014] Input the multidimensional data matrix corresponding to each frame of data into a pre-trained neural network model, and output the prediction result of the tracking target state, wherein the pre-trained neural network model includes a multi-layer convolution layer with a channel number equal to the number of attributes of the data in the multi-dimensional data matrix, an LSTM network layer and a fully connected classifier; the data of each attribute in the multi-dimensional data matrix corresponding to each frame of data is respectively input into the multi-layer convolution layer of the corresponding channel, and the feature sequence corresponding to each frame is output as the input of the LSTM network layer, and the output of the LSTM network layer is fused and input into the fully connected classifier, so that the fully connected classifier outputs the prediction result of the state of the tracking target, and the state of the tracking target includes multiple action states;

[0015] The final state of the tracking target is determined based on the tracking target state prediction result and a pre-established finite state machine.

[0016] In some embodiments of the present invention, the feature sequence corresponding to each frame output by the multi-layer convolutional layer is a one-dimensional feature sequence, and the prediction result of the state of the tracking target output by the fully connected classifier is the predicted probability value of each action state.

[0017] In some embodiments of the present invention, the action state is selected from part or all of the following states: standing, falling, walking, running, jumping, sitting and lying.

[0018] In some embodiments of the present invention, the method further includes: a neural network model training step.

[0019] In some embodiments of the present invention, the center coordinates of the multidimensional data matrix are the spatial coordinates of the tracking target, and generating the multidimensional data matrix corresponding to each frame of data based on the split data includes:

[0020] When the number of detection points in the point cloud data of the current frame is less than a predetermined number, some detection points of the previous frame closest to the target center point of the current frame are used to fill the missing data until the number of detection points in the point cloud data of the current frame reaches the predetermined number.

[0021] In some embodiments of the present invention, the pre-established finite state machine contains the transformation relationship between states with each action state as a node and the minimum threshold value of the state prediction probability.

[0022] In some embodiments of the present invention, determining the final state of the tracking target based on the tracking target state prediction result and a pre-established finite state machine includes:

[0023] The final state of the tracking target is a state that meets the transformation relationship between states and the predicted probability value is greater than a preset probability threshold, or is an unchanged state.

[0024] In some embodiments of the present invention, the method further includes: generating prompt information and / or sending prompt information to a communication terminal based on the final state of the tracking target determined by the finite state machine.

[0025] Another aspect of the present invention provides a tracking target state identification device based on millimeter wave sensing, the device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, when the computer instructions are executed by the processor, the device implements the steps of the method described above.

[0026] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program implements the steps of the method described above when executed by a processor.

[0027] The tracking target state recognition method and device based on millimeter wave perception of the present invention can recognize the motion state of the tracking target (such as a person) without being affected by light, without the need to wear sensors, and without the risk of privacy leakage.

[0028] As an example, the state recognition of the tracking target can include walking, running, sitting, jumping, lying, standing and falling, which basically covers all common action states.

[0029] In some embodiments of the present invention, real-time status monitoring of multiple persons can also be achieved, with high accuracy and strong practicality.

[0030] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0031] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0033] Figure 1 The figure is a flow chart of a method for identifying personnel status based on millimeter waves in one embodiment of the present invention.

[0034] Figure 2 The figure is a schematic diagram of a personnel status recognition process based on millimeter waves in one embodiment of the present invention.

[0035] Figure 3 The figure is a schematic diagram of the millimeter wave radar signal analysis process in the prior art.

[0036] Figure 4 This is a flow chart of the cluster tracking algorithm based on millimeter-wave radar.

[0037] Figure 5 Schematic diagram of a neural network model for state recognition in one embodiment of the present invention.

[0038] Figure 6 It is a schematic diagram of the structure of a finite state machine in one embodiment of the present invention.

[0039] Figure 7 Schematic diagram of three outgoing edges of a finite state machine in one embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0041] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0042] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0043] The present invention proposes a non-contact state recognition method based on millimeter waves and a corresponding device. Since millimeter waves are used to realize state perception, within the millimeter wave perception range, the user's action state can be recognized in real time without the user wearing any wearable device or sensor. The recognizable states can cover most common states, such as walking, running, sitting, lying, jumping, standing and falling. Figure 1FIG. 1 is a flow chart of a method for identifying personnel status based on millimeter waves in an embodiment of the present invention. Figure 2 FIG. 1 is a schematic diagram of a personnel status recognition process based on millimeter waves according to an embodiment of the present invention. Figure 2 As shown, this embodiment first generates point cloud data based on the original signal (millimeter wave signal) detected by the millimeter wave radar. The point cloud data contains original information such as the spatial coordinates of the detection point, Doppler data, and signal-to-noise ratio. Then, clustering is performed based on the point cloud data through a clustering tracking algorithm to identify each tracking target (such as the person being measured) for target positioning and tracking. Then, the neural network model (action classification algorithm) for state recognition established by the present invention is used to perform action classification to obtain the tracking target state prediction result (such as the probability of the target being in each state). Then, based on the tracking target state prediction result and the pre-established finite state machine, the final state of the tracking target is determined, thereby completing state recognition. Figure 1 As shown, a method for identifying personnel status based on millimeter waves according to an embodiment of the present invention includes the following steps S1-S5:

[0044] Step S1, acquiring point cloud data in real time based on the millimeter wave signal received by the millimeter wave radar.

[0045] The point cloud data may include spatial coordinate data, Doppler velocity information, signal-to-noise ratio and other information of each detection point.

[0046] As an example, the millimeter wave radar is the Texas Instruments (TI) millimeter wave radar IWR6843, which is a FMCW millimeter wave radar, but the present invention is not limited to this. The millimeter wave radar device has 3 transmitting antennas and 4 receiving antennas. The millimeter wave radar continuously transmits millimeter waves. The millimeter waves emitted by the radar are reflected by the human body and received by the receiving antenna of the radar. The received original signal is processed in the radar to generate point cloud data. The generated point cloud data contains some characteristic values ​​related to the detection points of the measured person.

[0047] Figure 3 Figure 1 is a schematic diagram of the analysis process of existing millimeter wave radar signals. Figure 3 As shown in the figure, after the steps of distance processing, Capon beamformer, target detection and Doppler estimation of TI millimeter wave radar IWR 6843, the original signal received by the millimeter wave radar is processed into point cloud data containing the spatial coordinates x, y, z of each detection point, Doppler velocity information and signal-to-noise ratio and other characteristic values. Figure 3 The existing point cloud data generation process will not be described in detail here.

[0048] In the embodiment of the present invention, the object reflecting radar millimeter waves may also be other animals, bionic robots, etc. besides humans, but the present invention is not limited to these examples, and may also be any object to be measured that needs to detect state changes.

[0049] Step S2: tracking the target based on the obtained point cloud data to determine the spatial coordinate data of the tracking target.

[0050] More specifically, the point cloud data is clustered, and the measured persons and the point cloud data corresponding to each measured person can be determined based on the clustering results. The measured persons can be located and tracked based on the point cloud data of each measured person.

[0051] Figure 4 The following is a flow chart of clustering and tracking algorithm for point cloud data. Figure 4 As shown in the figure, the point cloud data of each frame generated by the processing is sent to the cluster tracking algorithm, and the points of the new frame are associated with the trackable unit or clustered into a new category through the tracking algorithm, and the position of the target is corrected through the update algorithm based on the extended Kalman filter to achieve the distinction, positioning and tracking of the measured person. Then the target data is output through the UART serial port, including the identification ID and point cloud data of each measured person, where the point cloud data contains the spatial coordinates x, y, z of the point cloud of the tracking target (measured person), the Doppler velocity information of the point cloud, the signal-to-noise ratio of the point cloud, etc. After obtaining the position, Doppler velocity, signal-to-noise ratio and other information of the tracking target point cloud, the data can be input into the action classification algorithm frame by frame to realize the classification of the action; and the classification prediction results and prediction probabilities obtained are further input into the finite state machine, and the finite state machine performs logical judgment according to its own structure to obtain the final state of the target. Therefore, the above data obtained after clustering will be used as the data source of the state recognition algorithm. Since the tracking algorithm is a mature algorithm in the existing point cloud data processing, it will not be described in detail here.

[0052] Based on step S2, the present invention can also be used to confirm the number of people in the sensing area, calculate the flow of people, etc.

[0053] Step S3, splitting the point cloud data related to the tracking target within a time window of a predetermined length into frames, and generating a multidimensional data matrix corresponding to each frame of data based on the split data.

[0054] The multi-dimensional data matrix includes at least data of multiple attributes, and the data of multiple attributes include spatial coordinate attributes and at least one of the following attributes: Doppler velocity data and signal-to-noise ratio data.

[0055] In the embodiments of the present invention, the focus is on how to use the target point cloud data extracted by millimeter wave radar to accurately identify the status of personnel. In order to solve this problem, the present invention splits the point cloud data according to space and attributes, and designs a neural network (NN) model to extract the spatial and temporal features of the point cloud data, and finally uses a classifier to accurately identify the status. The point cloud data proposed by the present invention is split according to space and attributes, which can solve the problem of feature ambiguity caused by data mixing, and can make the neural network extract the features of the point cloud data in the spatial and temporal dimensions more efficient.

[0056] The present invention further aims to solve the problem of how to maintain the state information of the target in complex real-world situations, achieve the purpose of state maintenance and migration, realize long-term continuous state monitoring, and improve the accuracy of state recognition. To this end, the present invention also proposes a method of using a finite state machine (FSM) to maintain the target state information, abstracting the state of the user into a node of the FSM, abstracting the action that causes the state transition into an edge of the FSM, and reducing the state distinction problem to a classification problem for state transition actions. The state machine completes the state transition by means of logical judgment according to the output of the neural network and its own structure, and obtains the final state of the target, as described in the following step S5. The personnel state recognition method based on a finite state machine proposed by the present invention accurately and efficiently maintains the action state information of the personnel by abstracting the personnel state into a point of the FSM and the action that causes the state change into an edge of the FSM.

[0057] As an example, in this step, the point cloud data related to the tracking target within a time window of n frames in length can be split into frames, and a multidimensional data matrix (such as a 4-dimensional data matrix) corresponding to each frame of data is generated based on the split data as the input of the pre-trained neural network model, such as Figure 5 shown.

[0058] The time window is used to maintain the data of each measured person in a preset time period. The time window size is n frames, which contains the point cloud data information of the latest n frames related to each measured person, such as the spatial coordinates x, y, z of the point cloud data, Doppler velocity information, and information of the five attributes of the signal-to-noise ratio. In practice, the value of n is 10, which is relatively good, but the present invention is not limited to this, and can also be 5, 15, 20, etc., which are greater than or less than 10, and can achieve a balance between real-time performance and detection effect based on actual application scenarios. As an example, the point cloud clustering range of each measured person is a 2m*2m*2m cube (the present invention is not limited to this), and the resolution granularity is taken as 4cm (based on the resolution, it can also be other values), then there are 50 granularity units in each direction; the spatial position information of the tracking target of the measured person after the tracking algorithm is taken as the central coordinates of the multidimensional data matrix, recorded as x0, y0, z0. The point cloud data information of the person being tested is split and sorted frame by frame according to the spatial granularity, and a 50*50*50*5 4-dimensional data matrix is ​​established using the information generated by the frame-by-frame split according to the spatial granularity and the five attributes of the point cloud data. All point cloud data are traversed, and the spatial coordinates (i, j, k) of each detection point in the data matrix are calculated based on the difference between the spatial coordinates x, y, z of each detection point and the center coordinates x0, y0, z0, and the five attributes of spatial coordinates i, j, k, Doppler velocity, and signal-to-noise ratio are filled into the corresponding positions of the data matrix.

[0059] In one embodiment of the present invention, the generation of a multidimensional data matrix corresponding to each frame of data based on the split data includes: when the number of detection points in the point cloud data of the current frame is less than a predetermined number, the partial detection points of the previous frame closest to the target center point of the current frame are used to fill in the missing data until the number of detection points in the point cloud data of the current frame reaches the predetermined number. For example, the value of the position where there is no data to fill is set to 0; if the point cloud data of the current frame is too small, the data reuse method can be used to take the partial point cloud data of the previous frame closest to the target center point of the current frame (with the shortest Euclidean distance) to fill in the missing data until the number of points reaches the predetermined number, such as 64.

[0060] Step S4, input the multidimensional data matrix corresponding to each frame data into the pre-trained neural network model, and output the tracking target state prediction result.

[0061] Among them, the pre-trained neural network model includes a multi-layer convolution layer with the same number of channels as the number of attributes of the data in the multidimensional data matrix, a long short-term memory (LSTM) network layer and a fully connected classifier; the data of each attribute in the multidimensional data matrix corresponding to each frame of data is respectively input into the multi-layer convolution layer of the corresponding channel, and the feature sequence corresponding to each frame is output as the input of the LSTM network layer, and the output of the LSTM network layer is fused and input into the fully connected classifier, so that the fully connected classifier outputs the prediction result of the state of the tracking target, and the state of the tracking target includes multiple action states. In the neural network architecture of the embodiment of the present invention, the convolution layer is used to extract the features of each attribute of the point cloud in the spatial dimension, and the LSTM network is used to extract the features of the point cloud in the time dimension, which is used for more accurate classification.

[0062] As an example, the multidimensional data matrix (such as a 4-dimensional data matrix) with 5 attributes generated in step S3 is input into the 5-channel convolution layer, and the spatial features of each attribute of the point cloud data in each frame are extracted using a feature extraction method similar to that of RGB images. The data matrix with 5 attributes of the point cloud data is regarded as 5 channels, and the 5-channel convolution layer is used to perform feature extraction. The above convolution process is performed on each frame of data in the time window. After the 5-channel convolution layer performs convolution, the total feature sequence X0, X1...X is output. n-1 ; The convolved feature sequence enters the LSTM network layer. Since the actions of the person being tested are continuous in the time dimension, the convolved feature sequence X0, X1…X n-1 Inputting into the LSTM network layer can learn the characteristics of point cloud data in the time dimension and output time series features h0,h1,…,h n-1 ; The time series features of the output of the LSTM network layer are feature fused, connected into a vector and the vector is input into a fully connected classifier for classification, and the result is a specific classification result and its predicted probability value in one of multiple action states (the multiple action states can be selected from part or all of the following states: standing, falling, walking, running, jumping, sitting and lying).

[0063] In the embodiment of the present invention, the time window size n, the number of data matrix dimensions, the number of data matrix attributes, the resolution granularity of the point cloud clustering range, and the number of gap data fillings can be adaptively adjusted as needed, and are not limited to the above examples.

[0064] The point cloud data is spatially split and sorted according to the five attributes of x, y, z, Doppler velocity information, and signal-to-noise ratio according to the spatial coordinates, and convolution is performed separately to extract features. This can not only extract the features of the spatial dimension of the point cloud data, but also avoid interference between different attributes, ensure the correct and efficient extraction of the features of each attribute, and avoid feature ambiguity caused by the mixing of attributes.

[0065] In one embodiment of the present invention, the tracking target state recognition method further includes a training step for training the model based on pre-prepared training set data.

[0066] Step S5: determining the final state of the tracking target based on the tracking target state prediction result and the pre-established finite state machine.

[0067] The personnel status recognition method based on finite state machine proposed in the present invention abstracts the personnel status into the point of FSM (finite state machine), and abstracts the action that causes the status change into the edge of FSM. It can accurately and efficiently maintain the status information of the measured personnel and eliminate inaccurate classification prediction results in the neural network output of status recognition.

[0068] The pre-established finite state machine contains the transformation relationship between each state with each action state as a node and the minimum threshold value of the state prediction probability. Figure 6 FIG. 1 is a schematic diagram of a finite state machine structure diagram in an embodiment of the present invention. Figure 6 As shown in Figure 1, the finite state machine defines all possible states of the target within the perception scene, including walking, running, sitting, jumping, lying, standing and falling. Each state corresponds to a node on the finite state machine. The finite state machine also defines the conditions for state transition, such as Figure 7 As shown, each edge corresponds to a state transition condition. The unidirectional edge from A to B indicates that state A can only be transferred to state B. The bidirectional edge from C to D indicates that state C and state D can be transferred to each other if the conditions are met. The spin edge from E to E indicates that no state transition action occurs and the state remains unchanged. Except for the spin edge, each edge corresponds to an action and a minimum probability threshold θ. The minimum probability threshold can be set to 0.75 based on experience, but the present invention is not limited to this. The minimum probability threshold is an adjustable parameter and can be adjusted as needed.

[0069] After obtaining the output of the neural network model, the obtained classification prediction result and prediction probability are input into the finite state machine. The finite state machine will perform logical judgment based on its own structure to obtain the final state of the tracking target: the final state of the tracking target is a state that meets the transformation relationship between states and the predicted probability value is greater than the preset minimum probability threshold, or it is an unchanged state. The finite state machine will check whether the actions and probability values ​​corresponding to all outgoing edges of the node corresponding to the current state match the output of the obtained neural network model. When the match is successful (the actions corresponding to the outgoing edges meet the transformation relationship between states and the predicted probability value is greater than the preset minimum probability threshold), the classification prediction result is considered accurate, the state is transferred along the outgoing edges, and the motion state of the person being measured is output; when all outgoing edges are mismatched, the state remains unchanged along the spin edge, and the output state of the person being measured has not changed.

[0070] As can be seen from the above, the present invention realizes contactless state recognition through millimeter wave radar. The state recognition algorithm proposed in the present invention uses finite state machine (Finite State Machine, FSM) and neural network (Neural Network, NN) as tools, abstracts the state of the user into the node of FSM, abstracts the action that causes state transition into the edge of FSM, and reduces the state distinction problem to the classification problem of state transition action; uses neural network to classify the action, and uses the point cloud data collected by millimeter wave radar, including position, Doppler, signal-to-noise ratio, etc., as the input of the neural network module, and the neural network module gives the classification prediction result and prediction probability; the finite state machine completes the state migration by means of logical judgment according to the above output and its own structure, and obtains the final state of the target, so as to identify the real-time state of the user, including walking, running, sitting, lying, jumping, standing and falling.

[0071] Using the non-contact state recognition method based on millimeter waves proposed by the present invention, all the persons under test appearing in the scene are identified in the following steps: when the tracking unit appears in the perception scene for the first time, the system will locate a node in the finite state machine for the person under test, which is used to maintain the state information of the person under test, and its initial state is set to stand by default, corresponding to the initial state node on the finite state machine. When the acquired data of the person under test reaches the number of frames n set by the time window size, the relevant data of the person under test is processed and input into the neural network model for action classification. After classification by the neural network model, the prediction result of the action classification and its prediction probability are obtained, and the output of the neural network model is used as the input of the finite state machine; the finite state machine checks all the outgoing edges of the node corresponding to the current state, and matches the output of the neural network model with the conditions defined on the outgoing edge. If the match is successful, the state transfer is performed along the edge. If all edges are mismatched, they spin. After the above steps, the state machine outputs the final state of the target.

[0072] By selecting millimeter waves with a certain degree of penetration, the present invention can be unaffected by adverse environments such as light intensity, smoke, rain and snow, and can achieve multi-scenario applications; there is no need to wear sensors or wearable devices, which can provide users with a more comfortable user experience; and there is no need to install cameras in the sensing area, so there is no risk of privacy leakage; the detectable states include walking, running, sitting, jumping, lying, standing and falling, basically covering all common states, and can provide more comprehensive user activity data; because the millimeter wave radar can continuously send FMCW millimeter waves, it can also maintain the state information of the target in complex real-life situations, achieve the purpose of state maintenance and migration, realize long-term personnel trajectory tracking and continuous state monitoring, avoid the randomness of non-continuous monitoring, and provide more objective user activity data; at the same time, it can also realize real-time state monitoring of multiple people, with high accuracy and strong practicality, and solve some drawbacks of traditional methods.

[0073] The present invention can be applied to a variety of different scenarios. For example, these scenarios may include: 1) Analysis of home activity habits of family members. For example, by long-term recording of the activity trajectory and status of family members, through big data analysis, personalized health guidance suggestions can be made to the personnel, such as sedentary reminders, etc., to analyze the home activity habits of family members; 2) Smart home control and energy saving. For example, according to the user's location and status, combined with the Internet of Things control system, the lighting system can be automatically turned on when the user returns home or gets up at night, and the air conditioner or lighting system can be automatically turned off when the user goes to bed or leaves the room. The air conditioner gear and light intensity information can also be intelligently adjusted according to different numbers of people and activity states, providing users with a more convenient, intelligent and comfortable living environment, while also preventing power waste and achieving energy saving and environmental protection effects; 3) User health status monitoring. For example, by monitoring the number and time of users getting up at night, the user's sleep quality can be analyzed, and then their health status can be reflected, providing objective data for doctors to diagnose and treat. In addition, it can also be used to detect whether a family member has fallen, especially for elderly families living alone. After the elderly fall, they can send prompt information through software, text messages, etc. to notify family members or caregivers to provide assistance as soon as possible. The above are only some examples of the application scenarios of the present invention, and the present invention is not limited to this.

[0074] In some embodiments of the present invention, the tracking target state identification method based on millimeter wave sensing also includes: generating prompt information and / or sending prompt information to a communication terminal based on the final state of the tracking target determined by a finite state machine. That is, when the final state is determined to be a dangerous state such as a fall state or other state that requires prompting, prompt information will be automatically generated and / or prompt information will be automatically sent to a pre-associated communication terminal, so that the state can be known in the first place and corresponding measures can be taken, thereby playing a role in ensuring personnel safety.

[0075] In addition, in addition to home use, according to the characteristics of the present invention that it does not disclose privacy and has strong penetration, the present invention can also be combined with other equipment for monitoring the status of personnel in hospitals, such as toilets and wards where privacy needs to be protected, and can also be used in locker rooms or other areas where privacy needs to be protected, so as to provide timely assistance to personnel while protecting privacy; the present invention can also be used in other areas where cameras cannot be installed, such as confidentiality agencies, banks, etc., and can realize the status and trajectory detection of the persons being measured entering the perception area. By combining with other equipment, it can also be realized that when the preset conditions are not met, a prompt will be given when a person's movement trajectory or action is detected in the perception area.

[0076] Corresponding to the above method, the present invention also provides a tracking target state identification device based on millimeter wave sensing, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the method described above.

[0077] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned edge computing server deployment method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0078] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0079] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0080] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A tracking target state recognition method based on millimeter wave sensing, characterized in that: The method comprises the following steps: Based on the millimeter wave signal received by the millimeter wave radar, point cloud data is acquired in real time, wherein the point cloud data includes spatial coordinate data, Doppler velocity information and signal-to-noise ratio of each detection point; Perform target tracking based on the obtained point cloud data to determine the spatial coordinate data of the tracked target; Splitting the point cloud data related to the tracking target within a time window of a predetermined length by frame, generating a multidimensional data matrix corresponding to each frame of data based on the split data, wherein the multidimensional data matrix includes at least data of multiple attributes, and the data of the multiple attributes includes a spatial coordinate attribute and at least one of the following attributes: a Doppler velocity attribute and a signal-to-noise ratio attribute; Input the multidimensional data matrix corresponding to each frame of data into a pre-trained neural network model, and output the prediction result of the tracking target state, wherein the pre-trained neural network model includes a multi-channel multi-layer convolution layer, an LSTM network layer and a fully connected classifier, the number of channels of which is the same as the number of attributes of the data in the multi-dimensional data matrix; the data of each attribute in the multi-dimensional data matrix corresponding to each frame of data is respectively input into the multi-layer convolution layer of the corresponding channel, and the feature sequence corresponding to each frame is output as the input of the LSTM network layer, and the output of the LSTM network layer is fused and input into the fully connected classifier, so that the fully connected classifier outputs the prediction result of the state of the tracking target, and the state of the tracking target includes multiple action states; The final state of the tracking target is determined based on the tracking target state prediction result and a pre-established finite state machine.

2. The method according to claim 1, characterized in that The feature sequence corresponding to each frame output by the multi-layer convolutional layer is a one-dimensional feature sequence, and the prediction result of the state of the tracking target output by the fully connected classifier is the predicted probability value of each action state.

3. The method according to claim 1, characterized in that The action state is selected from part or all of the following states: standing, falling, walking, running, jumping, sitting and lying.

4. The method according to claim 1, characterized in that: The method also includes: a neural network model training step.

5. The method according to claim 1, characterized in that: The center coordinates of the multidimensional data matrix are the spatial coordinates of the tracking target, and the generation of the multidimensional data matrix corresponding to each frame of data based on the split data includes: When the number of detection points in the point cloud data of the current frame is less than a predetermined number, some detection points of the previous frame closest to the target center point of the current frame are used to fill the missing data until the number of detection points in the point cloud data of the current frame reaches the predetermined number.

6. The method according to claim 2, characterized in that The pre-established finite state machine contains the transformation relationship between various states with each action state as a node and the minimum threshold value of the state prediction probability.

7. The method according to claim 6, characterized in that The determining the final state of the tracking target based on the tracking target state prediction result and a pre-established finite state machine includes: The final state of the tracking target is a state that meets the transformation relationship between states and the predicted probability value is greater than a preset probability threshold, or is an unchanged state.

8. The method according to claim 1, characterized in that The method further comprises: Based on the final state of the tracking target determined by the finite state machine, prompt information is generated and / or sent to the communication terminal.

9. A tracking target state recognition device based on millimeter wave sensing, comprising a processor and a memory, characterized in that: The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the method as claimed in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images