Fall monitoring method, device, equipment and program product

Abstract

The invention discloses a fall monitoring method, device, equipment and program product, and relates to the technical field of behavior monitoring, and the fall monitoring method comprises the steps: obtaining the posture point cloud data of a human body, and carrying out the recognition of the posture point cloud data through a preset fall recognition model, and obtaining a recognition result. The tumble identification model is obtained by training the to-be-trained model based on the attitude sample data with different tumble durations corresponding to the real tumble label sequence. Therefore, the tumble identification model can better learn the mapping relationship between the tumble actions of different tumble durations and the tumble events, thereby improving the identification accuracy of the tumble actions of different tumble durations.

Description

Technical Field

[0001] The present application relates to the field of behavior monitoring technology, and in particular to fall monitoring methods, devices, equipment, and program products. Background Art

[0002] Fall monitoring is of great significance in the fields of elderly care, medical health monitoring, and public safety. Currently, fall monitoring is usually performed based on the rich spatial information provided by millimeter-wave point cloud data.

[0003] However, the falling actions and duration caused by different individuals and different types of falls are quite different (for example, falling while standing, falling while walking, and falling while sitting, the actions and duration of these types of falls are quite different), resulting in the low recognition accuracy of the current fall monitoring method based on millimeter wave point cloud data. Summary of the Invention

[0004] The main purpose of this application is to provide a fall monitoring method, device, equipment and program product, aiming to solve the technical problem of low recognition accuracy of current fall monitoring methods.

[0005] To achieve the above objectives, the present application proposes a fall monitoring method, which includes:

[0006] Obtain human body posture point cloud data;

[0007] The posture point cloud data is identified by a preset fall recognition model to obtain a recognition result, wherein the recognition result includes whether the event corresponding to the posture point cloud data is a fall event or not a fall event, and the fall recognition model is obtained by training a training model based on posture sample data with different fall durations corresponding to real fall label sequences.

[0008] In one embodiment, before the step of obtaining the human body posture point cloud data, the method further includes:

[0009] Obtain posture sample data of different fall durations;

[0010] Identifying the posture sample data of different fall durations using the to-be-trained model to obtain a target fall label sequence;

[0011] Based on a preset CTC (Connectionist Temporal Classification) algorithm, generating fall labels corresponding to the fall events in the posture sample data to obtain the true fall label sequence;

[0012] Calculating a CTC loss value based on the target fall label sequence and the real fall label sequence;

[0013] Based on the CTC loss value, the parameters of the model to be trained are adjusted to obtain a fall recognition model.

[0014] In one embodiment, the step of identifying the posture sample data of different fall durations by the to-be-trained model to obtain a target fall label sequence includes:

[0015] Identifying the posture sample data using the to-be-trained model, determining the probability of each time step in the posture sample data corresponding to a fall event and a non-fall event, and obtaining a fall probability sequence;

[0016] Based on the fall probability sequence and the CTC algorithm, determining different fall label sequences and the probabilities of each label sequence corresponding to the different fall label sequences;

[0017] The fall label sequence corresponding to the maximum label sequence probability among the label sequence probabilities is used as the target fall label sequence.

[0018] In one embodiment, the model to be trained includes a spatial feature extraction module, a temporal feature extraction module, and a fall monitoring classification module. The steps of identifying the posture sample data using the model to be trained and determining the probability of each time step in the posture sample data corresponding to a fall event and a non-fall event, and obtaining a fall probability sequence include:

[0019] Based on a preset first number of spatial feature extraction modules, performing spatial feature extraction on the posture sample data to obtain posture spatial feature data;

[0020] Based on a preset second number of time feature extraction modules, performing time feature extraction on the posture spatial feature data to obtain posture spatiotemporal feature data, wherein the first number is greater than the second number;

[0021] Based on the fall monitoring classification module and the posture spatiotemporal feature data, the probability of each time step in the posture spatiotemporal feature data corresponding to a fall event and a non-fall event is determined to obtain a fall probability sequence.

[0022] In one embodiment, the step of generating a fall label corresponding to a fall event in the posture sample data based on a preset CTC algorithm to obtain the real fall label sequence includes:

[0023] If a fall event exists in the posture sample data, a fall label is set for the portion of the posture sample data corresponding to the fall event, and a blank label is set for the remaining portion to obtain the true fall label sequence;

[0024] If there is no fall event in the posture sample data, one or more active labels are set in the posture sample data based on a preset interval, and blank labels are set in the rest to obtain the real fall label sequence.

[0025] In one embodiment, before the step of obtaining posture sample data of different fall durations, the method further includes:

[0026] Acquire millimeter wave signals associated with the target human body;

[0027] Based on the millimeter wave signal, the azimuth data, distance data, elevation data and speed data of each point on the target human body relative to the millimeter wave signal acquisition device are calculated;

[0028] Based on the azimuth data, the distance data, the elevation angle data and the speed data, the coordinate values ​​of each point of the target human body are calculated to obtain coordinate point cloud data.

[0029] In one embodiment, after the step of calculating the coordinate values ​​of each point of the target human body based on the orientation data, the distance data, the elevation data, and the speed data to obtain coordinate point cloud data, the method further includes:

[0030] Acquire the coordinate point cloud data;

[0031] If there are missing data frames in the coordinate point cloud data, the coordinate mean of the preset first number of frames is used as the coordinate value of the missing data frame to obtain continuous coordinate data;

[0032] Normalizing the continuous coordinate data based on a preset normalization method to obtain normalized coordinate data;

[0033] If there is an abnormal data frame in the normalized coordinate data, the coordinate mean of the preset second number of frames is used as the coordinate value of the abnormal data frame to obtain processed normalized coordinate data, wherein the difference between the coordinate value corresponding to the abnormal data frame and the coordinate mean of the adjacent multiple frames of data is greater than a preset abnormal threshold;

[0034] Based on a preset smoothing window, the processed normalized coordinate data is smoothed to obtain posture sample data.

[0035] In addition, to achieve the above objectives, the present application also proposes a fall monitoring device, which includes:

[0036] Data acquisition module, used to obtain human body posture point cloud data;

[0037] A fall recognition module is used to identify the posture point cloud data through a preset fall recognition model to obtain a recognition result, wherein the recognition result includes whether the event corresponding to the posture point cloud data is a fall event or not a fall event, and the fall recognition model is obtained by training the training model based on posture sample data with different fall durations corresponding to real fall label sequences.

[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a fall monitoring device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the fall monitoring method described above.

[0039] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the fall monitoring method described above.

[0040] One or more technical solutions proposed in this application have at least the following technical effects:

[0041] This application obtains human body posture point cloud data and identifies the posture point cloud data using a preset fall recognition model to obtain a recognition result. Because the fall recognition model is based on posture sample data of different fall durations corresponding to real fall label sequences, the training model is trained. Therefore, the fall recognition model can better learn the mapping relationship between fall actions of different fall durations and fall events, thereby improving the recognition accuracy of fall actions of different fall durations. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 A flowchart of the first embodiment of the fall monitoring method of this application is provided;

[0045] Figure 2 A flowchart of the second embodiment of the fall monitoring method of this application is provided;

[0046] Figure 3A schematic diagram of a scenario provided for Example 2 of the fall monitoring method of this application;

[0047] Figure 4 A flowchart of the third embodiment of the fall monitoring method of this application is provided;

[0048] Figure 5 A schematic diagram of a scenario provided in Example 3 of the fall monitoring method of this application;

[0049] Figure 6 This is a schematic diagram of the module structure of the fall monitoring device according to an embodiment of the present application;

[0050] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the fall monitoring method in the embodiment of the present application.

[0051] The purpose, features and advantages of this application will be further explained with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0053] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0054] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device capable of implementing the above functions, such as a fall monitoring device. The following uses a fall monitoring device as an example to illustrate this embodiment and the following embodiments.

[0055] Based on this, the embodiment of the present application provides a fall monitoring method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the fall monitoring method of the present application.

[0056] In this embodiment, the fall monitoring method includes steps S10 to S20:

[0057] Step S10, obtaining human body posture point cloud data;

[0058] It should be noted that the posture point cloud data in this embodiment is 3D point cloud data of the human body. The 3D point cloud data is composed of three-dimensional information of points on the surface of the human body at each time point. The change of the 3D point cloud data over time can represent the change of the human body's posture over time, thereby representing the falling action of the human body.

[0059] Fall monitoring is a technology that automatically detects whether a person has accidentally fallen. Current methods mainly use wearable devices, cameras, ultrasonic sensors, lidar, and millimeter waves to obtain human posture data, and perform human fall monitoring based on the acquired posture data.

[0060] It is understandable that falls can cause serious harm to the body for the elderly and some patients. If they do not receive help for a long time after falling, it may lead to serious consequences. Therefore, fall monitoring is of great significance in the fields of elderly care, medical health monitoring, and public safety.

[0061] Among current fall detection methods, wearable devices are invasive, require battery power, and may be forgotten, cameras have privacy issues, lidar is typically expensive, and ultrasonic sensors are sensitive to noise.

[0062] Millimeter-wave radar, due to its high range and high angular resolution, can provide more detailed motion information for fall detection. Its relatively low price can reduce the cost of fall detection. Therefore, human fall detection based on millimeter-wave signals is a relatively feasible method. This embodiment uses millimeter-wave radar to collect millimeter-wave signals, converts them into human posture point cloud data, and then performs human fall detection based on the acquired posture point cloud data.

[0063] Step S20: The posture point cloud data is identified by a preset fall recognition model to obtain a recognition result, wherein the recognition result includes whether the event corresponding to the posture point cloud data is a fall event or not a fall event. The fall recognition model is obtained by training the training model based on posture sample data of different fall durations corresponding to real fall label sequences.

[0064] It should be noted that the posture sample data is the training set data used to train the fall recognition model, including data with and without fall events. The duration of the fall action in each posture sample data with a fall event varies, and some posture sample data contain multiple fall time periods. Each fall event in the posture sample data is marked with the time point or time period when the fall occurred. The duration of the fall action is the total time from the start to the end of the human body falling action.

[0065] In this embodiment, each time step in the fall action will be marked with a fall label, and each time step of the non-fall action will be marked with other labels, thereby generating a real fall label sequence consisting of the fall label and other labels corresponding to each posture sample data.

[0066] It is understandable that the current method is to manually mark the fall time points and fall time periods in the posture sample data, and pre-set some thresholds. When the data is detected to exceed these thresholds, the system determines that a fall event has occurred.

[0067] Manual labeling requires significant human resources to review and label data, which is time-consuming and costly. Furthermore, because real-life fall scenarios are extremely diverse, it's difficult to capture all possible scenarios with limited manually labeled data. This can lead to inadequate model training or poor generalization of the trained model.

[0068] However, fall detection methods that use fixed thresholds are affected by individual differences (such as age, body shape, and health status) and can only capture the most direct changes in physical parameters. They cannot capture complex movement patterns or behavioral characteristics, resulting in false positives or missed positives. For example, a person may fall while walking, standing, or sitting. The movements and duration of these three types of falls are different. Simply judging whether the physical parameters exceed the fixed threshold is difficult to accurately identify these falls.

[0069] The above method cannot accurately identify the falling actions of different users and different fall durations. Therefore, in order to solve the above problem, it is necessary to enable the model to better learn the mapping relationship between the human body's fall duration and fall events, that is, during the model training process, it is necessary to associate the model output results with the fall duration.

[0070] Therefore, this embodiment generates a real fall label sequence corresponding to posture sample data with different fall durations, trains the training model through the real fall label sequence, obtains a trained fall recognition model, and then uses the fall recognition model to recognize the posture point cloud data to obtain a recognition result.

[0071] Specifically, when using a preset fixed threshold method for fall detection, the fall condition is set as when an acceleration exceeding 3g is detected and the body angle changes by more than 45 degrees (for example, from upright to horizontal) within 0.5 seconds. In some posture sample data, the person falls while sitting. In this case, because the body does not experience a large angle change, the model cannot accurately learn the fall action in this situation, resulting in a low recognition accuracy of the trained model.

[0072] The method of this embodiment generates a real fall label sequence corresponding to the above-mentioned sitting fall event, and trains the model based on the real fall label sequence, so that the model learns the mapping relationship between the falling action and the fall event when falling while sitting, thereby accurately monitoring falls through the trained model.

[0073] In summary, this embodiment first obtains human body posture point cloud data, then uses a preset fall recognition model to identify the posture point cloud data to obtain a recognition result. Because the fall recognition model is trained on posture sample data of different fall durations corresponding to real fall label sequences, the model to be trained can better learn the mapping relationship between fall actions of different fall durations and fall events, thereby improving the recognition accuracy of fall actions of different fall durations.

[0074] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 2 Before the step of obtaining the human body posture point cloud data, the fall monitoring method further includes steps S01 to S05:

[0075] Step S01, obtaining posture sample data of different fall durations;

[0076] It should be noted that the posture sample data in this embodiment is pre-processed posture sample data. The posture sample data is the posture data of the human body when performing continuous movements. The posture data of the human body when performing continuous movements can enable the model to better learn the human body's falling behavior.

[0077] Step S02, identifying the posture sample data of different fall durations by the to-be-trained model to obtain a target fall label sequence;

[0078] It should be noted that the model to be trained is a model that has not been trained or converged in the fall monitoring task. The fall label sequence is composed of the recognition results of the human fall behavior at each time step in the posture sample data by the model to be trained.

[0079] It is understandable that since the model to be trained is a model that has not yet been trained or has not converged, the accuracy of the posture sample data recognized by the model will be relatively low. In order to improve the accuracy, the model needs to be trained based on the recognition results of the model. Therefore, it is necessary to use the model to be trained to recognize the posture sample data of the different fall durations to obtain the target fall label sequence, and then train the model using the target fall label sequence.

[0080] Step S03: generating a fall label corresponding to the fall event in the posture sample data based on a preset CTC algorithm to obtain the real fall label sequence;

[0081] It should be noted that the CTC algorithm is an algorithm for processing sequence data and can process input sequences and output sequences of inconsistent lengths.

[0082] It's understandable that the length and position of the generated true fall labels vary based on posture sample data from different fall time periods and fall time points, resulting in different true fall label sequences. By training the model on these true fall label sequences, the model learns the mapping between fall actions of varying durations and fall events, enabling more accurate recognition of fall actions of varying durations.

[0083] Specifically, the first posture sample data consists of 500 frames, of which the fall event occurs at frame 100, and the fall duration is 60 frames. Therefore, the labels corresponding to the data from frames 100 to 160 are the fall labels, and the labels corresponding to the data of other frames are other labels. Based on this posture sample data, the corresponding first real fall label sequence can be obtained.

[0084] The second posture sample data has a total of 600 frames, among which the fall event occurs in the 300th frame and the fall time is 100 frames. Therefore, the label corresponding to the data from the 300th frame to the 400th frame is the fall label, and the labels corresponding to the data of other frames are other labels. Based on this posture sample data, the corresponding second real fall label sequence can be obtained.

[0085] Based on the above examples, it can be seen that the corresponding real fall label sequences are different for posture sample data of different fall lengths. Therefore, in order to learn the fall actions of different fall lengths in the posture sample data, it is necessary to generate fall labels corresponding to the fall events in the posture sample data based on the preset CTC algorithm to obtain the real fall label sequence.

[0086] In a feasible implementation manner, the specific implementation manner of generating a fall label corresponding to the fall event in the posture sample data based on a preset CTC algorithm to obtain the real fall label sequence may also be:

[0087] If there is a fall event in the posture sample data, a fall label is set in the corresponding part of the fall event in the posture sample data, and a blank label is set in the remaining part to obtain the real fall label sequence. If there is no fall event in the posture sample data, one or more active labels are set in the posture sample data based on a preset setting interval, and a blank label is set in the remaining part to obtain the real fall label sequence.

[0088] It should be noted that the blank label is a special character introduced by the CTC algorithm to fill the gaps between time steps and does not represent any actual label. The introduction of the blank label allows the consistency of the time series to be maintained even when the lengths of the input and output sequences are inconsistent.

[0089] Understandably, in traditional sequence-to-sequence models, the input and output sequences require strict time alignment. However, in real-world applications, there are often cases where the input and output sequences are of different lengths (e.g., in recognition tasks like speech and handwriting recognition). The CTC algorithm automatically aligns the input and output sequences by introducing a special blank label to fill the gaps between time steps.

[0090] It should also be noted that adding activity labels to posture sample data does not indicate a fall event. Placing activity labels in pure activity samples without falls can increase the robustness of the model. The preset interval can be modified based on the specific posture sample data. For longer posture sample data, a larger interval can be set, while for shorter posture sample data, a smaller interval can be set, thereby further improving the robustness of the model.

[0091] It is understandable that some daily human actions are easily identified as falls (such as sitting down quickly and bending over to pick up something). Therefore, to reduce false positives for such actions in fall detection tasks, this embodiment adds activity labels to pure activity samples to enable the model to better understand the characteristics of these common actions. The model is trained using samples of multiple different activities to improve its generalization ability.

[0092] Step S04: Calculate a CTC loss value based on the target fall label sequence and the real fall label sequence;

[0093] It should be noted that the CTC loss value is a value used to represent the error between the predicted label sequence and the true label sequence when the lengths of the input sequence and the output sequence are inconsistent. In this embodiment, it is used to calculate the error between the target fall label sequence and the true fall label sequence.

[0094] The CTC loss is calculated using the CTC loss function, which considers all possible predicted label sequences, adds the probabilities of each predicted label sequence, and calculates the final loss based on the summed probabilities. In this embodiment, the above-mentioned process of adding the probabilities of each predicted label sequence is to add the probabilities of all possible paths of the target label sequence.

[0095] It is understandable that the fall recognition model identifies fall events at each time step of the posture sample data, and ultimately obtains the final recognition result based on the recognition results of each time step. Therefore, there will be different fall label sequences, and each fall label sequence has a corresponding probability of generating the fall label sequence.

[0096] The CTC loss function can consider all possible fall label sequences, determine the target fall label sequence, and add the probabilities of all possible paths of the target label sequence. The loss value is calculated by the target fall label sequence and the real fall label sequence. The loss value calculated in this way can better reflect the error between the predicted result and the real result, so that the model can better learn the fall actions of different fall durations, thereby improving the recognition accuracy of the model. Therefore, in order to improve the recognition accuracy of the model, it is necessary to calculate the CTC loss value based on the target fall label sequence and the real fall label sequence.

[0097] Step S05: Based on the CTC loss value, adjust the parameters of the model to be trained to obtain a fall recognition model.

[0098] It should be noted that the model training in this embodiment is performed in an iterative manner, and the condition for the end of the iteration can be that the training reaches a preset round or that the loss value of the model is lower than a set loss value threshold.

[0099] The learning rate during model training can be a fixed learning rate, a step-down learning rate, or a simulated annealing learning rate. The optimizer for model training can be an SGD (Stochastic Gradient Descent) optimizer or an Adam (Adaptive Moment Estimation) optimizer. This embodiment uses the Adam optimizer and the learning rate is set to 0.001.

[0100] Understandably, the recognition accuracy of the trained model is low, necessitating adjustments to the model parameters based on the calculated error to achieve a better model. After multiple iterations of training, the calculated loss gradually decreases and stabilizes within a small range, resulting in a converged fall recognition model that can accurately identify human falls.

[0101] In a feasible implementation manner, before the step of obtaining posture sample data of different fall durations, the method further includes:

[0102] Acquire a millimeter wave signal associated with the target human body, and based on the millimeter wave signal, calculate the azimuth data, distance data, elevation data, and speed data of each point on the target human body relative to the millimeter wave signal acquisition device; based on the azimuth data, the distance data, the elevation data, and the speed data, calculate the coordinate values ​​of each point on the target human body to obtain coordinate point cloud data.

[0103] It should be noted that the millimeter wave acquisition device of this embodiment is a radar having multiple receiving antennas and multiple transmitting antennas. Each transmitting antenna transmits a frequency modulated continuous wave (FMCW) signal in turn. These signals have a frequency that varies linearly with time. The specific formula is:

[0104]

[0105] Where i represents different transmitting antennas (i = 1, 2, 3), A is the amplitude, f0 is the initial frequency, μ is the frequency modulation frequency, t is the time, φ 0,i is the initial phase of each transmitting antenna.

[0106] The signal received by each receiving antenna is the sum of multiple transmitted signals after reflection from objects, time delay, attenuation and phase shift.

[0107] It is understood that since the signal received by each receiving antenna can be represented as the sum of multiple reflected components of the transmitted signal, the intermediate frequency (IF) signal can be obtained by mixing the received signal with a copy of each transmitted signal. The IF signal reflects the frequency difference between the transmitted and received signals, and the frequency difference is proportional to the distance from the object to the radar. Therefore, the distance to the radar can be obtained from the IF signal.

[0108] This embodiment first performs an analog-to-digital converter (ADC) on the received analog signal to obtain discrete time series samples. Then, a windowing technique is applied (the window can be a rectangular window, a Hanning window, a Hamming window, etc., and this embodiment uses a Hanning window) to reduce edge effects, and the time domain signal is converted into a frequency domain signal through a discrete Fourier transform (DFT). The results of the distance-dimensional DFT reveal the intensity of different frequency components in the echo signal, where the IF peak frequency is directly related to the distance to the target. Therefore, the calculation formula for the distance R can be obtained as:

[0109]

[0110] Where c is the speed of light, f IF is the IF peak frequency in the DFT result, and μ is the bandwidth.

[0111] Static objects (such as walls and the ground) will generate clutter. Since the position of static objects remains unchanged in the radar view, while the position of moving objects changes over time, after obtaining the range spectrum, it is necessary to eliminate static clutter by subtracting the average range spectrum from consecutive frames, so that only the echo signals of moving objects are retained.

[0112] After calculating the distance, in order to determine the specific position of the target in terms of distance and azimuth, this embodiment applies Capon beamforming technology to generate a range-azimuth heat map by optimizing the weight vector to minimize interference and noise while maximizing the signal power in the desired direction. In this embodiment, the desired direction is the direction of the target object, where the calculation of the weight vector involves the inversion of the covariance matrix of the received signal and the calculation of the steering vector related to the azimuth angle. The signal power is used to generate the range-azimuth heat map. The specific calculation expression is:

[0113] P(θ,R)=|w H x(R)| 2

[0114] Where w is the beamformer weight vector, x(R) is the signal vector received at distance R, and w H is the conjugate transpose of the weight vector.

[0115] After obtaining the range-azimuth heat map, each point on the surface of the target object can be determined based on the obtained range-azimuth heat map. This embodiment detects the target point by a constant false alarm rate (CFAR) algorithm. The CFAR algorithm determines whether a target exists by comparing the statistical characteristics of the detection unit with those of its surrounding units. In millimeter-wave radar, a two-stage CFAR algorithm is usually adopted, that is, CFAR-CASO (Cell-Averaging and Selection of Optimum) detection is first performed in the distance domain, and then a secondary search is performed in the azimuth domain to confirm the detected points, thereby obtaining the azimuth data and distance data of each point on the target object from the radar.

[0116] For each point detected in the range-azimuth domain after two-stage CFAR detection, the Capon beamforming algorithm is applied to estimate the elevation angle. In the elevation angle estimation calculation, the steering vector is related to the elevation angle φ. By searching for the maximum peak in the elevation angle spectrum, the elevation angle of the detected point can be determined, thereby obtaining the elevation angle data for each point on the target object from the radar.

[0117] The radial velocity information of the target object is obtained through the Doppler dimension DFT. By performing DFT on the continuous linear frequency modulation signal (chirp), the Doppler spectrum can be extracted and the maximum peak in the spectrum can be searched to estimate the radial velocity. The calculation formula of Doppler velocity v is:

[0118]

[0119] Where λ is the wavelength of the millimeter wave, f d is the Doppler shift, and θ is the angle between the target and the radar beam, thereby obtaining the velocity of each point on the target object relative to the radar.

[0120] Furthermore, since the azimuth data, distance data, elevation data and speed data relative to the radar have been obtained, the coordinate values ​​of each point on the surface of the target human body can be calculated through the above data, thereby obtaining the coordinate point cloud data during the human body activity process.

[0121] In a feasible embodiment, after the step of calculating the coordinate values ​​of each point of the target human body based on the orientation data, the distance data, the elevation data and the speed data to obtain the coordinate point cloud data, the step further includes:

[0122] Acquire the coordinate point cloud data. If there are missing data frames in the coordinate point cloud data, use the coordinate mean of a preset first number of frames as the coordinate value of the missing data frame to obtain continuous coordinate data. Based on a preset normalization method, normalize the continuous coordinate data to obtain normalized coordinate data. If there are abnormal data frames in the normalized coordinate data, use the coordinate mean of a preset second number of frames as the coordinate value of the abnormal data frame to obtain processed normalized coordinate data, wherein the difference between the coordinate value corresponding to the abnormal data frame and the coordinate mean of adjacent multiple frames of data is greater than a preset abnormal threshold. Based on a preset smoothing window, smooth the processed normalized coordinate data to obtain posture sample data.

[0123] It should be noted that the preset first number of frames, the preset second number of frames and the preset abnormality threshold are set according to specific data processing requirements before data processing. The normalization method can be minimum-maximum normalization and standard score normalization (Z-score normalization) and other normalization methods. This embodiment uses Z-score normalization. The size of the sliding window can be determined according to specific data processing requirements. In this embodiment, the size of the sliding window is 5.

[0124] In this embodiment, the preset first frame number is the previous frame of the missing data frame and its two adjacent frames (e.g., if the missing data frame is the 100th frame, the preset first frame number is the two adjacent frames of the 99th frame, i.e., the 97th, 98th, 99th, 100th and 101st frames), the preset second frame number is the frame corresponding to the abnormal value and its two adjacent frames (e.g., if the abnormal data frame is the 100th frame, the preset second frame number is the two frames before and after the 100th frame, i.e., the 98th, 99th, 100th, 101st and 102nd frames), and the preset abnormal threshold is 3 times the standard deviation of the average value of the current frame coordinates.

[0125] It is understandable that in the process of calculating the coordinate point cloud signal, there may be missing data, resulting in missing data in some frames, or because the collected signal value has a large deviation from the true value, the calculated coordinate value is abnormal. Therefore, this embodiment uses the coordinate mean of the preset first frame number as the coordinate value of the missing data frame to obtain continuous coordinate values, and reduces the difference of some values ​​through Z-score normalization. Then, the coordinate mean of the preset second frame number is used to replace the coordinate value that deviates from the current frame coordinate mean by more than 3 standard deviations. Finally, a sliding window of size 5 is used to smooth out small fluctuations in the data, making the overall data smoother and more coherent. For the specific processing flow, refer to Figure 3 .

[0126] By processing outliers and missing values ​​in the data and normalizing and smoothing the data, the impact of noise and outliers on the model during training can be reduced, thereby improving the recognition accuracy of the model.

[0127] In summary, this embodiment first obtains the millimeter wave signal associated with the target human body, obtains the point cloud data of the human body posture through calculation, and preprocesses the point cloud data to obtain smooth and coherent posture sample data without abnormal data and missing data. The posture sample data of different fall durations are identified by the to-be-trained model to obtain a target fall label sequence. Based on the preset CTC algorithm, fall labels corresponding to the fall events in the posture sample data are generated to obtain the true fall label sequence. Based on the target fall label sequence and the true fall label sequence, a CTC loss value is calculated. Based on the CTC loss value, the parameters of the to-be-trained model are adjusted to obtain a fall recognition model.

[0128] Different fall durations and posture sample data at different fall time points can generate different sequences of true fall labels. Training the model with these true fall label sequences and the target predicted label sequences predicted by the training model ensures that the calculated loss value better reflects the error between the predicted and true results, enabling the model to better learn fall actions of varying durations, ultimately yielding a model with higher recognition accuracy.

[0129] Based on the first and second embodiments of this application, the same or similar contents as those of the above-mentioned first and second embodiments can be referred to the above introduction and will not be repeated hereafter. Figure 4 The step of identifying the posture sample data of different fall durations by the to-be-trained model to obtain a target fall label sequence further includes steps S021 to S023:

[0130] Step S021, identifying the posture sample data by the to-be-trained model, determining the probability of each time step in the posture sample data corresponding to a fall event and a non-fall event, and obtaining a fall probability sequence;

[0131] It should be noted that the fall probability sequence is a set of probabilities that each time step is a fall event or not a fall event in the posture sample data predicted by the training model, that is, each time step has two probabilities, one of which is that the time step is a fall event, and the other is that the time step is not a fall event.

[0132] It is understood that falling is a process involving multiple continuous actions. There may be a transition phase between normal activity and falling. For each time step in the posture sample data, there is a probability that it is a time point at which a fall occurs. Therefore, when performing recognition, it is necessary to use the to-be-trained model to identify the posture sample data, determine the probability of each time step in the posture sample data corresponding to a fall event and a non-fall event, and obtain a fall probability sequence, so that the model can better capture the characteristics of the fall action.

[0133] Step S022: determining different fall label sequences and the label sequence probabilities corresponding to the different fall label sequences based on the fall probability sequence and the CTC algorithm;

[0134] It is understandable that since each time step in the fall probability sequence has a probability of a fall event and a probability of not a fall event, each time step in the fall probability sequence that is determined to be a fall event or a non-fall event will generate a different fall label sequence.

[0135] Specifically, the first time step of a fall probability sequence of length 5 has a probability of 0.8 being a fall event and a probability of 0.2 being a non-fall event. When the time step is determined to be a fall event, the first label of the corresponding fall label sequence is a fall label, and when the time step is determined to be a non-fall event, the first label of the corresponding fall label sequence is a blank label. Therefore, the above two situations will produce different fall label sequences, and each different fall label sequence has a corresponding generation probability.

[0136] Step S023: taking the fall label sequence corresponding to the maximum label sequence probability among the label sequence probabilities as the target fall label sequence.

[0137] It can be understood that each different fall label sequence has a corresponding generation probability, and the fall label sequence with the largest generation probability is the fall label sequence that may be generated. Therefore, the fall label sequence corresponding to the maximum label sequence probability among the label sequence probabilities is used as the target fall label sequence, and the target fall label sequence is used as the prediction result of the model to be trained.

[0138] In a feasible embodiment, the model to be trained includes a spatial feature extraction module, a temporal feature extraction module, and a fall monitoring classification module. The step of identifying the posture sample data by the model to be trained and determining the probability of each time step in the posture sample data corresponding to a fall event and a non-fall event, and obtaining a fall probability sequence includes:

[0139] Based on a preset first number of spatial feature extraction modules, spatial feature extraction is performed on the posture sample data to obtain posture spatial feature data. Based on a preset second number of time feature extraction modules, time feature extraction is performed on the posture spatial feature data to obtain posture spatiotemporal feature data, wherein the first number is greater than the second number. Based on the fall monitoring classification module and the posture spatiotemporal feature data, the probability of each time step in the posture spatiotemporal feature data corresponding to a fall event and a non-fall event is determined to obtain a fall probability sequence.

[0140] It should be noted that the preset first number and the preset second number are set according to specific needs before model training. In this embodiment, the first number is 5 and the second number is 1. The spatial feature extraction module can be a normal convolution module, a residual convolution module, or other modules for extraction. In this embodiment, a residual convolution module is used to extract spatial features. The temporal feature extraction module can be a gated recurrent unit (GRU), a simple recursive unit, an encoder (Transformer), and a bidirectional encoder (Bidirectional Encoder Representations from Transformers, BERT). In this embodiment, a GRU is used to extract temporal features.

[0141] It's understandable that the process of a person falling is a time- and space-dependent process. The posture sample data contains data related to both time and space. The 3D coordinate point cloud data of the human body is related to space, and each time step in the posture sample data is related to time. To enable the model to learn the human falling motion, it's necessary to extract temporal and spatial features from the posture sample data.

[0142] Therefore, this embodiment first extracts spatial features through multiple residual layers. Since each residual layer contains a one-dimensional convolution operation, it can well capture the spatial pattern in the data. As the number of network layers increases, the number of feature maps (number of channels) gradually increases, allowing the network to learn more complex and abstract spatial features. In the task of human fall monitoring, spatial features contain richer action information. Therefore, this embodiment extracts air conditioning features through multiple spatial feature extraction modules.

[0143] Since the GRU updates its internal state based on the output of the previous time step and the input of the current time step, it extracts features that change over time. Therefore, after extracting spatial features, this embodiment uses the GRU to iteratively process the feature vectors of each time step to generate a feature representation that contains temporal dynamic information.

[0144] Finally, this embodiment outputs the data after temporal feature extraction and spatial feature extraction to the classifier. The classifier consists of two fully connected layers, with LeakyReLU (Leaky Rectified Linear Unit) used as the activation function in the middle, and finally normalized by LogSoftmax (logarithmic soft maximum function), outputting the probability distribution of fall events and non-fall events at each time step, and obtaining the fall probability sequence. For the specific process, refer to Figure 5 .

[0145] In summary, this embodiment extracts the temporal and spatial features of the posture samples, determines the probabilities of each time step in the posture temporal and spatial feature data corresponding to a fall event and not a fall event, and obtains a fall probability sequence. Based on the fall probability sequence and the CTC algorithm, different fall label sequences and the label sequence probabilities corresponding to the different fall label sequences are determined, and the fall label sequence corresponding to the maximum label sequence probability among the label sequence probabilities is used as the target fall label sequence.

[0146] The human fall process is a time- and space-dependent process. Therefore, this embodiment extracts temporal and spatial features from the posture sample data. Furthermore, since falling involves multiple continuous movements, there may be a transitional phase between normal activity and falling. Therefore, based on the extracted temporal and spatial features, this embodiment outputs the probability distribution of whether a fall event is considered a fall or not at each time step, generating a fall probability sequence. This allows the model to better learn the fall action and improves the accuracy of fall action recognition.

[0147] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the fall monitoring method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0148] This application also provides a fall monitoring device, please refer to Figure 6 , the fall monitoring device comprises:

[0149] The data acquisition module 10 is used to obtain the posture point cloud data of the human body;

[0150] The fall recognition module 20 is used to recognize the posture point cloud data through a preset fall recognition model to obtain a recognition result, wherein the recognition result includes whether the event corresponding to the posture point cloud data is a fall event or not a fall event, and the fall recognition model is obtained by training the training model based on posture sample data with different fall durations corresponding to real fall label sequences. The fall monitoring device provided in the present application adopts the fall monitoring method in the above embodiment, which can solve the technical problem of low recognition accuracy of the current fall monitoring method. Compared with the prior art, the beneficial effects of the fall monitoring device provided in the present application are the same as the beneficial effects of the fall monitoring method provided in the above embodiment, and the other technical features in the fall monitoring device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0151] In one embodiment, the fall monitoring device further includes:

[0152] A sample data acquisition module is used to obtain posture sample data of different fall durations;

[0153] a label sequence generation module, configured to identify the posture sample data of different fall durations using a to-be-trained model to obtain a target fall label sequence;

[0154] a label conversion module, configured to generate a fall label corresponding to the fall event in the posture sample data based on a preset CTC algorithm, and obtain the real fall label sequence;

[0155] A loss value calculation module, configured to calculate a CTC loss value based on the target fall label sequence and the real fall label sequence;

[0156] The parameter adjustment module adjusts the parameters of the model to be trained based on the CTC loss value to obtain a fall recognition model.

[0157] In one embodiment, the first identification module includes:

[0158] a probability sequence generation submodule, configured to identify the posture sample data using the to-be-trained model, determine the probability of each time step in the posture sample data corresponding to a fall event or not, and obtain a fall probability sequence;

[0159] A probability sequence determination submodule, configured to determine different fall label sequences and the probabilities of each label sequence corresponding to the different fall label sequences based on the fall probability sequence and the CTC algorithm;

[0160] The target sequence determination submodule is used to take the fall label sequence corresponding to the maximum label sequence probability among the label sequence probabilities as the target fall label sequence.

[0161] In one embodiment, the probability sequence determination submodule includes:

[0162] A spatial feature extraction unit, configured to extract spatial features from the posture sample data based on a preset first number of spatial feature extraction modules to obtain posture spatial feature data;

[0163] a time feature extraction unit, configured to perform time feature extraction on the posture spatial feature data based on a preset second number of time feature extraction modules to obtain posture spatiotemporal feature data, wherein the first number is greater than the second number;

[0164] A probability sequence generation unit is used to determine the probability of each time step in the posture spatiotemporal feature data corresponding to a fall event and a non-fall event based on the fall monitoring classification module and the posture spatiotemporal feature data, and obtain a fall probability sequence.

[0165] In one embodiment, the label conversion module includes:

[0166] A fall label conversion submodule is configured to, if a fall event exists in the posture sample data, set a fall label for the portion corresponding to the fall event in the posture sample data and set a blank label for the remaining portion, thereby obtaining the true fall label sequence;

[0167] The active label setting submodule is used to set one or more active labels in the posture sample data based on a preset setting interval if there is no fall event in the posture sample data, and set blank labels in the rest to obtain the real fall label sequence.

[0168] In one embodiment, the fall monitoring device further includes:

[0169] A signal acquisition module, used to acquire millimeter wave signals associated with the target human body;

[0170] A data calculation module is used to calculate the azimuth data, distance data, elevation data and speed data of each point on the target human body relative to the millimeter wave signal acquisition device based on the millimeter wave signal;

[0171] The point cloud generation module is used to calculate the coordinate values ​​of each point of the target human body based on the orientation data, the distance data, the elevation angle data and the speed data to obtain coordinate point cloud data.

[0172] In one embodiment, the point cloud generation module includes:

[0173] A point cloud acquisition submodule, used to acquire the coordinate point cloud data;

[0174] a missing value supplement submodule, configured to, if there are missing data frames in the coordinate point cloud data, use the coordinate mean of a preset first number of frames as the coordinate value of the missing data frame to obtain continuous coordinate data;

[0175] A data normalization submodule, configured to normalize the continuous coordinate data based on a preset normalization method to obtain normalized coordinate data;

[0176] an abnormal value processing submodule, configured to, if an abnormal data frame exists in the normalized coordinate data, use the coordinate mean of a preset second number of frames as the coordinate value of the abnormal data frame to obtain processed normalized coordinate data, wherein the difference between the coordinate value corresponding to the abnormal data frame and the coordinate mean of the adjacent multiple frames of data is greater than a preset abnormal threshold;

[0177] The data smoothing submodule is used to smooth the processed normalized coordinate data based on a preset smoothing window to obtain posture sample data.

[0178] The present application provides a fall monitoring device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the fall monitoring method of the above-mentioned embodiment 1.

[0179] Reference below Figure 7 , which shows a schematic structural diagram of a fall monitoring device suitable for implementing an embodiment of the present application. The fall monitoring device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, tablet computers, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The fall monitoring device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0180] like Figure 7 As shown, the fall monitoring device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. Various programs and data required for the operation of the fall monitoring device are also stored in RAM 1004. The processing device 1001, ROM 1002, and RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the fall monitoring device to communicate with other devices wirelessly or wired to exchange data. Although the figure shows a fall monitoring device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems can be implemented or provided instead.

[0181] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0182] The fall monitoring device provided by this application, which utilizes the fall monitoring method described in the above-mentioned embodiment, can address the technical issue of low recognition accuracy in current fall monitoring methods. Compared to the prior art, the beneficial effects of the fall monitoring device provided by this application are the same as those of the fall monitoring method described in the above-mentioned embodiment. The other technical features of this fall monitoring device are the same as those disclosed in the above-mentioned embodiment, and are not further elaborated here.

[0183] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0184] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0185] The present application also provides a computer program product, comprising a computer program, which implements the steps of the above-mentioned fall monitoring method when executed by a processor.

[0186] The computer program product provided in this application can solve the technical problem of low recognition accuracy of current fall detection methods. Compared with the existing technology, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the fall detection method provided in the above embodiment, and will not be repeated here.

[0187] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A fall monitoring method, characterized in that: The method includes: Obtain human body posture point cloud data; The posture point cloud data is identified by a preset fall recognition model to obtain a recognition result, wherein the recognition result includes whether the event corresponding to the posture point cloud data is a fall event or not a fall event, and the fall recognition model is obtained by training a training model based on posture sample data with different fall durations corresponding to real fall label sequences.

2. The method according to claim 1, wherein Before the step of obtaining the human body posture point cloud data, the method further includes: Obtain posture sample data of different fall durations; Identifying the posture sample data of different fall durations using the to-be-trained model to obtain a target fall label sequence; Based on a preset CTC algorithm, generating a fall label corresponding to the fall event in the posture sample data to obtain the real fall label sequence; Calculating a CTC loss value based on the target fall label sequence and the real fall label sequence; Based on the CTC loss value, the parameters of the model to be trained are adjusted to obtain a fall recognition model.

3. The method according to claim 2, wherein The step of identifying the posture sample data of different fall durations by the to-be-trained model to obtain a target fall label sequence comprises: Identifying the posture sample data using the to-be-trained model, determining the probability of each time step in the posture sample data corresponding to a fall event and a non-fall event, and obtaining a fall probability sequence; Based on the fall probability sequence and the CTC algorithm, determining different fall label sequences and the probabilities of each label sequence corresponding to the different fall label sequences; The fall label sequence corresponding to the maximum label sequence probability among the label sequence probabilities is used as the target fall label sequence.

4. The method according to claim 3, wherein The model to be trained includes a spatial feature extraction module, a temporal feature extraction module, and a fall monitoring classification module. The steps of identifying the posture sample data by the model to be trained, determining the probability of each time step in the posture sample data corresponding to a fall event and a non-fall event, and obtaining a fall probability sequence include: Based on a preset first number of spatial feature extraction modules, performing spatial feature extraction on the posture sample data to obtain posture spatial feature data; Based on a preset second number of time feature extraction modules, performing time feature extraction on the posture spatial feature data to obtain posture spatiotemporal feature data, wherein the first number is greater than the second number; Based on the fall monitoring classification module and the posture spatiotemporal feature data, the probability of each time step in the posture spatiotemporal feature data corresponding to a fall event and a non-fall event is determined to obtain a fall probability sequence.

5. The method according to claim 2, wherein: The step of generating a fall label corresponding to the fall event in the posture sample data based on a preset CTC algorithm to obtain the real fall label sequence includes: If a fall event exists in the posture sample data, a fall label is set for the portion of the posture sample data corresponding to the fall event, and a blank label is set for the remaining portion to obtain the true fall label sequence; If there is no fall event in the posture sample data, one or more active labels are set in the posture sample data based on a preset interval, and blank labels are set in the rest to obtain the real fall label sequence.

6. The method according to claim 2, wherein: Before the step of obtaining posture sample data of different fall durations, the method further includes: Acquire millimeter wave signals associated with the target human body; Based on the millimeter wave signal, the azimuth data, distance data, elevation data and speed data of each point on the target human body relative to the millimeter wave signal acquisition device are calculated; Based on the azimuth data, the distance data, the elevation angle data and the speed data, the coordinate values ​​of each point of the target human body are calculated to obtain coordinate point cloud data.

7. The method according to claim 6, wherein: After the step of calculating the coordinate values ​​of each point of the target human body based on the orientation data, the distance data, the elevation data and the speed data to obtain coordinate point cloud data, the method further includes: Acquire the coordinate point cloud data; If there are missing data frames in the coordinate point cloud data, the coordinate mean of the preset first number of frames is used as the coordinate value of the missing data frame to obtain continuous coordinate data; Normalizing the continuous coordinate data based on a preset normalization method to obtain normalized coordinate data; If there is an abnormal data frame in the normalized coordinate data, the coordinate mean of the preset second number of frames is used as the coordinate value of the abnormal data frame to obtain processed normalized coordinate data, wherein the difference between the coordinate value corresponding to the abnormal data frame and the coordinate mean of the adjacent multiple frames of data is greater than a preset abnormal threshold; Based on a preset smoothing window, the processed normalized coordinate data is smoothed to obtain posture sample data.

8. A fall monitoring device, characterized in that: The device comprises: Data acquisition module, used to obtain human body posture point cloud data; A fall recognition module is used to identify the posture point cloud data through a preset fall recognition model to obtain a recognition result, wherein the recognition result includes whether the event corresponding to the posture point cloud data is a fall event or not a fall event, and the fall recognition model is obtained by training the training model based on posture sample data with different fall durations corresponding to real fall label sequences.

9. A fall monitoring device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the fall monitoring method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the fall monitoring method according to any one of claims 1 to 7 are implemented.

Images