Fall detection method, apparatus and device based on laser point cloud

Abstract

The application discloses a fall detection method, device and equipment based on laser point cloud. The method comprises the following steps: acquiring multiple frames of point cloud data including current frame point cloud data; for each frame of point cloud data, identifying the key parts of a human body from the point cloud data, determining the key points of the key parts, calculating the displacement of the key points in the vertical direction within the time length of the multiple frames of point cloud data, and judging whether the displacement is greater than or equal to a threshold value; when the displacement is greater than or equal to the threshold value, it is determined that a fall event occurs, and a fall alarm is sent. The method can achieve non-sensing detection, is easy to be accepted by the elderly, and the point cloud data does not collect the biological characteristics of the human body, so there is no risk of privacy leakage. In addition, by identifying the key parts of the human body and judging whether a fall event occurs according to the displacement of the key points of the key parts in the vertical direction, the whole human body does not need to be modeled for the fall action, and the detection efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to laser point cloud technology, and in particular to a fall detection method, device and equipment based on laser point cloud. BACKGROUND

[0002] Old people are prone to fall due to the decline of physical function. If the fall is not discovered and treated in time, it may cause permanent loss of body tissue of the old people, and even death. Therefore, it is necessary to detect the fall of the old people.

[0003] An existing fall detection method uses a wearable device (for example, a smart watch) to detect the fall action. However, due to the complicated use process of the wearable device, such as charging, wearing and device maintenance, it is difficult for the old people to accept. Another fall detection method uses a camera to shoot video images to track the trajectory of the human body, so as to realize fall action detection. However, due to the privacy protection problem of the camera shooting video images, it cannot be applied to some private places (for example, toilets, bathrooms). SUMMARY

[0004] The present application provides a fall detection method, device and equipment based on laser point cloud, so as to realize non-inductive detection, and the point cloud data will not collect the biological characteristics of the human body, without the risk of privacy leakage.

[0005] In a first aspect, the present application provides a fall detection method based on laser point cloud, comprising:

[0006] Obtaining multiple frames of point cloud data including current frame point cloud data;

[0007] For each frame of point cloud data, identifying the key parts of the human body from the point cloud data;

[0008] Determining the key points of the key parts;

[0009] Calculating the displacement amount of the key points in the vertical direction within the time length of the multiple frames of point cloud data;

[0010] Judging whether the displacement amount is greater than or equal to a threshold value;

[0011] When the displacement amount is greater than or equal to the threshold value, it is determined that a fall event occurs, and a fall alarm is issued.

[0012] Optionally, for each frame of point cloud data, identifying the key parts of the human body from the point cloud data, comprising:

[0013] Eliminating the environmental point cloud data from the point cloud data, and retaining the human body point cloud data;

[0014] The human body point cloud data is encoded to obtain an encoded feature representing semantic similarity between points in the human body point cloud data.

[0015] The encoded feature is pooled to obtain a global feature of the human body point cloud data.

[0016] The encoded feature and the global feature are spliced in a spatial dimension to obtain a fusion feature.

[0017] The fusion feature is subjected to semantic segmentation to identify key parts of the human body.

[0018] Optionally, the human body point cloud data is encoded to obtain an encoded feature representing semantic similarity between points in the human body point cloud data, including:

[0019] The human body point cloud data is embedded to convert the human body point cloud data into an embedded feature.

[0020] The embedded feature is processed based on a self-attention mechanism to obtain attention features of multiple different feature scales.

[0021] The attention features of multiple different feature scales are spliced in a spatial dimension to obtain a spliced feature.

[0022] The spliced feature is linearized, batch-normalized, and activated non-linearly to obtain an encoded feature representing semantic similarity between points in the human body point cloud data.

[0023] Optionally, the human body point cloud data is embedded to convert the human body point cloud data into an embedded feature, including:

[0024] The human body point cloud data is linearized, batch-normalized, and activated non-linearly to obtain a regularized human body point cloud feature.

[0025] The regularized human body point cloud feature is down-sampled to obtain a down-sampled point cloud feature.

[0026] For each sampling point in the down-sampled point cloud feature, the sampling point is determined from the regularized human body point cloud feature.

[0027] From the regularized human body point cloud feature, K nearest points closest to the sampling point are calculated to determine a K-neighborhood feature of each sampling point.

[0028] The down-sampled point cloud feature and the K-neighborhood features of all the sampling points are fused to obtain a neighborhood fusion feature.

[0029] The neighborhood fusion feature is linearized, batch-normalized, and activated non-linearly to obtain an embedded feature of the human body point cloud data.

[0030] Optionally, the embedding features are processed based on a self-attention mechanism to obtain attention features of multiple different feature scales, including:

[0031] The embedding features are input into a self-attention module for processing, the self-attention module including multiple cascaded self-attention units, an input feature of a first self-attention unit being the embedding features, an input feature of an i-th self-attention unit being an output feature of an (i-1)-th self-attention unit, each self-attention unit being configured to output an attention feature of a different feature scale, i being a positive integer greater than 1.

[0032] Optionally, the processing procedure of the self-attention unit is as follows:

[0033] The input feature is linearly transformed to obtain a query matrix, a key matrix and a value matrix;

[0034] The dot product of the query matrix and the key matrix is calculated and normalized to obtain an attention weight matrix;

[0035] The dot product of the attention weight matrix and the value matrix is calculated to obtain a self-attention matrix;

[0036] The difference between the input feature and the self-attention matrix is calculated to obtain a bias matrix;

[0037] The bias matrix is linearized, batch-normalized and activated to obtain a bias compensation matrix of the input feature;

[0038] The sum of the input feature and the bias compensation matrix is calculated to obtain an output feature of the self-attention unit.

[0039] Optionally, the key points of the key part are determined, including:

[0040] The point cloud data of the key part is clustered to calculate a cluster center of the point cloud data of the key part;

[0041] The cluster center is taken as a key point of the key part.

[0042] Optionally, the fall detection method based on laser point cloud further includes:

[0043] When the displacement is less than a threshold, the next frame of point cloud data is received as the current frame of point cloud data, and the step of acquiring multiple frames of point cloud data including the current frame of point cloud data is returned to be executed.

[0044] In a second aspect, the application further provides a fall detection device based on laser point cloud, including:

[0045] a point cloud data acquisition module, configured to acquire multi-frame point cloud data including current frame point cloud data;

[0046] a key position identification module, configured to identify a key position of a human body from the point cloud data for each frame of point cloud data;

[0047] a key point determination module, configured to determine a key point of the key position;

[0048] a displacement amount calculation module, configured to calculate a displacement amount of the key point in a vertical direction in a time length of the multi-frame point cloud data;

[0049] a judgment module, configured to judge whether the displacement amount is greater than or equal to a threshold value;

[0050] a fall alarm module, configured to determine that a fall event occurs and issue a fall alarm when the displacement amount is greater than or equal to the threshold value.

[0051] In a third aspect, the present application further provides an electronic device, comprising:

[0052] one or more processors;

[0053] a storage device configured to store one or more programs;

[0054] when the one or more programs are executed by the one or more processors, the one or more processors implement the laser point cloud-based fall detection method provided in the first aspect of the present application.

[0055] In a fourth aspect, the present application further provides a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the laser point cloud-based fall detection method provided in the first aspect of the present application.

[0056] The laser point cloud-based fall detection method provided by the present application acquires multi-frame point cloud data including current frame point cloud data, identifies a key position of a human body from the point cloud data for each frame of point cloud data, determines a key point of the key position, calculates a displacement amount of the key point in a vertical direction in a time length of the multi-frame point cloud data, judges whether the displacement amount is greater than or equal to a threshold value, determines that a fall event occurs and issues a fall alarm when the displacement amount is greater than or equal to the threshold value, can achieve non-invasive detection, is easy to be accepted by the elderly, and the point cloud data does not collect biological characteristics of the human body, without the risk of privacy leakage. In addition, by identifying the key position of the human body and judging whether a fall event occurs according to the displacement amount of the key point of the key position in the vertical direction, the whole human body does not need to be modeled for a fall action, and the detection efficiency is improved.

[0057] It is to be understood that the details set forth herein do not limit the scope of the embodiments of the application to the specific embodiments described. The foregoing detailed description has set forth various embodiments of the devices and / or processes via the use of specific terminology. However, embodiments of the application are not necessarily limited to those described, but can be practiced with the BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0059] Figure 1 A flow chart of a fall detection method based on laser point cloud provided by the present application;

[0060] Figure 2 A structural schematic diagram of a key part recognition model provided by the present application;

[0061] Figure 3 A schematic diagram of an embedding processing provided by the present application;

[0062] Figure 4 A processing flow chart of a self-attention unit provided by the present application;

[0063] Figure 5 A structural schematic diagram of a fall detection device based on laser point cloud provided by the present application;

[0064] Figure 6 A structural schematic diagram of an electronic device provided by the present application.

[0065] The above drawings have shown the specific embodiments of the present application, and will be described in more detail hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0066] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort should be within the scope of protection of the present application.

[0067] It should be noted that the terms "first", "second", and the like in the description and in the claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0068] Figure 1 A flowchart of a laser point cloud-based fall detection method provided by the present application, the present embodiment can be applicable to human fall detection using a laser radar, and is particularly suitable for human fall detection in a privacy space. The method can be executed by a laser point cloud-based fall detection device provided by the present embodiment of the present application, which can be implemented in software and / or hardware, and is usually configured in an electronic device, such as a server. Figure 1 As shown, the laser point cloud-based fall detection method can include the following steps:

[0069] S101, acquiring multiple frames of point cloud data including current frame point cloud data.

[0070] For example, in the present embodiment, the laser radar can be arranged indoors, such as in a bedroom, bathroom, washroom, living room, etc., which is not limited herein. The laser radar continuously emits laser, scans the surrounding environment, and receives the reflected laser of external objects to obtain point cloud data. Specifically, the laser radar emits a laser beam including multiple (for example, 32 lines or 64 lines) laser beams, and receives the reflected laser reflected by the surface of an object. The laser radar accurately measures the propagation time of the laser from emission to reflection. Since the speed of light is known, the propagation time can be converted into a distance measurement. Combined with the height of the laser radar and the laser scanning angle, the three-dimensional coordinates of each light spot (i.e., the points in the point cloud) in the coordinate system of the laser radar can be accurately calculated.

[0071] In the present embodiment, the electronic device (which can be a server) continuously receives the point cloud data uploaded by the laser radar. When the current frame point cloud data is received, multiple frames of point cloud data including the current frame point cloud data (i.e., the current frame point cloud data and the previous multiple frames of point cloud data) are acquired.

[0072] S102, for each frame of point cloud data, identifying the key parts of the human body from the point cloud data.

[0073] In the embodiments of the present application, for each frame of point cloud data, semantic segmentation can be used to identify the key part of the human body from the point cloud data. For example, the key part can be the head, and the displacement of the head in the vertical direction changes most obviously during the fall of the human body. Therefore, the head is selected as the key part. Of course, in other embodiments of the present application, other parts can also be selected as the key part, for example, the torso, which is not limited in the embodiments of the present application.

[0074] In some embodiments of the present application, the point cloud data can be processed in combination with a deep learning algorithm to identify the key part of the human body. For example, for each frame of point cloud data, the key part of the human body is identified from the point cloud data, comprising:

[0075] S1021, removing the environmental point cloud data from the point cloud data and retaining the human point cloud data.

[0076] For each frame of point cloud data, the environmental point cloud data is removed from the point cloud data and the human point cloud data is retained. For example, the laser radar can scan the environment when there is no one in the room to obtain the environmental point cloud data and prestore it in the electronic device. During detection, only the received point cloud data needs to be compared with the pre-stored environmental point cloud data to remove the environmental point cloud data in the point cloud data. After removing the environmental point cloud data, the human point cloud data is retained, avoiding the influence of the environmental point cloud data on the subsequent detection results, and at the same time, reducing the data amount and improving the processing efficiency.

[0077] S1022, encoding the human point cloud data to obtain an encoding feature representing the semantic similarity between points in the human point cloud data.

[0078] In the embodiments of the present application, the human point cloud data is encoded and converted to a high-dimensional encoding feature, which can represent the semantic similarity between points in the human point cloud data. For example, the unordered human point cloud data can be first vectorized to obtain a vector representation (or feature) of the human point cloud data, and then the vector is input into an encoder for encoding to obtain an encoding feature representing the semantic similarity between points in the human point cloud data. The encoder can use common encoder structures and other structures in the field of deep learning, which is not limited in the embodiments of the present application.

[0079] Figure 2 A structure diagram of a key part identification model provided by the present application is shown in Figure 2 As shown in the figure, the key part identification model mainly includes an encoding part (Encoder) and a semantic recognition part (Segmentation), the encoding part is used for encoding the human point cloud data, and the semantic recognition part is used for identifying the key part. In some embodiments of the present application, referring to Figure 2The process of encoding the human body point cloud data to obtain the encoded features is as follows:

[0080] 1. The human body point cloud data is embedded to convert the human body point cloud data into embedded features.

[0081] Reference Figure 2 The human body point cloud data is input into an embedding layer for embedding processing, and the discrete and non-continuous human body point cloud data is converted into continuous embedded features (i.e., vectors or matrices).

[0082] Figure 3 An embedding processing schematic provided by the present application is shown in FIG. 1, and the process of embedding processing is as follows: Figure 3

[0083] 1.1. The human body point cloud data is linearized, batch normalized, and nonlinearly activated to obtain regularized human body point cloud features.

[0084] The human body point cloud data is input into an LBR unit for processing. The LBR unit includes a linearization layer (Linear), a batch normalization layer (Batch Normalization), and a Rule function activation layer connected in sequence. The human body point cloud data is linearized, then batch normalized, and finally nonlinearly activated using the Rule function to obtain regularized human body point cloud features.

[0085] 1.2. The regularized human body point cloud features are down-sampled to obtain down-sampled point cloud features.

[0086] The regularized human body point cloud features are sampled to obtain down-sampled point cloud features P1. In some embodiments of the present application, for example, a farthest point sampling algorithm (FPS) can be used for sampling processing. The farthest point sampling algorithm is an algorithm for down-sampling point cloud data. Its purpose is to select a subset from a large-scale point set so that the point cloud in the subset can represent the shape features of the original point cloud as much as possible. The advantage of the farthest point sampling algorithm is that it can maintain the coverage range and shape features of the point cloud, because it preferentially selects those points that are more isolated in geometry. This makes the algorithm particularly suitable for non-uniformly distributed point cloud data, especially when the point cloud contains details and is unevenly distributed.

[0087] 1.3. For each sampling point in the down-sampled point cloud features, the sampling point is determined from the regularized human body point cloud features.

[0088] In embodiments of the present application, for each sampling point in the down-sampled point cloud features P1, the sampling point is determined from the regularized human body point cloud features by tracing back to the regularized human body point cloud features.​

[0089] 1.4、Calculate the K nearest neighbors of the sampling point from the regularized human point cloud features to determine the K neighborhood features of each sampling point.

[0090] Calculate the K nearest neighbors of the sampling point from the regularized human point cloud features to determine the K neighborhood features of each sampling point. For example, as shown in the figure, Figure 3 the K nearest neighbor algorithm (KNN) can be used to calculate the K nearest neighbors of the sampling point from the regularized human point cloud features to determine the K neighborhood features P2 of each sampling point.

[0091] 1.5、Fuse the down-sampled point cloud features and the K neighborhood features of all sampling points to obtain neighborhood fusion features.

[0092] In an embodiment of the present application, the down-sampled point cloud features and the K neighborhood features of all sampling points are fused to obtain neighborhood fusion features. For example, as shown in the figure, Figure 3 the down-sampled point cloud features are repeated K times to obtain a matrix with the same spatial dimension as the matrix composed of the K neighborhood features of all sampling points, then the matrix is added to the matrix composed of the K neighborhood features of all sampling points, and then the features after K times of replication are spliced to obtain neighborhood fusion features P3.

[0093] 1.6、Linearize, batch normalize and nonlinearly activate the neighborhood fusion features to obtain embedding features of the human point cloud data.

[0094] In an embodiment of the present application, the neighborhood fusion features are linearized, batch normalized and nonlinearly activated to obtain embedding features of the human point cloud data. For example, as shown in the figure, Figure 3 the neighborhood fusion features are input into an LBR unit for linearization, batch normalization and nonlinear activation to obtain embedding features P4 of the human point cloud data.

[0095] In an embodiment of the present application, the embedding features of the human point cloud data fuse the K neighborhood features of the sampling points, introduce local feature information, and are beneficial to improve the accuracy of subsequent key part recognition.

[0096] 2、Process the embedding features based on a self-attention mechanism to obtain attention features of multiple different feature scales.

[0097] In an embodiment of the present application, the embedding features are processed based on a self-attention mechanism to capture the correlation between elements in the embedding features, and attention features of multiple different feature scales are obtained, which is beneficial to improve the accuracy of subsequent key part recognition. For example, as shown in the figure, Figure 2As shown, the embedding feature is input into a self-attention module for processing, the self-attention module includes a plurality of cascaded self-attention units Attention, the input feature of the first self-attention unit is the embedding feature, the input feature of the i-th self-attention unit is the output feature of the (i-1)-th self-attention unit, and the output feature of each self-attention unit is simultaneously used as the attention feature output by the self-attention unit, each self-attention unit outputs attention features of different feature scales, and i is a positive integer greater than 1.

[0098] Each self-attention unit can process the input feature in a common self-attention processing manner, that is, the input feature is linearly transformed respectively to obtain a query matrix, a key matrix and a value matrix, the dot product of the query matrix and the key matrix is calculated and normalized to obtain an attention weight matrix, the dot product of the attention weight matrix and the value matrix is calculated to obtain a self-attention matrix, and finally the sum of the self-attention matrix and the input feature is calculated to obtain the output feature of the self-attention unit.

[0099] In the embodiment of the application, in order to better capture the details and local geometric structures in the point cloud, and further improve the accuracy of subsequent key part recognition, the self-attention unit is improved, Figure 4 A processing flowchart of the self-attention unit provided by the application is shown in Figure 4 As shown, the processing process of the self-attention unit is as follows:

[0100] (1) The input feature is linearly transformed respectively to obtain a query matrix, a key matrix and a value matrix.

[0101] In the embodiment of the application, the input feature is first linearly transformed three times (Linear) to obtain a query matrix Query, a key matrix Key and a value matrix Value respectively.

[0102] (2) The dot product of the query matrix and the key matrix is calculated and normalized to obtain an attention weight matrix.

[0103] For example, as shown in Figure 4 After the dot product of the query matrix and the key matrix is calculated, the dot product is input into an LBR unit for linearization, batch normalization and nonlinear activation to obtain an attention weight matrix.

[0104] (3) The dot product of the attention weight matrix and the value matrix is calculated to obtain a self-attention matrix.

[0105] For example, as shown in Figure 4 The dot product of the attention weight matrix and the value matrix is calculated to obtain a self-attention matrix.

[0106] (4) The difference between the input feature and the self-attention matrix is calculated to obtain a bias matrix.

[0107] As shown in Figure 4 , the difference between the input feature and the self-attention matrix is calculated to obtain a deviation matrix. The deviation matrix represents the deviation of the input feature from the self-attention matrix.

[0108] (5) Linearization, batch normalization and non-linear activation are performed on the deviation matrix to obtain a deviation compensation matrix of the input feature.

[0109] As shown in Figure 4 , the deviation matrix is input into an LBR unit for linearization, batch normalization and non-linear activation to obtain a deviation compensation matrix of the input feature.

[0110] (6) The sum of the input feature and the deviation compensation matrix is calculated to obtain the output feature of the self-attention unit.

[0111] As shown in Figure 4 , the sum of the input feature and the deviation compensation matrix is calculated to obtain the output feature of the self-attention unit. That is, the input feature is adjusted using the calculated deviation compensation matrix, so that the model can better capture the details and local geometric structure in the point cloud, thereby improving the accuracy of subsequent key part recognition.

[0112] 3. Concatenate the attention features of different feature scales in the spatial dimension to obtain a concatenated feature.

[0113] Referring back to Figure 2 , the attention features of different feature scales output by the multiple self-attention units are concatenated (indicated by C in the figure) in the spatial dimension to obtain a concatenated feature.

[0114] 4. Linearization, batch normalization and non-linear activation are performed on the concatenated feature to obtain an encoding feature representing the semantic similarity between points in the human point cloud data.

[0115] As shown in Figure 2 , the concatenated feature is input into an LBR unit for linearization, batch normalization and non-linear activation to obtain an encoding feature representing the semantic similarity between points in the human point cloud data.

[0116] S1023, Pooling processing is performed on the encoding feature to obtain a global feature of the human point cloud data.

[0117] As shown in Figure 2The encoding features are subjected to a pooling processing (MA-Pooling) to obtain global features of the human point cloud data, as shown in the semantic recognition part. In an example embodiment, the encoding features are first input into a max-pooling layer (Max-Pooling), and the output of the max-pooling layer is input into an average-pooling layer (Average-Pooling), and finally the global features of the human point cloud data are obtained.

[0118] S1024, the encoding features and the global features are concatenated in the spatial dimension to obtain fusion features.

[0119] In an example embodiment, the encoding features and the global features are concatenated (Concatenate) in the spatial dimension to obtain fusion features. Since the encoding features contain a large amount of local feature information, the fusion features obtained finally fuse the local feature information and the global feature information, which is beneficial to improve the recognition accuracy of the subsequent key parts.

[0120] S1025, the fusion features are subjected to semantic segmentation to identify the key parts of the human body.

[0121] As shown in Figure 2 the fusion features are subjected to semantic segmentation to identify the key parts of the human body. In an example, the fusion features are input into an LBRD unit for processing, the LBRD unit includes a linearization layer (Linear), a batch normalization layer (Batch Normalization), a Rule function activation layer and a dropout layer (Dropout) connected in sequence, the features output by the LBRD unit are sent into an LBR unit for processing, the LBR unit includes a linearization layer (Linear), a batch normalization layer (Batch Normalization) and a Rule function activation layer connected in sequence, the features output by the LBR unit are sent into a linearization layer (Linear) for processing to obtain each part of the human body, and the key parts of the human body are determined therefrom, for example, the head.

[0122] S103, determining the key points of the key parts.

[0123] After the key parts of the human body are identified, the key points of the key parts are determined. In an example, the geometric center or the centroid of the key parts can be calculated as the key points of the key parts, which is not limited in the example embodiment. In an example embodiment, the point cloud data of the key parts is clustered to calculate the cluster center of the point cloud data of the key parts, and the cluster center is taken as the key point of the key parts. The clustering algorithm can use K-means, FCM, etc., which is not limited in the example embodiment.

[0124] S104, calculate the displacement amount of the key point in the vertical direction in the duration of the multi-frame point cloud data.

[0125] For example, each frame of point cloud data carries a timestamp, and the duration of the multi-frame point cloud data can be calculated according to the difference between the timestamps of the last frame and the first frame of the multi-frame point cloud data. According to the coordinate information of the key point in the world coordinate system in the multi-frame point cloud data, the coordinate information of the key point in the world coordinate system is calculated through coordinate system conversion, and the coordinate value in the vertical direction is determined, and then the displacement amount of the key point in the vertical direction is calculated according to the coordinate value in the vertical direction.

[0126] S105, judge whether the displacement amount is greater than or equal to the threshold value.

[0127] In the embodiment of the application, the calculated displacement amount of the key point in the vertical direction is compared with the preset threshold value, and it is judged whether the displacement amount is greater than or equal to the threshold value.

[0128] S106, determine that a fall event occurs and issue a fall alarm.

[0129] When the displacement amount is greater than or equal to the threshold value, it is determined that a fall event occurs, and a fall alarm is issued to the mobile terminal of the guardian, so that the guardian can discover and arrive at the scene in time for rescue.

[0130] The prior art establishes a human action model through the key points of each part of the human body after recognizing each part of the human body, and analyzes whether a fall event occurs through the action model. However, the modeling process has a large amount of calculation, and therefore the detection efficiency is low. The present application judges whether a fall event occurs according to the displacement amount of the key point in the vertical direction of the key part of the human body, without modeling the action of the whole human body, thereby improving the detection efficiency.

[0131] S107, receive the next frame of point cloud data as the current frame of point cloud data.

[0132] When the displacement amount is less than the threshold value, it is considered that a fall event does not occur, and the next frame of point cloud data is continuously received as the current frame of point cloud data, and the step of acquiring multi-frame point cloud data including the current frame of point cloud data is returned to be executed, and the above fall detection process is continuously performed.

[0133] The application provides a fall detection method based on laser point cloud, acquires multiple frames of point cloud data including current frame point cloud data, identifies a key part of a human body from the point cloud data for each frame of point cloud data, determines a key point of the key part, calculates a displacement amount of the key point in a vertical direction within a time length of the multiple frames of point cloud data, judges whether the displacement amount is greater than or equal to a threshold value, and when the displacement amount is greater than or equal to the threshold value, determines that a fall event occurs and issues a fall alarm, so that the fall detection can be achieved without feeling, is easy to be accepted by the elderly, and the point cloud data does not collect biological characteristics of the human body, and there is no risk of privacy leakage. In addition, the key part of the human body is identified, and whether a fall event occurs is judged according to the displacement amount of the key point of the key part in the vertical direction, so that the whole human body does not need to be modeled for a fall action, and the detection efficiency is improved.

[0134] Figure 5 A structure diagram of a fall detection device based on laser point cloud provided by the application is shown in Figure 5 The fall detection device based on laser point cloud comprises:

[0135] A point cloud data acquisition module 201 is configured to acquire multiple frames of point cloud data including current frame point cloud data.

[0136] A key part identification module 202 is configured to identify a key part of a human body from the point cloud data for each frame of point cloud data.

[0137] A key point determination module 203 is configured to determine a key point of the key part.

[0138] A displacement amount calculation module 204 is configured to calculate a displacement amount of the key point in a vertical direction within a time length of the multiple frames of point cloud data.

[0139] A judgment module 205 is configured to judge whether the displacement amount is greater than or equal to a threshold value.

[0140] A fall alarm module 206 is configured to determine that a fall event occurs and issue a fall alarm when the displacement amount is greater than or equal to the threshold value.

[0141] In some embodiments of the application, the key part identification module 202 comprises:

[0142] A point cloud elimination sub-module is configured to eliminate environmental point cloud data from the point cloud data and retain human body point cloud data.

[0143] An encoding sub-module is configured to perform encoding processing on the human body point cloud data to obtain an encoding feature representing semantic similarity between points in the human body point cloud data.

[0144] A pooling processing sub-module is configured to perform pooling processing on the encoding feature to obtain a global feature of the human body point cloud data.

[0145] a feature splicing submodule configured to splice the encoded features and the global features in a spatial dimension to obtain fused features;

[0146] a semantic segmentation submodule configured to perform semantic segmentation on the fused features to identify key parts of the human body.

[0147] In some embodiments of the present application, the encoding submodule comprises:

[0148] an embedding processing unit configured to perform embedding processing on the human point cloud data to convert the human point cloud data into embedding features;

[0149] an attention processing unit configured to process the embedding features based on a self-attention mechanism to obtain attention features of different feature scales;

[0150] a feature splicing unit configured to splice the attention features of different feature scales in a spatial dimension to obtain spliced features;

[0151] an encoded feature determination unit configured to linearize, batch-normalize and nonlinearly activate the spliced features to obtain encoded features representing semantic similarities between points in the human point cloud data.

[0152] In some embodiments of the present application, the embedding processing unit comprises:

[0153] a regularization subunit configured to linearize, batch-normalize and nonlinearly activate the human point cloud data to obtain regularized human point cloud features;

[0154] a down-sampling subunit configured to down-sample the regularized human point cloud features to obtain down-sampled point cloud features;

[0155] a sampling point determination subunit configured to determine, for each sampling point in the down-sampled point cloud features, the sampling point from the regularized human point cloud features;

[0156] a neighborhood calculation subunit configured to calculate K nearest points closest to the sampling point from the regularized human point cloud features to determine K-neighborhood features of each sampling point;

[0157] a feature fusion subunit configured to fuse the down-sampled point cloud features and the K-neighborhood features of all the sampling points to obtain neighborhood fused features;

[0158] an embedding feature determination subunit configured to linearize, batch-normalize and nonlinearly activate the neighborhood fused features to obtain embedding features of the human point cloud data.

[0159] In some embodiments of the present application, the attention processing unit comprises:

[0160] An attention processing subunit configured to input the embedding feature into a self-attention module for processing, the self-attention module comprising a plurality of cascaded self-attention units, an input feature of a first self-attention unit being the embedding feature, an input feature of an i-th self-attention unit being an output feature of an (i-1)-th self-attention unit, each of the self-attention units being configured to output an attention feature of a different feature scale, i being a positive integer greater than 1.

[0161] In some embodiments of the present application, the processing procedure of the self-attention unit is as follows:

[0162] linearly transforming the input feature to obtain a query matrix, a key matrix and a value matrix;

[0163] calculating a dot product of the query matrix and the key matrix and performing normalization processing to obtain an attention weight matrix;

[0164] calculating a dot product of the attention weight matrix and the value matrix to obtain a self-attention matrix;

[0165] calculating a difference between the input feature and the self-attention matrix to obtain a bias matrix;

[0166] linearizing, batch normalizing and performing nonlinear activation on the bias matrix to obtain a bias compensation matrix of the input feature;

[0167] calculating a sum of the input feature and the bias compensation matrix to obtain an output feature of the self-attention unit.

[0168] In some embodiments of the present application, the key point determination module 203 comprises:

[0169] a clustering sub-module configured to cluster the point cloud data of the key part and calculate a clustering center of the point cloud data of the key part;

[0170] a key point determination sub-module configured to take the clustering center as a key point of the key part.

[0171] In some embodiments of the present application, the laser point cloud-based fall detection device further comprises:

[0172] a return execution module configured to, when the displacement is less than a threshold, receive a next frame of point cloud data as a current frame of point cloud data and return to execute the step of acquiring a plurality of frames of point cloud data including the current frame of point cloud data.

[0173] The above laser point cloud-based fall detection device can execute the laser point cloud-based fall detection method provided by the foregoing embodiments of the present application, and has corresponding function modules and beneficial effects of executing the laser point cloud-based fall detection method.

[0174] Figure 6 A block diagram of an electronic device is provided. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices (e.g., headgear, eyewear, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the applications described and / or claimed in this document.

[0175] As shown in Figure 6 The electronic device includes at least one processor 11 and memory, such as read-only memory (ROM) 12, random access memory (RAM) 13, etc., communicatively connected to the at least one processor 11, where the memory stores computer programs executable by the at least one processor 11, and the processor 11 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 12 or loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for operation of the electronic device can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0176] Various components in the electronic device are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0177] The processor 11 can be various general and / or special purpose processing components having processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the laser point cloud based fall detection method.

[0178] In some embodiments, the laser point cloud based fall detection method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the laser point cloud based fall detection method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the laser point cloud based fall detection method by any other appropriate means, such as by means of firmware.

[0179] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0180] Computer programs used to implement the methods of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor of the machine, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0181] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0182] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0183] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0184] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. Servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are a host product in the cloud computing service system to solve the defects of great management difficulty and weak business scalability in traditional physical hosts and VPS services.

[0185] The embodiment of the present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the fall detection method based on laser point cloud as provided in any embodiment of the present application.

[0186] The computer program product can be written in one or more programming languages or combinations of languages including object-oriented languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0187] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present application. For example, the steps recited in the present application can be performed in parallel, in series, or in a different order, and the present application is not limited in this regard.

[0188] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for fall detection based on laser point cloud, characterized in that, The method comprises the following steps: acquiring multi-frame point cloud data including current frame point cloud data; for each frame of point cloud data, identifying a key part of a human body from the point cloud data; determining a key point of the key part; calculating a displacement amount of the key point in a vertical direction within a time length of the multi-frame point cloud data; judging whether the displacement amount is greater than or equal to a threshold value; when the displacement amount is greater than or equal to the threshold value, determining that a fall event occurs and issuing a fall alarm; wherein, for each frame of point cloud data, identifying a key part of a human body from the point cloud data comprises: removing environmental point cloud data from the point cloud data and retaining human body point cloud data; performing encoding processing on the human body point cloud data to obtain an encoding feature representing semantic similarity between points in the human body point cloud data; performing pooling processing on the encoding feature to obtain a global feature of the human body point cloud data; splicing the encoding feature and the global feature in a spatial dimension to obtain a fusion feature; performing semantic segmentation on the fusion feature to identify the key part of the human body; the encoding processing on the human body point cloud data to obtain the encoding feature representing the semantic similarity between points in the human body point cloud data comprises: performing embedding processing on the human body point cloud data to convert the human body point cloud data into an embedding feature; processing the embedding feature based on a self-attention mechanism to obtain attention features of multiple different feature scales; splicing the attention features of multiple different feature scales in a spatial dimension to obtain a spliced feature; performing linearization, batch normalization and non-linear activation on the spliced feature to obtain the encoding feature representing the semantic similarity between points in the human body point cloud data; the embedding processing on the human body point cloud data to convert the human body point cloud data into an embedding feature comprises: performing linearization, batch normalization and non-linear activation on the human body point cloud data to obtain a regularized human body point cloud feature; down-sampling the regularized human body point cloud feature to obtain a down-sampled point cloud feature; for each sampling point in the down-sampled point cloud feature, determining the sampling point from the regularized human body point cloud feature; calculating K nearest points of each sampling point from the regularized human body point cloud feature to determine a K-neighborhood feature of each sampling point; fusing the down-sampled point cloud feature and the K-neighborhood features of all the sampling points to obtain a neighborhood fusion feature; performing linearization, batch normalization and non-linear activation on the neighborhood fusion feature to obtain the embedding feature of the human body point cloud data.

2. The laser point cloud based fall detection method of claim 1, wherein, processing the embedding feature based on a self-attention mechanism to obtain attention features of multiple different feature scales comprises: inputting the embedding feature into a self-attention module for processing, the self-attention module comprising a plurality of cascaded self-attention units, the input feature of a first self-attention unit being the embedding feature, the input feature of an i-th self-attention unit being the output feature of an i-1-th self-attention unit, each self-attention unit being used to output attention features of different feature scales, i being a positive integer greater than 1.

3. The laser point cloud based fall detection method of claim 2, wherein, the processing process of the self-attention unit is as follows: Linearly transforming the input features respectively to obtain a query matrix, a key matrix and a value matrix; Calculating the dot product of the query matrix and the key matrix and performing normalization processing to obtain an attention weight matrix; Calculating the dot product of the attention weight matrix and the value matrix to obtain a self-attention matrix; Calculating the difference between the input features and the self-attention matrix to obtain a bias matrix; Linearizing, batch normalizing and nonlinearly activating the bias matrix to obtain a bias compensation matrix of the input features; Calculating the sum of the input features and the bias compensation matrix to obtain the output features of the self-attention unit.

4. The laser point cloud based fall detection method according to any one of claims 1-3, characterized in that, Determining the key points of the key part, comprising: Clustering the point cloud data of the key part to calculate the cluster centers of the point cloud data of the key part; Taking the cluster centers as the key points of the key part.

5. The laser point cloud based fall detection method according to any one of claims 1-3, characterized in that, Further comprising: When the displacement amount is less than a threshold, receiving the next frame of point cloud data as the current frame of point cloud data, and returning to perform the step of obtaining multiple frames of point cloud data including the current frame of point cloud data.

6. A fall detection device based on laser point cloud, characterized in that, Comprise: A point cloud data acquisition module for acquiring multiple frames of point cloud data including the current frame of point cloud data; A key part identification module for identifying the key part of the human body from the point cloud data for each frame of point cloud data; A key point determination module for determining the key points of the key part; A displacement amount calculation module for calculating the displacement amount of the key points in the vertical direction in the time length of multiple frames of point cloud data; A judgment module for judging whether the displacement amount is greater than or equal to a threshold; A fall alarm module for determining that a fall event occurs and issuing a fall alarm when the displacement amount is greater than or equal to the threshold; The key part identification module comprises: A point cloud elimination submodule for eliminating environmental point cloud data from the point cloud data and retaining human body point cloud data; An encoding submodule for encoding the human body point cloud data to obtain encoding features representing the semantic similarity between points in the human body point cloud data; A pooling processing submodule for performing pooling processing on the encoding features to obtain global features of the human body point cloud data; A feature splicing submodule for splicing the encoding features and the global features in the spatial dimension to obtain fusion features; A semantic segmentation submodule for performing semantic segmentation on the fusion features to identify the key part of the human body; The encoding submodule comprises: An embedding processing unit for embedding the human body point cloud data to convert the human body point cloud data into embedding features; An attention processing unit for processing the embedding features based on a self-attention mechanism to obtain attention features of multiple different feature scales; A feature splicing unit for splicing the attention features of multiple different feature scales in the spatial dimension to obtain spliced features; An encoding feature determination unit for linearizing, batch normalizing and nonlinearly activating the spliced features to obtain encoding features representing the semantic similarity between points in the human body point cloud data; The embedding processing unit comprises: a regularization subunit configured to linearize, batch-normalize, and non-linearly activate the human point cloud data to obtain regularized human point cloud features; a down-sampling subunit configured to down-sample the regularized human point cloud features to obtain down-sampled point cloud features; a sampling point determination subunit configured to determine, for each sampling point in the down-sampled point cloud features, the sampling point from the regularized human point cloud features; a neighborhood calculation subunit configured to calculate K nearest points closest to the sampling point from the regularized human point cloud features to determine a K-neighborhood feature for each sampling point; a feature fusion subunit configured to fuse the down-sampled point cloud features and the K-neighborhood features of all the sampling points to obtain neighborhood fused features; an embedding feature determination subunit configured to linearize, batch-normalize, and non-linearly activate the neighborhood fused features to obtain embedding features of the human point cloud data.

7. An electronic device, comprising: comprising: one or more processors; a memory device storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the laser point cloud based fall detection method according to any one of claims 1-5.

Images