A method and system for elderly safety monitoring

Abstract

The present invention provides a method and system for elderly safety monitoring, which relates to the field of intelligent detection technology, including: collecting millimeter wave radar data in a private scene, obtaining first point cloud data of the monitored object in the private scene, and judging whether the monitored object is in a falling state in the private scene based on the first point cloud data; collecting video stream information in a public scene, obtaining a human image of the monitored object, identifying key points of the human skeleton, and determining the position of the key points of the human skeleton; collecting millimeter wave radar data in a public scene, obtaining second point cloud data of the monitored object in the public scene; inputting the position of the key points of the human skeleton, the second point cloud data and the RGB image data into a multi-stream neural network, using an attention mechanism for feature fusion, and judging whether the monitored object is in a falling state in the public scene. While fully protecting the privacy of the monitored object, the present invention also solves the problem of high false alarm rate due to complex environment in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of intelligent detection technology, and in particular to a method and system for elderly safety monitoring. Background Art

[0002] The proportion of the elderly population is growing rapidly. From the perspective of medical health, the elderly are prone to falling due to their physical functions. Falling of the elderly will not only cause and aggravate their diseases, but may even endanger their lives. Traditional manual monitoring methods require a dedicated person to monitor the same monitored object in real time. Due to limited manpower, manual monitoring methods cannot achieve all-weather monitoring, so the elderly cannot get timely assistance when they fall, which increases the harm of falling events. At the same time, manual monitors cannot monitor in some private places such as toilets and bathrooms. Because of the shortcomings of traditional manual monitoring methods, machine monitoring methods have gradually emerged.

[0003] At present, machine monitoring methods usually use cameras, environmental sensors and wearable devices. The mainstream monitoring methods based on cameras have problems such as high false alarm rate, high computing overhead, and privacy infringement. Cameras also infringe on users' privacy in some scenarios such as toilets and bathrooms, and toilets and bathrooms are often the places where people are most likely to fall, so real-time monitoring must be implemented. In addition, the home environment is often more complex and prone to occlusion, poor lighting and other problems. In addition, mainstream monitoring methods based on cameras often rely on deep learning technology, and the computing resource overhead required for deep learning is huge.

[0004] Monitoring methods based on environmental sensors generally use millimeter-wave radar as an environmental sensor. The millimeter-wave radar detects the point cloud data of the monitored object and further determines whether the elderly have fallen. Although the monitoring method based on millimeter-wave radar technology environmental sensors can solve the problem of privacy infringement, it is similar to the mainstream monitoring method of cameras and will also encounter problems such as occlusion and poor lighting in complex environments, resulting in high rates of missed reports and false alarms. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a method and system for elderly safety monitoring, which aims to fully protect the privacy of the monitored object while solving the problem of high false alarm rate due to complex environment in the prior art.

[0006] The embodiment of the present application provides the following technical solution: a method for elderly safety monitoring, comprising:

[0007] Collecting and preprocessing millimeter-wave radar data in a privacy scene to obtain first point cloud data of the monitored object in the privacy scene, and performing behavior recognition based on the first point cloud data to determine whether the monitored object is in a falling state in the privacy scene;

[0008] Collect and pre-process video stream information in a public scene to obtain a human body image of the monitored object, identify key points of the human skeleton in the human body image, and determine the positions of the key points of the human skeleton;

[0009] Collect and pre-process the millimeter-wave radar data in the public scene to obtain the second point cloud data of the monitored object in the public scene;

[0010] The positions of the key points of the human skeleton, the second point cloud data and the RGB color space image data are input into a multi-stream neural network, and an attention mechanism is used in the multi-stream neural network to perform feature fusion to determine whether the monitored object is in a falling state in a public scene.

[0011] According to an embodiment of the present application, video stream information in a public scene is collected and preprocessed to obtain a human body image of a monitored object, including:

[0012] Performing image enhancement and denoising processing on the video stream information to obtain a processed picture frame;

[0013] Performing background subtraction processing on the picture frame to separate the moving human object from the environmental background;

[0014] Binarizing the moving human object to obtain a binary image;

[0015] The shape of the binary image is completed by using a morphological closing operation, and the completed maximum envelope space is used as the human body boundary box to obtain the human body image.

[0016] According to an embodiment of the present application, identifying key points of human skeleton in the human body image and determining positions of the key points of human skeleton include:

[0017] The RTMPose key point detection algorithm is used to identify the key points of the human skeleton in the human body image and determine the positions of the key points of the human skeleton. According to an embodiment of the present application, millimeter wave radar data in a privacy scene is collected and preprocessed to obtain first point cloud data of the monitored object in the privacy scene, and behavior recognition is performed based on the first point cloud data to determine whether the human body is in a falling state in the privacy scene, including:

[0018] Performing Range-FFT processing on the collected millimeter-wave radar data, including: performing 1D windowing and 1D fast Fourier transform on the millimeter-wave radar data;

[0019] The Capon BF algorithm is used to create a heat map, including: first, static carrier removal, then calculation of the spatial correlation matrix, and finally, the Capon BF is used to generate a heat map in the horizontal direction;

[0020] The CFAR algorithm is used to perform target detection, including: detecting target points by a two-dimensional CFAR algorithm according to the thermal map;

[0021] Calculate the spatial correlation matrix at each target point of [distance, horizontal angle] to generate a vertical heat map, and obtain the vertical angle of the target by peak search on the vertical heat map;

[0022] Convert [distance, horizontal angle, vertical angle] into (X, Y, Z) coordinates through coordinate transformation, with Z max As the height of the human body, when the height change of the human body in the privacy scene is greater than the set threshold, it is determined that the monitored object is in a falling state in the privacy scene.

[0023] According to an embodiment of the present application, millimeter wave radar data in a public scene is collected and preprocessed to obtain second point cloud data of a monitored object in the public scene, including:

[0024] The collected millimeter-wave radar data is collected by frequency-modulated continuous wave radar and the callback data is preprocessed to obtain a distance-speed image. According to the distance-speed image, the range gate signal where the monitored object is located is coherently superimposed and short-time Fourier transform is performed to obtain a Doppler time-frequency spectrum image. The Doppler time-frequency spectrum images are combined to obtain the second point cloud data.

[0025] According to an embodiment of the present application, the positions of the key points of the human skeleton are taken as sparse data, and the second point cloud data and the RGB color space image data are input into a multi-stream neural network as dense data, and an attention mechanism is used in the multi-stream neural network for feature fusion. The specific steps include:

[0026] Using a point cloud feature extractor to perform feature extraction on the second point cloud data, including: using a CNN-Lstm network to extract a speed distance feature f1 from the distance-speed image, using an EffcientNet network to extract a frequency feature f2 from the Doppler time-frequency spectrum image, using a Vision-Transformer to extract an image feature f3 from the RGB color space image data, and using a multi-layer perceptron to extract a bone key point feature f4 from the human bone key points, the multi-layer perceptron comprising three fully connected layers, and each fully connected layer has a ReLU function and a normalization layer between them;

[0027] The skeleton key point feature f4 is used as sparse data, and a fully connected layer is used to map the skeleton key point feature into a Query feature matrix. Three fully connected layers are used to map the second point cloud data and the RGB color space image data into a Key feature matrix and a Value feature matrix, respectively, and the speed distance feature f1, the frequency feature f2 and the image feature f3 are enhanced to obtain the enhanced speed distance feature f1', the frequency feature f2' and the image feature f3';

[0028] The speed distance feature f1', the frequency feature f2' and the image feature f3' are added bit by bit to obtain the fusion feature f'. The specific formula is as follows:

[0029] f'=αf1'+βf2'+γf3'

[0030] Among them, α, β and γ are learning parameters, and the initial values ​​of α, β and γ are

[0031] A fully connected layer is used to receive the fusion feature f' and output the confidence falldown , when the confidence falldown When it is greater than the set threshold, it is judged that the monitored object is in a fall state in a public scene, and a fall signal is issued.

[0032] According to an embodiment of the present application, determining whether the monitored object is in a falling state in a public scene also includes:

[0033] The fall video corresponding to the fall signal is saved, and the saved fall video is manually verified for a second time. If a fall event is confirmed, a notification message is manually sent to the guardian of the care object, and the fall signal data is saved in the database;

[0034] Among them, when the notification information is sent manually to the guardian of the care object, a notification text message is first sent to the mobile device of the guardian, and real-time monitoring is carried out to see whether the guardian has viewed the notification text message and the timing is started. If the guardian has viewed the notification text message within the set time range, the verification is ended. If the guardian has not viewed the notification text message within the set time range, a phone call is made to notify the guardian.

[0035] According to an embodiment of the present application, it also includes: when the fall signal data entry in the database exceeds a set value, autonomously training the data in the database and optimizing the learning parameters.

[0036] The present application also provides an elderly safety monitoring system using the above-mentioned elderly safety monitoring method, comprising:

[0037] A privacy scene monitoring device, the privacy scene monitoring device comprising a millimeter wave radar and a server, the millimeter wave radar is used to collect millimeter wave radar data in the privacy scene, the server is used to pre-process the millimeter wave radar data to obtain first point cloud data of the monitored object in the privacy scene, and perform behavior recognition based on the first point cloud data to determine whether the monitored object is in a falling state in the privacy scene;

[0038] A public scene monitoring device, the public scene monitoring device comprising an image acquisition device, a millimeter wave radar and a server, the image acquisition device is used to collect video stream information in the public scene, and the millimeter wave radar is used to collect millimeter wave radar data in the public scene;

[0039] The server is used to preprocess the video stream information, obtain a human body image of the monitored object, identify key points of the human skeleton in the human body image, and determine the positions of the key points of the human skeleton; it is also used to preprocess the millimeter wave radar data, obtain second point cloud data of the monitored object in a public scene, input the positions of the key points of the human skeleton, the second point cloud data, and the RGB color space image data into a multi-stream neural network, use an attention mechanism in the multi-stream neural network to perform feature fusion, and determine whether the monitored object is in a falling state in the public scene.

[0040] According to one embodiment of the present application, it also includes a manual remote monitoring platform, which is used to perform a secondary verification on the judgment result of the server. If a fall event is confirmed to have occurred, the manual remote monitoring platform sends a notification message to the guardian of the care object and saves the fall signal data to the database.

[0041] Compared with the prior art, the beneficial effects that can be achieved by at least one of the above-mentioned technical solutions adopted in the embodiments of this specification include at least the following: the embodiments of the present invention can effectively protect the privacy of users in special occasions by designing a fall detection method based on millimeter-wave radar and a fall detection method based on skeletal key points; at the same time, by adopting a method combining millimeter-wave radar with images, when a fall occurs, the real-time status can be checked in time through monitoring images, which greatly increases the convenience of the guardian / user. In addition, the embodiments of the present invention design a method combining machine monitoring and manual monitoring, which fully reduces the time cost and economic cost required by the manual monitoring method while also improving the accuracy of the machine monitoring method. The embodiments of the present invention can improve the accuracy of algorithm monitoring by designing a post-verification optimization strategy to adapt to the behavioral characteristics of different elderly people. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A process deployment scheme for the elderly safety monitoring method in an embodiment of the present invention;

[0044] Figure 2 This is the process of the elderly safety monitoring system in an embodiment of the present invention;

[0045] Figure 3 This is a process for detecting elderly falls based on millimeter wave radar in an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram of the point cloud obtained by the elderly fall detection method based on millimeter wave radar;

[0047] Figure 5 The key points of the skeleton are obtained by the elderly fall detection method based on the key points of the skeleton;

[0048] Figure 6 Schematic diagram of a multi-stream neural network in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0050] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0051] like Figure 1-Figure 2 As shown, an embodiment of the present invention provides a method for elderly safety monitoring, including:

[0052] In a privacy scenario: collecting and preprocessing millimeter-wave radar data in the privacy scenario to obtain first point cloud data of the monitored object in the privacy scenario, and performing behavior recognition based on the first point cloud data to determine whether the monitored object is in a falling state in the privacy scenario;

[0053] In a public scene: video stream information in the public scene is collected and preprocessed to obtain a human body image of the monitored object, key points of the human skeleton in the human body image are identified, and the positions of the key points of the human skeleton are determined; millimeter-wave radar data in the public scene is collected and preprocessed to obtain second point cloud data of the monitored object in the public scene; the positions of the key points of the human skeleton, the second point cloud data, and the RGB color space image data are input into a multi-stream neural network, and an attention mechanism is used in the multi-stream neural network for feature fusion to determine whether the monitored object is in a falling state in the public scene.

[0054] The embodiment of the present invention proposes an innovative multi-stream neural network, which uses an attention mechanism to fuse multimodal data (the positions of the key points of the human skeleton, the second point cloud data, and the RGB color space image data) to determine whether the elderly is in a falling state, effectively improving the detection accuracy.

[0055] The embodiment of the present invention can effectively protect the privacy of users in special occasions by designing a fall detection method based on millimeter wave radar and a fall detection method based on skeleton key points. In addition, in public scenes, the millimeter wave radar and video stream information are fused, and the fusion result is used to jointly determine whether the monitored object is in a fall state in the public scene, which can further improve the accuracy of detection and avoid missed judgments and misjudgments.

[0056] In the specific implementation, the video stream information in the public scene is collected and pre-processed to obtain the human body image of the monitored object, including the following steps:

[0057] (1) Performing image enhancement and denoising on the video stream information to obtain processed picture frames; specifically, performing image enhancement and denoising on the video stream information to improve picture quality and reduce the impact of dark pictures collected in complex environments. Specifically, it mainly includes two stages of processing: adaptive histogram equalization and mean filtering. Adaptive histogram equalization is performed on the picture to suppress the impact of low light.

[0058] (2) performing background subtraction processing on the image frame to separate the moving human object from the environmental background; specifically, this embodiment separates the moving person from the background through KNN (K-Nearest Neighbors) background subtraction.

[0059] (3) Binarizing the moving human object to obtain a binary image; specifically, binarizing the moving person using the Otsu method. In computer vision and image processing, the Otsu binarization method is used to automatically binarize clustering-based images, that is, to degenerate a grayscale image into a binary image.

[0060] (4) A morphological closing operation is used to complete the shape of the binary image, and the completed maximum envelope space is used as the human body boundary box to obtain the human body image.

[0061] In this embodiment, the key points of the human skeleton in the human body image are identified, and the positions of the key points of the human skeleton are determined, including: using the RTMPose key point detection algorithm to identify the key points of the human skeleton in the human body image, and determine the positions of the key points of the human skeleton, such as Figure 5 shown.

[0062] In this embodiment, millimeter-wave radar data in a privacy scene is collected and preprocessed to obtain first point cloud data of the monitored object in the privacy scene, and behavior recognition is performed based on the first point cloud data to determine whether the human body is in a falling state in the privacy scene. Figure 3-Figure 4 As shown, the following steps are included:

[0063] (1) performing Range-FFT processing on the collected millimeter-wave radar data, including: performing 1D windowing and 1D fast Fourier transform (FFT) on the millimeter-wave radar data;

[0064] (2) Using the Capon BF algorithm to create a heat map, including: first, static carrier removal, then calculating the spatial correlation matrix, and finally generating a horizontal heat map through Capon BF;

[0065] (3) using the CFAR algorithm to perform target detection, including: detecting the target point using the two-dimensional CFAR algorithm according to the thermal map;

[0066] (4) calculating the spatial correlation matrix at each target point of [distance, horizontal angle], generating a vertical heat map, and obtaining the vertical angle of the target by peak search on the vertical heat map;

[0067] (5) Convert [distance, horizontal angle, vertical angle] into (X, Y, Z) coordinates through coordinate transformation. max As the human body height, when the human body height change in the privacy scene is greater than the set threshold, it is determined that the monitored object is in a falling state in the privacy scene; otherwise it is determined to be normal.

[0068] In this embodiment, the millimeter wave radar data in the public scene is collected and preprocessed to obtain the second point cloud data of the monitored object in the public scene, including:

[0069] The collected millimeter-wave radar data is collected by frequency-modulated continuous wave radar and the callback data is preprocessed to obtain a distance-speed image. According to the distance-speed image, the range gate signal where the monitored object is located is coherently superimposed and short-time Fourier transform is performed to obtain a Doppler time-frequency spectrum image. The Doppler time-frequency spectrum images are combined to obtain the second point cloud data.

[0070] like Figure 6 As shown, in this embodiment, the multimodal data of elderly safety monitoring (including the position of key points of human skeleton, second point cloud data and RGB color space image) is input into the multi-stream neural network for processing to determine whether the monitored object is in a falling state in a public scene. At the same time, the present invention uses the position of the key points of the human skeleton as sparse data (low data density), and the second point cloud data and RGB color space image data as dense data into the multi-stream neural network, and uses the attention mechanism in the multi-stream neural network for feature fusion. The specific steps include:

[0071] Using a point cloud feature extractor to perform feature extraction on the second point cloud data, specifically, using a CNN-Lstm network to extract a speed distance feature f1 from the distance-speed image, using an EffcientNet network to extract a frequency feature f2 from the Doppler time-frequency spectrum image, using a Vision-Transformer to extract an image feature f3 from the RGB color space image data, and using a multi-layer perceptron to extract a bone key point feature f4 from the human skeleton key points, the multi-layer perceptron comprising three fully connected layers, and each fully connected layer has a ReLU function and a normalization layer between them;

[0072] The skeleton key point feature f4 is used as sparse data, and a fully connected layer is used to map the skeleton key point feature into a Query feature matrix. Three fully connected layers are used to map the second point cloud data and the RGB color space image data into a Key feature matrix and a Value feature matrix, respectively, and the speed distance feature f1, the frequency feature f2 and the image feature f3 are enhanced to obtain the enhanced speed distance feature f1', the frequency feature f2' and the image feature f3';

[0073] The speed distance feature f1', the frequency feature f2' and the image feature f3' are added bit by bit to obtain the fusion feature f'. The specific formula is as follows:

[0074] f'=αf1'+βf2'+γf3'(1)

[0075] Among them, α, β and γ are learning parameters, and the initial values ​​of α, β and γ are

[0076] Use a fully connected layer to receive the fused feature f' and output the confidence falldown , when the confidence falldown When the value is greater than the set threshold, it is determined that the monitored object is in a falling state in the public scene, and a falling signal is sent. In this embodiment, the threshold is set to 0.5.

[0077] In this embodiment, determining whether the monitored object is in a fallen state in a public scene further includes:

[0078] The fall video corresponding to the fall signal is saved, and the saved fall video is manually verified for a second time. If a fall event is confirmed, a notification message is manually sent to the guardian of the care object, and the fall signal data is saved in the database.

[0079] This embodiment also includes that when a notification message is sent manually to the guardian of the care object, a notification text message is first sent to the mobile device of the guardian, and real-time monitoring is performed to determine whether the guardian has viewed the notification text message and a timer is started. If the guardian has viewed the notification text message within the set time range, the verification is terminated. If the guardian has not viewed the notification text message within the set time range, a phone call is made to notify the guardian.

[0080] This embodiment also includes: when the fall signal data entries in the database exceed a set value, autonomous training is performed on the data in the database, and the learning parameters are optimized.

[0081] In specific implementation, after the verification process is completed, when the data entries in the database exceed 100, the system will conduct autonomous training on the data in the database to continuously improve the accuracy of algorithm monitoring and optimize three learnable parameters (α, β and γ) to adapt to the behavioral characteristics of different elderly people. Specifically, linear regression is used to optimize the parameters α, β and γ in formula (1).

[0082] A method for elderly safety monitoring in an embodiment of the present invention is a combination of a machine monitoring method and a manual monitoring method; the machine monitoring method includes: using a multi-stream neural network to perform human fall detection and millimeter-wave radar human fall detection, directly using millimeter-wave radar human fall detection in private occasions, and fusing the modalities of the two methods in public occasions to improve accuracy.

[0083] The present application also provides a system for elderly safety monitoring, including:

[0084] A privacy scene monitoring device, the privacy scene monitoring device comprising a millimeter wave radar and a server, the millimeter wave radar is used to collect millimeter wave radar data in the privacy scene, the server is used to pre-process the millimeter wave radar data to obtain first point cloud data of the monitored object in the privacy scene, and perform behavior recognition based on the first point cloud data to determine whether the monitored object is in a falling state in the privacy scene;

[0085] A public scene monitoring device, the public scene monitoring device comprising an image acquisition device, a millimeter wave radar and a server, the image acquisition device is used to collect video stream information in the public scene, and the millimeter wave radar is used to collect millimeter wave radar data in the public scene;

[0086] The server is used to pre-process the video stream information, obtain a human body image of the monitored object, identify key points of the human skeleton in the human body image, and determine the positions of the key points of the human skeleton; it is also used to pre-process the millimeter-wave radar data, obtain second point cloud data of the monitored object in a public scene, input the positions of the key points of the human skeleton, the second point cloud data, and the RGB color space image data into a multi-stream neural network, use an attention mechanism in the multi-stream neural network to perform feature fusion, and determine whether the monitored object is in a falling state in the public scene.

[0087] This embodiment also includes a manual remote monitoring platform, which is used to perform a secondary verification on the judgment result of the server. If a fall event is confirmed, the manual remote monitoring platform sends a notification message to the guardian of the care object and saves the fall signal data to the database.

[0088] In this embodiment, the part of the human fall detection based on millimeter-wave radar is mainly deployed in some private places involving the monitored object, and the recognition algorithm is deployed on the server; the human fall detection based on skeletal key points adopts a camera to collect video signals, and the recognition algorithm is deployed on the server; when the fall signal is output, the corresponding fall video will be saved for manual inspection personnel to view. The deployment strategy of the above-mentioned machine monitoring will be differentiated in two different occasions (public occasions and private occasions): the machine monitoring deployment strategy uses millimeter-wave radar for direct perception in private occasions, and the machine monitoring deployment strategy uses cameras and millimeter-wave radar for joint perception in public occasions. In addition, in public scenes, the method of manual monitoring is further combined to further improve the detection accuracy and ensure that there will be no omissions or false alarms. The manual monitoring mainly relies on manual inspection personnel to view the corresponding fall video to determine the danger of the fall, and notify the emergency contact of the monitored object through SMS / phone.

[0089] The working process of the elderly safety monitoring system of the embodiment of the present invention is as follows:

[0090] In a privacy scenario:

[0091] (1) The millimeter-wave radar collects data and sends it to the server;

[0092] (2) The detection algorithm in the server preprocesses the millimeter wave data into point cloud data;

[0093] (3) Perform behavior recognition based on point cloud data to determine whether the elderly person has fallen.

[0094] In a public scene:

[0095] (1) The camera first collects the guardian's video stream;

[0096] (2) The collected video stream of the guardian is transmitted to the server;

[0097] (3) The search algorithm deployed on the server finds the location of the elderly and cuts out the image block containing the elderly;

[0098] (4) The RTMPose skeleton key point detection algorithm deployed on the server detects the key points of the human body from the image block containing the elderly in step (3);

[0099] (5) The millimeter-wave radar collects data and sends it to the server;

[0100] (6) The detection algorithm in the server preprocesses the millimeter wave data into point cloud data;

[0101] (7) Use a multi-stream neural network to fusion analyze the three modalities of point cloud data, skeleton key points, and RGB images to determine whether the elderly person has fallen; if the elderly person has fallen, a fall signal is output, and the corresponding elderly person falling video is saved to the server;

[0102] (8) The verification personnel follow the verification process to determine whether the elderly person is in danger, and notify the elderly person's emergency contact by text message or phone call. At the same time, the verification personnel record data entries, which are speed distance features f1', frequency features f2', and image features f3', to determine whether the elderly person has actually fallen.

[0103] (9) When the data items recorded in step (8) exceed 100, the server optimizes the algorithm parameters based on the data set to adapt to the behavioral characteristics of different elderly people.

[0104] The embodiment of the present invention can effectively protect the privacy of users in special occasions by designing a fall detection method based on millimeter wave radar and a fall detection method based on skeleton key points; at the same time, by adopting a method combining millimeter wave radar with images, when a fall occurs, the real-time status can be checked in time through monitoring images, which greatly increases the convenience of the guardian / user. In addition, the embodiment of the present invention designs a method combining machine monitoring and manual monitoring, which fully reduces the time cost and economic cost required by the manual monitoring method while also improving the accuracy of the machine monitoring method.

[0105] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for elderly safety monitoring, characterized in that: include: Collecting and preprocessing millimeter-wave radar data in a privacy scene to obtain first point cloud data of the monitored object in the privacy scene, and performing behavior recognition based on the first point cloud data to determine whether the monitored object is in a falling state in the privacy scene; Collect and pre-process video stream information in a public scene to obtain a human body image of the monitored object, identify key points of the human skeleton in the human body image, and determine the positions of the key points of the human skeleton; Collect and pre-process the millimeter-wave radar data in the public scene to obtain the second point cloud data of the monitored object in the public scene; Inputting the positions of the key points of the human skeleton, the second point cloud data, and the RGB color space image data into a multi-stream neural network, using an attention mechanism in the multi-stream neural network to perform feature fusion, and determining whether the monitored object is in a falling state in a public scene; Collect and pre-process the millimeter-wave radar data in the public scene to obtain the second point cloud data of the monitored object in the public scene, including: The collected millimeter-wave radar data is collected by frequency-modulated continuous wave radar and the callback data is preprocessed to obtain a distance-speed image. According to the distance-speed image, the range gate signal where the monitored object is located is coherently superimposed and then short-time Fourier transform is performed to obtain a Doppler time-frequency spectrum image. The Doppler time-frequency spectrum images are combined to obtain the second point cloud data; The positions of the key points of the human skeleton are used as sparse data, and the second point cloud data and the RGB color space image data are used as dense data to input into a multi-stream neural network, and an attention mechanism is used in the multi-stream neural network to perform feature fusion. The specific steps include: Using a point cloud feature extractor to perform feature extraction on the second point cloud data, including: using a CNN-Lstm network to extract a speed distance feature f1 from the distance-speed image, using an EffcientNet network to extract a frequency feature f2 from the Doppler time-frequency spectrum image, using a Vision-Transformer to extract an image feature f3 from the RGB color space image data, and using a multi-layer perceptron to extract a bone key point feature f4 from the human bone key points, the multi-layer perceptron comprising three fully connected layers, and each fully connected layer has a ReLU function and a normalization layer between them; The skeleton key point feature f4 is used as sparse data, and a fully connected layer is used to map the skeleton key point feature into a Query feature matrix. Three fully connected layers are used to map the second point cloud data and the RGB color space image data into a Key feature matrix and a Value feature matrix, respectively, and the speed distance feature f1, the frequency feature f2 and the image feature f3 are enhanced to obtain the enhanced speed distance feature f1', the frequency feature f2' and the image feature f3'; The speed distance feature f1', the frequency feature f2' and the image feature f3' are added bit by bit to obtain the fusion feature f'. The specific formula is as follows: f'=αf1'+βf2'+γf3' Among them, α, β and γ are learning parameters, and the initial values ​​of α, β and γ are A fully connected layer is used to receive the fusion feature f' and output the confidence falldown , when the confidence falldown When it is greater than the set threshold, it is judged that the monitored object is in a fall state in a public scene, and a fall signal is issued.

2. The elderly safety monitoring method according to claim 1, characterized in that: Collect and pre-process video stream information in public scenes to obtain human body images of monitored subjects, including: Performing image enhancement and denoising processing on the video stream information to obtain a processed picture frame; Performing background subtraction processing on the picture frame to separate the moving human object from the environmental background; Binarizing the moving human object to obtain a binary image; The shape of the binary image is completed by using a morphological closing operation, and the completed maximum envelope space is used as a human body boundary box to obtain the human body image.

3. The elderly safety monitoring method according to claim 1, characterized in that: Identifying key points of human skeleton in the human body image and determining positions of the key points of human skeleton include: The RTMPose key point detection algorithm is used to identify the key points of the human skeleton in the human body image and determine the positions of the key points of the human skeleton.

4. The elderly safety monitoring method according to claim 1, characterized in that: Collecting and preprocessing millimeter-wave radar data in a privacy scene to obtain first point cloud data of a monitored object in the privacy scene, and performing behavior recognition based on the first point cloud data to determine whether a human body is in a falling state in the privacy scene, including: Performing Range-FFT processing on the collected millimeter-wave radar data, including: performing 1D windowing and 1D fast Fourier transform on the millimeter-wave radar data; The Capon BF algorithm is used to create a heat map, including: first, static carrier removal, then calculation of the spatial correlation matrix, and finally, the Capon BF is used to generate a heat map in the horizontal direction; The CFAR algorithm is used to perform target detection, including: detecting target points by a two-dimensional CFAR algorithm according to the thermal map; Calculate the spatial correlation matrix at each target point of [distance, horizontal angle] to generate a vertical heat map, and obtain the vertical angle of the target by peak search on the vertical heat map; Convert [distance, horizontal angle, vertical angle] into (X, Y, Z) coordinates through coordinate transformation, with Z max As the height of the human body, when the height change of the human body in the privacy scene is greater than the set threshold, it is determined that the monitored object is in a falling state in the privacy scene.

5. The elderly safety monitoring method according to claim 1, characterized in that: Determining whether the subject of supervision is in a state of falling in a public scene also includes: The fall video corresponding to the fall signal is saved, and the saved fall video is manually verified for a second time. If a fall event is confirmed, a notification message is manually sent to the guardian of the care object, and the fall signal data is saved in the database; Among them, when the notification information is sent manually to the guardian of the care object, a notification text message is first sent to the mobile device of the guardian, and real-time monitoring is carried out to see whether the guardian has viewed the notification text message and the timing is started. If the guardian has viewed the notification text message within the set time range, the verification is ended. If the guardian has not viewed the notification text message within the set time range, a phone call is made to notify the guardian.

6. The elderly safety monitoring method according to claim 5, characterized in that: Also includes: When the fall signal data entries in the database exceed a set value, autonomous training is performed on the data in the database, and the learning parameters are optimized.

7. An elderly safety monitoring system using the elderly safety monitoring method according to any one of claims 1 to 6, characterized in that: include: A privacy scene monitoring device, the privacy scene monitoring device comprising a millimeter wave radar and a server, the millimeter wave radar is used to collect millimeter wave radar data in the privacy scene, the server is used to pre-process the millimeter wave radar data to obtain first point cloud data of the monitored object in the privacy scene, and perform behavior recognition based on the first point cloud data to determine whether the monitored object is in a falling state in the privacy scene; A public scene monitoring device, the public scene monitoring device comprising an image acquisition device, a millimeter wave radar and a server, the image acquisition device is used to collect video stream information in the public scene, and the millimeter wave radar is used to collect millimeter wave radar data in the public scene; The server is used to preprocess the video stream information, obtain a human body image of the monitored object, identify key points of the human skeleton in the human body image, and determine the positions of the key points of the human skeleton; it is also used to preprocess the millimeter wave radar data, obtain second point cloud data of the monitored object in a public scene, input the positions of the key points of the human skeleton, the second point cloud data, and the RGB color space image data into a multi-stream neural network, use an attention mechanism in the multi-stream neural network to perform feature fusion, and determine whether the monitored object is in a falling state in the public scene.

8. The elderly safety monitoring system according to claim 7, characterized in that: It also includes a manual remote monitoring platform, which is used to perform a second verification on the judgment result of the server. If a fall event is confirmed, the manual remote monitoring platform sends a notification message to the guardian of the care object and saves the fall signal data to the database.

Images