A lightweight home care system, electronic device and medium for fall recognition

Abstract

The present invention provides a lightweight family care system, electronic device and medium for fall recognition, including: a picture acquisition and video encoding module acquires picture frames from a lens sensor and sends them to a fall detection and recognition module, and at the same time performs h265 video encoding on the picture frames and saves them in a database; when the fall detection and recognition module detects a fall behavior while running a fall detection and recognition algorithm, it extracts the human skeleton key point information of the fall person and when using the human skeleton key point information for enhanced recognition to confirm the existence of the fall behavior, it reports the fall behavior event to the business processing module; when the business processing module receives the reported fall behavior event, it notifies the pre-configured family-related personnel. The lightweight fall recognition scheme of the system of the present invention can be well deployed on a low-cost, limited computing power home surveillance camera, and plays a role in protecting the safety of the elderly and children in the family.

Description

Technical Field

[0001] The present invention relates to the technical field of fall recognition, and in particular to a lightweight home care system for fall recognition, an electronic device and a readable storage medium. Background Art

[0002] The topic of family care has always attracted everyone's attention and discussion, especially for left-behind children and the elderly in rural areas. With the use of a large number of low-cost home cameras, the problem of people away from home knowing the family situation remotely anytime and anywhere has been basically solved. However, it is impossible to watch the home surveillance camera all the time, especially when the elderly fall and endanger their lives, timely intervention and treatment are needed. Because home cameras are generally low-cost and have limited computing power, there is an urgent need for a lightweight fall detection and recognition system and equipment to care for the family. Summary of the invention

[0003] In view of the above problems, the present invention is proposed to provide a lightweight fall recognition home care system, an electronic device and a readable storage medium that overcome the above problems or at least partially solve the above problems.

[0004] The present invention provides a lightweight home care system for fall recognition, comprising:

[0005] The image acquisition and video encoding module is used to acquire image frames from the lens sensor and send them to the fall detection and recognition module, and at the same time perform h265 video encoding on the image frames and save them in the database;

[0006] A fall detection and recognition module is used to extract the human skeleton key point information of the fallen person when a fall behavior is detected during the operation of the fall detection and recognition algorithm, and report the fall behavior event to the business processing module when the human skeleton key point information is used for enhanced recognition to confirm the existence of the fall behavior;

[0007] The business processing module is used to notify pre-configured family-related personnel when receiving a reported fall behavior event.

[0008] Optionally, the fall detection and identification module is further used to:

[0009] Run the fall detection and recognition algorithm to detect whether there is a fall behavior;

[0010] If a fall is detected, the human skeleton key point information of the person who fell is extracted and when the human skeleton key point information is used for enhanced identification to confirm the fall, the fall event is reported to the business processing module;

[0011] If no falling behavior is detected, wait for the image acquisition and video encoding module to send a new image frame.

[0012] Optionally, the fall detection and identification module is further used to:

[0013] Using the human skeleton key point information to enhance recognition and analyze whether there is a fall behavior;

[0014] If a fall is confirmed, the fall event is reported to the business processing module;

[0015] If it is confirmed that there is no fall behavior, the fall detection and recognition algorithm is re-run to detect whether there is a fall behavior.

[0016] Optionally, the lightweight fall detection neural network of the fall detection and recognition algorithm is composed of a depthwise separable convolution unit, which includes pointwise convolution, depthwise convolution, batch normalization BN, and activation function Relu. The number of parameters and the amount of calculation are about 1 / 9 of the 3x3 convolution kernel, which is suitable for deployment on a home surveillance camera platform with limited computing resources.

[0017] The lightweight fall detection neural network is divided into three parts:

[0018] Backbone network: adopts VGG straight-tube network structure, and uses the maximum pooling operation with minimal computational effort for downsampling; uses the standard 3x3 convolution kernel in the first stage of stage 1, and after 3x3 convolution with a step size of 2 and maximum pooling, the height and width of the output feature map become 1 / 4 of the input image;

[0019] Feature pyramid fusion layer: The upsampling operation adopts the nearest neighbor interpolation method; the Stage n layer is upsampled and simply added to the Stage n-1 layer to perform feature fusion, so that the low-level network has high-level semantic information;

[0020] Multi-task decoupling head layer: including target category, target box, and skeleton key points;

[0021] The loss function Loss includes category loss, object confidence loss, box loss, and key point loss.

[0022] Optionally, the fall detection and identification module is further used to:

[0023] Extracting the human skeleton key point information of the fallen person and converting the human skeleton key point information into a skeleton feature vector;

[0024] Inputting the bone feature vector into a support vector machine to analyze whether there is a falling behavior;

[0025] If a fall is confirmed, the fall event is reported to the business processing module.

[0026] Optionally, the business processing module is also used to: support remote online viewing and management of home surveillance cameras.

[0027] Optionally, the business processing module notifies pre-configured family-related personnel in a manner including: sending an Internet text message or pushing via a WeChat applet.

[0028] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the electronic device executes the computer program, it loads a lightweight home care system for fall recognition as described in any one of the embodiments of the present invention.

[0029] The present invention also provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the lightweight fall recognition home care system as described in any one of the embodiments of the present invention is loaded.

[0030] The present invention includes the following advantages:

[0031] The lightweight fall recognition family care system of the present invention obtains picture frames from the lens sensor through the picture acquisition and video encoding module, and sends them to the fall detection and recognition module, and at the same time, performs h265 video encoding on the picture frames and saves them in the database. When the fall detection and recognition algorithm is running to detect the fall behavior, the fall detection and recognition module extracts the human skeleton key point information of the fall person and when the human skeleton key point information is used to strengthen the recognition and confirm the fall behavior, the fall behavior event is reported to the business processing module. When the business processing module receives the reported fall behavior event, it notifies the pre-configured family-related personnel. The lightweight fall recognition scheme of the system of the present invention can be well deployed on low-cost, limited computing power home surveillance cameras, and plays a role in protecting the safety of the elderly and children in the family. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a structural block diagram of a lightweight fall recognition home care system provided by an embodiment of the present invention;

[0033] Figure 2 A schematic diagram of a lightweight fall detection neural network for a fall detection and recognition algorithm provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] Reference Figure 1, shows a structural block diagram of a lightweight fall recognition home care system provided in an embodiment of the present invention, which may specifically include the following modules:

[0036] The image acquisition and video encoding module is used to acquire image frames from the lens sensor and send them to the fall detection and recognition module, and at the same time perform h265 video encoding on the image frames and save them in the database;

[0037] A fall detection and recognition module is used to extract the human skeleton key point information of the fallen person when a fall behavior is detected during the operation of the fall detection and recognition algorithm, and report the fall behavior event to the business processing module when the human skeleton key point information is used for enhanced recognition to confirm the existence of the fall behavior;

[0038] The business processing module is used to notify pre-configured family-related personnel when receiving a reported fall behavior event.

[0039] The lightweight fall recognition home care system of the present invention can be applied to home surveillance cameras. The main hardware options for home surveillance cameras can be: the CPU is cheap and has good performance, dual-core CPU, and 1T computing power NPU is used to run AI neural network algorithms to accelerate the effect. A 2 million resolution camera, 1G memory, the total hardware cost can be controlled at around 200 yuan, which meets the low-cost usage scenario and meets everyone's consumption budget.

[0040] The lightweight home care system of the present invention focuses on the fall recognition function and is designed for home surveillance cameras. The system mainly includes an image acquisition and video encoding module, which is responsible for capturing image frames from the camera lens sensor and sending them to the fall detection and recognition module for processing in real time. At the same time, it also performs h265 video encoding on these image frames and saves them in the database, which can save storage space and optimize video playback quality.

[0041] The fall detection and recognition module is the core module of the system of the present invention. The fall detection and recognition module runs the fall detection and recognition algorithm, and can monitor and analyze the received picture frames in real time to detect whether a fall has occurred. Once a possible fall event is detected, the module will further extract the human skeleton key point information of the person who fell, and strengthen the recognition based on this information to confirm the existence of the fall behavior. After confirmation, the module will immediately report the fall behavior event to the business processing module.

[0042] The business processing module is responsible for receiving and responding to events reported by the fall detection and identification module. Once a fall event notification is received, the module will immediately trigger the preset emergency process, such as sending a notification to pre-configured family members so that they can take timely response measures to ensure the safety and health of their family members.

[0043] The lightweight fall recognition home care system of the present invention realizes real-time monitoring and rapid response to fall behaviors in the home environment through the joint action of the image acquisition and video encoding module, the fall detection and recognition module, and the business processing module, thereby improving the safety of family members and bringing peace of mind and convenience to more families.

[0044] In one embodiment of the present invention, the fall detection and identification module is further used for:

[0045] Run the fall detection and recognition algorithm to detect whether there is a fall behavior;

[0046] If a fall is detected, the human skeleton key point information of the person who fell is extracted and when the human skeleton key point information is used for enhanced identification to confirm the fall, the fall event is reported to the business processing module;

[0047] If no falling behavior is detected, wait for the image acquisition and video encoding module to send a new image frame.

[0048] In this embodiment, the picture acquisition and video encoding module continuously collects picture frames in real time and sends them to the fall detection and recognition module in real time, so the fall detection and recognition module always maintains a high state of alert during operation. It runs the fall detection and recognition algorithm uninterruptedly, and carefully analyzes each frame of the received picture to determine whether there is a fall behavior. Once the algorithm detects signs of a fall behavior, the fall detection and recognition module will immediately start the enhanced recognition process, extract the key point information of the human skeleton of the person who falls, and further confirm the existence of the fall behavior based on this information. After confirmation, the module will quickly report the fall behavior event to the business processing module so that timely measures can be taken. If the algorithm fails to detect the fall behavior in the current picture frame, the fall detection and recognition module will wait for the picture acquisition and video encoding module to send the next frame of the picture, and continue to perform its monitoring task to ensure real-time detection of the home environment.

[0049] In one embodiment of the present invention, the fall detection and identification module is further used for:

[0050] Using the human skeleton key point information to enhance recognition and analyze whether there is a fall behavior;

[0051] If a fall is confirmed, the fall event is reported to the business processing module;

[0052] If it is confirmed that there is no fall behavior, the fall detection and recognition algorithm is re-run to detect whether there is a fall behavior.

[0053] In this embodiment, the fall detection and recognition module uses the extracted human skeleton key point information for enhanced recognition, and can more accurately determine whether there is a fall behavior by deeply analyzing key information such as the human body's posture changes, the relative position between the skeleton key points and the movement trajectory. If the fall behavior is confirmed after enhanced recognition, the module will immediately report the fall behavior event to the business processing module, triggering the corresponding emergency response mechanism. If it is confirmed that there is no fall behavior, the fall detection and recognition module will re-run the fall detection and recognition algorithm, and perform a new round of detection and analysis on the next frame of the picture to ensure continuous and efficient monitoring of possible falls in the home environment.

[0054] In one embodiment of the present invention, the lightweight fall detection neural network of the fall detection and recognition algorithm is composed of a depth-separable convolution unit, which includes point-by-point convolution, depth-by-depth convolution, batch normalization BN, and activation function Relu. The number of parameters and the amount of calculation are about 1 / 9 of the 3x3 convolution kernel, which is suitable for deployment on a home surveillance camera platform with limited computing resources.

[0055] The lightweight fall detection neural network is divided into three parts:

[0056] Backbone network: adopts VGG straight-tube network structure, and uses the maximum pooling operation with minimal computational effort for downsampling; uses the standard 3x3 convolution kernel in the first stage of stage 1, and after 3x3 convolution with a step size of 2 and maximum pooling, the height and width of the output feature map become 1 / 4 of the input image;

[0057] Feature pyramid fusion layer: The upsampling operation adopts the nearest neighbor interpolation method; the Stage n layer is upsampled and simply added to the Stage n-1 layer to perform feature fusion, so that the low-level network has high-level semantic information;

[0058] Multi-task decoupling head layer: including target category, target box, and skeleton key points;

[0059] The loss function Loss includes category loss, object confidence loss, box loss, and key point loss.

[0060] Reference Figure 2The network of the fall detection and recognition algorithm designed by the present invention is basically composed of depthwise separable convolution PDunit units, which include point-by-point convolution, depth-by-depth convolution, batch normalization BN, and fast activation function Relu. Its parameter volume and calculation amount are about 1 / 9 of the 3x3 convolution kernel, which is very suitable for deployment on the platform of home surveillance cameras with limited computing resources. This network mainly includes the following three parts:

[0061] Backbone:

[0062] The backbone network is designed to be very concise, drawing on the VGG straight-tube network structure. Compared with multi-branch network structures such as residual networks, the straight-tube design is more conducive to hardware computing acceleration and reducing the number of memory accesses, which generally improves the speed of network reasoning. At the same time, downsampling uses a maximum pooling operation with minimal computational effort. In the front part of the stage 1 stage, a standard 3x3 convolution kernel is used to retain the shallow information of the image as much as possible, which is conducive to subsequent high-order feature extraction. After a 3x3 convolution with a step size of 2 and maximum pooling, the height and width of the output feature map becomes 1 / 4 of the input image, reducing the computational effort of subsequent operations.

[0063] FPN:

[0064] The feature pyramid fusion layer is also very streamlined. The upsampling operation uses the nearest neighbor interpolation method, which requires almost no computation. The Stage n layer is upsampled and the Stage n-1 layer is simply added to perform feature fusion, so that the low-level network has high-level semantic information.

[0065] Head:

[0066] The decoupled head layer is designed to be multi-task, which is conducive to the stability and rapid convergence of the training process. It includes: target category, target box, and skeleton key points.

[0067] The loss function Loss includes category loss, object confidence loss, box loss, and key point loss, as shown below:

[0068]

[0069] Among them, N is the number of positive samples, and the specific loss functions used are: cross entropy, L1 loss, IOU intersection-over-union ratio, and smooth-L1 smoothing loss function.

[0070] In one embodiment of the present invention, the fall detection and identification module is further used for:

[0071] Extracting the human skeleton key point information of the fallen person and converting the human skeleton key point information into a skeleton feature vector;

[0072] Inputting the bone feature vector into a support vector machine to analyze whether there is a falling behavior;

[0073] If a fall is confirmed, the fall event is reported to the business processing module.

[0074] In order to further improve the accuracy of fall recognition, the key points of the human skeleton are used for enhanced recognition while using the detection method. The steps are as follows:

[0075] Step 1: Run the fall detection and recognition algorithm. If a fall is detected, go to step 2.

[0076] Step 2: Extract the key points of the human skeleton of the person who fell, convert them into skeleton feature vectors, and input them into the support vector machine (SVM) to further confirm the fall behavior. If the fall is confirmed, go to step 3, otherwise go to step 1;

[0077] Step 3: Report the fall incident.

[0078] In one embodiment of the present invention, the business processing module is also used to support remote online viewing and management of home surveillance cameras.

[0079] In this embodiment, the service processing module not only notifies relevant family members after a fall occurs, but also provides the function of remote online viewing and management of home surveillance cameras. Users can access the service processing module anytime and anywhere through smartphones, tablets or other remote devices to view the images of home surveillance cameras in real time to ensure the safety of their homes. In addition, the service processing module also supports users to remotely manage surveillance cameras, including adjusting the camera's monitoring angle, controlling the camera's on / off status, setting recording and alarm parameters, etc., so as to achieve comprehensive control of the home monitoring system.

[0080] In an embodiment of the present invention, the business processing module notifies pre-configured family-related personnel in a manner including: sending an Internet text message and pushing through a WeChat applet.

[0081] In this embodiment, after detecting a fall event, the service processing module uses flexible and diverse methods to notify pre-configured family members. In order to ensure the timely transmission of information, the service processing module can send network text messages to the mobile devices of family members, so that they can quickly receive emergency notifications about the fall event no matter where they are. In addition, the service processing module also integrates the WeChat applet push function, which pushes detailed fall event information and monitoring images through the user's WeChat applet account, so that family members can more intuitively understand the emergency situation at home and take timely measures to rescue.

[0082] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to load a lightweight fall recognition home care system as described in any one of the embodiments of the present invention.

[0083] Specifically, the electronic device includes: a memory and a processor, the memory and the processor are connected via a bus communication, a computer program is stored in the memory, and the computer program can be run on the processor, thereby loading the lightweight fall recognition home care system described in any one of the first aspects of the embodiments of the present invention.

[0084] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0085] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0086] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, it loads a lightweight fall recognition home care system as described in any one of the first aspects of the embodiments of the present invention.

[0087] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0088] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, devices, electronic devices, storage media or computer program products. Therefore, embodiments of the present invention may take the form of complete hardware embodiments, complete software embodiments, or embodiments combining software and hardware. Moreover, embodiments of the present invention may take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CDROM, optical storage, etc.) containing computer-usable program codes.

[0089] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0090] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0091] The above is a detailed introduction to a lightweight fall recognition home care system, electronic device and medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application. The above embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art on the basis of the present invention are all within the protection scope of the present invention.

Claims

1. A lightweight fall recognition home care system, characterized in that: Applied to home surveillance cameras, including: The image acquisition and video encoding module is used to acquire image frames from the lens sensor and send them to the fall detection and recognition module, and at the same time perform h265 video encoding on the image frames and save them in the database; A fall detection and recognition module is used to extract the human skeleton key point information of the fallen person when a fall behavior is detected during the operation of the fall detection and recognition algorithm, and report the fall behavior event to the business processing module when the human skeleton key point information is used for enhanced recognition to confirm the existence of the fall behavior; The business processing module is used to notify pre-configured family-related personnel when receiving a reported fall behavior event.

2. The lightweight fall recognition home care system according to claim 1, characterized in that: The fall detection and identification module is also used for: Run the fall detection and recognition algorithm to detect whether there is a fall behavior; If a fall is detected, the human skeleton key point information of the person who fell is extracted and when the human skeleton key point information is used for enhanced identification to confirm the fall, the fall event is reported to the business processing module; If no falling behavior is detected, wait for the image acquisition and video encoding module to send a new image frame.

3. The lightweight fall recognition home care system according to claim 1, characterized in that: The fall detection and identification module is also used for: Using the human skeleton key point information to enhance recognition and analyze whether there is a fall behavior; If a fall is confirmed, the fall event is reported to the business processing module; If it is confirmed that there is no fall behavior, the fall detection and recognition algorithm is re-run to detect whether there is a fall behavior.

4. The lightweight fall recognition home care system according to claim 1, characterized in that: The lightweight fall detection neural network of the fall detection and recognition algorithm is composed of a depth-separable convolution unit, which includes point-by-point convolution, depth-by-depth convolution, batch normalization BN, and activation function Relu. The number of parameters and the amount of calculation are about 1 / 9 of the 3x3 convolution kernel, which is suitable for deployment on a home surveillance camera platform with limited computing resources. The lightweight fall detection neural network is divided into three parts: Backbone network: adopts VGG straight-tube network structure, and uses the maximum pooling operation with minimal computational effort for downsampling; uses the standard 3x3 convolution kernel in the first stage of stage 1, and after 3x3 convolution with a step size of 2 and maximum pooling, the height and width of the output feature map become 1 / 4 of the input image; Feature pyramid fusion layer: The upsampling operation adopts the nearest neighbor interpolation method; the Stage n layer is upsampled and simply added to the Stage n-1 layer to perform feature fusion, so that the low-level network has high-level semantic information; Multi-task decoupling head layer: including target category, target box, and skeleton key points; The loss function Loss includes category loss, object confidence loss, box loss, and key point loss.

5. The lightweight fall recognition home care system according to claim 1, characterized in that: The fall detection and identification module is also used for: Extracting the human skeleton key point information of the fallen person and converting the human skeleton key point information into a skeleton feature vector; Inputting the bone feature vector into a support vector machine to analyze whether there is a falling behavior; If a fall is confirmed, the fall event is reported to the business processing module.

6. The lightweight fall recognition home care system according to claim 1, characterized in that: The business processing module is also used to support remote online viewing and management of home surveillance cameras.

7. The lightweight fall recognition home care system according to claim 1, characterized in that: The business processing module notifies pre-configured family-related personnel in the following ways: sending an Internet text message and pushing through a WeChat applet.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the electronic device executes the computer program, it loads the lightweight fall recognition home care system as described in any one of claims 1 to 7.

9. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the lightweight fall recognition home care system as described in any one of claims 1 to 7 is loaded.

Images