Personnel fall monitoring system and method

Abstract

The present invention provides a personnel fall monitoring system and method, belonging to the field of safety monitoring technology. The present invention provides a new personnel fall monitoring system, which uses video and audio to detect suspected fall areas, and then drives the ball camera monitoring the scene to automatically adjust to the best observation angle for secondary confirmation and subsequent processing, greatly improving the response speed and accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of safety monitoring, and in particular to a personnel fall monitoring system and method. Background Art

[0002] As they age, the physical functions of the elderly gradually decline, such as weakened muscle strength, reduced balance ability, vision and hearing loss, etc. These factors greatly increase their risk of falling. Falls are one of the most common accidental injuries in nursing homes, which may lead to serious consequences such as fractures and head trauma. In addition, the demand for safety in public places is also gradually increasing, and it is particularly important to monitor and respond to falls in a timely and effective manner.

[0003] In current nursing homes, routine fall monitoring methods include:

[0004] 1. Manual monitoring: including regular inspections by personnel, real-time viewing of surveillance videos, etc. This method has the problems of delayed response and low efficiency.

[0005] 2. Wearing various sensors: including accelerometers, gyroscopes and other sensors. Although this method can capture body dynamics in real time and responds relatively quickly, it also has the following defects: 1) Motion misidentification and sensor sensitivity issues may lead to false alarms and missed alarms; 2) Continuous monitoring will consume device power and affect the user experience; 3) Wearable devices with integrated multiple sensors are relatively expensive; 4) Active cooperation is required, and it is usually difficult to wear them in real time 24 hours a day.

[0006] 3. Millimeter wave technology: Using 3D millimeter waves, millimeter waves can penetrate clothing and provide depth information, which helps to accurately identify fall events. However, this method also has the following defects: 1) The on-site environment is complex, and the depth information of millimeter waves cannot distinguish between people or other objects, which is easy to cause false alarms; 2) The unit cost is expensive and difficult to maintain; 3) Due to the need for privacy protection, some areas (such as toilets and showers) cannot be directly covered due to partitions; 4) A large amount of hardware needs to be specially installed, involving a large number of on-site modifications. For welfare institutions such as nursing homes, there are problems such as high costs and inability to popularize.

[0007] 4. Based on video analysis: This can be achieved by using the surveillance cameras installed on site. Through a large amount of data training, it can adapt to different fall scenarios and individual differences, and integrate multiple features for comprehensive judgment to improve the accuracy of judgment. However, this method also has the following defects: 1) Due to the need for privacy protection, some areas (such as toilets and showers) cannot be directly covered due to partitions, etc., and there is a risk of underreporting in such cases; 2) As a public welfare institution, the video coverage of the nursing home is usually based on what can be seen (video analysis requires "clear vision"), and then some ball cameras are supplemented for manual control to achieve clear vision, that is, the scene is covered by a large number of gun cameras + some ball cameras. In large scenes, due to the small image range of the target and the fact that some areas are almost invisible due to the changing light source, this method usually has a certain risk of underreporting.

[0008] How to optimize and improve the above-mentioned defects of the existing fall monitoring system to provide a fall monitoring system with higher response speed and response accuracy is a technical problem that urgently needs to be solved. Summary of the invention

[0009] In order to at least solve the technical problems existing in the above-mentioned background technology, the present invention provides a method, system, electronic device, storage medium and computer program product for monitoring a person falling.

[0010] The present invention provides a personnel fall monitoring system, the system comprises a video acquisition module, an audio acquisition module, a data fusion module, a ball camera control module, an alarm verification module, and an alarm handling module;

[0011] The video acquisition module is used to analyze the behavioral characteristics of the monitored object within the monitoring range by using a deep learning model to obtain the first behavioral characteristics of the suspected person falling;

[0012] The audio acquisition module is used to use audio analysis to identify the audio features of suspected heavy objects falling or people calling for help, that is, to obtain the second behavior features of suspected people falling;

[0013] The data fusion module is used to perform weighted fusion of the first behavior feature and the second behavior feature in combination with scene characteristics to obtain a third behavior feature, and determine the occurrence area of ​​the first behavior feature or the second behavior feature as the occurrence area of ​​the third behavior feature;

[0014] The ball camera control module is used to dispatch at least one ball camera to shoot the area where the third behavior feature occurs, so as to obtain a verification image of the suspected person falling;

[0015] The alarm verification module is used to confirm whether a person has fallen according to the verification image and obtain a person fall verification conclusion;

[0016] The alarm handling module is used to immediately trigger the alarm mechanism when the alarm verification module's conclusion on the person falling is that a person has confirmed a fall.

[0017] Optionally, the video acquisition module is specifically used to:

[0018] S11. Data collection and preprocessing;

[0019] S12. Video feature extraction;

[0020] S13. Deep learning model design;

[0021] S14. Deep learning model training;

[0022] S15. Deep learning model evaluation and generation;

[0023] S16. Real-time monitoring and response: Use the trained deep learning model to process the real-time video stream, add a non-maximum suppression mechanism to reduce repeated detection, and use the sliding window strategy to detect the first behavioral features of suspected people falling in consecutive frames.

[0024] Optionally, the audio acquisition module is specifically used to:

[0025] S21. Audio acquisition and preprocessing: After acquiring the live audio data from the camera, preprocess the live audio data;

[0026] S22. Audio feature extraction: extracting time domain features, frequency domain features, and time-frequency domain features of the pre-processed live audio data;

[0027] S23. Audio classification and recognition algorithm design: Build and train a neural network classifier, use the neural network classifier to identify the characteristic patterns of people falling or people calling for help, and use convolutional neural networks to perform deeper abstraction and learning of the time domain or frequency domain features of the collected on-site audio data;

[0028] S24. Real-time monitoring and response: Real-time analysis is implemented. Once the audio classification and recognition algorithm identifies the audio features of a suspected person falling or a person calling for help, further confirmation mechanisms are immediately initiated.

[0029] Optionally, the data fusion module is specifically used to:

[0030] Determine whether the scene corresponding to the occurrence area of ​​the first behavior feature or the second behavior feature is an environment that is easily blocked, or determine whether the scene is a large scene environment, or determine whether the relative size of the monitored object in the scene is greater than a specified value;

[0031] If so, setting the first weight of the first behavior feature to be less than the second weight of the second behavior feature; otherwise, setting the first weight of the first behavior feature to be equal to the second weight of the second behavior feature;

[0032] The first behavior feature and the second behavior feature are weightedly fused based on the first weight and the second weight to obtain the third behavior feature, and the occurrence area of ​​the first behavior feature or the second behavior feature is determined as the occurrence area of ​​the third behavior feature.

[0033] Optionally, the ball camera control module is specifically used for:

[0034] S31. Data calibration;

[0035] S32. Matching algorithm design;

[0036] S33. Global scheduling of ball machines;

[0037] S34. Dome camera control.

[0038] Optionally, the alarm verification module is specifically used to:

[0039] dispatching at least one ball camera to shoot the area where the third behavior feature occurs, so as to obtain a verification image of the suspected person falling;

[0040] The deep learning model is used to analyze the behavioral characteristics of the monitored object in the verification image to verify whether the monitored object has fallen.

[0041] Optionally, the alarm handling module is specifically used to:

[0042] When the personnel fall verification conclusion of the alarm verification module is confirmed to be a person fall, or an emergency signal is received, the alarm mechanism is immediately triggered;

[0043] The emergency signal is a signal generated when the scene is a private environment.

[0044] The present invention also provides a method for monitoring a person falling, which is based on any of the above-mentioned systems and comprises the following steps:

[0045] The video acquisition module uses a deep learning model to analyze the behavioral characteristics of the monitored object within the monitoring range to obtain the first behavioral characteristics of the suspected person falling;

[0046] The audio collection module uses audio analysis to identify the audio features of suspected heavy objects falling or people calling for help, that is, to obtain the second behavior features of suspected people falling;

[0047] The data fusion module performs weighted fusion on the first behavior feature and the second behavior feature in combination with scene characteristics to obtain a third behavior feature, and determines an occurrence area of ​​the first behavior feature or the second behavior feature as an occurrence area of ​​the third behavior feature;

[0048] The ball camera control module dispatches at least one ball camera to shoot the area where the third behavior feature occurs, so as to obtain a verification image of the suspected person falling;

[0049] The alarm verification module confirms whether a person has fallen according to the verification image, and obtains a person fall verification conclusion;

[0050] The alarm handling module immediately triggers the alarm mechanism when the alarm verification module's personnel fall verification conclusion is that a person has fallen.

[0051] The present invention also provides an electronic device, comprising: a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the method as described in any of the preceding items.

[0052] The present invention also provides a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, any of the above methods is executed.

[0053] The present invention also provides a computer program product, comprising a computer program stored on a non-transitory computer-readable medium, characterized in that: when the computer program is executed by a processor, the method described in any one of the above items is implemented.

[0054] The present invention provides a new personnel fall monitoring system. The system uses video + audio to detect the suspected fall area, and then drives the dome camera at the monitoring site to automatically adjust to the optimal observation angle for secondary confirmation and subsequent processing, which greatly improves the response speed and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 It is a structural schematic diagram of a personnel fall monitoring system disclosed in an embodiment of the present invention.

[0057] Figure 2It is a flow chart of a method for monitoring a person falling disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0060] In the description of the present invention, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear to indicate an orientation or position relationship, it is based on the orientation or position relationship shown in the accompanying drawings, or is the orientation or position relationship in which the product of the invention is usually placed when used. It is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the system or component referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0061] The terms "first", "second", "third", and "fourth" in the specification and claims of the present invention are used to distinguish different objects rather than to describe a specific order of the objects. For example, the first input, the second input, the third input, and the fourth input are used to distinguish different inputs rather than to describe a specific order of the inputs.

[0062] In the embodiments of the present invention, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0063] In the description of the embodiments of the present invention, unless otherwise specified, “multiple” means two or more than two. For example, multiple processing units refer to two or more processing units; multiple elements refer to two or more elements, etc.

[0064] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0065] like Figure 1As shown, a personnel fall monitoring system according to an embodiment of the present invention includes a video acquisition module, an audio acquisition module, a data fusion module, a ball camera control module, an alarm verification module, and an alarm handling module;

[0066] The video acquisition module is used to analyze the behavioral characteristics of the monitored object within the monitoring range by using a deep learning model to obtain the first behavioral characteristics of the suspected person falling;

[0067] The audio acquisition module is used to use audio analysis to identify the audio features of suspected heavy objects falling or people calling for help, that is, to obtain the second behavior features of suspected people falling;

[0068] The data fusion module is used to perform weighted fusion of the first behavior feature and the second behavior feature in combination with scene characteristics to obtain a third behavior feature, and determine the occurrence area of ​​the first behavior feature or the second behavior feature as the occurrence area of ​​the third behavior feature;

[0069] The ball camera control module is used to dispatch at least one ball camera to shoot the area where the third behavior feature occurs, so as to obtain a verification image of the suspected person falling;

[0070] The alarm verification module is used to confirm whether a person has fallen according to the verification image and obtain a person fall verification conclusion;

[0071] The alarm handling module is used to immediately trigger the alarm mechanism when the alarm verification module's conclusion on the person falling is that a person has confirmed a fall.

[0072] In response to the technical problems in the above-mentioned background technology, the present invention provides a new personnel fall monitoring system. The system uses video + audio to detect suspected fall areas, and then drives the dome camera at the monitoring site to automatically adjust to the optimal observation angle for secondary confirmation and subsequent processing, which greatly improves the response speed and accuracy.

[0073] Using deep learning models to detect falls usually includes key steps such as data collection and preprocessing, feature extraction, model design, model training, model evaluation and generation, and real-time monitoring and response. Through this step, the first behavioral characteristics and corresponding areas of suspected falls can be analyzed.

[0074] Optionally, the video acquisition module is specifically used to:

[0075] S11. Data collection and preprocessing;

[0076] Collect various video data containing falling behaviors, including confusing scenes such as normal activities, falling, lying down, and sitting down quickly, to ensure data diversity, and annotate the video data at the frame or fragment level to clearly indicate which behaviors belong to falling behaviors; preprocessing includes image standardization, cropping, scaling, etc. to ensure consistent data format for easy model training.

[0077] S12. Video feature extraction;

[0078] Video feature extraction mainly includes several steps such as human body detection and spatiotemporal features. Human body detection can use the YOLOv8-Pose key point detection algorithm to extract human skeleton information and obtain joint coordinates; spatiotemporal features can be extracted from video sequences using a spatiotemporal convolutional network (Conv3D), that is, to capture the dynamic behavior information of the monitored object.

[0079] S13. Deep learning model design;

[0080] The deep learning model design mainly includes the skeleton model (selecting a deep learning model suitable for action recognition (YOLOv8)), model optimization (for the fall detection task, adding a fall feature detection layer, designing a specific loss function (cross entropy loss) to optimize the accuracy of fall detection), and building an attention mechanism (introducing an attention mechanism to allow the model to focus on the key parts or dynamic features of the human body when falling, thereby improving recognition accuracy).

[0081] S14. Deep learning model training;

[0082] The training of deep learning models mainly includes building training strategies (using batch normalization, learning rate decay and other techniques to improve model generalization capabilities), multi-model fusion (training multiple models and fusing their prediction results to improve the final detection performance), etc.

[0083] S15. Deep learning model evaluation and generation;

[0084] During the training process, cross-validation and independent test sets are used to evaluate model performance, focusing on indicators such as model accuracy, recall rate, ROC curve, etc., with special attention paid to the false alarm rate and missed alarm rate of the model in actual application scenarios, so as to generate the most optimized model that adapts to the actual on-site environment.

[0085] S16. Real-time monitoring and response: Use the trained deep learning model to process the real-time video stream, add the non-maximum suppression (NMS) mechanism to reduce repeated detection, and use the sliding window strategy to detect the first behavioral features of suspected people falling in consecutive frames.

[0086] The first behavior feature includes the corresponding behavior feature and the fall prediction probability.

[0087] The method of using audio analysis to identify suspected heavy objects falling or people calling for help usually involves key steps such as audio acquisition and preprocessing, feature extraction, audio classification and recognition algorithm design, real-time monitoring and response. Through this step, the second behavioral characteristics and corresponding areas of suspected people falling can be analyzed.

[0088] Optionally, the audio acquisition module is specifically used to:

[0089] S21. Audio acquisition and preprocessing: After acquiring the live audio data from the camera, preprocess the live audio data;

[0090] Preprocessing operations include noise reduction (such as background noise removal, environmental noise filtering), gain control and other conventional methods to improve signal quality. At the same time, low-pass filtering can be used to remove high-frequency noise, and high-pass filtering can be used to highlight human voices or impact sounds.

[0091] S22. Audio feature extraction: extracting the time domain features, frequency domain features, and time-frequency domain features of the pre-processed live audio data.

[0092] Time domain features include sound energy, peak frequency, and other features used to identify the intensity and dynamic changes of sound; and, by converting the time domain signal to the frequency domain through Fourier transform, the spectrum energy distribution, i.e., frequency domain features, can be extracted; and, short-time Fourier transform (STFT) and Mel-frequency cepstral coefficients (MFCCs) can be used to capture the time-frequency domain features of sound.

[0093] S23. Audio classification and recognition algorithm design: Build and train a neural network classifier, use the neural network classifier to identify the characteristic patterns of people falling or people calling for help, and use convolutional neural networks to perform deeper abstraction and learning of the time domain or frequency domain features of the collected on-site audio data;

[0094] For sudden and atypical events such as people falling, a statistical method is used to identify sounds that are significantly different from normal ambient sounds. To this end, the present invention first constructs and trains a neural network classifier to identify the characteristic patterns of people falling or people calling for help, and uses a convolutional neural network to perform a deeper level of abstraction and learning on the time domain or frequency domain features of the collected audio.

[0095] S24. Real-time monitoring and response: Real-time analysis is implemented. Once the audio classification and recognition algorithm identifies the audio features of a suspected person falling or a person calling for help, further confirmation mechanisms are immediately initiated.

[0096] Optionally, the data fusion module is specifically used to:

[0097] Determine whether the scene corresponding to the occurrence area of ​​the first behavior feature or the second behavior feature is an environment that is easily blocked, or determine whether the scene is a large scene environment, or determine whether the relative size of the monitored object in the scene is greater than a specified value;

[0098] If so, setting the first weight of the first behavior feature to be less than the second weight of the second behavior feature; otherwise, setting the first weight of the first behavior feature to be equal to the second weight of the second behavior feature;

[0099] The first behavior feature and the second behavior feature are weightedly fused based on the first weight and the second weight to obtain the third behavior feature, and the occurrence area of ​​the first behavior feature or the second behavior feature is determined as the occurrence area of ​​the third behavior feature.

[0100] In this embodiment, it is first determined whether the scene characteristics of the suspected fall area belong to an environment that is easily obstructed, such as a private environment such as a toilet or bathroom, a night environment, an environment where the target area of ​​the fall is too dark, etc. If it is an environment that is easily obstructed, it is more inclined to make a fall judgment based on sound characteristics, for example, the first weight of the first behavior feature is set to 0.3, and the second weight of the second behavior feature is set to 0.7; if the scene belongs to a large scene environment, or the relative size of the monitored object in the scene is greater than the specified value, that is, the target is too large, it is more inclined to video characteristics, for example, the first weight of the first behavior feature is set to 0.7, and the second weight of the second behavior feature is set to 0.3. For other conventional environments, the same confidence is given to both.

[0101] Among them, the first behavior feature and the second behavior feature are used to describe the probability of a person falling. The first behavior feature is predicted by performing motion detection and human posture recognition on the video data of the monitored object, and the second behavior feature is predicted by analyzing the audio features of suspected heavy objects falling or people calling for help in the audio data of the monitored object.

[0102] Optionally, the ball camera control module is specifically used for:

[0103] S31. Data calibration:

[0104] First, define the coordinate system. One of the coordinate systems is the "video coordinate system", which is the coordinate system used by the gun camera or ball camera to capture the picture; the other coordinate system is the "physical coordinate system", which refers to the physical space coordinate system based on which the ball camera pan / tilt actually rotates. Set some reference objects with known positions (such as calibration points) in the monitoring area, and then record the pixel positions of these reference objects in the video coordinate system and their actual positions in the physical coordinate system, and record multiple sets of such corresponding relationships to complete data calibration.

[0105] S32. Matching algorithm design:

[0106] Using multiple groups of corresponding relationships in the above step S31, a mapping model can be established, so that the system can infer the coordinates in the physical space based on the position information in the video image. Commonly used mapping models include perspective transformation, polar coordinate transformation, etc. A mapping function from video coordinates to physical coordinates is established through polynomial regression, direct linear transformation (DLT) or more complex machine learning methods. This mapping function can convert the pixel coordinates of the suspected falling area identified in the video into coordinates in the ball camera coordinate system. In the present invention, considering the need for real-time monitoring and the fact that the accuracy requirements for matching in actual use are not very high (as long as the person is within the image range after the ball camera is rotated), the perspective transformation model selected for the mapping model.

[0107] S33. Global scheduling of ball machine:

[0108] Considering that the number of gun cameras is usually much larger than the number of ball cameras in the actual scene, that is, there may be M alarms, and N ball cameras need to be dispatched for confirmation. In actual use, when M=1, the surrounding N ball cameras that can see the area are directly called for confirmation to ensure that the alarm is verified from multiple angles; when M>1, the ball camera with the best angle is selected (the face direction determined by the human skeleton information in the video feature extraction is the first priority, and the ball camera with the angle facing the direction of the personnel is preferred) to ensure that most of the verification angles after the ball camera is rotated are optimal. At the same time, if there are multiple alarms in the same area at the same time, and no other ball cameras can be called (for example, only one ball camera can cover a certain area, but multiple alarms occur at the same time, or the number of alarms at the same time is greater than the number of ball cameras), the system needs to quickly capture and retain the real scene after the corresponding ball camera rotates to the first alarm, and then immediately rotate to another alarm area for confirmation, so as to ensure that each alarm can be confirmed in the first time.

[0109] S34. Dome camera control:

[0110] According to the ball camera selected in step S33 and the rotation coordinate position of the ball camera calculated in step S32, the ball camera control protocol (such as onvif or a camera manufacturer's private control protocol) is used to control the ball camera to rotate to the corresponding position.

[0111] Optionally, the alarm verification module is specifically used to:

[0112] dispatching at least one ball camera to shoot the area where the third behavior feature occurs, so as to obtain a verification image of the suspected person falling;

[0113] The deep learning model is used to analyze the behavioral characteristics of the monitored object in the verification image to verify whether the monitored object has fallen.

[0114] In this embodiment, since the ball camera has enlarged the suspected alarm area in the center, compared with the large scene, on the one hand, the target is larger and easier to distinguish; on the other hand, the target will be clearer: the visible scene becomes smaller, and the change of the light on the scene has a smaller impact, avoiding the problem of a certain place being too dark to be seen due to obvious strong light sources, backlighting, overexposure, etc. in the large scene image (the wide dynamic principle of the camera, for a small scene, the range of light intensity changes is smaller, and it is easier to see the target in the dark). Therefore, the present invention dispatches at least one ball camera to shoot the occurrence area of ​​the third behavior feature (i.e., the occurrence area of ​​the first behavior feature and the second behavior feature) to obtain a verification image of a suspected person falling, and continues to use the aforementioned deep learning model to confirm whether there is a person falling in the verification image.

[0115] Optionally, the alarm handling module is specifically used to:

[0116] When the personnel fall verification conclusion of the alarm verification module is confirmed to be a person fall, or an emergency signal is received, the alarm mechanism is immediately triggered;

[0117] The emergency signal is a signal generated when the scene is a private environment.

[0118] In this embodiment, after confirming the fall event, the alarm mechanism is immediately triggered to notify relevant personnel to deal with it on site. Furthermore, the voice interaction function can also be combined to conduct preliminary communication with the person who fell and assess his condition. At the same time, the system stores the corresponding warning data (pictures, videos, etc.).

[0119] In addition, when it is confirmed that the area where the fall occurred is a private environment, such as a toilet or shower room, since the camera cannot capture these private environments, once the suspected fall is confirmed based on the second behavioral feature, an emergency signal can be generated to skip the alarm verification module and directly enter the alarm handling module.

[0120] Optionally, when the video acquisition module detects that the monitored object is about to enter the blocked area, a deep learning model is used to detect the shadow features of the monitored object in the closest area to the blocked area;

[0121] It is determined based on the shadow feature whether the monitored object has fallen down, and if so, an emergency signal is generated and sent to the alarm handling module.

[0122] In this embodiment, if the monitored object falls in an obstructed area (such as a corridor or other blind spot in monitoring), and there is no audio feature of a heavy object falling or a person calling for help (for example, after falling, the person is too seriously injured to call for help), the alarm cannot be realized in the above manner. In this regard, the present invention is arranged to continuously track the monitored object when it is detected that the monitored object is about to enter the obstructed area. After the monitored object enters the monitoring area, the closest area outside the obstructed area can still be detected by the video acquisition module, and the shadow feature of the monitored object is extracted in the closest area. In addition, the shadow feature is input into a pre-constructed prediction large model to analyze whether the shadow feature is consistent with the person's falling action. If so, an emergency signal is directly generated to trigger the alarm handling module to perform alarm handling. The prediction large model can be a BERT model, a GPT model, etc.

[0123] It should be noted that if there is light in the obstruction area, the shadow of the monitored object will be projected in the closest area. If the monitored object walks normally, the shadow is stable and gradually moves toward the obstruction area until it disappears. If the monitored object falls, the shadow changes very quickly and disappears quickly (the height of the monitored object after falling is insufficient, resulting in the shadow not being projected in the closest area). Based on the above characteristics, the fall of a person can be identified.

[0124] like Figure 2 As shown, the present invention also discloses a method for monitoring a person falling, which is based on any of the above-mentioned systems and comprises the following steps:

[0125] The video acquisition module uses a deep learning model to analyze the behavioral characteristics of the monitored object within the monitoring range to obtain the first behavioral characteristics of the suspected person falling;

[0126] The audio collection module uses audio analysis to identify the audio features of suspected heavy objects falling or people calling for help, that is, to obtain the second behavior features of suspected people falling;

[0127] The data fusion module performs weighted fusion on the first behavior feature and the second behavior feature in combination with scene characteristics to obtain a third behavior feature, and determines an occurrence area of ​​the first behavior feature or the second behavior feature as an occurrence area of ​​the third behavior feature;

[0128] The ball camera control module dispatches at least one ball camera to shoot the area where the third behavior feature occurs, so as to obtain a verification image of the suspected person falling;

[0129] The alarm verification module confirms whether a person has fallen according to the verification image, and obtains a person fall verification conclusion;

[0130] The alarm handling module immediately triggers the alarm mechanism when the alarm verification module's personnel fall verification conclusion is that a person has fallen.

[0131] The present invention also discloses an electronic device, comprising: a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the method as described in any of the preceding items.

[0132] The present invention also discloses a computer storage medium, on which a computer program is stored. When the computer program is run by a processor, any of the above methods is executed.

[0133] The present invention also discloses a computer program product, comprising a computer program stored on a non-transitory computer-readable medium, wherein the computer program implements any of the above methods when executed by a processor.

[0134] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0136] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0137] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0138] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0139] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A personnel fall monitoring system, characterized in that: The system includes a video acquisition module, an audio acquisition module, a data fusion module, a ball camera control module, an alarm verification module, and an alarm handling module; The video acquisition module is used to analyze the behavioral characteristics of the monitored object within the monitoring range by using a deep learning model to obtain the first behavioral characteristics of the suspected person falling; The audio acquisition module is used to use audio analysis to identify the audio features of suspected heavy objects falling or people calling for help, that is, to obtain the second behavior features of suspected people falling; The data fusion module is used to perform weighted fusion of the first behavior feature and the second behavior feature in combination with scene characteristics to obtain a third behavior feature, and determine the occurrence area of ​​the first behavior feature or the second behavior feature as the occurrence area of ​​the third behavior feature; The ball camera control module is used to dispatch at least one ball camera to shoot the area where the third behavior feature occurs, so as to obtain a verification image of the suspected person falling; The alarm verification module is used to confirm whether a person has fallen according to the verification image and obtain a person fall verification conclusion; The alarm handling module is used to immediately trigger the alarm mechanism when the personnel fall verification conclusion of the alarm verification module is confirmed to have a person fall; When the video acquisition module detects that the monitored object is about to enter the blocked area, the deep learning model is used to detect the shadow features of the monitored object in the closest area of ​​the blocked area; the shadow features are used to analyze whether the monitored object has fallen, and if so, an emergency signal is generated and sent to the alarm handling module; The data fusion module is specifically used for: Determine whether the scene corresponding to the occurrence area of ​​the first behavior feature or the second behavior feature is an environment that is easily blocked, or determine whether the scene is a large scene environment, or determine whether the relative size of the monitored object in the scene is greater than a specified value; If so, setting the first weight of the first behavior feature to be less than the second weight of the second behavior feature; otherwise, setting the first weight of the first behavior feature to be equal to the second weight of the second behavior feature; The first behavior feature and the second behavior feature are weightedly fused based on the first weight and the second weight to obtain the third behavior feature, and the occurrence area of ​​the first behavior feature or the second behavior feature is determined as the occurrence area of ​​the third behavior feature.

2. A personnel fall monitoring system according to claim 1, characterized in that: The video acquisition module is specifically used for: S11. Data collection and preprocessing; S12. Video feature extraction; S13. Deep learning model design; S14. Deep learning model training; S15. Deep learning model evaluation and generation; S16. Real-time monitoring and response: Use the trained deep learning model to process the real-time video stream, add a non-maximum suppression mechanism to reduce repeated detection, and use the sliding window strategy to detect the first behavioral features of suspected people falling in consecutive frames.

3. A personnel fall monitoring system according to claim 2, characterized in that: The audio acquisition module is specifically used for: S21. Audio acquisition and preprocessing: After acquiring the live audio data from the camera, preprocess the live audio data; S22. Audio feature extraction: extracting time domain features, frequency domain features, and time-frequency domain features of the pre-processed live audio data; S23. Audio classification and recognition algorithm design: Build and train a neural network classifier, use the neural network classifier to identify the characteristic patterns of people falling or people calling for help, and use convolutional neural networks to perform deeper abstraction and learning of the time domain or frequency domain features of the collected on-site audio data; S24. Real-time monitoring and response: Real-time analysis is implemented. Once the audio classification and recognition algorithm identifies the audio features of a suspected person falling or a person calling for help, further confirmation mechanisms are immediately initiated.

4. A personnel fall monitoring system according to claim 1, characterized in that: The ball machine control module is specifically used for: S31. Data calibration; S32. Matching algorithm design; S33. Global scheduling of ball machines; S34. Dome camera control.

5. A personnel fall monitoring system according to claim 4, characterized in that: The alarm verification module is specifically used for: dispatching at least one ball camera to shoot the area where the third behavior feature occurs, so as to obtain a verification image of the suspected person falling; The deep learning model is used to analyze the behavioral characteristics of the monitored object in the verification image to verify whether the monitored object has fallen.

6. A personnel fall monitoring system according to claim 5, characterized in that: The alarm handling module is specifically used for: When the personnel fall verification conclusion of the alarm verification module is confirmed to be a person fall, or an emergency signal is received, the alarm mechanism is immediately triggered; The emergency signal is a signal generated when the scene is a private environment.

7. A method for monitoring a person falling, the method being based on the system according to any one of claims 1 to 6, characterized in that: The method comprises the following steps: The video acquisition module uses a deep learning model to analyze the behavioral characteristics of the monitored object within the monitoring range to obtain the first behavioral characteristics of the suspected person falling; The audio collection module uses audio analysis to identify the audio features of suspected heavy objects falling or people calling for help, that is, to obtain the second behavior features of suspected people falling; The data fusion module performs weighted fusion on the first behavior feature and the second behavior feature in combination with scene characteristics to obtain a third behavior feature, and determines an occurrence area of ​​the first behavior feature or the second behavior feature as an occurrence area of ​​the third behavior feature; The ball camera control module dispatches at least one ball camera to shoot the area where the third behavior feature occurs, so as to obtain a verification image of the suspected person falling; The alarm verification module confirms whether a person has fallen according to the verification image, and obtains a person fall verification conclusion; The alarm handling module immediately triggers the alarm mechanism when the alarm verification module's personnel fall verification conclusion is that a person has fallen.

8. An electronic device, comprising: A memory storing executable program code; A processor coupled to the memory; characterized in that: the processor calls the executable program code stored in the memory to execute the method as claimed in claim 7.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to claim 7 is performed.

Images