Person fall detection method and device, electronic equipment and storage medium

Abstract

The present application provides a kind of personnel fall detection method, device, electronic equipment and storage medium, method includes: obtaining person falls classification model and behavior classification model;Wherein, person falls classification model is obtained by the standing image set of personnel, falls classification model is obtained by the multiple behavior image set of personnel training;Real-time personnel image in the set period is classified based on person falls classification model, to classify whether personnel falls;In response to personnel falls, the image sequence of real-time personnel image in the set period is obtained;Image sequence is classified based on behavior classification model, to classify the behavior sequence of personnel;According to behavior sequence, the probability of personnel fall is calculated, and whether personnel fall behavior exists is determined based on the probability of fall. Therefore, through the personnel falls and behavior sequence classified by person falls classification model and behavior classification model, it is comprehensively confirmed whether personnel fall behavior occurs, and the accuracy of fall detection is improved.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a method, apparatus, electronic device, and storage medium for detecting human falls. Background Technology

[0002] Fall detection is a crucial issue in the application research of surveillance systems. In recent years, with the rapid aging of the population, the elderly population has shown a trend towards greater aging and empty-nest lifestyles. Falls among the elderly are not only a concern for every family but also a societal problem. Furthermore, falls have resulted in injuries and deaths in daily life and work settings. Therefore, researching effective fall detection methods is essential to improving the safety of life for the elderly and other people with mobility impairments, and to ensuring the safety of people in production settings. Summary of the Invention

[0003] The present invention aims to at least partially solve one of the technical problems in the related art.

[0004] Therefore, the first objective of this invention is to provide a method for detecting falls.

[0005] The second objective of this invention is to provide a fall detection device.

[0006] The third objective of this invention is to provide an electronic device.

[0007] The fourth objective of this invention is to provide a non-transitory computer-readable storage medium storing computer instructions.

[0008] To achieve the above objectives, a first aspect of the present invention proposes a method for detecting human falls. The method includes: acquiring a human fall classification model and a behavior classification model; wherein the human fall classification model is trained using a set of standing images and a set of fallen images of a person, and the behavior classification model is trained using a set of images of various behaviors of a person; classifying real-time human images within a set period based on the human fall classification model to determine whether a person has fallen; in response to a person falling, acquiring an image sequence of real-time human images within the set period; classifying the image sequence based on the behavior classification model to determine a behavior sequence of the person; calculating the probability of a person falling based on the behavior sequence, and determining whether a person has fallen based on the fall probability.

[0009] To achieve the above objectives, a second aspect of the present invention provides a person fall detection device, the device comprising: a first acquisition module, configured to acquire a person fall classification model and a behavior classification model; wherein the person fall classification model is trained using a set of standing images and a set of fallen images of a person, and the behavior classification model is trained using a set of images of various behaviors of a person; a first classification module, configured to classify real-time person images within a set period based on the person fall classification model to classify whether a person has fallen; a second acquisition module, configured to acquire an image sequence of real-time person images within a set period in response to a person falling; a second classification module, configured to classify the image sequence based on the behavior classification model to classify a behavior sequence of a person; and a calculation module, configured to calculate the probability of a person falling based on the behavior sequence, and determine whether a person has fallen based on the fall probability.

[0010] To achieve the above objectives, a third aspect of the present invention provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0011] To achieve the above objectives, a fourth aspect of the present invention provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method described in the first aspect.

[0012] The present invention provides a method, apparatus, electronic device, and storage medium for detecting falls, which acquires a fall classification model and a behavior classification model. The fall classification model is trained using a set of standing images and a set of fallen images of individuals, while the behavior classification model is trained using a set of images showing various behaviors of individuals. Based on the fall classification model, real-time images of individuals within a set period are classified to determine whether an individual has fallen. In response to a fall, an image sequence of real-time images of individuals within the set period is acquired. Based on the behavior classification model, the image sequence is classified to determine a behavior sequence of the individual. Based on the behavior sequence, the probability of a fall is calculated, and based on the fall probability, it is determined whether a fall has occurred. Therefore, by comprehensively confirming whether a fall has occurred using the fall and behavior sequences classified by the fall classification model and the behavior classification model, the accuracy of fall detection is improved.

[0013] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0014] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart illustrating a method for detecting falls by persons provided in an embodiment of the present invention. Figure 2 This is a flowchart illustrating another method for detecting falls provided in an embodiment of the present invention. Figure 3 This is a functional structure diagram of a method for detecting falls provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a personnel fall detection device provided in an embodiment of the present invention. Detailed Implementation

[0015] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0016] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this invention all comply with the relevant provisions of national laws and regulations.

[0017] The following description, with reference to the accompanying drawings, outlines a method, apparatus, electronic device, and storage medium for detecting personnel falls according to embodiments of the present invention.

[0018] Figure 1 This is a flowchart illustrating a method for detecting falls by persons provided in an embodiment of the present invention.

[0019] like Figure 1 As shown, the method includes the following steps: Step 101: Obtain the human fall classification model and the behavior classification model; wherein, the human fall classification model is trained by the image set of people standing and the image set of people falling, and the behavior classification model is trained by the image set of people in various behaviors.

[0020] In some possible implementations, the human fall classification model is constructed using the following steps: acquiring a set of standing images and a set of fallen images of a person; extracting and normalizing the human keypoint set from the standing image set and the fallen image set to obtain the standing image keypoint set and the fallen image keypoint set; wherein the human keypoint set corresponds to multiple parts of the human body; dividing the standing image keypoint set and the fallen image keypoint set into a first training, validation, and test dataset; and training, validating, and testing the convolutional neural network based on the first training, validation, and test datasets to obtain the human fall classification model.

[0021] To improve the accuracy of the keypoint sets for standing and fallen images, a human posture detection model is used to extract multiple key points of the human body from each image in the standing and fallen image sets, corresponding to multiple parts of the human body, to obtain the initial keypoint sets for standing and fallen images. Based on the width and height of the images in the standing and fallen image sets, the initial keypoint sets for standing and fallen images are normalized to obtain normalized keypoint sets for standing and fallen images. The data normalization includes dividing the x-coordinate of all key points in the initial keypoint sets for standing and fallen images by the width of the image, and dividing the y-coordinate of all key points in the initial keypoint sets for standing and fallen images by the height of the image.

[0022] In addition, the human fall classification model uses a pre-trained convolutional neural network, freezes the convolutional layers of the convolutional neural network, and sets two hidden layers containing 32 neurons each; the behavior classification model uses a pre-trained hybrid architecture model, freezes the convolutional layers of the hybrid architecture model, and sets two hidden layers containing 36 neurons each.

[0023] Specifically, as an example, obtaining the key point set of standing images and the key point set of fallen images includes: acquiring N images of people standing via surveillance cameras and the Internet to form a standing image set. A set of images of people falling to the ground consists of M images. .in, For the i-th image of a person standing, Let i be the image of a person lying on the ground. Iterate through the set of standing images for each image. and images of people falling to the ground For the i-th image of a person standing... The image of the person lying on the ground. Using a human pose detection model (e.g., BlazePose), the set of key points for the i-th standing image is detected. Or the key point of the i-th fallen image Finally, a set of key points for all standing images is obtained. , With all the keypoint sets of the images of the fallen figure ,in:

[0024]

[0025]

[0026]

[0027]

[0028] in, It is the position of the j-th key point corresponding to the i-th standing image; It is the coordinate x of the j-th key point corresponding to the i-th standing image. The coordinate y is the j-th key point corresponding to the i-th standing image. It is the coordinate of the j-th key point corresponding to the i-th image of a person falling to the ground. It is the coordinate x of the j-th key point corresponding to the i-th image of a person falling to the ground. It is the coordinate y of the j-th key point corresponding to the i-th image of a person falling to the ground.

[0029] It should be noted that the key point set for standing images and the key point set for fallen images include 33 human body key points. The 1st to 33rd key points correspond to the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle, left toe, right toe, inner corner of left eye, inner corner of right eye, outer corner of left eye, outer corner of right eye, tip of left ear, tip of right ear, tip of left shoulder, tip of right shoulder, tip of left elbow, tip of right elbow, tip of left wrist, tip of right wrist, front of left knee, and front of right knee.

[0030] As an example, data normalization includes: obtaining the width of images in the standing image set and the fallen image set. Image height Iterate through all keypoint sets of the standing images. and the set of key points for all images of people falling to the ground For the key point set of the i-th standing image ; and the key point set of the i-th fallen image The coordinates of the j-th keypoint corresponding to the i-th standing image are calculated using the following formula. The coordinates of the j-th key point corresponding to the i-th image of a person falling to the ground. After normalization, the coordinates of the j-th normalized keypoint corresponding to the i-th standing image are obtained. The coordinates of the j-th normalized keypoint corresponding to the i-th image of a person falling to the ground. Finally, the normalized keypoint set of the i-th standing image is obtained. Normalized keypoint set of the i-th fallen image All normalized standing image keypoint sets Normalized keypoint set of the fallen image ,in:

[0031]

[0032]

[0033]

[0034]

[0035]

[0036]

[0037]

[0038] .

[0039] in, It is the coordinate of the j-th normalized keypoint corresponding to the i-th standing image. It is the normalized keypoint coordinate x corresponding to the j-th normalized keypoint in the i-th standing image. It is the normalized keypoint coordinate y of the j-th image corresponding to the i-th standing person image. It is the coordinate of the j-th normalized keypoint corresponding to the i-th image of a person falling to the ground. It is the normalized keypoint coordinate x corresponding to the j-th image of the i-th person falling to the ground. It is the coordinate y of the j-th normalized keypoint corresponding to the i-th image of a person falling to the ground.

[0040] As an example, a convolutional neural network is constructed as follows: For instance, using the EfficientNet-L2 model as the chosen convolutional neural network, cross-entropy as the loss function, and Adam as the optimizer, the EfficientNet-L2 model is pre-trained. The convolutional layers of the EfficientNet-L2 are frozen. The first hidden layer contains 32 neurons with ReLU activation. The second hidden layer contains 32 neurons with ReLU activation. The output layer contains 2 neurons with softmax activation.

[0041] As an example, the first training, validation, and test dataset includes: a set of normalized stand-up image keypoints. Normalized keypoint set of the fallen image The dataset for classifying people falling to the ground is obtained by randomly dividing the dataset into three parts: training set (70%), test set (10%), and validation set (20%). And a verification test was conducted by setting the category label "label1" for all images (pics), as shown below, to obtain the personnel fall classification label file. : .

[0042] As an example, training, validating, and testing convolutional neural networks includes: using a dataset for classifying people who have fallen to the ground. Category label file for people falling to the ground ,use The EfficientNet-L2 model is trained to obtain a human fall classification model. .

[0043] In some possible implementations, the behavior classification model is constructed using the following steps: acquiring multiple sets of behavioral images of people; dividing the multiple sets of behavioral images into a second training, validation, and test dataset; training, validating, and testing the hybrid architecture model based on the second training, validation, and test datasets to obtain the behavior classification model; wherein, the hybrid architecture model is a hybrid model that combines the self-attention mechanism and the characteristics of convolutional neural networks.

[0044] It should be noted that the multiple behavioral image sets include at least one of the following: images of people going downhill, running, being in mid-air, standing on one leg, standing upside down, collision, imbalance, falling, and throwing.

[0045] As an example, acquiring a set of images depicting various human behaviors includes: acquiring N1 images of people going downhill to form a downhill image set. N2 images of people running constitute a running image set. The N3 images of people in mid-air constitute a set of images of people in mid-air. A set of N4 images of a person standing on one leg constitutes a single-leg standing image set. The set of N5 images of people standing upside down constitutes an inverted image set. The collision image set consists of N6 images of human collisions. The set of 7 images of people losing their center of gravity constitutes an image of imbalance. The set of N8 images of people falling to the ground constitutes a set of images of people falling down. The N9 images of people falling and scattering objects constitute a scattering image set. All images The category label label2 is set as follows to obtain the behavior category label file. : .

[0046] As an example, a hybrid architecture model is constructed as follows: For instance, using the BoTNet model as the hybrid architecture model, with cross-entropy as the loss function and Adam as the optimizer, the BoTNet model is pre-trained, and the convolutional layers of BoTNet are frozen. The first hidden layer contains 36 neurons with ReLU activation. The second hidden layer contains 36 neurons with ReLU activation. The output layer contains 9 neurons with softmax activation.

[0047] As an example, the second training, validation, and test dataset includes: a traversal of the downhill image set. Running Image Collection Aerial Image Collection Image collection of images of people standing on one leg Inverted Image Set Collision image set Imbalanced Image Set Image collection of falling down Scattered image set The dataset is randomly divided into three parts: training set (70%), test set (10%), and validation set (20%), to obtain the behavior classification dataset. .

[0048] As an example, training, validating, and testing a hybrid architecture model includes: training on a behavior classification dataset. and behavior classification label files The behavior classification model is trained using the BoTNet model. .

[0049] Step 102: Classify real-time images of people within a set period based on a human fall classification model to determine whether a person has fallen.

[0050] In some possible implementations, real-time images of monitored personnel are acquired every set period T. Using the BlazePose human pose detection model to detect real-time human images Obtain the real-time initial standing image keypoint set or the real-time initial falling image keypoint set. . ,

[0051] in yes The coordinates of the i-th key point yes The coordinates of the i-th key point, x, yes The coordinates y of the i-th key point.

[0052] Real-time traversal According to the following formula Coordinates of the i-th key point Normalization is performed to obtain real-time data. The coordinates of the i-th normalized key point Finally, the real-time standing image keypoint set or the real-time fallen image keypoint set is obtained after normalization of the real-time personnel image. ,in,

[0053]

[0054]

[0055] .

[0056] in, It is real-time The coordinates x of the i-th normalized keypoint. It is real-time The coordinates y of the i-th normalized key point.

[0057] Will Input the trained human fall classification model The output is the real-time fall detection result. .when If the time is right, it indicates that someone has fallen to the ground; otherwise, no one has fallen to the ground.

[0058] Step 103: In response to a person falling to the ground, acquire an image sequence of real-time images of the person within a set period.

[0059] In some possible implementations, real-time images of monitored personnel are acquired. The first T1(s) of the video, with a frame rate of f, yields the image sequence used for inspection. , .

[0060] Step 104: Classify the image sequence based on the behavior classification model to identify the behavior sequence of the people.

[0061] In some possible implementations, for image sequences Use the trained behavior classification model The classification process yields a sequence of behavioral classification results, which represents the sequence of human behaviors. .

[0062] Step 105: Calculate the probability of a person falling based on the behavior sequence, and determine whether there is a person falling based on the fall probability.

[0063] In some possible implementations, the weights of various behaviors in the behavior sequence are determined, and each behavior includes at least one of the following: downhill, running, airborne, standing on one leg, handstand, collision, imbalance, fall, and spillage; the frequency of occurrence of each behavior in the behavior sequence is counted, and the probability of falling is calculated by weighted summation based on the weight and the frequency of occurrence; when the probability of falling is greater than a preset judgment threshold, a person has fallen; when the probability of falling is less than or equal to the preset judgment threshold, it is determined that no person has fallen.

[0064] As an example, traversing the sequence of behavior classification results The number of downhill sections was counted. Number of times running occurred Number of times it appears in mid-air Number of times one person stands on one leg Number of times inverted position Number of collisions Number of times imbalance occurs Number of times falling Number of times items were thrown Determine the weights of each type of behavior in the behavior sequence. Then, calculate the probability of falling using the following formula (weighted summation). For the judgment threshold ,when If a person falls, it is determined that a fall has occurred; otherwise, it is determined that no fall has occurred.

[0065]

[0066] in, These are the weights corresponding to various behaviors.

[0067] The method for detecting falls in this invention acquires a fall classification model and a behavior classification model. The fall classification model is trained using a set of standing images and a set of fallen images of individuals, while the behavior classification model is trained using a set of images showing various behaviors of individuals. Based on the fall classification model, real-time images of individuals within a set period are classified to determine whether an individual has fallen. In response to a fall, an image sequence of real-time images of individuals within the set period is acquired. Based on the behavior classification model, the image sequence is classified to determine a behavior sequence of the individual. Based on the behavior sequence, the probability of a fall is calculated, and based on the fall probability, it is determined whether a fall has occurred. Therefore, by comprehensively confirming whether a fall has occurred using the fall classification model and the behavior sequence classified by the behavior classification model, the accuracy of fall detection is improved.

[0068] To clearly illustrate the above embodiment, Figure 2This is a flowchart illustrating another method for detecting falls provided in an embodiment of the present invention. Specifically, the method involves acquiring a set of standing images and a set of fallen images of a person; traversing the standing and fallen image sets, and inputting multiple key points of the human body in each image from the standing and fallen image sets into a human posture detection model to obtain an initial set of key points for standing images and an initial set of key points for fallen images; acquiring the width and height of the images in the standing and fallen image sets, traversing the initial set of key points for standing images and the initial set of key points for fallen images, and performing data normalization to obtain a normalized set of key points for standing images and a set of key points for fallen images; constructing an EfficientNet-L2 model, traversing the set of key points for standing images and the set of key points for fallen images, dividing the set of key points for standing images and the set of key points for fallen images into a first training, validation, and test dataset, setting a human fall classification label file, and training, validating, and testing the EfficientNet-L2 model to obtain a human fall classification model. Acquire multiple sets of images depicting various human behaviors; these sets should include at least one of the following: images of people descending a slope, running, in mid-air, standing on one leg, inverted, colliding, losing balance, falling, or throwing. Set up a behavior classification label file. Construct a BOTNet model. Iterate through the multiple behavior image sets, partition the dataset, and train the BOTNet model to obtain a behavior classification model. Acquire real-time human images within a set period. Use the BlazePose human pose detection model to detect real-time human images, obtaining an initial set of real-time image key points (including the initial real-time standing image key point set or the initial real-time falling image key point set). Iterate through the initial set of real-time image key points and normalize them to obtain a normalized set of real-time image key points (including the real-time standing image key point set or the real-time falling image key point set). Input the real-time image key point set into the human fall classification model to classify whether a person has fallen. )when When a person falls, it indicates that a person has fallen. In response to a person falling, an image sequence of real-time images of the person within a set period is acquired. The image sequence is traversed, and a behavior classification model is used to classify the image sequence to identify the person's behavior sequence. The behavior sequence is traversed, and the frequency of each type of behavior is counted. Based on the weight and frequency of each type of behavior, a weighted sum is calculated to determine the probability of falling. When the probability of falling is greater than a preset judgment threshold, a person has fallen. When the probability of falling is less than or equal to the preset judgment threshold, it is determined that no person has fallen.

[0069] Figure 3This is a functional structure diagram of a person fall detection method provided in an embodiment of the present invention. Specifically, the person fall detection method based on EfficientNet-L2 and BoTNet in this invention, after acquiring a set of standing and falling images of people through a camera internet, uses BlazePose to extract the human body keypoint set from all images. After normalizing the keypoint sets of standing and falling images, a person fall classification model is trained using EfficientNet-L2. After acquiring multiple sets of behavior images, a behavior classification model is trained using BoTNet. During fall detection, real-time monitoring images of people are acquired at regular intervals (real-time monitoring images), and human body keypoint sets are extracted using BlazePose. The trained person fall classification model is used to determine whether a person has fallen. After a person falls, an image sequence from a period of time prior to the real-time personnel images is acquired, and the trained behavior classification model is used to classify the behavior corresponding to each image sequence to obtain a behavior sequence. Based on the behavior sequence, the fall probability is calculated comprehensively. When the fall probability is greater than a judgment threshold, a fall behavior is considered to have been detected.

[0070] In summary, this method for detecting falls based on EfficientNet-L2 and BoTNet triggers confirmation of a fall by detecting the act of falling. To reduce the resource consumption of routine fall detection, it utilizes methods with fewer parameters, high computational efficiency, and accurate classification. Specifically, EfficientNet-L2 is used to classify the act of falling. It considers the correlation between falling behavior and dangerous behavior, including downhill running, jumping (airborne), standing on one leg, inversion, and collision in the behavior classification list. This increases the dimensionality of the fall probability assessment and helps identify falls in dangerous patterns. For the behavior classification model, BoTNet is used, which can better understand the global context of images when processing image classification tasks, thus handling complex image classification tasks and improving the accuracy of behavior classification. A weighted average is used to calculate the fall probability, which allows for customization of weights according to specific groups or situations, improving the accuracy of fall detection.

[0071] To achieve the above embodiments, the present invention also proposes a person fall detection device.

[0072] Figure 4 This is a schematic diagram of a personnel fall detection device provided in an embodiment of the present invention.

[0073] like Figure 4 As shown, the personnel fall detection device 400 includes: a first acquisition module 401, a first classification module 402, a second acquisition module 403, a second classification module 404, and a calculation module 405.

[0074] The system comprises the following modules: a first acquisition module 401, used to acquire a person falling classification model and a behavior classification model; wherein the person falling classification model is trained using a set of standing images and a set of falling images of a person, and the behavior classification model is trained using a set of images of various behaviors of a person; a first classification module 402, used to classify real-time images of people within a set period based on the person falling classification model to determine whether a person has fallen; a second acquisition module 403, used to acquire an image sequence of real-time images of people within a set period in response to a person falling; a second classification module 404, used to classify the image sequence based on the behavior classification model to classify the behavior sequence of a person; and a calculation module 405, used to calculate the probability of a person falling based on the behavior sequence, and to determine whether a person has fallen based on the probability of falling.

[0075] Furthermore, in one possible implementation of this invention, the human fall classification model is constructed using the following modules: a third acquisition module, used to acquire a set of standing images and a set of fallen images of a person; an extraction module, used to extract human keypoint sets and normalize the data from the standing image set and the fallen image set to obtain a set of standing image keypoints and a set of fallen image keypoints; wherein, the set of human keypoints corresponds to multiple parts of the human body; a first partitioning module, used to partition the set of standing image keypoints and the set of fallen image keypoints into a first training, validation, and test dataset; and a first training module, used to train, validate, and test a convolutional neural network based on the first training, validation, and test datasets to obtain the human fall classification model.

[0076] Further, in one possible implementation of this invention, the extraction module is specifically used to extract multiple key points of the human body from each image in the standing image set and the fallen image set using a human posture detection model, corresponding to multiple parts of the human body, to obtain an initial standing image key point set and an initial fallen image key point set; based on the width and height of the images in the standing image set and the fallen image set, the initial standing image key point set and the initial fallen image key point set are normalized to obtain a normalized standing image key point set and a fallen image key point set; wherein, the data normalization includes: dividing the abscissa of all key points in the initial standing image key point set and the initial fallen image key point set by the width of the image, and dividing the ordinate of all key points by the height of the image.

[0077] Furthermore, in one possible implementation of this invention, the behavior classification model is constructed using the following modules: a fourth acquisition module for acquiring a set of multiple behavior images of a person; a second partitioning module for partitioning the set of multiple behavior images into a second training, validation, and test dataset; and a second training module for training, validating, and testing the hybrid architecture model based on the second training, validation, and test dataset to obtain the behavior classification model; wherein the hybrid architecture model is a hybrid model that combines a self-attention mechanism with the characteristics of a convolutional neural network.

[0078] Furthermore, in one possible implementation of the present invention, the multiple behavioral image sets include at least one of the following: a set of images of people going downhill, a set of images of people running, a set of images of people in mid-air, a set of images of people standing on one leg, a set of images of people standing upside down, a set of images of people colliding, a set of images of people losing balance, a set of images of people falling, and a set of images of people throwing things.

[0079] Furthermore, in one possible implementation of this invention, the calculation module is specifically used to: determine the weights corresponding to various behaviors in the behavior sequence, wherein each behavior includes at least one of the following: downhill, running, airborne, standing on one leg, handstand, collision, imbalance, fall, and spillage; count the number of occurrences of each behavior in the behavior sequence, and calculate the fall probability by weighted summation based on the weights and the number of occurrences; when the fall probability is greater than a preset judgment threshold, a person has fallen; when the fall probability is less than or equal to the preset judgment threshold, it is determined that no person has fallen.

[0080] Furthermore, in one possible implementation of this invention, the human fall classification model uses a pre-trained convolutional neural network, freezes the convolutional layers of the convolutional neural network, and sets two hidden layers containing 32 neurons each; the behavior classification model uses a pre-trained hybrid architecture model, freezes the convolutional layers of the hybrid architecture model, and sets two hidden layers containing 36 neurons each.

[0081] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of this embodiment, and will not be repeated here.

[0082] The personnel fall detection device of this invention acquires a personnel fall classification model and a behavior classification model. The personnel fall classification model is trained using a set of standing images and a set of fallen images of personnel, while the behavior classification model is trained using a set of images showing various behaviors of personnel. Based on the personnel fall classification model, real-time personnel images within a set period are classified to determine whether a person has fallen. In response to a person falling, an image sequence of real-time personnel images within the set period is acquired. Based on the behavior classification model, the image sequence is classified to determine a behavior sequence of the person. According to the behavior sequence, the probability of a person falling is calculated, and based on the fall probability, it is determined whether a person has fallen. Therefore, by comprehensively confirming whether a person has fallen using the personnel fall and behavior sequence classified by the personnel fall classification model and the behavior classification model, the accuracy of fall detection is improved.

[0083] To achieve the above embodiments, the present invention also proposes an electronic device, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the aforementioned method.

[0084] To implement the above embodiments, the present invention also proposes a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the aforementioned method.

[0085] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0086] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0087] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.

[0088] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0089] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0090] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0091] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0092] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for detecting falls, characterized in that, The method includes: A human fall classification model and a behavior classification model are obtained; wherein, the human fall classification model is trained by a set of images of people standing and images of people falling, and the behavior classification model is trained by a set of images of people engaging in various behaviors. Based on the aforementioned human fall classification model, real-time human images within a set period are classified to determine whether a person has fallen to the ground. In response to a person falling to the ground, acquire an image sequence of real-time images of the person within a set period; The image sequence is classified based on the behavior classification model to identify the behavior sequence of the person. Based on the behavioral sequence, the probability of a person falling is calculated, and based on the falling probability, it is determined whether a person has fallen.

2. The method according to claim 1, characterized in that, The human fall classification model is constructed using the following steps: Acquire standing and fallen images of personnel; Human body key point set extraction and data normalization are performed on the standing image set and the fallen image set to obtain the standing image key point set and the fallen image key point set; wherein, the human body key point set corresponds to multiple parts of the human body; The keypoint sets of standing images and falling images are divided into the first training, validation, and test datasets. The convolutional neural network is trained, validated, and tested based on the first training, validation, and test datasets to obtain a human fall classification model.

3. The method according to claim 2, characterized in that, Human keypoint set extraction and data normalization are performed on the standing image set and the fallen image set to obtain the standing image keypoint set and the fallen image keypoint set, including: The human posture detection model extracts multiple key points of the human body from each image in the standing image set and the fallen image set, corresponding to multiple parts of the human body, to obtain the initial key point set of the standing image and the initial key point set of the fallen image. Based on the width and height of the images in the standing image set and the fallen image set, the initial standing image key point set and the initial fallen image key point set are normalized to obtain the normalized standing image key point set and the fallen image key point set. The data normalization includes dividing the x-coordinate of all key points in the initial standing image key point set and the initial falling image key point set by the width of the image, and dividing the y-coordinate of all key points by the height of the image.

4. The method according to claim 1, characterized in that, The behavior classification model is constructed using the following steps: Acquire a set of images depicting various human behaviors; The image set of multiple behaviors is divided into a second training, validation, and test dataset; The hybrid architecture model is trained, validated, and tested based on the second training, validation, and testing dataset to obtain a behavior classification model; wherein the hybrid architecture model is a hybrid model that combines the self-attention mechanism and the characteristics of convolutional neural networks.

5. The method according to claim 4, characterized in that, The multiple behavioral image sets include at least one of the following: images of people going downhill, running, in mid-air, standing on one leg, standing upside down, collision, imbalance, falling, and throwing.

6. The method according to claim 5, characterized in that, The step of calculating the probability of a person falling based on the behavioral sequence, and determining whether a person has fallen based on the falling probability, includes: Determine the weights of each type of behavior in the behavior sequence. Each type of behavior includes at least one of the following: downhill, running, airborne, standing on one leg, handstand, collision, imbalance, fall, and spillage. The probability of falling is calculated by counting the number of occurrences of each type of behavior in the behavioral sequence, and by weighting and summing the occurrences based on the weights. When the probability of falling is greater than a preset threshold, a person is considered to have fallen; when the probability of falling is less than or equal to the preset threshold, it is determined that no person has fallen.

7. The method according to claim 2 or 4, characterized in that, in: The human fall classification model uses a pre-trained convolutional neural network, freezes the convolutional layers of the convolutional neural network, and sets two hidden layers containing 32 neurons each. The behavior classification model uses a pre-trained hybrid architecture model, freezes the convolutional layers of the hybrid architecture model, and sets two hidden layers containing 36 neurons each.

8. A device for detecting falls, characterized in that, The device includes: The first acquisition module is used to acquire a human fall classification model and a behavior classification model; wherein, the human fall classification model is trained by a set of images of people standing and images of people falling, and the behavior classification model is trained by a set of images of people engaging in various behaviors; The first classification module is used to classify real-time images of people within a set period based on the human fall classification model, so as to classify whether a person has fallen to the ground. The second acquisition module is used to acquire an image sequence of real-time images of a person within a set period in response to a person falling to the ground. The second classification module is used to classify the image sequence based on the behavior classification model to classify the behavior sequence of the person. The calculation module is used to calculate the probability of a person falling based on the behavior sequence, and to determine whether a person has fallen based on the fall probability.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

Images