Fall behavior detection method, apparatus, device, and computer-readable storage medium

Abstract

The application discloses a fall behavior detection method, device and equipment and a computer readable storage medium, and belongs to the technical field of machine vision. The fall behavior detection method comprises the following steps: acquiring a video stream, sampling the video stream to obtain a video frame; identifying the video frame through a preset deep learning algorithm model to obtain feature information of an identified object; wherein the feature information comprises historical feature information and current feature information; determining a historical behavior state of the identified object according to the historical feature information, determining a current behavior state of the identified object according to the historical behavior state and the current feature information; and analyzing the historical behavior state and the current behavior state to obtain a fall behavior detection result. The application analyzes and identifies the target behavior state in the fall event through the preset deep learning algorithm model, identifies the key behavior state in the fall event under the condition that the model accuracy is limited, and further identifies the whole fall event process.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, and in particular to methods, apparatus, devices and computer-readable storage media for fall behavior detection. Background Technology

[0002] In recent years, with the continuous development of behavior recognition technology, fall behavior recognition, which is a popular field in visual scenarios, has also received widespread attention.

[0003] Currently, most fall behavior recognition algorithms in related technologies are based on instantaneous abnormal behavior recognition at a spatial point or over a time interval. However, in some scenarios where the entire process of abnormal behavior events is the target, these algorithms cannot meet the requirements. In these scenarios, it is desirable for the algorithm to continuously and uninterruptedly identify the target behavior and further obtain key state information of the event. This requires the algorithm model to maintain its detection capability at the baseline at all times. However, few models can currently guarantee that they can maintain a high accuracy rate in recognizing various fall postures under any background. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, device, and computer-readable storage medium for detecting fall behavior, aiming to solve the technical problem of how to identify key behavioral states in a fall event under the condition of limited model accuracy, and thus identify the entire fall event process.

[0005] To achieve the above objectives, this application provides a fall behavior detection method, the fall behavior detection method comprising:

[0006] Acquire a video stream and sample the video stream to obtain video frames;

[0007] The video frames are identified using a pre-defined deep learning algorithm model to obtain feature information of the identified object; wherein, the feature information includes historical feature information and current feature information;

[0008] The historical behavior state of the identified object is obtained from the historical feature information, and the current behavior state of the identified object is determined based on the historical behavior state and the current feature information;

[0009] The continuous action consisting of the historical behavior state and the current behavior state is compared with the preset fall standard to obtain the fall behavior detection result.

[0010] Optionally, the preset deep learning algorithm model is a human pose estimation model; the feature information includes: bounding boxes, behavior categories, and skeleton sequences; the step of identifying the video frames using the preset deep learning algorithm model to obtain the feature information of the identified object includes:

[0011] The video frames are identified using a human pose estimation model to obtain the bounding box, behavior category, and skeleton sequence of the identified object.

[0012] Optionally, the preset deep learning algorithm model includes: an object detection algorithm and a skeleton recognition algorithm; the feature information includes: bounding boxes, behavior categories, and skeleton sequences; the step of recognizing the video frame using the preset deep learning algorithm model to obtain the feature information of the object includes:

[0013] The video frames are identified using an object detection algorithm to obtain the bounding box and behavior category of the identified object;

[0014] Based on the coordinate frame, a coordinate region including the identified object is cropped from the video frame;

[0015] The coordinate region is identified using a skeleton recognition algorithm to obtain the skeleton sequence of the identified object.

[0016] Optionally, the bounding box includes: a current bounding box and a historical bounding box; the behavior category includes: a current behavior category and a historical behavior category; before the step of obtaining the historical behavior state of the identified object from the historical feature information, the fall behavior detection method further includes:

[0017] Map the historical coordinate frame that has the greatest overlap with the current coordinate frame to the same recognition object;

[0018] The step of obtaining the historical behavioral state of the identified object from the historical feature information includes:

[0019] The historical behavior category corresponding to the historical coordinate frame is obtained from the historical feature information, and the historical behavior category is used as the historical behavior state of the identified object.

[0020] Optionally, the skeleton sequence includes: a current skeleton sequence and a historical skeleton sequence; after the step of mapping the historical coordinate box with the largest overlap with the current coordinate box to the same recognition object, the fall behavior detection method further includes:

[0021] The historical skeleton sequence corresponding to the historical coordinate frame is obtained from the historical feature information and used as the historical skeleton sequence of the identified object.

[0022] The temporal leg offset distance sequence of the identified object is determined based on the current skeleton sequence and the historical skeleton sequence of the identified object.

[0023] Optionally, the step of determining the temporal leg offset distance sequence of the identified object based on the current skeleton sequence and the historical skeleton sequence includes:

[0024] Determine the coordinates of the standard point based on the head coordinates in the current skeleton sequence;

[0025] The vector difference between the coordinates of the standard point and the leg coordinate sequence in the current skeleton sequence is used as the current leg offset distance sequence;

[0026] The sum of the squares of the differences between the current leg offset distance sequence and the historical leg offset distance sequence in the historical skeleton sequence is taken as the temporal leg offset distance sequence of the identified object.

[0027] Optionally, the current behavior category and the historical behavior state include: squatting state, falling state, and standing state; the step of determining the current behavior state of the identified object based on the historical behavior state and the current feature information includes:

[0028] If the current behavior category is a squatting state, the current behavior category is changed to a falling state or a standing state based on the historical behavior categories.

[0029] If the current behavior category is a fallen state, the current behavior state of the identified object is determined based on the continuous occurrence frequency of the standing state in the historical behavior states;

[0030] When the current behavior category is standing, the current behavior state of the identified object is determined based on the continuous occurrence frequency of falling states in the historical behavior states and the time-domain leg offset distance sequence.

[0031] Furthermore, to achieve the above objectives, this application also provides a fall behavior detection device, the fall behavior detection device comprising:

[0032] The acquisition module is used to acquire a video stream and sample the video stream to obtain video frames;

[0033] The identification module is used to identify the video frame using a preset deep learning algorithm model to obtain the feature information of the identified object; wherein, the feature information includes historical feature information and current feature information;

[0034] The calculation module is used to obtain the historical behavior state of the identified object from the historical feature information, and determine the current behavior state of the identified object based on the historical behavior state and the current feature information;

[0035] The analysis module is used to compare the continuous action consisting of the historical behavior state and the current behavior state with the preset fall standard to obtain the fall behavior detection result.

[0036] In addition, to achieve the above objectives, this application also provides a fall behavior detection device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the fall behavior detection method as described above.

[0037] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the fall behavior detection method as described above.

[0038] This application proposes a fall behavior detection method, apparatus, device, and computer-readable storage medium. In the fall behavior detection method, a video stream is first acquired, and the video stream is sampled to obtain video frames. Then, a preset deep learning algorithm model is used to identify the video frames, obtaining feature information of the identified object. The feature information includes historical feature information and current feature information. Next, the historical behavior state of the identified object is obtained from the historical feature information, and the current behavior state of the identified object is determined based on the historical behavior state and the current feature information. Finally, the continuous action composed of the historical behavior state and the current behavior state is compared with a preset fall standard to obtain a fall behavior detection result. This application identifies the feature information of the identified object in consecutive video frames by recognizing each video frame in the target scene, thereby achieving the goal of continuously and uninterruptedly recognizing the target behavior. By jointly calculating the historical behavior state and the current feature information, the current behavior state of the identified object can be obtained more accurately. By combining the continuous action composed of the historical behavior state and the current behavior state, the key state of the fall event can be accurately obtained, and then the entire fall event process can be identified. This application achieves the recognition of the key behavior state in the fall event under the condition of limited model accuracy, and thus the recognition of the entire fall event process, overcoming the technical defects in related technologies. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only a part of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1A flowchart illustrating a fall behavior detection method provided in an embodiment of this application;

[0041] Figure 2 for Figure 1 A flowchart illustrating one implementation of step S20;

[0042] Figure 3 for Figure 1 A flowchart illustrating another implementation of step S20;

[0043] Figure 4 A flowchart illustrating a fall behavior detection method provided in another embodiment of this application;

[0044] Figure 5 for Figure 4 A detailed flowchart of step S05;

[0045] Figure 6 for Figure 5 A flowchart illustrating one implementation of the corresponding embodiment;

[0046] Figure 7 for Figure 1 A detailed flowchart of some steps in step S30;

[0047] Figure 8 for Figure 7 A flowchart illustrating one implementation of the corresponding embodiment;

[0048] Figure 9 This is a schematic diagram of the structure of a fall behavior detection device provided in an embodiment of this application;

[0049] Figure 10 This is a schematic diagram of the hardware structure of a fall behavior detection device provided in an embodiment of this application. Detailed Implementation

[0050] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that the embodiments of this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the embodiments of this application with unnecessary detail.

[0051] It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0052] It should also be understood that references to "one embodiment" or "some embodiments" in the specification of embodiments of this application mean that one or more embodiments of this application include the specific features, structures, or characteristics described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0053] In recent years, with the continuous development of behavior recognition technology, fall behavior recognition, which is a popular field in visual scenarios, has also received widespread attention.

[0054] There are two main methods for fall behavior recognition: visual and depth information. Depth information methods primarily rely on depth sensors or depth cameras as algorithm input, while visual methods mainly use cameras or multi-dimensional cameras to capture video streams of a certain area as algorithm input. Currently, mature fall behavior recognition technologies based on the visual dimension mainly combine detection with pose recognition. This method first identifies human targets in the current video frame using algorithms such as target detection and instance segmentation. Then, it calculates the skeletal information of the target person using skeletal algorithms such as OpenPose (a pose estimation network). Finally, it further determines whether a fall posture exists based on the features of the skeletal information in the temporal or spatial domains.

[0055] Currently, most fall behavior recognition algorithms in related technologies are based on instantaneous abnormal behavior recognition at a spatial point or over a time interval. However, in some scenarios where the entire process of abnormal behavior events is the target, these algorithms cannot meet the requirements. In these scenarios, it is desirable for the algorithm to continuously and uninterruptedly identify the target behavior and further obtain key state information of the event (such as whether someone is falling, whether the fall lasts for t minutes, and whether the person has stood up after falling). This requires the algorithm model to maintain its detection capability at the baseline at all times. However, few models can currently guarantee a high accuracy rate of over 95% when facing various fall postures in any context.

[0056] Based on this, embodiments of this application provide a fall behavior detection method, apparatus, device, and computer-readable storage medium. By combining target detection algorithms and skeleton recognition algorithms, the key states of a fall event are analyzed and identified, enabling continuous identification of the key states of a fall event under the condition of limited model accuracy, thereby achieving the purpose of identifying the entire complete process of the fall event.

[0057] The fall behavior detection method, apparatus, device, and computer-readable storage medium provided in this application are specifically described through the following embodiments. First, the fall behavior detection method in this application is described.

[0058] This application provides a fall behavior detection method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating a fall behavior detection method according to an embodiment of this application. This fall behavior detection method can be applied to fall behavior detection devices, such as... Figure 1 As shown, the fall behavior detection method provided in this embodiment includes steps S10 to S40.

[0059] Step S10: Obtain the video stream and sample the video stream to obtain video frames;

[0060] In this embodiment, the execution entity is a fall behavior detection device. The video stream can be a video stream pre-stored in the storage space of the fall behavior detection device, or it can be a video stream captured by a camera connected to the fall behavior detection device for capturing a specified scene. The fall behavior detection device will sample the video stream at a certain sampling frequency to obtain multiple video frames as images to be processed. By recognizing these images to be processed, it can be determined whether a fall behavior has occurred in the video stream.

[0061] As an example, the fall detection method provided in this embodiment can be applied to elevator scenarios. By acquiring the video stream captured by the camera inside the elevator, sampling the video stream to obtain video frames, and then identifying each video frame, it can be determined whether there is a person falling inside the elevator in the time domain corresponding to the video stream.

[0062] Step S20: The video frame is identified using a preset deep learning algorithm model to obtain the feature information of the identified object; wherein, the feature information includes current feature information and historical feature information;

[0063] In this embodiment, a deep learning algorithm model with target detection and skeleton recognition functions is used to identify multiple video frames provided in the aforementioned steps to obtain feature information of the identified object (e.g., the target person). Since each video frame is sampled from the same video stream, there will be a chronological order in the temporal domain. In this embodiment, the video frame being identified at the current moment is taken as the current video frame. The feature information obtained after the algorithm model identifies the current video frame is the current feature information. Video frames that are earlier than the current video frame in the temporal domain are historical video frames. The feature information obtained after the algorithm model identifies the historical video frames is the historical feature information.

[0064] In some feasible embodiments, the preset deep learning algorithm model is a human pose estimation model; the feature information includes: coordinate boxes, behavior categories and skeleton sequences; the above step S20 includes: identifying video frames through the human pose estimation model to obtain the coordinate boxes, behavior categories and skeleton sequences of the identified objects.

[0065] In this embodiment, the coordinate frame, behavior category, and skeleton sequence are in one-to-one correspondence. That is, a coordinate frame must correspond to a behavior category and a set of skeleton sequence coordinate frames. The coordinate frame can be a rectangle that selects the location of a person in a video frame. The behavior category can be text, characters, or codes that represent the behavior state of a person and are marked inside or on the coordinate frame. The skeleton sequence can be a set of coordinates that mark the human skeleton points inside the coordinate frame.

[0066] This embodiment provides an implementation of a preset deep learning algorithm model, namely a human pose estimation model, which represents a multi-output deep learning algorithm similar to YOLOpose, possessing both object detection and skeleton point recognition capabilities, such as... Figure 2 As shown, taking video frames as input to this algorithm model will simultaneously output multiple results such as bounding boxes, behavior categories, and skeleton sequences.

[0067] In some feasible embodiments, the preset deep learning algorithm model includes: an object detection algorithm and a skeleton recognition algorithm; the feature information includes: a bounding box, a behavior category, and a skeleton sequence; the above step S20 includes: recognizing the video frame using the object detection algorithm to obtain the bounding box and behavior category of the object to be recognized; cropping the coordinate region including the object to be recognized from the video frame based on the bounding box; recognizing the coordinate region using the skeleton recognition algorithm to obtain the skeleton sequence of the object to be recognized.

[0068] In this embodiment, the coordinate frame, behavior category, and skeleton sequence are in one-to-one correspondence. That is, a coordinate frame must correspond to a behavior category and a set of skeleton sequence coordinate frames. The coordinate frame can be a rectangle that selects the location of a person in a video frame. The behavior category can be text, characters, or codes that represent the behavior state of a person and are marked inside or on the coordinate frame. The skeleton sequence can be a set of coordinates that mark the human skeleton points inside the coordinate frame.

[0069] This embodiment provides another implementation of a preset deep learning algorithm model, namely, the object detection algorithm and the skeleton recognition algorithm are implemented sequentially, such as... Figure 3 As shown, when a video frame is used as input to the algorithm model, the object detection algorithm is first executed to output the coordinate box and behavior category. Then, the coordinate regions selected by each coordinate box are cropped from the video frame, and the skeleton recognition algorithm is executed on each coordinate region to output the skeleton sequence.

[0070] It should be noted that the above two preset deep learning algorithm models are only examples. Other deep learning algorithm models with object detection and skeleton recognition functions can also be applied to this embodiment, as long as the algorithm model can output a one-to-one corresponding coordinate box, behavior category and skeleton sequence after processing the video frame.

[0071] Step S30: Obtain the historical behavior state of the object from the historical feature information, and determine the current behavior state of the object based on the historical behavior state and the current feature information;

[0072] It should be noted that although the current feature information given in the aforementioned steps already includes the behavior category, in order to improve the recognition accuracy of the current behavior state and avoid occasional insufficient accuracy of the algorithm model, this embodiment will take the historical behavior state in the historical feature information, the behavior category in the current feature information, and the skeleton sequence as input, and calculate the current behavior state of the object to be identified through a preset state calculation function.

[0073] In some feasible embodiments, the coordinate frame includes: a current coordinate frame and a historical coordinate frame; the behavior category includes: a current behavior category and a historical behavior category; prior to step S30 above, the fall behavior detection method further includes:

[0074] Step S03: Map the historical coordinate frame with the largest overlap with the current coordinate frame to the same recognition object.

[0075] It should be noted that when there is only one person in the video frame, it is obvious that the historical feature information and the current feature information correspond to the same person. However, in general, there are multiple people in the same frame in the scene captured by the camera. Therefore, there are multiple sets of feature information in each video frame, namely multiple bounding boxes, multiple behavior categories and multiple skeleton sequences. In this case, when obtaining the historical behavior category from the historical feature information, it is necessary to confirm whether the obtained historical behavior category matches the current person. To address this, this embodiment will traverse all bounding boxes in two adjacent video frames in the temporal domain. Arranging these bounding boxes in the temporal domain will result in a matrix. The two bounding boxes with the largest IOU (Intersection over Union, also known as the crossover ratio, is usually used in the evaluation of current object detection algorithms. The higher the IOU value, the higher the prediction accuracy of the algorithm.) values. These two bounding boxes are used as the mapping of the same person, so that the feature information between consecutive video frames in a short time domain interval is coherent, and the feature information between people in multi-person scenes is prevented from interfering with each other.

[0076] The step S30 above, which involves obtaining the historical behavior state of the identified object from the historical feature information, includes: obtaining the historical behavior category corresponding to the historical coordinate frame from the historical feature information, and using the historical behavior category as the historical behavior state of the identified object.

[0077] It is understandable that, when it is determined that the historical coordinate frame and the current coordinate frame correspond to the same recognition object, when obtaining the historical behavior state of the recognition object, the historical coordinate frame mapped to the recognition object can be used as the basis to obtain the historical behavior category corresponding to the historical coordinate frame as the historical behavior state of the recognition object.

[0078] In some feasible embodiments, the skeleton sequence includes: a current skeleton sequence and a historical skeleton sequence; refer to Figure 4 Following step S03, the fall behavior detection method further includes:

[0079] Step S04: Obtain the historical skeleton sequence corresponding to the historical coordinate frame from the historical feature information as the historical skeleton sequence of the object to be identified;

[0080] Similarly to the aforementioned embodiments, when it is determined that the historical coordinate frame and the current coordinate frame correspond to the same identification object, when obtaining the historical skeleton sequence of the identification object, the historical coordinate frame mapped to the identification object can be used as the basis to obtain the historical skeleton sequence corresponding to the historical coordinate frame as the historical skeleton sequence of the identification object.

[0081] Step S05: Determine the temporal leg offset distance sequence of the identified object based on the current skeleton sequence and the historical skeleton sequence of the identified object.

[0082] In this embodiment, in order to further improve the accuracy of identifying the current behavior state, the time domain change characteristics of the skeleton sequence of the identified object, i.e., the time domain leg offset distance sequence, will be used as an input to the state calculation function in the aforementioned embodiment. This represents the changes in the legs of the identified object from the historical video frame to the current video frame.

[0083] Reference Figure 5 In some feasible embodiments, step S05 above includes:

[0084] Step S051: Determine the coordinates of the standard point based on the head coordinates in the current skeleton sequence;

[0085] In this embodiment, it is first determined whether the head coordinates are not all 0 and there is no out-of-bounds phenomenon. If so, it means that the head coordinates can be used as a reference. At this time, the average value of the nose, eye and ear coordinates in the skeleton sequence is used as the standard point coordinates. If not, it means that the head coordinates cannot be used as a reference. At this time, the average value of the left shoulder and right shoulder coordinates in the skeleton sequence is used as the standard point coordinates.

[0086] Step S052: The vector difference between the standard point coordinates and the leg coordinate sequence in the current skeleton sequence is used as the current leg offset distance sequence;

[0087] In this embodiment, the vector difference between the leg coordinate x sequence and the standard point x coordinate is calculated as the x sequence in the current leg offset distance sequence, and the vector difference between the leg coordinate y sequence and the standard point y coordinate is calculated as the y sequence in the current leg offset distance sequence.

[0088] Step S053: The sum of the squares of the differences between the current leg offset distance sequence and the historical leg offset distance sequence in the historical skeleton sequence is used as the temporal leg offset distance sequence of the identified object.

[0089] In this embodiment, the leg offset distance sequence before n samplings is selected as the historical leg offset distance sequence (n can be a positive integer such as 1, 2, 3, etc.), and the sum of the squares of the differences obtained by subtracting it from the current leg offset distance sequence is used as the temporal leg offset distance sequence of the object from before n samplings to the current video frame.

[0090] As an example, one implementation of steps S051 to S053 above is as follows: Figure 6 As shown.

[0091] Reference Figure 7 In some feasible embodiments, the current behavior category and historical behavior state include: squatting state, falling state, and standing state; the step of determining the current behavior state of the identified object based on the historical behavior state and current feature information in step S30 above includes:

[0092] Step S31: If the current behavior category is squatting, change the current behavior category to falling or standing based on the historical behavior categories.

[0093] In this embodiment, when the current behavior category is a squatting state, the historical behavior state of the identified object in the historical video frame adjacent to the current video frame (i.e., the previous frame of the current video frame) needs to be used as a reference to change the squatting state category. Specifically, if the historical behavior state of the identified object in the previous frame is a falling state, the current behavior category is changed from squatting state to falling state; if the historical behavior state of the identified object in the previous frame is not a falling state, the current behavior category is changed from squatting state to standing state.

[0094] Step S32: If the current behavior category is a fallen state, determine the current behavior state of the identified object based on the continuous occurrence frequency of the standing state in the historical behavior states.

[0095] In this embodiment, if the current behavior category is a fallen state, and if there are m consecutive historical video frames in the time domain starting from the current video frame in which the historical behavior category is a standing state, then the current behavior category is corrected and the current behavior category is finally determined to be a standing state; otherwise, if there are no m consecutive historical video frames in the time domain starting from the current video frame in which the historical behavior category is a standing state, then the current behavior category is not corrected and the current behavior category is finally determined to be a fallen state.

[0096] Step S33: If the current behavior category is standing, determine the current behavior state of the identified object based on the continuous occurrence frequency of falling state in historical behavior states and the time-domain leg offset distance sequence.

[0097] In this embodiment, if the current behavior category is standing, and if, starting from the current video frame, there are n consecutive historical video frames in the time domain where the historical behavior is in a falling state, then the current behavior category is corrected, and the current behavior state is finally determined to be a falling state. Conversely, if, starting from the current video frame, there are no n consecutive historical video frames in the time domain where the historical behavior is in a falling state, then the time domain leg offset distance sequence needs to be considered. If the time domain leg offset distance sequence satisfies the falling change condition, then the current behavior category is corrected, and the current behavior state is finally determined to be a falling state. If, starting from the current video frame, there are no n consecutive historical video frames in the time domain where the historical behavior is in a falling state, and the time domain leg offset distance sequence does not satisfy the falling change condition, then the current behavior category is not corrected, and the current behavior state is finally determined to be standing.

[0098] As an example, m, n, and the fall change conditions in steps S31 to S33 above can be adjusted according to actual detection needs. One implementation of steps S31 to S33 is as follows: Figure 8 As shown.

[0099] Furthermore, it should be noted that when the current video frame is the initial video frame in the video stream, the behavior category identified by the algorithm model is directly used as the behavior state of the identified object in that video frame. However, when the current video frame is not the initial video frame in the video stream, historical behavior states from previous video frames must be used as a reference, and the current behavior category must be corrected based on the current feature information to arrive at the most accurate current behavior state.

[0100] Step S40: Compare the continuous action consisting of historical behavior state and current behavior state with the preset fall standard to obtain the fall behavior detection result.

[0101] In this embodiment, the preset fall criteria can include situations such as the previous frame being standing and the next frame being a fall (the starting point of the fall event), the previous frame being a fall and the next frame being a fall (the middle process of the fall event), and the previous frame being a fall and the next frame being standing (the end point of the fall event). Therefore, if the continuous action composed of the historical behavior state and the current behavior state meets the preset fall criteria, it can be determined that a fall event exists in the video stream, and the corresponding fall behavior detection results can be obtained as: someone is falling, the fall lasts for several minutes, and the person has stood up after falling, etc., which are key states of the fall event. If the continuous action composed of the historical behavior state and the current behavior state does not meet the preset fall criteria, it can be determined that no fall event exists in the video stream.

[0102] This embodiment provides a fall behavior detection method. By using a deep learning algorithm model that integrates target detection and skeleton recognition algorithms, it identifies abnormal behaviors of people in a specified scene in terms of both target detection and skeleton dimensions. By combining the results of the two algorithms, it calculates the key behavioral states that a person will exhibit when a fall occurs. Then, it combines a temporal coordinate matching algorithm to ensure that the feature information of adjacent video frames belonging to different people in multi-person scenes is not confused, thereby identifying the key state nodes of the fall behavior event and ultimately achieving the goal of continuously and uninterruptedly identifying fall behavior events.

[0103] Furthermore, this application also proposes a fall behavior detection device, referring to... Figure 9 , Figure 9 This is a schematic diagram of the structure of a fall behavior detection device provided in an embodiment of this application, as shown below. Figure 9As shown, in this embodiment, the fall behavior detection device includes: an acquisition module 10, an identification module 20, a calculation module 30, and an analysis module 40.

[0104] The acquisition module 10 is used to acquire the video stream and sample the video stream to obtain video frames;

[0105] The recognition module 20 is used to recognize video frames through a preset deep learning algorithm model to obtain the feature information of the object to be recognized; wherein, the feature information includes historical feature information and current feature information;

[0106] The calculation module 30 is used to obtain the historical behavior state of the object to be identified from the historical feature information, and to determine the current behavior state of the object to be identified based on the historical behavior state and the current feature information.

[0107] The analysis module 40 is used to compare the continuous actions composed of historical behavior states and current behavior states with preset fall standards to obtain fall behavior detection results.

[0108] In some feasible embodiments, the preset deep learning algorithm model is a human pose estimation model; the feature information includes: coordinate boxes, behavior categories and skeleton sequences; the recognition module 20 is also used to recognize video frames through the human pose estimation model to obtain the coordinate boxes, behavior categories and skeleton sequences of the recognized objects.

[0109] In some feasible embodiments, the preset deep learning algorithm model includes: object detection algorithm and skeleton recognition algorithm; the feature information includes: coordinate box, behavior category and skeleton sequence; the recognition module 20 is also used to recognize video frames through the object detection algorithm to obtain the coordinate box and behavior category of the object to be recognized; to crop the coordinate region of the object to be recognized from the video frame based on the coordinate box; and to recognize the coordinate region through the skeleton recognition algorithm to obtain the skeleton sequence of the object to be recognized.

[0110] In some feasible embodiments, the coordinate frame includes: the current coordinate frame and the historical coordinate frame; the behavior category includes: the current behavior category and the historical behavior category; the fall behavior detection device also includes: a mapping module, which is used to map the historical coordinate frame with the largest overlap with the current coordinate frame to the same recognition object; the calculation module 30 is also used to obtain the historical behavior category corresponding to the historical coordinate frame from the historical feature information, and use the historical behavior category as the historical behavior state of the recognition object.

[0111] In some feasible embodiments, the skeleton sequence includes: a current skeleton sequence and a historical skeleton sequence; the calculation module 30 is also used to obtain the historical skeleton sequence corresponding to the historical coordinate frame from the historical feature information as the historical skeleton sequence of the object to be identified; and to determine the temporal leg offset distance sequence of the object to be identified based on the current skeleton sequence and the historical skeleton sequence of the object to be identified.

[0112] In some feasible embodiments, the calculation module 30 is further configured to determine the standard point coordinates based on the head coordinates in the current skeleton sequence; use the vector difference between the standard point coordinates and the leg coordinate sequence in the current skeleton sequence as the current leg offset distance sequence; and use the sum of the squares of the differences between the current leg offset distance sequence and the historical leg offset distance sequences in the historical skeleton sequence as the temporal leg offset distance sequence of the identified object.

[0113] In some feasible embodiments, the current behavior category and historical behavior state include: squatting state, falling state, and standing state; the calculation module 30 is further configured to, when the current behavior category is squatting state, convert the current behavior category to falling state or standing state according to the historical behavior state; when the current behavior category is falling state, determine the current behavior state of the identified object according to the continuous occurrence frequency of the standing state in the historical behavior state; when the current behavior category is standing state, determine the current behavior state of the identified object according to the continuous occurrence frequency of the falling state in the historical behavior state and the time-domain leg offset distance sequence.

[0114] The fall behavior detection device provided in this embodiment belongs to the same inventive concept as the fall behavior detection method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in any of the above embodiments. Furthermore, this embodiment has the same beneficial effects as the fall behavior detection method.

[0115] Furthermore, this application also provides a fall behavior detection device. The fall behavior detection method applied to the fall behavior detection device can be executed by the fall behavior detection device, which can be implemented by software and / or hardware and integrated into the fall behavior detection device. The fall behavior detection device can be a mobile device capable of communicating with the network side, such as a PC (Personal Computer, including desktop computers, laptops, mini-laptops, tablets, and ultrabooks), a mobile phone, or a smart wearable device.

[0116] Reference Figure 10 , Figure 10 This is a schematic diagram of the hardware structure of a fall behavior detection device provided in an embodiment of this application. Figure 10As shown, the fall detection device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0117] Those skilled in the art will understand that Figure 10 The structure shown does not constitute a limitation on the fall behavior detection device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0118] like Figure 10 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and computer programs.

[0119] exist Figure 10 In the fall detection device shown, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in this embodiment can be set in the fall detection device. The fall detection device calls the computer program stored in the memory 1005 through the processor 1001 and performs the following operations:

[0120] Acquire the video stream and sample it to obtain video frames;

[0121] The video frames are identified using a pre-defined deep learning algorithm model to obtain the feature information of the identified object; the feature information includes historical feature information and current feature information.

[0122] The historical behavior state of the identified object is obtained from historical feature information, and the current behavior state of the identified object is determined based on the historical behavior state and the current feature information.

[0123] The continuous actions consisting of historical and current behavioral states are compared with preset fall criteria to obtain fall behavior detection results.

[0124] Furthermore, the preset deep learning algorithm model is a human pose estimation model; the feature information includes: bounding box, behavior category, and skeleton sequence; the processor 1001 can call the computer program stored in the memory 1005 and also perform the following operations:

[0125] By using a human pose estimation model to identify video frames, the coordinate boxes, behavior categories, and skeleton sequences of the identified objects are obtained.

[0126] Furthermore, the preset deep learning algorithm model includes: an object detection algorithm and a skeleton recognition algorithm; the feature information includes: bounding boxes, behavior categories, and skeleton sequences; the processor 1001 can call the computer program stored in the memory 1005 and also perform the following operations:

[0127] Video frames are identified using object detection algorithms to obtain the bounding boxes and behavior categories of the identified objects;

[0128] The coordinate region of the object to be identified is cropped from the video frame based on the coordinate frame;

[0129] The skeleton recognition algorithm identifies the coordinate region and obtains the skeleton sequence of the object.

[0130] Furthermore, the coordinate frame includes: the current coordinate frame and the historical coordinate frame; the behavior category includes: the current behavior category and the historical behavior category; the processor 1001 can call the computer program stored in the memory 1005 and also perform the following operations:

[0131] Map the historical bounding box that has the greatest overlap with the current bounding box to the same recognition object;

[0132] The historical behavior category corresponding to the historical coordinate frame is obtained from the historical feature information, and the historical behavior category is used as the historical behavior state of the object to be identified.

[0133] Furthermore, the skeleton sequence includes: the current skeleton sequence and the historical skeleton sequence; the processor 1001 can call the computer program stored in the memory 1005 and also perform the following operations:

[0134] The historical skeleton sequence corresponding to the historical coordinate frame is obtained from the historical feature information and used as the historical skeleton sequence of the object to be identified.

[0135] The temporal leg offset distance sequence of the identified object is determined based on the current skeleton sequence and the historical skeleton sequence of the identified object.

[0136] Furthermore, the processor 1001 can call a computer program stored in the memory 1005 and also perform the following operations:

[0137] Determine the coordinates of the standard point based on the head coordinates in the current skeleton sequence;

[0138] The vector difference between the standard point coordinates and the leg coordinate sequence in the current skeleton sequence is used as the current leg offset distance sequence;

[0139] The sum of the squares of the differences between the current leg offset distance sequence and the historical leg offset distance sequences in the historical skeleton sequence is used as the temporal leg offset distance sequence of the identified object.

[0140] Furthermore, the current behavior category and historical behavior state include: squatting state, falling state, and standing state; the processor 1001 can call the computer program stored in the memory 1005 and also perform the following operations:

[0141] If the current behavior category is squatting, the current behavior category will be changed to falling or standing based on the historical behavior categories.

[0142] When the current behavior category is a fallen state, the current behavior state of the identified object is determined based on the continuous frequency of the standing state in the historical behavior states;

[0143] When the current behavior category is standing, the current behavior state of the identified object is determined based on the continuous frequency of falling in the historical behavior states and the time-domain leg offset distance sequence.

[0144] The fall behavior detection device proposed in this embodiment and the fall behavior detection method applied to the fall behavior detection device proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in any of the above embodiments, and this embodiment has the same beneficial effects as the fall behavior detection method.

[0145] Furthermore, this application also proposes a computer-readable storage medium for use in a computer. The computer-readable storage medium can be a non-volatile computer-readable storage medium, and a computer program is stored on the computer program. When the computer program is executed by a processor, it implements the fall behavior detection method of any of the embodiments described above.

[0146] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0147] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.

Claims

1. A method for detecting fall behavior, characterized in that, The fall behavior detection method includes: Acquire a video stream and sample the video stream to obtain video frames; The video frames are identified using a preset deep learning algorithm model to obtain feature information of the identified object; wherein, the feature information includes historical feature information and current feature information, and the feature information also includes current coordinate frame, historical coordinate frame, current behavior category, historical behavior category, current skeleton sequence and historical skeleton sequence, and the historical behavior states corresponding to the current behavior category and the historical feature categories both include squatting state, falling state and standing state; Map the historical coordinate frame that has the greatest overlap with the current coordinate frame to the same recognition object; The historical skeleton sequence corresponding to the historical coordinate frame is obtained from the historical feature information and used as the historical skeleton sequence of the identified object. The standard point coordinates are determined based on the head coordinates in the current skeleton sequence: if the head coordinates are not all 0 and there is no out-of-bounds phenomenon, the average value of the nose, eye and ear coordinates in the current skeleton sequence is used as the standard point coordinates; if the head coordinates are not all 0 or there is an out-of-bounds phenomenon, the average value of the left shoulder and right shoulder coordinates in the current skeleton sequence is used as the standard point coordinates. The vector difference between the coordinates of the standard point and the leg coordinate sequence in the current skeleton sequence is used as the current leg offset distance sequence; The sum of the squares of the differences between the current leg offset distance sequence and the historical leg offset distance sequences in the historical skeleton sequence is taken as the temporal leg offset distance sequence of the identified object. The historical behavior state of the identified object is obtained from the historical feature information, and the current behavior state of the identified object is determined according to the historical behavior state and the current feature information: if the current behavior category is a squatting state, the current behavior category is converted to a falling state or a standing state based on the historical behavior state corresponding to the historical video frame adjacent to the current video frame; if the current behavior category is a standing state, and if there are multiple consecutive historical video frames in the time domain that are all in a falling state, or if the time domain leg offset distance sequence satisfies the falling change condition, then the current behavior state is corrected to a falling state. The continuous action consisting of the historical behavior state and the current behavior state is compared with the preset fall criteria to obtain the fall behavior detection result; wherein, the preset fall criteria include: the previous frame is standing and the next frame is falling, the previous frame is standing and the next frame is falling, and the previous frame is falling and the next frame is standing; the fall behavior detection result includes: falling, falling continuously, and standing up after falling.

2. The fall behavior detection method as described in claim 1, characterized in that, The preset deep learning algorithm model is a human pose estimation model; the step of identifying the video frame using the preset deep learning algorithm model to obtain the feature information of the identified object includes: The video frames are identified using a human pose estimation model to obtain the bounding box, behavior category, and skeleton sequence of the identified object.

3. The fall behavior detection method as described in claim 1, characterized in that, The preset deep learning algorithm model includes: an object detection algorithm and a skeleton recognition algorithm; the step of recognizing the video frame using the preset deep learning algorithm model to obtain the feature information of the object to be recognized includes: The video frames are identified using an object detection algorithm to obtain the bounding box and behavior category of the identified object; Based on the coordinate frame, a coordinate region including the identified object is cropped from the video frame; The coordinate region is identified using a skeleton recognition algorithm to obtain the skeleton sequence of the identified object.

4. The fall behavior detection method as described in claim 1, characterized in that, The step of determining the current behavior state of the identified object based on the historical behavior state and the current feature information further includes: If the current behavior category is a fallen state, the current behavior state of the identified object is determined based on the continuous occurrence frequency of the standing state in the historical behavior states.

5. A fall behavior detection device, characterized in that, The fall behavior detection device includes: The acquisition module is used to acquire a video stream and sample the video stream to obtain video frames; The recognition module is used to recognize the video frame through a preset deep learning algorithm model to obtain the feature information of the object to be recognized; wherein, the feature information includes historical feature information and current feature information, and the feature information also includes current coordinate frame, historical coordinate frame, current behavior category, historical behavior category, current skeleton sequence and historical skeleton sequence, and the historical behavior states corresponding to the current behavior category and the historical feature categories both include squatting state, falling state and standing state; The calculation module is used to map the historical coordinate frame with the largest overlap with the current coordinate frame to the same recognition object; The historical skeleton sequence corresponding to the historical coordinate frame is obtained from the historical feature information and used as the historical skeleton sequence of the identified object. The standard point coordinates are determined based on the head coordinates in the current skeleton sequence: if the head coordinates are not all 0 and there is no out-of-bounds phenomenon, the average value of the nose, eye and ear coordinates in the current skeleton sequence is used as the standard point coordinates; if the head coordinates are not all 0 or there is an out-of-bounds phenomenon, the average value of the left shoulder and right shoulder coordinates in the current skeleton sequence is used as the standard point coordinates. The vector difference between the coordinates of the standard point and the leg coordinate sequence in the current skeleton sequence is used as the current leg offset distance sequence; The sum of the squares of the differences between the current leg offset distance sequence and the historical leg offset distance sequences in the historical skeleton sequence is taken as the temporal leg offset distance sequence of the identified object. The historical behavior state of the identified object is obtained from the historical feature information, and the current behavior state of the identified object is determined according to the historical behavior state and the current feature information: if the current behavior category is a squatting state, the current behavior category is converted to a falling state or a standing state based on the historical behavior state corresponding to the historical video frame adjacent to the current video frame; if the current behavior category is a standing state, and if there are multiple consecutive historical video frames in the time domain that are all in a falling state, or if the time domain leg offset distance sequence satisfies the falling change condition, then the current behavior state is corrected to a falling state. An analysis module is used to compare the continuous action consisting of the historical behavior state and the current behavior state with a preset fall standard to obtain a fall behavior detection result; wherein, the preset fall standard includes: the previous frame is standing and the next frame is falling, the previous frame is standing and the next frame is falling, and the previous frame is falling and the next frame is standing; the fall behavior detection result includes: falling, falling continuously, and standing up after falling.

6. A fall behavior detection device, characterized in that, The fall behavior detection device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the fall behavior detection method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the fall behavior detection method as described in any one of claims 1 to 4.

Images