Multi-view fall detection method, device and storage medium

Abstract

The present invention discloses a multi-view fall detection method, device and storage medium based on spatiotemporal adaptation and human posture estimation, which comprises: obtaining video stream segments collected by a camera; preprocessing the video sequence through a lightweight convolutional network to obtain a global feature map sequence; performing spatiotemporal adaptive processing on each global feature map in the global feature map sequence to obtain an image block containing the behavior and action of the elderly; obtaining the position coordinates of the key points of the human body from the image block using a human posture estimation algorithm; judging whether the elderly in the image block under different visual angles has fallen according to the position coordinates of the key points of the human body using a threshold method; and generating an alarm signal when N consecutive image blocks are judged as the elderly having fallen. The present invention can reduce redundant calculations and reduce the calculation overhead of the model while ensuring the real-time and accuracy of the elderly fall detection.

Description

Technical Field

[0001] The present invention relates to a multi-view fall detection method, device and storage medium based on spatiotemporal adaptation and human body posture estimation, and belongs to the technical field of computer vision. Background Art

[0002] With the continuous development of society, the problem of population aging is becoming increasingly serious. More and more young people will live apart from their parents after work, resulting in an increasing number of empty-nest elderly people and elderly people living alone. Therefore, the living conditions of the elderly at home are receiving more and more attention from the society. As the elderly age, their various physiological functions continue to decline, and the degree of damage caused by falls to the elderly's body is becoming more and more serious. Falls have become the primary threat to the health of the elderly in daily life. Traditional fall detection for the elderly is achieved through wearable sensor devices. For the elderly with declining memory, not only may they forget to wear the detection equipment, but the wearable contact detection equipment may make the elderly feel uncomfortable or make it more inconvenient for the elderly to move, affecting their daily life, and is not suitable for use by elderly people at home.

[0003] With the continuous development of computer vision technology, there are more and more elderly fall detection technologies based on intelligent video surveillance. Intelligent video surveillance no longer requires manual real-time viewing. Some lightweight algorithms can be used to detect whether the elderly in the video have fallen in real time. This will not only not interfere with the daily life of the elderly, but also send out alarms in time when unexpected situations occur. At present, most video-based fall detection algorithms either have high computing costs and hardware requirements, making it difficult to implement applications in actual scenarios; or in order to ensure the real-time performance of video detection and reduce computing overhead, the algorithm model is lightweight, but it is difficult to ensure the accuracy of video detection or generate good visualization effects, thus affecting the determination of fall posture. For example, the classic human posture estimation algorithm OpenPose has a large amount of calculation and is difficult to ensure real-time performance on embedded devices, but the lightweight OpenPose has a low recognition accuracy for complex postures. Although there are many methods for fall determination based on 2D human posture, it is impossible to accurately determine whether a person has fallen based on different visual angles, and it is difficult to solve the problem of judging three-dimensional postures with two-dimensional images. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention proposes a multi-perspective fall detection method, device and storage medium based on spatiotemporal adaptation and human posture estimation, which can not only reduce redundant calculations and lower the computational overhead of the model, realize application in actual scenarios, but also can detect in real time and accurately whether the elderly indoors fall.

[0005] In order to solve the above technical problems, the present invention adopts the following technical means:

[0006] In a first aspect, the present invention proposes a multi-view fall detection method based on spatiotemporal adaptation and human posture estimation, comprising the following steps:

[0007] Get the video stream segments captured by the camera to obtain a video sequence;

[0008] Preprocess the video sequence through a lightweight convolutional network to obtain a global feature map sequence;

[0009] Performing spatiotemporal adaptive processing on each global feature map in the global feature map sequence to obtain an image block containing the elderly's behavior and actions;

[0010] The human body posture estimation algorithm is used to obtain the position coordinates of the key points of the human body from the image block;

[0011] According to the position coordinates of the key points of the human body, the threshold method is used to determine whether the elderly in the image block at different visual angles have fallen;

[0012] When N consecutive image blocks are judged as an elderly person falling down, an alarm signal is generated.

[0013] In combination with the first aspect, further, each global feature map in the global feature map sequence is processed using a spatially adaptive reward function and a temporally adaptive reward function to select an image block containing the elderly's behavior actions, wherein the spatially adaptive reward function and the temporally adaptive reward function are expressed as follows:

[0014]

[0015]

[0016] in, represents the spatially adaptive reward function, Represents the number of frames from the i-th video frame O i The sampled image blocks, express The softmax prediction of the image block containing the elderly’s behavior, represents the expectation obtained by sampling from a Gaussian distribution, R' i (B i |B1,...,B i-1 ) represents the time-adaptive reward function, B i represents the Bernoulli parameter of the i-th video frame output, P' iy (B1,...,B i-1 ,1) indicates B i = 1, softmax prediction, P' iy (B1,...,B i-1 ,0) indicates Bi =0, λ is a predefined hyperparameter, L represents the length of the image block, ω represents the threshold for activating video frames, i=1,2,…,n, n is the number of video frames.

[0017] In combination with the first aspect, further, the human body posture estimation algorithm adopts the MoveNet algorithm.

[0018] In combination with the first aspect, further, a method for obtaining position coordinates of key points of a human body from an image block using a human posture estimation algorithm is as follows:

[0019] After horizontally connecting the ResNet network with the feature pyramid network, the image blocks in the key frame are processed to obtain the key feature map;

[0020] The position coordinates of the key points of the human body in the key feature map are predicted through four prediction heads;

[0021] Through post-processing, a set of human key point position coordinates closest to the center of the image block are accurately located and output.

[0022] In combination with the first aspect, further, the key points of the human body include left shoulder, right shoulder, left hip, right hip, left knee, right knee, left ankle and right ankle.

[0023] In combination with the first aspect, further, according to the position coordinates of the key points of the human body, a threshold method is used to determine whether the elderly in the image block at different visual angles has fallen, including:

[0024] When 10 -y 11 When <0, it is judged that the old man fell down facing the camera, where y 10 Indicates the ordinate of the midpoint between the two hips of the human body in the current frame image block, y 11 Indicates the ordinate of the midpoint of the two ankles of the human body in the current frame image block;

[0025] When 10 -y 11 <y min and When the old man falls sideways to the camera, y min represents the minimum vertical height between the midpoints of the two hips and the two ankles when the elderly walk upright, θ represents the deviation angle of the torso, (x9, y9) represents the position coordinates of the midpoint of the two shoulders of the human body in the current frame image block, x 10 Represents the horizontal coordinate of the midpoint between the two hips of the human body in the current frame image block;

[0026] when and When the old man falls with his back to the camera, (x5, y5) represents the position coordinates of the left knee of the human body in the current frame image block, (x6, y6) represents the position coordinates of the right knee of the human body in the current frame image block, (x7, y7) represents the position coordinates of the left ankle of the human body in the current frame image block, (x8, y8) represents the position coordinates of the right ankle of the human body in the current frame image block, and x 11 represents the horizontal coordinate of the midpoint of the two ankles of the human body in the current frame image block; based on the original video sequence collected, (x'5, y'5) represents the position coordinate of the left knee of the human body in the mth frame image from the current frame image block forward, (x'6, y'6) represents the position coordinate of the right knee of the human body in the mth frame image from the current frame image block forward, (x'7, y'7) represents the position coordinate of the left ankle of the human body in the mth frame image from the current frame image block forward, (x'8, y'8) represents the position coordinate of the right ankle of the human body in the mth frame image from the current frame image block forward, (x'9, y'9) represents the position coordinate of the midpoint of the two shoulders of the human body in the mth frame image from the current frame image block forward, (x' 10 ,y' 10 ) represents the position coordinates of the midpoint between the two hips of the human body in the mth frame image from the current frame image block forward, (x' 11 ,y' 11 ) represents the position coordinates of the midpoints of the two ankles of the human body in the mth frame image starting from the current frame image block.

[0027] Combined with the first aspect, further, the lightweight convolutional network adopts the ShuffleNet-V2 network.

[0028] In a second aspect, the present invention proposes a multi-view fall detection device based on spatiotemporal adaptation and human posture estimation, comprising:

[0029] The video stream acquisition module is used to acquire video stream segments through a camera to obtain a video sequence;

[0030] The first-level feature extraction module is used to preprocess the video sequence through a lightweight convolutional network to obtain a global feature map sequence;

[0031] The second-level feature extraction module is used to perform spatiotemporal adaptive processing on each global feature map in the global feature map sequence to obtain an image block containing the elderly's behavior and actions;

[0032] The third-level feature extraction module is used to obtain the position coordinates of the key points of the human body from the image block using the human posture estimation algorithm;

[0033] The fall judgment module is used to judge whether the elderly in the image block at different visual angles has fallen based on the position coordinates of the key points of the human body and the threshold method;

[0034] The alarm model is used to generate an alarm signal when N consecutive image blocks are judged as an elderly person falling down.

[0035] In a third aspect, the present invention proposes a multi-view fall detection device based on spatiotemporal adaptation and human posture estimation, including a processor and a storage medium;

[0036] The storage medium is used to store instructions;

[0037] The processor is used to operate according to the instructions to execute the steps of the method described in the first aspect of the present invention.

[0038] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect of the present invention.

[0039] The following advantages can be obtained by using the above technical means:

[0040] The present invention proposes a multi-view fall detection method, device and storage medium based on spatiotemporal adaptation and human posture estimation. The image blocks containing the elderly's behavior and actions in the key frames are selected from the original video sequence through spatiotemporal adaptive processing, which can remove redundant information in the video stream from the two dimensions of time and space. Not only does it not require frame-by-frame processing, but the spatial size of the image block is smaller than the original image, which can significantly reduce the computational overhead of the network model and ensure the real-time performance of the algorithm when running on an embedded device or mobile device. At the same time, the present invention uses a human posture estimation algorithm to obtain the coordinates of the key points of the human body from the image block. The human posture estimation algorithm uses a ResNet network with a large amount of calculation to improve the accuracy of the key point coordinates and achieve a good key point visualization effect. Finally, the present invention uses a threshold method to determine whether the person in the image block falls at different visual angles, which can effectively distinguish normal standing, squatting and lying postures, and accurately determine whether the elderly fall.

[0041] The present invention can reduce redundant calculations and lower the computational overhead of the model while ensuring the real-time and accuracy of elderly fall detection, and further optimizes the elderly fall detection technology based on intelligent video monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of the steps of a multi-view fall detection method based on spatiotemporal adaptation and human posture estimation according to the present invention;

[0043] Figure 2 Schematic diagram of the process of a multi-view fall detection method according to an embodiment of the present invention;

[0044] Figure 3 A schematic diagram of key points of a human body in an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of an elderly person walking upright in an embodiment of the present invention;

[0046] Figure 5 This is a schematic diagram of an old man falling down facing the camera in an embodiment of the present invention;

[0047] Figure 6 This is a schematic diagram of an elderly person falling down with his side facing the camera in an embodiment of the present invention;

[0048] Figure 7 This is a schematic diagram of an old man falling down with his back to the camera in an embodiment of the present invention;

[0049] Figure 8 Schematic diagram of the structure of a multi-view fall detection device based on spatiotemporal adaptation and human posture estimation in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The technical solution of the present invention is further described below in conjunction with the accompanying drawings:

[0051] The present invention proposes a multi-view fall detection method based on spatiotemporal adaptation and human posture estimation. Figure 1 , 2 As shown, the specific steps include:

[0052] Step A: Obtain video stream segments captured by the camera to obtain a video sequence.

[0053] Step B: Use the lightweight convolutional network ShuffleNet-V2 to quickly and roughly preprocess the video sequence, obtain the global feature information of each frame in the video sequence, and obtain a global feature map sequence.

[0054] Step C: perform spatiotemporal adaptive processing on each global feature map in the global feature map sequence, determine the key area of ​​the key frame in the video stream, and obtain an image block containing the elderly's behavior and actions.

[0055] Each global feature in the global feature map sequence is taken as input, and the image blocks containing the elderly's behavior actions in each key frame are output through spatiotemporal adaptive processing.

[0056] Spatiotemporal adaptation is a policy network that takes the cheap global feature map obtained in step B as input and integrates the information of all video frames so far. It uses the policy gradient method to optimize the goal of maximizing the discounted reward function. It uses the spatial adaptive reward function and the temporal adaptive reward function to process each global feature map in the global feature map sequence. The reward function value obtained is used to determine whether each frame of the video is processed and select the image block containing the elderly's behavior actions.

[0057] In the embodiment of the present invention, the expressions of the spatial adaptive reward function and the temporal adaptive reward function are as follows:

[0058]

[0059]

[0060] in, represents the spatially adaptive reward function, Represents the number of frames from the i-th video frame O i The sampled image blocks, express The softmax prediction of the image block containing the elderly’s behavior, represents the expectation obtained by sampling from a Gaussian distribution, R' i (B i |B1,...,B i-1 ) represents the time-adaptive reward function, B i represents the Bernoulli parameter of the i-th video frame output, P' iy (B1,...,B i-1 ,1) indicates B i = 1, softmax prediction, P' iy (B1,...,B i-1 ,0) indicates B i =0, λ is a predefined hyperparameter, L represents the length of the image block, ω represents the threshold for activating video frames, i=1,2,…,n, n is the number of video frames.

[0061] The expectation obtained by sampling the Gaussian distribution in formula (3) is used as the benchmark optimization loss function, which can encourage the model to select image blocks that can produce reliable predictions. Formula (4) compares the confidence increase effect on the label category with the computational overhead required for the image blocks in the video frame, and selects key frames through thresholds to avoid redundant calculations.

[0062] Step D: using a human body posture estimation algorithm to obtain the position coordinates of key points of the human body from the image block.

[0063] The human body posture estimation algorithm adopts the MoveNet algorithm. The specific operation of using the human body posture estimation algorithm to obtain the position coordinates of the key points of the human body from the image block is:

[0064] ResNet is used as a local convolutional network. After the ResNet network is horizontally connected with the feature pyramid network, the image blocks in the key frame are processed to output a high-resolution key feature map with rich semantic information. The position coordinates of the human key points in the key feature map are predicted through four prediction heads to roughly locate the range of the key points. The key point coordinates are refined through post-processing weighting, and a set of human key point position coordinates closest to the center of the image block are output.

[0065] In the embodiment of the present invention, Figure 3 As shown, there are 8 key points of the human body, namely 1: left shoulder, 2: right shoulder, 3: left hip, 4: right hip, 5: left knee, 6: right knee, 7: left ankle, 8: right ankle.

[0066] Step E: Based on the position coordinates of the key points of the human body, the threshold method is used to determine whether the elderly person in the image block at different visual angles has fallen. The specific operations are as follows:

[0067] Step E01: extract the first 8 frames of images after the elderly person enters the video detection area from the video sequence, and calculate the minimum vertical height between the midpoints of the two hips and the midpoints of the two ankles in the 8 frames, denoted as y min ,like Figure 4 As shown, y min It represents the minimum vertical height of the legs of an elderly person when walking upright normally, and is an important threshold in the present invention.

[0068] Step E02, calculate the position coordinates of the midpoints of the shoulders, hips and ankles of the human body in the current frame image block according to the position coordinates of the key points of the human body. The current frame here refers to the frame position of an image block in the acquired original video sequence.

[0069] Step E03, when y 10 -y 11 When <0, it is judged that the old man fell down facing the camera, where y 10 Indicates the ordinate of the midpoint between the two hips of the human body in the current frame image block, y 11 Indicates the vertical coordinate of the midpoint of the two ankles of the human body in the current frame image block. Figure 5 As shown, y 10 -y 11 <0 means that the coordinate height of the midpoints of the two ankles in the video frame image block is higher than the coordinate height of the midpoints of the two hips.

[0070] Step E04: When y 10 -y 11 <y min and When the old man falls sideways to the camera, y minrepresents the minimum vertical height between the midpoints of the two hips and the two ankles when the elderly walk upright, θ represents the deviation angle of the torso, (x9, y9) represents the position coordinates of the midpoint of the two shoulders of the human body in the current frame image block, x 10 Indicates the horizontal coordinate of the midpoint between the two hips of the human body in the current frame image block. Figure 6 As shown, y 10 -y 11 <y min Indicates that the vertical height between the midpoints of the two hips and the midpoints of the two ankles in the video frame image block is less than y min , the elderly are in an abnormal walking or standing state, that is, the elderly may be lying down or squatting. In order to exclude the normal squatting posture, the present invention uses the angle of the trunk for judgment. If the vertical angle of the trunk is too large (that is, θ < 45°), it is a fall.

[0071] Step E05: and When the old man falls with his back to the camera, (x5, y5) represents the position coordinates of the left knee of the human body in the current frame image block, (x6, y6) represents the position coordinates of the right knee of the human body in the current frame image block, (x7, y7) represents the position coordinates of the left ankle of the human body in the current frame image block, (x8, y8) represents the position coordinates of the right ankle of the human body in the current frame image block, and x 11 represents the horizontal coordinate of the midpoint of the two ankles of the human body in the current frame image block; based on the original video sequence collected, (x'5, y'5) represents the position coordinate of the left knee of the human body in the mth frame image from the current frame image block forward, (x'6, y'6) represents the position coordinate of the right knee of the human body in the mth frame image from the current frame image block forward, (x'7, y'7) represents the position coordinate of the left ankle of the human body in the mth frame image from the current frame image block forward, (x'8, y'8) represents the position coordinate of the right ankle of the human body in the mth frame image from the current frame image block forward, (x'9, y'9) represents the position coordinate of the midpoint of the two shoulders of the human body in the mth frame image from the current frame image block forward, (x' 10 ,y' 10 ) represents the position coordinates of the midpoint between the two hips of the human body in the mth frame image from the current frame image block forward, (x' 11 ,y' 11 ) represents the position coordinates of the midpoints of the two ankles of the human body in the mth frame image from the current frame image block forward. Figure 7 As shown in the figure, the above condition means that when the elderly person moves a short range within the m-frame time interval, the length of the body trunk and the length of the two calves in the current frame are both reduced compared to the m-frame forward. In order to prevent a small range of position deviation when falling, if the coordinates of the midpoints of the two ankles in the current frame (x 11 ,y 11) is located at the midpoint coordinates of the two ankles before the m frame (x' 11 , y' 11 ) is the center of the circle, In the circular range with a radius of , determine whether the length of the body trunk and limbs has shrunk. If they have shrunk, it is judged as a fall.

[0072] In the embodiment of the present invention, m=8, that is, the position coordinates of the key points of the human body in the current frame in the video sequence are compared with the position coordinates of the key points of the human body in the 8th frame forward to perform fall judgment.

[0073] Step F: When N consecutive image blocks are judged as the elderly falling down, an alarm signal is generated. In the embodiment of the present invention, N=3.

[0074] Based on the method of the present invention, the present invention also proposes a multi-view fall detection device based on spatiotemporal adaptation and human posture estimation, such as Figure 8 As shown, it mainly includes a video stream acquisition module, a first-level feature extraction module, a second-level feature extraction module, a third-level feature extraction module, a fall judgment module and an alarm module.

[0075] The video stream acquisition module is mainly used to acquire video stream segments through the camera to obtain a video sequence.

[0076] The first-level feature extraction module is mainly used to preprocess the video sequence through a lightweight convolutional network to obtain a global feature map sequence. The lightweight convolutional network adopts the ShuffleNet-V2 network.

[0077] The second-level feature extraction module is mainly used to perform spatiotemporal adaptive processing on each global feature map in the global feature map sequence to obtain an image block containing the elderly's behavior and actions. The operation of the second-level feature extraction module is consistent with step C of the method of the present invention.

[0078] The third-level feature extraction module is mainly used to obtain the position coordinates of the key points of the human body from the image block using the human posture estimation algorithm. The human posture estimation algorithm adopts MoveNet. The operation of the third-level feature extraction module is consistent with step D of the method of the present invention.

[0079] The fall judgment module is mainly used to judge whether the elderly person in the image block falls down at different visual angles based on the position coordinates of the key points of the human body and the threshold method. The fall judgment module can judge whether the elderly person falls down facing the camera direction, sideways to the camera direction or with his back to the camera direction. The specific operation is consistent with step E of the method of the present invention.

[0080] The alarm model is mainly used to generate an alarm signal when N consecutive image blocks are judged as an elderly person falling down, where N=3.

[0081] Based on the method of the present invention, the present invention also proposes a multi-perspective fall detection device based on spatiotemporal adaptation and human posture estimation, including a processor and a storage medium; wherein the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the method of the present invention.

[0082] Based on the method of the present invention, the present invention further proposes a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the method of the present invention are implemented.

[0083] Compared with the prior art, the present invention first selects the image blocks containing the elderly's behavior and actions in the key frames from the original video sequence through spatiotemporal adaptive processing, which can remove redundant information in the video stream from the two dimensions of time and space. Not only does it not need frame-by-frame processing, but the spatial size of the image block is smaller than the original image, which can significantly reduce the computational overhead of the network model and ensure the real-time performance of the algorithm when running on an embedded device or mobile device. Secondly, the present invention uses a human posture estimation algorithm to obtain the coordinates of the key points of the human body from the image block. The human posture estimation algorithm uses a ResNet network with a large amount of calculation to improve the accuracy of the key point coordinates and achieve a good key point visualization effect. Finally, the present invention uses a threshold method to determine whether the person in the image block falls under different visual angles. First, the vertical height of the person's legs is used to determine whether the elderly is in a normal walking, standing or squatting state, and then the trunk inclination angle is used to accurately determine whether the elderly have fallen. When the distance from the camera is a certain distance, the length of the body trunk and limbs of the person at the same position in the camera's perspective is reduced when the person lies down with his back to the camera compared to when standing normally, which can effectively distinguish normal standing, squatting and lying postures, and accurately determine whether the elderly have fallen.

[0084] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0085] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0086] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0087] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0088] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multi-view fall detection method based on spatiotemporal adaptation and human posture estimation, characterized in that: The steps include: Get the video stream segments captured by the camera to obtain a video sequence; Preprocess the video sequence through a lightweight convolutional network to obtain a global feature map sequence; Performing spatiotemporal adaptive processing on each global feature map in the global feature map sequence to obtain an image block containing the elderly's behavior and actions; The human body posture estimation algorithm is used to obtain the position coordinates of the key points of the human body from the image block; According to the position coordinates of the key points of the human body, the threshold method is used to determine whether the elderly in the image block at different visual angles have fallen; When N consecutive image blocks are judged as an elderly person falling, an alarm signal is generated; Among them, according to the position coordinates of the key points of the human body, the threshold method is used to determine whether the elderly in the image block at different visual angles has fallen, including: when When the old man falls down facing the camera, Indicates the vertical coordinate of the midpoint between the two hips of the human body in the current frame image block, Indicates the ordinate of the midpoint of the two ankles of the human body in the current frame image block; when and When the old man falls with his side facing the camera, It indicates the minimum vertical height between the midpoint of the two hips and the midpoint of the two ankles when the elderly walk upright. represents the torso deviation angle, Indicates the position coordinates of the midpoints of the shoulders of the human body in the current frame image block. Represents the horizontal coordinate of the midpoint between the two hips of the human body in the current frame image block; when , , and When the old man falls with his back to the camera, Indicates the position coordinates of the left knee of the human body in the current frame image block, Indicates the position coordinates of the right knee of the human body in the current frame image block, Indicates the position coordinates of the left ankle of the human body in the current frame image block, Indicates the position coordinates of the right ankle of the human body in the current frame image block. Indicates the horizontal coordinate of the midpoint of the two ankles of the human body in the current frame image block; based on the original video sequence collected, Indicates the position coordinates of the left knee of the human body in the mth frame image starting from the current frame image block. Indicates the position coordinates of the right knee of the human body in the mth frame image from the current frame image block forward. Indicates the position coordinates of the left ankle of the human body in the mth frame image from the current frame image block forward, Indicates the position coordinates of the right ankle of the human body in the mth frame image from the current frame image block forward, Indicates the position coordinates of the midpoint of the shoulders of the human body in the mth frame image from the current frame image block forward. Indicates the position coordinates of the midpoint between the two hips of the human body in the mth frame image from the current frame image block forward. Represents the position coordinates of the midpoints of the two ankles of the human body in the mth frame image starting from the current frame image block.

2. The multi-view fall detection method based on spatiotemporal adaptation and human posture estimation according to claim 1 is characterized in that: Each global feature map in the global feature map sequence is processed using the spatial adaptive reward function and the temporal adaptive reward function to select the image block containing the elderly's behavior. The expressions of the spatial adaptive reward function and the temporal adaptive reward function are as follows: ； ； in, represents the spatially adaptive reward function, Indicates the number of frames from the i-th video The sampled image blocks, express The softmax prediction of the image block containing the elderly’s behavior, represents the expectation obtained by sampling from a Gaussian distribution, represents the time-adaptive reward function, represents the Bernoulli parameter of the i-th video frame output, Indicates B i =1 when softmax prediction, Indicates B i =0 when softmax prediction, is a predefined hyperparameter, L represents the length of the image block, represents the threshold for activating video frames, , n is the number of video frames.

3. The multi-view fall detection method based on spatiotemporal adaptation and human posture estimation according to claim 1, characterized in that: The human body posture estimation algorithm adopts the MoveNet algorithm.

4. The multi-view fall detection method based on spatiotemporal adaptation and human posture estimation according to claim 3 is characterized in that: The method of obtaining the position coordinates of the key points of the human body from the image block using the human posture estimation algorithm is: After horizontally connecting the ResNet network with the feature pyramid network, the image blocks in the key frame are processed to obtain the key feature map; The position coordinates of the key points of the human body in the key feature map are predicted through four prediction heads; Through post-processing, a set of human key point position coordinates closest to the center of the image block are accurately located and output.

5. The multi-view fall detection method based on spatiotemporal adaptation and human posture estimation according to claim 1, characterized in that: The key points of the human body include left shoulder, right shoulder, left hip, right hip, left knee, right knee, left ankle and right ankle.

6. The multi-view fall detection method based on spatiotemporal adaptation and human posture estimation according to claim 1, characterized in that: The lightweight convolutional network uses the ShuffleNet-V2 network.

7. A multi-view fall detection device based on spatiotemporal adaptation and human posture estimation, characterized in that: include: The video stream acquisition module is used to acquire video stream segments through a camera to obtain a video sequence; The first-level feature extraction module is used to preprocess the video sequence through a lightweight convolutional network to obtain a global feature map sequence; The second-level feature extraction module is used to perform spatiotemporal adaptive processing on each global feature map in the global feature map sequence to obtain an image block containing the elderly's behavior and actions; The third-level feature extraction module is used to obtain the position coordinates of the key points of the human body from the image block using the human posture estimation algorithm; The fall judgment module is used to judge whether the elderly in the image block at different visual angles has fallen based on the position coordinates of the key points of the human body and the threshold method; An alarm model is used to generate an alarm signal when N consecutive image blocks are judged as an elderly person falling down; Among them, the fall judgment module is specifically used for: when When the old man falls down facing the camera, Indicates the vertical coordinate of the midpoint between the two hips of the human body in the current frame image block, Indicates the ordinate of the midpoint of the two ankles of the human body in the current frame image block; when and When the old man falls with his side facing the camera, It indicates the minimum vertical height between the midpoint of the two hips and the midpoint of the two ankles when the elderly walk upright. represents the torso deviation angle, Indicates the position coordinates of the midpoints of the shoulders of the human body in the current frame image block. Represents the horizontal coordinate of the midpoint between the two hips of the human body in the current frame image block; when , , and When the old man falls with his back to the camera, Indicates the position coordinates of the left knee of the human body in the current frame image block, Indicates the position coordinates of the right knee of the human body in the current frame image block, Indicates the position coordinates of the left ankle of the human body in the current frame image block, Indicates the position coordinates of the right ankle of the human body in the current frame image block. Indicates the horizontal coordinate of the midpoint of the two ankles of the human body in the current frame image block; based on the original video sequence collected, Indicates the position coordinates of the left knee of the human body in the mth frame image starting from the current frame image block. Indicates the position coordinates of the right knee of the human body in the mth frame image from the current frame image block forward. Indicates the position coordinates of the left ankle of the human body in the mth frame image from the current frame image block forward, Indicates the position coordinates of the right ankle of the human body in the mth frame image from the current frame image block forward, Indicates the position coordinates of the midpoint of the shoulders of the human body in the mth frame image from the current frame image block forward. Indicates the position coordinates of the midpoint between the two hips of the human body in the mth frame image from the current frame image block forward. Represents the position coordinates of the midpoints of the two ankles of the human body in the mth frame image starting from the current frame image block.

8. A multi-view fall detection device based on spatiotemporal adaptation and human posture estimation, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

Images