Fall detection method and apparatus for a companion robot
By employing a two-stage approach of coarse screening and fine screening in the fall detection method, and utilizing a posture detection model and a multi-head self-attention module of an improved Transformer model to calculate the posture difference matrix, the problem of insufficient capture of posture sequence features by traditional models is solved, and rapid and accurate detection of fall behavior in the elderly is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING UNIV OF CIVIL ENG & ARCHITECTURE
- Filing Date
- 2026-01-27
- Publication Date
- 2026-06-02
AI Technical Summary
In existing fall detection methods, traditional sequence models are insufficient in capturing the long-term dependencies and differential features of posture sequences, resulting in a high false detection rate for falls and similar behaviors such as sitting down or picking up objects, making it difficult to achieve accurate fall detection.
A two-stage method of coarse screening and fine inspection is adopted. The position coordinates of key points of the human body are extracted by the posture detection model, the posture difference matrix is calculated, and it is integrated into the multi-head self-attention module of the improved Transformer model to enhance difference perception and improve the accuracy of fall detection.
It improves the accuracy of fall detection, reduces the false alarm rate, and adapts to the movement speed of different individuals, especially the slow falling movements of the elderly, to ensure timely detection of falls.
Smart Images
Figure CN122135258A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of computer vision and behavior recognition technology, and in particular to a fall detection method and apparatus for companion robots. Background Technology
[0002] With the increasing aging of the population, the application of companion robots in the daily monitoring of the elderly is becoming more and more widespread. Fall detection, as one of the core functions, is directly related to the safety of the elderly in terms of its real-time performance and accuracy—if a fall is not detected and treated in time, it can easily lead to serious consequences. Currently, non-contact detection is achieved by acquiring image data through visual cameras and using computer vision algorithms. This method does not require the active cooperation of the person being monitored and better meets the actual application needs of companionship scenarios, thus becoming the mainstream research direction.
[0003] In related technologies, fall detection is performed through a two-stage process of coarse screening and fine detection. The coarse screening stage extracts human posture features and determines whether a fall is suspected based on these features. For suspected falls, the fine detection stage further analyzes the posture sequence using traditional sequence models such as CNN (Convolutional Neural Network) or RNN (Recurrent Neural Network) to determine whether it is a genuine fall. However, the traditional sequence models such as CNN and RNN used in the fine detection stage are insufficient at capturing long-term dependencies and differential features in posture sequences, making it difficult to effectively uncover the differential features between falls and similar behaviors such as sitting down or picking up objects, resulting in a high false detection rate. Summary of the Invention
[0004] To address the aforementioned technical problems, this application provides a fall detection method, device, electronic device, storage medium, and computer program product for companion robots.
[0005] According to a first aspect of this application, a fall detection method for a companion robot is provided, comprising: Real-time acquisition of video data of monitored individuals, and extraction of video frames at a fixed frame rate; The pose detection model is used to extract the position coordinates of human key points from multiple consecutive video frames, including the current video frame. Based on the position coordinates of key points of the human body in the multiple video frames, it is determined whether the monitored person is suspected of falling. If the monitored person is suspected of falling, then the posture difference matrix between the multiple video frames is calculated based on the position coordinates of key points of the human body in the multiple video frames; The position coordinates of key human body points in the multiple video frames are input into the improved Transformer model, and the fall detection results are output. The improved Transformer model's multi-head self-attention module is used to transform the position coordinates of human keypoints in the multiple video frames through different linear transformations to obtain the original query matrix, the original key matrix, and the original value matrix, respectively; to generate a target query matrix based on the original query matrix and the pose difference matrix; to generate a target key matrix based on the original key matrix and the pose difference matrix; and to determine the self-attention weights based on the target query matrix, the target key matrix, and the original value matrix.
[0006] Optionally, the multi-head self-attention module generates a target query matrix based on the original query matrix and the pose difference matrix, and generates a target key matrix based on the original key matrix and the pose difference matrix through the following steps: The attitude difference matrix D is symmetrically normalized to obtain the target difference matrix. ; According to the formula: Determine the target query matrix ; According to the formula: Determine the target key matrix ; Where Q represents the original query matrix, K represents the original key matrix, and I represents the identity matrix. This represents the modulation intensity coefficient.
[0007] Optionally, determining whether the monitored person is suspected of falling based on the position coordinates of key human body points in the plurality of video frames includes: Based on the position coordinates of key points of the human body in each video frame, determine multiple posture features of the human body in the video frame, and determine the coarse screening judgment value based on the multiple posture features. The speed of movement of the monitored person is determined based on the position coordinates of key points of the human body in each video frame. Based on the movement speed, a corresponding fall threshold is determined; wherein, the movement speed is positively correlated with the fall threshold; If the coarse screening value is greater than the fall threshold, it is determined to be a suspected fall; if the coarse screening value is less than or equal to the fall threshold, it is determined to be normal behavior.
[0008] Optionally, determining the corresponding fall threshold based on the action speed includes: According to the following formula: Determine the fall threshold ; in, Here, f represents the baseline fall threshold, f represents a pre-set constant, and v represents the movement speed of the monitored individual. This indicates the standard speed of movement.
[0009] Optionally, calculating the pose difference matrix between the multiple video frames based on the position coordinates of human key points in the multiple video frames includes: Assume the coordinates of the m-th human body keypoint in the i-th video frame are... The coordinates of the m-th human body keypoint in the j-th video frame are: ; Then, according to the following formula: ; Determine the elements in the attitude difference matrix D .
[0010] According to a second aspect of this application, a fall detection device for a companion robot is provided, comprising: The video frame extraction module is used to collect video data of the monitored person in real time and extract video frames at a fixed frame rate. The key point location coordinate extraction module is used to extract the location coordinates of human key points from multiple consecutive video frames containing the current video frame using a pose detection model. The suspected fall detection module is used to determine whether the monitored person is suspected of falling based on the position coordinates of key points of the human body in the multiple video frames. The posture difference matrix determination module is used to calculate the posture difference matrix between the multiple video frames based on the position coordinates of key points of the human body in the multiple video frames if the monitored person is suspected of falling. The fall detection result output module is used to input the position coordinates of key human body points in the multiple video frames into the improved Transformer model and output the fall detection result. The improved Transformer model's multi-head self-attention module is used to transform the position coordinates of human keypoints in the multiple video frames through different linear transformations to obtain the original query matrix, the original key matrix, and the original value matrix, respectively; to generate a target query matrix based on the original query matrix and the pose difference matrix; to generate a target key matrix based on the original key matrix and the pose difference matrix; and to determine the self-attention weights based on the target query matrix, the target key matrix, and the original value matrix.
[0011] Optionally, the multi-head self-attention module generates a target query matrix based on the original query matrix and the pose difference matrix, and generates a target key matrix based on the original key matrix and the pose difference matrix through the following steps: The attitude difference matrix D is symmetrically normalized to obtain the target difference matrix. ; According to the formula: Determine the target query matrix ; According to the formula: Determine the target key matrix ; Where Q represents the original query matrix, K represents the original key matrix, and I represents the identity matrix. This represents the modulation intensity coefficient.
[0012] Optionally, the suspected fall detection module is specifically used to determine multiple posture features of the human body in each video frame based on the position coordinates of key points in the human body; determine a coarse screening value based on the multiple posture features; determine the movement speed of the monitored person based on the position coordinates of key points in each video frame; determine a corresponding fall threshold based on the movement speed; the movement speed is positively correlated with the fall threshold; if the coarse screening value is greater than the fall threshold, it is determined to be a suspected fall; if the coarse screening value is less than or equal to the fall threshold, it is determined to be normal behavior.
[0013] Optionally, the suspected fall detection module is specifically used to determine the corresponding fall threshold based on the action speed in the following manner: According to the following formula: Determine the fall threshold ; in, Here, f represents the baseline fall threshold, f represents a pre-set constant, and v represents the movement speed of the monitored individual. This indicates the standard speed of movement.
[0014] Optionally, the posture difference matrix determination module is specifically used to determine the position coordinates of the m-th human body key points in the i-th video frame if the monitored person is suspected of falling. The coordinates of the m-th human body keypoint in the j-th video frame are: ; Then, according to the following formula: ; Determine the elements in the attitude difference matrix D .
[0015] According to a third aspect of this application, an electronic device is provided, comprising: a processor configured to execute a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the method described in the first aspect.
[0016] According to a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.
[0017] According to a fifth aspect of this application, a computer program product is provided that, when the computer program product is run on a computer, causes the computer to perform the method described in the first aspect.
[0018] The technical solution provided in this application has the following advantages compared with the prior art: This application employs a two-stage approach of coarse screening and fine inspection for fall detection. In the fine inspection stage, a pose difference matrix is calculated between multiple video frames. This matrix is then integrated into the self-attention weight calculation of the multi-head self-attention module in the Transformer model, strengthening the attention between video frames with significant differences. Furthermore, the pose difference matrix is directly used in the generation process of the query matrix and key matrix, ensuring that the final target query matrix and target key matrix carry pose difference information. This strengthens difference perception from the source, thereby improving the accuracy of fall detection. Attached Figure Description
[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 This is a flowchart of a fall detection method for a companion robot in an embodiment of this application; Figure 2 This is a flowchart illustrating a method for determining whether a person under supervision is suspected of having fallen, as described in this application embodiment. Figure 3 This is a schematic diagram of a fall detection device for a companion robot in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device in an embodiment of this application. Detailed Implementation
[0022] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.
[0023] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.
[0024] To address the technical problem of low accuracy in existing fall detection methods, this application provides a fall detection method with a two-stage progressive architecture of "coarse screening-fine inspection". This method can be used in companion robots and is suitable for rapid and accurate detection of elderly people's fall behavior by relying solely on video images acquired by a vision camera. It can be widely applied in scenarios such as home care and nursing home monitoring.
[0025] See Figure 1 , Figure 1 This is a flowchart of a fall detection method for a companion robot in this application embodiment, which may include the following steps: Step S102: Collect video data of the monitored person in real time and extract video frames at a fixed frame rate.
[0026] The companion robot uses a vision camera to collect video data of the monitored person in the monitored area in real time, and extracts single frame images at a fixed frame rate (e.g., 30fps).
[0027] Step S104: Use a pose detection model to extract the position coordinates of human key points from multiple consecutive video frames containing the current video frame.
[0028] For each extracted video frame, preprocessing can be performed, such as noise reduction (Gaussian filtering), illumination enhancement (adaptive histogram equalization), and region of interest (ROI) extraction (retaining only the areas where the human body may appear), reducing the computational load of subsequent modules. Then, a lightweight pose detection model (e.g., MobileNet+OpenPose) is used to extract the position coordinates of 17 human keypoints (head, neck, shoulder, elbow, wrist, hip, knee, and ankle) from the preprocessed image.
[0029] Step S106: Based on the position coordinates of key points of the human body in multiple video frames, determine whether the monitored person is suspected of falling.
[0030] For the initial screening stage, various posture features related to fall detection can be extracted based on the position coordinates of 17 key human body points. These posture features can then be used to determine whether the monitored individual is suspected of having fallen. (See also...) Figure 2 , Figure 2 This is a flowchart illustrating a method for determining whether a person under supervision is suspected of having fallen, as described in this application embodiment. The method includes the following steps: Step S202: Based on the position coordinates of key points of the human body in each video frame, determine multiple pose features of the human body in the video frame, and determine the coarse screening judgment value based on the multiple pose features.
[0031] Based on the position coordinates of 17 key human body points, various posture features can be determined, including: The relative height of the center of gravity h is calculated by weighting the coordinates of the human body's center of gravity from key points (head weight 0.1, torso weight 0.5, limb weight 0.4). The relative height of the center of gravity h is the ratio of the y-coordinate of the center of gravity to the image height (the y-axis is positive downwards). When falling, h will increase rapidly.
[0032] The ratio of upper limb length to lower limb length, k, is the ratio of the total length of the upper limb (shoulder-elbow-wrist) to the total length of the lower limb (hip-knee-ankle). When falling, the limbs are mostly in an extended state, and k is relatively stable. However, when sitting down or picking something up, k fluctuates significantly with limb flexion. k(t) is the characteristic limb extension coefficient at time t.
[0033] trunk tilt angle The trunk baseline is formed by the three points: the neck joint, the right hip, and the left hip. The trunk tilt angle is... The angle between this baseline and the horizontal direction is the angle at which the fall occurs. It will usually quickly exceed the preset angle (e.g., 60°).
[0034] The rate of change of the relative height of the center of gravity, *maxh*, can be calculated based on the relative height *h* of the center of gravity, reflecting the speed of descent of the center of gravity. During a fall, the relative height *h* of the center of gravity decreases, and the rate of change of this relative height is positive and relatively large. This can be calculated using the following formula: ; Calculate the rate of change of the relative height of the center of gravity at time t. ;in, ; Let be the time interval t and (t-(n-1)). The relative height difference of the center of gravity at time ) y(t) is the frame interval, n is the number of video frames; h(t) is the relative height of the centroid at time t, h(t) = y(t) / H, y(t) is the y-coordinate of the centroid, and H is the image height.
[0035] The limb stability index maxk can be calculated based on the ratio of upper to lower limb length k. When falling, limbs extend, and maxk is stable; when sitting down / picking something up, maxk fluctuates more significantly. This can be done using the following formula: ; Calculate the limb stability index at time t ;in, ; The difference between the ratio of upper and lower limb lengths at time t and at time (t-(n-1) 𝛥𝑡); when falling. Approaching 1, Nearly half.
[0036] For trunk tilt angle The degree of trunk tilt can be quantified by sinθ(t). Since the trunk tilts significantly when falling, sinθ(t) is close to 1.
[0037] Based on the rate of change of the relative height of the center of gravity, maxh(t), the limb stability index maxk(t), and the quantitative index of trunk tilt, sinθ(t), the following formula can be constructed: ; S(t) is the coarse screening judgment value at time t. a, b, and c are all weighting coefficients used to balance the importance of the three posture features. The sum of a, b, and c is 1. For example, a=0.4, b=0.4, and c=0.2. The larger S(t) is, the higher the probability of a suspected fall.
[0038] Step S204: Determine the movement speed of the monitored person based on the position coordinates of key points of the human body in each video frame, and determine the corresponding fall threshold based on the movement speed.
[0039] In related technologies, a fixed fall threshold is used. This method does not take into account the slow movements and small changes in posture of the elderly, resulting in a high rate of missed detection for slow falls. To solve this problem, this application adopts a dynamic fall threshold method to improve the accuracy of fall detection. Specifically, the movement speed can be calculated based on the position coordinates of key points on the human body. Then, the fall threshold is adjusted in real time according to the movement speed. For example, the threshold can be lowered for slow movements, thereby effectively avoiding missed detections.
[0040] For example, the center of gravity of the human body in a video frame can be determined based on the coordinates of the key points of the human body, and the motion speed v can be determined based on the displacement of the center of gravity of the human body in adjacent video frames, expressed as the formula: v=Δh / Δt, where Δh is the displacement of the center of gravity of the human body in adjacent video frames.
[0041] It's understandable that the slower the movement, the lower the fall threshold should be; that is, movement speed and fall threshold are positively correlated. This helps avoid missing some cases. Alternatively, the following formula can be used: Determine the fall threshold ; in, This represents the baseline fall threshold, for example, 0.6; f represents a pre-set constant, for example, 0.3; and v represents the movement speed of the monitored individual. This indicates the standard motion speed, for example, 0.1 pixels per frame. This can also be confirmed through calibration when the companion robot is put into use. The basic fall threshold is determined using the motion speed V. Fine-tuning is performed to obtain a dynamic fall threshold to adapt to different individuals and improve the accuracy of fall detection.
[0042] Step S206: Determine whether the coarse screening judgment value is greater than the fall threshold.
[0043] If S(t) is greater than the fall threshold, then proceed to step S208; if S(t) is less than or equal to the fall threshold, then proceed to step S210.
[0044] Step S208: It is determined to be a suspected fall.
[0045] Step S210: Determined to be normal behavior.
[0046] If a fall is suspected, step S108 is executed, which triggers the subsequent accurate identification process; otherwise, no action is taken.
[0047] Step S108: Calculate the pose difference matrix between multiple video frames based on the position coordinates of key human body points in multiple video frames.
[0048] Falling down, sitting down, and picking something up are similar behaviors. Traditional Transformer models' self-attention mechanisms distribute weights equally across all sequence positions, making it difficult to highlight the differentiated features of similar behaviors. This application improves the Multi-Head Self-Attention (MSA) module of the Transformer model by introducing a "behavior difference-aware factor" to strengthen the attention weights for regions with drastic pose changes. Specifically, the "behavior difference-aware factor" is a pose difference matrix between multiple video frames, where each element represents the difference between the corresponding two video frames. Assuming the number of video frames is n, then the pose difference matrix... .
[0049] Alternatively, the pose difference matrix between multiple video frames can be calculated in the following manner: Assume the coordinates of the m-th human body keypoint in the i-th video frame are... The coordinates of the m-th human body keypoint in the j-th video frame are: ; Then, according to the following formula: ; Determine the elements in the attitude difference matrix D The greater the difference between the i-th video frame and the j-th video frame, the better. The larger the value.
[0050] Step S110: Input the position coordinates of human key points in multiple video frames into the improved Transformer model and output the fall detection results.
[0051] The improved Transformer model's multi-head self-attention module is used to transform the position coordinates of human keypoints in multiple video frames through different linear transformations to obtain the original query matrix, original key matrix, and original value matrix, respectively. A target query matrix is generated based on the original query matrix and pose difference matrix, and a target key matrix is generated based on the original key matrix and pose difference matrix. Self-attention weights are determined based on the target query matrix, target key matrix, and original value matrix.
[0052] Optionally, in the multi-head self-attention module, the pose difference matrix D can be symmetrically normalized first to obtain the target difference matrix. To ensure The symmetry of can be expressed as: = (D + D ^T ) / 2. Then through Modulation target query matrix and target key matrix The generation of . For example, it can be based on the formula: Determine the target query matrix According to the formula: Determine the target key matrix ; Where Q represents the original query matrix, K represents the original key matrix, and I represents the identity matrix. This represents the modulation intensity coefficient, which can be, for example, 0.3.
[0053] It can be seen that by directly involving the pose difference matrix D in the generation process of the query matrix and the key matrix, the query matrix and the key matrix carry pose difference information, thereby strengthening the difference perception from the source.
[0054] The fall detection method for companion robots in this application employs a two-stage approach: coarse screening and fine screening. In the coarse screening stage, the movement speed of the monitored individual is calculated in real time, and the fall threshold is adaptively adjusted based on the movement speed to accommodate different individuals. For example, considering the slow movements and small posture changes of elderly individuals, the fall threshold can be lowered to avoid missing slow falls. In the fine screening stage, a posture difference matrix is calculated between multiple video frames. This matrix is then integrated into the self-attention weight calculation of the multi-head self-attention module in the Transformer model, strengthening attention between video frames with significant differences. Furthermore, the posture difference matrix is directly used in the generation process of the query matrix and key matrix, ensuring that the final target query matrix and target key matrix carry posture difference information, thus enhancing difference perception from the source and improving the accuracy of fall detection.
[0055] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.
[0056] This application also provides a fall detection device for a companion robot, see [link to relevant documentation]. Figure 3 The fall detection device 300 for companion robots includes: The video frame extraction module 302 is used to collect video data of the monitored person in real time and extract video frames at a fixed frame rate. The key point location coordinate extraction module 304 is used to extract the location coordinates of human key points from multiple consecutive video frames containing the current video frame using a pose detection model. The suspected fall detection module 306 is used to determine whether the monitored person is suspected of falling based on the position coordinates of key points of the human body in multiple video frames. The posture difference matrix determination module 308 is used to calculate the posture difference matrix between multiple video frames based on the position coordinates of key points of the human body in multiple video frames if the monitored person is suspected of falling. The fall detection result output module 310 is used to input the position coordinates of human key points in multiple video frames into the improved Transformer model and output the fall detection result. The improved Transformer model's multi-head self-attention module is used to transform the position coordinates of human keypoints in multiple video frames through different linear transformations to obtain the original query matrix, original key matrix, and original value matrix, respectively; the target query matrix is generated based on the original query matrix and pose difference matrix, and the target key matrix is generated based on the original key matrix and pose difference matrix; the self-attention weights are determined based on the target query matrix, target key matrix, and original value matrix.
[0057] Optionally, the multi-head self-attention module generates the target query matrix based on the original query matrix and the pose difference matrix, and generates the target key matrix based on the original key matrix and the pose difference matrix through the following steps: The attitude difference matrix D is symmetrically normalized to obtain the target difference matrix. ; According to the formula: Determine the target query matrix ; According to the formula: Determine the target key matrix ; Where Q represents the original query matrix, K represents the original key matrix, and I represents the identity matrix. This represents the modulation intensity coefficient.
[0058] Optionally, the suspected fall detection module 306 is specifically used to determine multiple posture features of the human body in each video frame based on the position coordinates of key points of the human body in each video frame; determine a coarse screening value based on the multiple posture features; determine the movement speed of the monitored person based on the position coordinates of key points of the human body in each video frame; determine the corresponding fall threshold based on the movement speed; the movement speed is positively correlated with the fall threshold; if the coarse screening value is greater than the fall threshold, it is determined to be a suspected fall; if the coarse screening value is less than or equal to the fall threshold, it is determined to be normal behavior.
[0059] Optionally, the suspected fall detection module 306 is specifically used to determine the corresponding fall threshold based on the speed of the movement in the following manner: According to the following formula: Determine the fall threshold ; in, Here, f represents the baseline fall threshold, f represents a pre-set constant, and v represents the movement speed of the monitored individual. This indicates the standard speed of movement.
[0060] Optionally, the posture difference matrix determination module is specifically used to determine the position coordinates of the m-th key points of the human body in the i-th video frame if the monitored person is suspected of falling. The coordinates of the m-th human body keypoint in the j-th video frame are: ; Then, according to the following formula: ; Determine the elements in the attitude difference matrix D .
[0061] The specific details of each module or unit in the above-mentioned device have been described in detail in the corresponding methods, so they will not be repeated here.
[0062] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.
[0063] This application also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the above-described fall detection method for a companion robot.
[0064] Reference Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device in an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.
[0065] like Figure 4 As shown, the electronic device may include: a processor 402, a communication interface 404, a memory 406, and a communication bus 408.
[0066] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408.
[0067] Communication interface 404 is used to communicate with other electronic devices or servers.
[0068] The processor 402 is used to execute program 410, specifically the relevant steps in the above method embodiments.
[0069] Specifically, program 410 may include program code that includes computer operation instructions.
[0070] Processor 402 may be a central processing unit, a specific integrated circuit, or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.
[0071] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.
[0072] Specifically, program 410 can be used to cause processor 402 to execute the steps in the above embodiment of the fall detection method for companion robots.
[0073] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.
[0074] In this embodiment of the application, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the above-described fall detection method for a companion robot.
[0075] It should be noted that the computer-readable storage medium shown in this application can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, radio frequency, etc., or any suitable combination thereof.
[0076] In this embodiment of the application, a computer program product is also provided, which, when run on a computer, causes the computer to execute the above-described fall detection method.
[0077] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0078] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A fall detection method for a companion robot, characterized in that, include: Real-time acquisition of video data of monitored individuals, and extraction of video frames at a fixed frame rate; The pose detection model is used to extract the position coordinates of human key points from multiple consecutive video frames, including the current video frame. Based on the position coordinates of key points of the human body in the multiple video frames, it is determined whether the monitored person is suspected of falling. If the monitored person is suspected of falling, then the posture difference matrix between the multiple video frames is calculated based on the position coordinates of key points of the human body in the multiple video frames; The position coordinates of key human body points in the multiple video frames are input into the improved Transformer model, and the fall detection results are output. The improved Transformer model's multi-head self-attention module is used to transform the position coordinates of human key points in the multiple video frames through different linear transformations to obtain the original query matrix, the original key matrix, and the original value matrix, respectively; a target query matrix is generated based on the original query matrix and the pose difference matrix, and a target key matrix is generated based on the original key matrix and the pose difference matrix. The self-attention weights are determined based on the target query matrix, the target key matrix, and the original value matrix.
2. The method according to claim 1, characterized in that, The multi-head self-attention module generates a target query matrix based on the original query matrix and the pose difference matrix, and generates a target key matrix based on the original key matrix and the pose difference matrix through the following steps: The attitude difference matrix D is symmetrically normalized to obtain the target difference matrix. ; According to the formula: Determine the target query matrix ; According to the formula: Determine the target key matrix ; Where Q represents the original query matrix, K represents the original key matrix, and I represents the identity matrix. This represents the modulation intensity coefficient.
3. The method according to claim 1, characterized in that, The step of determining whether the monitored person is suspected of falling based on the position coordinates of key points of the human body in the multiple video frames includes: Based on the position coordinates of key points of the human body in each video frame, determine multiple posture features of the human body in the video frame, and determine the coarse screening judgment value based on the multiple posture features. The speed of movement of the monitored person is determined based on the position coordinates of key points of the human body in each video frame. Based on the movement speed, a corresponding fall threshold is determined; wherein, the movement speed is positively correlated with the fall threshold; If the coarse screening value is greater than the fall threshold, it is determined to be a suspected fall; if the coarse screening value is less than or equal to the fall threshold, it is determined to be normal behavior.
4. The method according to claim 3, characterized in that, Determining the corresponding fall threshold based on the action speed includes: According to the following formula: Determine the fall threshold ; in, Here, f represents the baseline fall threshold, f represents a pre-set constant, and v represents the movement speed of the monitored individual. This indicates the standard speed of movement.
5. The method according to claim 1, characterized in that, The step of calculating the pose difference matrix between the multiple video frames based on the position coordinates of human key points in the multiple video frames includes: Assume the coordinates of the m-th human body keypoint in the i-th video frame are... The coordinates of the m-th human body keypoint in the j-th video frame are: ; Then, according to the following formula: ; Determine the elements in the attitude difference matrix D .
6. A fall detection device for a companion robot, characterized in that, The device includes: The video frame extraction module is used to collect video data of the monitored person in real time and extract video frames at a fixed frame rate. The key point location coordinate extraction module is used to extract the location coordinates of human key points from multiple consecutive video frames containing the current video frame using a pose detection model. The suspected fall detection module is used to determine whether the monitored person is suspected of falling based on the position coordinates of key points of the human body in the multiple video frames. The posture difference matrix determination module is used to calculate the posture difference matrix between the multiple video frames based on the position coordinates of key points of the human body in the multiple video frames if the monitored person is suspected of falling. The fall detection result output module is used to input the position coordinates of key human body points in the multiple video frames into the improved Transformer model and output the fall detection result. The improved Transformer model's multi-head self-attention module is used to transform the position coordinates of human keypoints in the multiple video frames through different linear transformations to obtain the original query matrix, the original key matrix, and the original value matrix, respectively; to generate a target query matrix based on the original query matrix and the pose difference matrix; to generate a target key matrix based on the original key matrix and the pose difference matrix; and to determine the self-attention weights based on the target query matrix, the target key matrix, and the original value matrix.
7. The apparatus according to claim 6, characterized in that, The multi-head self-attention module generates a target query matrix based on the original query matrix and the pose difference matrix, and generates a target key matrix based on the original key matrix and the pose difference matrix through the following steps: The attitude difference matrix D is symmetrically normalized to obtain the target difference matrix. ; According to the formula: Determine the target query matrix ; According to the formula: Determine the target key matrix ; Where Q represents the original query matrix, K represents the original key matrix, and I represents the identity matrix. This represents the modulation intensity coefficient.
8. The apparatus according to claim 6, characterized in that, The suspected fall detection module is specifically used to determine multiple posture features of the human body in each video frame based on the position coordinates of key points in the human body; determine a coarse screening value based on the multiple posture features; determine the movement speed of the monitored person based on the position coordinates of key points in each video frame; determine a corresponding fall threshold based on the movement speed; the movement speed is positively correlated with the fall threshold; if the coarse screening value is greater than the fall threshold, it is determined as a suspected fall; if the coarse screening value is less than or equal to the fall threshold, it is determined as normal behavior.
9. The apparatus according to claim 8, characterized in that, The suspected fall detection module is specifically used to determine the corresponding fall threshold based on the speed of the movement in the following manner: According to the following formula: Determine the fall threshold ; in, Here, f represents the baseline fall threshold, f represents a pre-set constant, and v represents the movement speed of the monitored individual. This indicates the standard speed of movement.
10. The apparatus according to claim 6, characterized in that, The posture difference matrix determination module is specifically used to determine the position coordinates of the m-th human body key points in the i-th video frame if the monitored person is suspected of falling. The coordinates of the m-th human body keypoint in the j-th video frame are: ; Then, according to the following formula: ; Determine the elements in the attitude difference matrix D .