Falling posture detection method based on graph convolutional network and detection system thereof

Abstract

The embodiment of the invention discloses a tumble posture detection method based on a graph convolutional network and a detection system thereof, which can improve the accuracy and robustness of tumble behavior recognition. The detection method comprises the following steps: S1, extracting skeleton joint information of a human body based on an MMPose model: firstly, identifying a portrait in a video by using a Faster-RCNN network to obtain a target image, and then extracting the skeleton joint information of the human body in the target image by using an HRNet network to generate corresponding skeleton sequence data; s2, tumble behavior recognition: tumble behavior recognition is performed on the skeleton sequence data based on a multi-stream attention-enhanced adaptive convolutional network MS-AAGCN model, and the MS-AAGCN model divides the input data into four streams by adopting a multi-stream mechanism: a node stream J-Stream, a skeleton stream B-Stream, a node-motion stream JM-Stream and a skeleton-motion stream BM-Stream, the node flow and the skeleton flow pay attention to position change information of joint points and skeletons composed of the joint points, and the node-motion flow and the skeleton-motion flow pay attention to motion information of a human body; and S3, if an abnormal behavior is recognized, a video corresponding to the abnormal behavior is output, and the abnormal behavior comprises a falling behavior.

Description

Technical Field

[0001] The embodiments of the present invention relate to the fields of image recognition and neural network technology, and in particular to a fall posture detection method and a detection system thereof based on a graph convolutional network. Background Art

[0002] The fall posture detection method based on graph convolutional network mainly combines human joint feature extraction and graph convolutional network, uses the information of human skeletal joints to build a skeleton graph, and performs graphical processing and fall pattern recognition on this basis. This technology enhances the sensitivity to dynamic changes and accurately distinguishes normal behavior from fall behavior by learning the spatiotemporal features of joints.

[0003] However, this method faces the problem of how to further improve the accuracy and robustness of fall behavior recognition, especially when dealing with various complex backgrounds, scenes with different clothing, and the unique action patterns of the elderly and disabled groups. Due to the limitations of data set diversity and changes in lighting conditions, the adaptability of the training model in different environments still needs to be improved. In addition, for subtle posture changes and fast-moving scenarios, the model needs to have more accurate posture tracking capabilities and faster response mechanisms to reduce the misjudgment rate. These aspects constitute technical challenges that need to be overcome urgently. Summary of the invention

[0004] In order to overcome the above-mentioned technical defects and improve the accuracy and robustness of fall behavior recognition, a fall posture detection method and a detection system based on a graph convolutional network are provided in an embodiment of the present application.

[0005] In the first aspect, an embodiment of the present application provides a fall posture detection method based on a graph convolutional network, which is applicable to the open source posture estimation algorithm library MMPose model based on the Faster-RCNN network and HRNet network architecture design, and includes the following steps:

[0006] S1. Extract human skeletal joint information based on MMPose model:

[0007] First, use the Faster-RCNN network to identify the portrait in the video to obtain the target image, and then use the HRNet network to extract the skeletal joint information of the human body in the target image to generate the corresponding skeletal sequence data;

[0008] S2. Falling behavior recognition:

[0009] Based on the multi-stream attention enhanced adaptive graph convolutional network MS-AAGCN model, fall behavior recognition is performed on skeleton sequence data. The MS-AAGCN model uses a multi-stream mechanism to divide the input data into four streams: node stream J-Stream, skeleton stream B-Stream, node-motion stream JM-Stream, and skeleton-motion stream BM-Stream. The node stream and skeleton stream focus on the position change information of the joints and the skeletons composed of the joints, while the node-motion stream and skeleton-motion stream focus on the movement information of the human body.

[0010] S3. If abnormal behavior is identified, output the video corresponding to the abnormal behavior, where the abnormal behavior includes falling behavior.

[0011] Optionally, in a possible implementation manner of the first aspect, in the above-mentioned fall behavior recognition, the following formula is used in the MS-AAGCN model:

[0012] Where Y out and Y in They represent the feature map tensors of the output and input networks, n represents the number of layers, β represents the preset parameter to avoid empty rows, and X n represents the kernel size of the spatial dimension, Represents the weight vector of the 1×1 convolution operation, L n represents the global graph learned from the data for action recognition tasks, M n Represent a single graph and learn a unique topology for each sample;

[0013] M n The vertex feature similarity is calculated by normalizing the embedded Gaussian function, and the solution formula is: n =SoftMax(Y in T W θn T W φn Y out ), where θ and represents two embedding functions with two kernel sizes of (1×1), and denote the embedding function θ and Parameters.

[0014] Optionally, in a possible implementation of the first aspect, the STC attention mechanism is introduced into the node stream J-Stream, the skeleton stream B-Stream, the node-motion stream JM-Stream, and the skeleton-motion stream BM-Stream in the MS-AAGCN model, and a new STC attention mechanism module including three sub-modules: a spatial attention sub-module, a temporal attention sub-module, and a channel attention sub-module is added between the activation function layer and the transposed convolution layer corresponding to each stream.

[0015] Optionally, in a possible implementation manner of the first aspect, the Faster-RCNN network includes three parts: a basic convolutional layer, a region proposal network RPN layer, and a RoI pooling and classification network layer;

[0016] The specific steps to identify and obtain the target image include:

[0017] S1.1: Perform image classification tasks in the base convolutional layer to output feature maps containing spatial and semantic information;

[0018] S1.2: Generate candidate detection boxes through the neural network at the RPN layer;

[0019] S1.3: In the RoI pooling and classification network layers, RoI pooling is used to map candidate boxes of different sizes to the feature map and extract fixed-size feature vectors. The classification network is used to process through two fully connected layers, and then the sister fully connected layer outputs coordinate fine-tuning and classification information.

[0020] Optionally, in a possible implementation manner of the first aspect, the process of extracting skeletal joint information of a human body in a target image using the HRNet network includes the following four stages of operations:

[0021] The first stage: extract high-resolution features to preserve image details;

[0022] The second stage: exchange and complementation of features with different resolutions through multi-scale fusion;

[0023] The third stage: using horizontal connections to fuse features of different resolutions to enhance network expressiveness and performance;

[0024] Stage 4: The fused features are used to output task results, such as image segmentation and pose estimation.

[0025] In a second aspect, an embodiment of the present invention provides a fall posture detection system based on a graph convolutional network, including: a feature extraction module constructed based on the MMPose model of the open source posture estimation algorithm library, and a recognition module constructed based on the multi-stream attention enhanced adaptive graph convolutional network MS-AAGCN;

[0026] The feature extraction module is used to perform the following steps:

[0027] S1. Extract human skeleton joint information based on MMPose model: First use Faster-RCNN network to identify the human figure in the video to obtain the target image, then use HRNet network to extract the human skeleton joint information in the target image and generate the corresponding skeleton sequence data;

[0028] The identification module is used to perform the following steps:

[0029] S2. Fall behavior recognition: Based on the multi-stream attention enhanced adaptive graph convolutional network MS-AAGCN model, fall behavior recognition is performed on skeleton sequence data. The MS-AAGCN model uses a multi-stream mechanism to divide the input data into four streams: node stream J-Stream, skeleton stream B-Stream, node-motion stream JM-Stream, and skeleton-motion stream BM-Stream. The node stream and skeleton stream focus on the position change information of the joints and the skeletons composed of the joints, and the node-motion stream and skeleton-motion stream focus on the movement information of the human body.

[0030] S3. If abnormal behavior is identified, output the video corresponding to the abnormal behavior, where the abnormal behavior includes falling behavior.

[0031] In a third aspect, an embodiment of the present invention provides a computer device, including:

[0032] at least one processor; and

[0033] a memory communicatively coupled to at least one processor; wherein:

[0034] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the method of any one of the first aspect and its implementation methods as described above.

[0035] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed by a processor, the method of any one of the first aspect and its implementation methods is implemented.

[0036] In a fifth aspect, a computer program product is provided in an embodiment of the present invention, including a computer program, characterized in that when the computer program is executed by a processor, the steps of the method of any one of the first aspect and its implementation methods are implemented.

[0037] The embodiments of the present application can achieve the following technical effects: in the technical solution of the present invention, the MMPose model constructed by the Faster-RCNN network and the HRNet network is used to extract the skeletal joint information of the human body. The model uses multi-scale feature fusion technology, which can effectively improve the accuracy and robustness of joint detection and realize the precise positioning of skeletal joints. At the same time, a multi-stream mechanism is used in the MS-AAGCN model to learn the spatiotemporal information of human skeletal joints and identify the characteristics of falling behavior. Each stream can capture joint information of different scales and types, such as joint position and motion information, and joint itself and motion information, thereby further improving the accuracy and robustness of falling behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0039] Figure 1 1 is a flow chart of a fall posture detection method based on a graph convolutional network in an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of a network architecture of an MS-AAGCN network in an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of a network architecture of the STC attention mechanism in an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of a network architecture of a Faster-RCNN network in an embodiment of the present invention;

[0043] Figure 5 A schematic diagram of a recognition process for obtaining a target image in an embodiment of the present invention;

[0044] Figure 6 A schematic diagram of a network architecture of the HRNet network in an embodiment of the present invention;

[0045] Figure 7 Schematic diagram of a fall posture detection system based on a graph convolutional network in an embodiment of the present invention;

[0046] Figure 8 FIG. 4 is a schematic diagram of a hardware structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0048] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0049] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0050] First, the following terms are explained:

[0051] The MMPose model, an open source posture estimation algorithm library model, is a posture estimation algorithm based on deep learning. It uses a convolutional neural network (CNN) to detect the skeletal joints of pedestrians in images. The model uses multi-scale feature fusion technology to accurately detect the skeletal joints of pedestrians in videos. Through this model, the joints of pedestrians in videos are detected and the skeleton is reconstructed, thereby obtaining the spatiotemporal information of the skeletal joints of the human body.

[0052] MS-AAGCN, Multi-Stream Attention-enhanced Adaptive Graph Convolutional Network, is designed for processing skeleton data and realizing action recognition tasks;

[0053] Faster-RCNN is a deep learning model with wide application in the field of object detection. It is designed to improve detection speed and accuracy. The entire Faster R-CNN model can be decomposed into three main parts: the basic convolutional layer backbone, the region proposal network RPN, and the RoI pooling and classification network;

[0054] HRNet, the full name of which is High-Resolution Network, is a deep neural network architecture used for tasks such as image semantic segmentation and pose estimation.

[0055] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below:

[0056] Embodiment 1

[0057] The present invention discloses a fall posture detection method based on graph convolutional network, which combines Faster-RCNN, HRNet network and multi-stream attention enhanced adaptive convolutional network (MS-AAGCN) to effectively identify fall behavior, and its purpose is to improve the accuracy and robustness of recognition. The method consists of three main steps: extraction of skeletal joint information, fall behavior recognition and result output. Specifically, by integrating advanced visual technology and complex algorithm processing, the system can quickly and accurately mark the body joints of a person under monitoring video or real-time video feed, and judge whether a fall event has occurred based on this.

[0058] See also Figure 1 In the embodiment of the present invention, the fall posture detection method based on graph convolutional network specifically includes the following steps:

[0059] S1. Extract human skeletal joint information based on the MMPose model.

[0060] Specifically, the Faster-RCNN network is first used to identify the portrait in the video to obtain the target image, and then the HRNet network is used to extract the skeletal joint information of the human body in the target image to generate the corresponding skeletal sequence data.

[0061] In S1, the Faster-RCNN module embedded in the MMPose model is used to separate and identify the image area containing people from the background clutter in the video frame; once determined, the advanced functions of HRNet, which is also one of the MMPose components, are further used to analyze the actions and postures in the selected image area; the final product of this step will be sequence point data representing the form of human activities; for ease of understanding, in an example scenario, you can imagine that there is a camera that is monitoring the safety environment of an elderly person; in the static pictures captured at any given time, it can be found that HRNet successfully marks and connects the joints representing a person's body structure to form a contour graph, and such results can be used in subsequent behavior classification.

[0062] S2. Fall behavior recognition is performed on skeleton sequence data based on the multi-stream attention enhanced adaptive convolutional network MS-AAGCN model.

[0063] Specifically, the MS-AAGCN model uses a multi-stream mechanism to divide the input data into four streams: node stream J-Stream, bone stream B-Stream, node-motion stream JM-Stream, and bone-motion stream BM-Stream. The node stream and bone stream focus on the position change information of the joints and the bones composed of the joints, while the node-motion stream and bone-motion stream focus on the movement information of the human body.

[0064] At this stage, the MS-AAGCN framework distributes the previously extracted image data into four small streams customized for optimizing the detection process (i.e., node stream J-Stream, skeleton stream B-Stream, node-motion stream JM-Stream, and skeleton-motion stream BM-Stream) for independent calculations; these sub-pipelines focus on the analysis of short-term instant trends and long-term overall changes, respectively, to provide more comprehensive and in-depth results; for example, assuming that in the elderly care scenario mentioned above, when a standing body gradually disappears from the camera's field of view and is accompanied by a rapid downward posture change, this series of data fluctuations will be quickly captured and synchronously transmitted and fed back to the central system by various paths to evaluate the type of dangerous action it may represent; this meticulous and comprehensive approach ensures that even the smallest changes in details will not be missed, thereby greatly improving the sensitivity and reliability of fall behavior detection.

[0065] S3. If abnormal behavior is identified, output the video corresponding to the abnormal behavior, where the abnormal behavior includes falling behavior.

[0066] Finally, if all the above steps determine that an unexpected action pattern has occurred in the observation area, that is, a fall, the warning mechanism will be triggered. This means that the various algorithms previously set up have achieved results; for example, in the case of the elderly mentioned above, if the system determines that an unexpected fall has occurred, it will immediately generate a warning and provide the video record to relevant personnel for further processing.

[0067] This approach of integrating multiple latest deep learning models and technical means to improve the ability to respond quickly and judge in potential high-risk situations such as falls reflects a highly responsible attitude and advanced technical concept to protect the value of fragile life. Through careful design of every detail, this solution can not only achieve accurate and timely alarms, but also show one of the possible ways to make innovative contributions to the field of elderly life safety protection in the context of an increasingly intelligent world. By adopting this comprehensive system, the system can effectively filter noise signals from massive data streams and focus on important dynamics, thereby providing valuable health monitoring services at the first time.

[0068] In the technical solution implemented by the present invention, the MMPose model constructed by the Faster-RCNN network and the HRNet network is used successively to extract the skeletal joint information of the human body. The model uses multi-scale feature fusion technology, which can effectively improve the accuracy and robustness of joint detection and realize the precise positioning of skeletal joints. At the same time, a multi-stream mechanism is used in the MS-AAGCN model to learn the spatiotemporal information of human skeletal joints and identify the characteristics of falling behavior. Each stream can capture joint information of different scales and types, such as joint position and motion information, joint itself and motion information, thereby improving the accuracy and robustness of falling behavior recognition.

[0069] Embodiment 2

[0070] The STC attention mechanism is introduced into the MS-AAGCN model of the present invention to improve the model and enhance the model attention. Figure 2 and Figure 3 .

[0071] MS-AAGCN is designed specifically for processing skeleton data and implementing action recognition tasks. In the MS-AAGCN model, the input skeleton sequence data, including three-dimensional skeleton coordinates and time dimension, will go through a series of preprocessing steps, such as normalization of joint coordinates and sampling of time steps, to enhance the stability and generalization ability of the model. Subsequently, these data are converted into a graph structure, in which joints are nodes and spatial relationships between joints are connections. This conversion helps to capture the skeleton graph of the action, such as Figure 2 As shown in the figure, in order to utilize data diversity, MS-AAGCN adopts a multi-stream mechanism to divide the input data into node stream J-Stream, bone stream B-Stream, node-motion stream JM-Stream, and bone-motion stream BM-Stream. The node stream and bone stream focus on the position change information of the joints and the bones composed of the joints, and the node-motion stream and bone-motion stream focus on the motion information of the human body.

[0072] For each of the above streams, the MS-AAGCN network is applied in the present invention, so that the MS-AAGCN model can learn and adapt to different action skeleton structures. Graph convolutional neural networks are particularly suitable for processing graph structure data, such as human skeletons, and can effectively capture the spatial relationship between nodes. The formula of the adaptive graph convolutional neural network is:

[0073] Where Y out and Y in They represent the feature map tensors of the output and input networks, n represents the number of layers, β represents the preset parameter to avoid empty rows, and X n represents the kernel size of the spatial dimension, Represents the weight vector of the 1×1 convolution operation, Ln represents the global graph learned from the data for action recognition tasks, M n Represents a single graph and learns a unique topological structure for each sample; during training, L n The elements of L can be updated together with other parameters without specific constraints and are learned entirely based on training data. The model learns the optimal graph structure through data-driven learning. n The uniqueness of M is that each layer has personalized characteristics and adapts to different semantic levels. n is a single graph that learns a unique topology for each sample.

[0074] Specifically, M. n The vertex feature similarity is calculated by normalizing the embedded Gaussian function, and the solution formula is: n =SoftMax(Y in T W θn T W φn Y out ), where θ and represents two embedding functions with two kernel sizes of (1×1), and denote the embedding function θ and Parameters.

[0075] After each stream is processed sequentially through multiple layers of convolution, normalization, activation function, attention, transposed convolution, normalization, and activation function, the processing results of the four streams are combined and predicted to determine whether a fall occurs. It should be noted that the network layer corresponding to the above attention is newly added in the present invention to introduce the STC attention mechanism. Its specific structure is as follows: Figure 3 As shown in the figure, the STC attention mechanism is introduced into the node stream J-Stream, the skeleton stream B-Stream, the node-motion stream JM-Stream, and the skeleton-motion stream BM-Stream. Specifically, a new STC attention mechanism module is added between the activation function layer and the transposed convolution layer, which includes three sub-modules: spatial attention sub-module, temporal attention sub-module, and channel attention sub-module.

[0076] First, in the STC attention mechanism, the attention submodule focuses on the spatial correlation and importance distribution of different human nodes. For example, in a fall behavior recognition scenario, the system can optimize by assigning higher weights to certain body parts (such as the waist and legs); this weight assignment can be determined by learning, and weighted adjustment is performed before input to the convolution layer, making it more sensitive and accurate to the joint positions in the input skeleton data. Secondly, the temporal attention submodule function is added to the STC attention mechanism, which focuses on analyzing the changing laws of human skeleton posture. Giving more important action frames higher attention in the time series is similar to the sequence weight in natural language processing. Assuming that each action clip in a fitness video is to be analyzed, this module will identify and annotate the time frame weights of the key movement stages; finally, the channel attention submodule is added, whose role is to process and highlight the significant features in different data dimensions-usually referring to the specific sensor channel information of the skeleton sequence or the channel mapping transmitted from the previous layer. In an application case involving multiple cameras or sensor fusion, it can help strengthen the information provided by certain input sources, thereby assisting the model to selectively filter and emphasize important factors from complex data sources.

[0077] Therefore, the introduction of the STC attention mechanism allows the MS-AAGCN model to automatically focus on important parts and ignore parts with smaller contributions when processing data. Figure 2 After all four streams are processed by multiple layers such as convolution, normalization, and attention mechanism, the features of different streams are fused to integrate information. Finally, the fused features are used by the classifier for action recognition and output abnormal behavior classification. This series of steps enables the MS-AAGCN network to extract useful information from the skeleton data and accurately identify the action type.

[0078] Embodiment 3

[0079] See also Figure 4 , 5 6 describe the Faster-RCNN network, the Faster-RCNN network, and the specific steps of obtaining the target image in the present invention, which are as follows:

[0080] like Figure 4 As shown, the Faster-RCNN network in the embodiment of the present invention includes three parts: basic convolution layer, region proposal network RPN layer, RoI pooling and classification network layer. The backbone network of the basic convolution layer part has 13 convolution layers, 13 activation function layers and 4 pooling layers, and the fully connected layer is removed; the region proposal network RPN layer part changes the dimension, normalizes, and processes the candidate frame on the feature map; the RoI pooling and classification network layer part performs RoI pooling, bounding box prediction, and normalization on the image features output in the candidate frame to obtain the corresponding classification probability.

[0081] like Figure 5 As shown, the specific steps of identifying and acquiring the target image in S1 include:

[0082] S1.1: Perform image classification tasks in the base convolutional layer to output feature maps containing spatial and semantic information.

[0083] The image classification task is performed in the basic convolutional layer to output feature maps containing spatial and semantic information, which means that the input original image first passes through a series of pre-trained basic networks, such as ResNet, VGG16 or GoogLeNet, and the top layer of these networks (usually the previous layer) produces new feature maps containing rich semantic content and location information. The convolution operations in these basic convolutional networks help identify different hierarchical patterns in the image and establish spatial hierarchical expressions.

[0084] S1.2: Generate candidate detection boxes through the neural network at the RPN layer.

[0085] RPN is a key part of Faster R-CNN. It generates candidate detection boxes or regions of interest (RoI) through neural networks to improve detection speed. RPN slides on the feature map to generate anchor boxes, then adjusts the coordinates to generate candidate detection boxes. The region proposal network generates candidate detections of multiple scales and aspect ratios at the same time, and predicts a bounding box and the probability that the box contains an object for each position on the feature map. This step improves the speed of object detection and ensures good accuracy.

[0086] S1.3: In the RoI pooling and classification network layers, RoI pooling is used to map candidate boxes of different sizes to the feature map and extract fixed-size feature vectors. The classification network is used to process through two fully connected layers, and then the sister fully connected layer outputs coordinate fine-tuning and classification information.

[0087] These candidate detection windows or RoI regions are passed to subsequent layers for RoI pooling. The purpose of RoI pooling is to ensure that even if the size of the RoI region is different, the extracted features still have a fixed size. This is conducive to input into subsequent network layers such as fully connected layers to ensure the consistency of the output size of each RoI; in the classification layer, each RoI will pass through a series of fully connected layers to obtain the category recognition result and its confidence score, and also pass through a branch parallel to the classification to perform precise target box regression adjustment to ensure that the positioning is as accurate as possible.

[0088] Therefore, the Faster-RCNN network achieves efficient and accurate target detection through three parts: basic convolutional layer, region proposal network RPN layer, RoI pooling and classification network layer.

[0089] like Figure 6 As shown in the figure, the HRNet network extracts the skeletal joint information of the human body in the target image through fully connected layers, convolution operations, upsampling and downsampling, etc., which includes the following four stages of operations: the first stage: extracting high-resolution features to retain image details; the second stage: realizing the exchange and complementation of features of different resolutions through multi-scale fusion; the third stage: using lateral connections to fuse features of each resolution to enhance the network's expressiveness and performance; the fourth stage: the fused features are used to output task results, such as image segmentation and posture estimation.

[0090] In the first stage, the input human target image is processed and feature extraction is performed using a convolutional branch structure with parallel branches to ensure the retention of features at different levels so that even the finest details can be maintained. This process is crucial because the accuracy of skeletal joint detection is closely related to the quality of the extracted features. To understand this intuitively, suppose there is a picture of a fall action and the joints need to be detected. The job of this stage is to ensure that the various shape information of the arms, legs, etc. can be clearly passed to the subsequent stages, even if they may be affected by different angles, lighting conditions or clothing occlusion.

[0091] The second stage involves interoperability between feature maps of different resolutions. Feature exchange between different feature maps is achieved by interleaving resolution reduction and upsampling, which helps ensure that even low-level features can play a role in high-level feature maps and help supplement deeper information to more accurately locate human joints. It can help the model better identify joints in local areas of the human body that are difficult to locate due to image size or complex background, thereby further improving its accuracy and robustness.

[0092] In the third stage, horizontal residual connections are used to combine feature maps at multiple scales. This process is essentially a reintegration of feature maps of different resolutions obtained in the second stage to enhance the learning ability and prediction effect of the entire network, because different body parts may be suitable for different types of information representation. For example, higher resolution data is required to identify the joints on the fingers, but a larger visual context may be required to accurately detect the position of the shoulder. This method enhances the network's ability to understand various scale patterns in the image.

[0093] Finally, after the completion of the above three stages, the fourth stage is entered, which is the fusion of features and their use to complete the predetermined task such as pose estimation. In this process, all the improvements made in the previous stage are integrated and combined to generate a map that clearly shows the location of key nodes or the prediction of pose.

[0094] Embodiment 4

[0095] See also Figure 7 A fall posture detection system based on a graph convolutional network of the present invention is described as follows:

[0096] The fall posture detection system 70 based on graph convolutional network mainly includes two core parts: a feature extraction module 701 built based on the open source posture estimation algorithm library MMPose model and a recognition module 702 built based on the multi-stream attention enhanced adaptive convolutional network MS-AAGCN;

[0097] The feature extraction module 701 is used to perform the following steps S1:

[0098] S1. Extract human skeleton joint information based on MMPose model: First use Faster-RCNN network to identify the human figure in the video to obtain the target image, then use HRNet network to extract the human skeleton joint information in the target image and generate the corresponding skeleton sequence data;

[0099] The identification module 702 is used to perform the following steps S2 and S3:

[0100] S2. Fall behavior recognition: Fall behavior recognition is performed on skeleton sequence data based on the multi-stream attention enhanced adaptive convolutional network MS-AAGCN model. The MS-AAGCN model adopts a multi-stream mechanism to divide the input data into four streams: node stream J-Stream, skeleton stream B-Stream, node-motion stream JM-Stream, and skeleton-motion stream BM-Stream. The node stream and skeleton stream focus on the position change information of the joints and the skeletons composed of the joints, and the node-motion stream and skeleton-motion stream focus on the movement information of the human body.

[0101] S3. If abnormal behavior is identified, output the video corresponding to the abnormal behavior, where the abnormal behavior includes falling behavior.

[0102] It should be understood that the workflow of the entire system can be further broken down into three main stages: first, MMPose processes the characters in the video to obtain their skeletal structure information; then the MS-AAGCN network is used to make a fall posture recognition decision based on this information; and finally the result is output. Specifically:

[0103] First, in step S1, feature extraction is performed based on the MMPose model, and the classic Faster R-CNN model in computer vision is used to locate the position of the person from the original video screen and cut out the screen of a single object (target image). Next, the HRNet architecture with high joint estimation accuracy is applied to accurately mark the position information of multiple joints of the human body (skeletal joint data) from the above target image. This step generates a set of data sequences that describe the changes in the posture of a person in each frame, called a skeleton sequence.

[0104] Then enter step S2, which is the key link of fall detection. MS-AAGCN uses four parallel processing streams with different focuses for motion analysis. These include the node stream J-Stream that only considers the state changes of the current joint points, the node-motion stream JM-Stream that combines the motion information of adjacent joint points, the bone stream B-Stream that specializes in the study of bone change patterns, and the bone-motion stream BM-Stream that contains the bone movement trend component. These four types of analysis angles together constitute a multi-dimensional information capture framework. This complex architecture ensures that the model can more finely distinguish the behavioral characteristics of different time dimensions without losing spatial distribution information. Through this multi-perspective approach, the MS-AAGCN model can effectively identify complex and sudden behaviors such as falls.

[0105] Once the recognition is completed and if the detection algorithm determines that an unexpected situation such as a fall has occurred, the third step, step S3, will prompt the output of the original recorded video segment carrying the abnormal behavior event mark, so that the monitor or system administrator can respond quickly based on this signal, such as sending an emergency signal to the security department or marking this video in the background management system for later review and confirmation.

[0106] Embodiment 5

[0107] Figure 8 The schematic diagram of the hardware architecture of a computer device 10000 suitable for implementing a fall posture detection method according to the third embodiment of the present application is schematically shown. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle-mounted terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Figure 8 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them:

[0108] The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of the computer device 10000, such as a hard disk or a memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 may also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store the operating system and various application software installed in the computer device 10000, such as the program code of the fall posture detection method, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or will be output.

[0109] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0110] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0111] It should be pointed out that Figure 8 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0112] In this embodiment, the fall posture detection method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0113] Embodiment 6

[0114] A computer-readable storage medium is also provided in an embodiment of the present application, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the fall posture detection method in the embodiment are implemented.

[0115] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the fall posture detection method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.

[0116] Embodiment 7

[0117] A computer program product is also provided in an embodiment of the present application, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0118] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0119] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A fall posture detection method based on graph convolutional network, characterized in that: The method is applicable to the open source posture estimation algorithm library MMPose model based on the Faster-RCNN network and HRNet network architecture design, and includes the following steps: S1. Extract human skeletal joint information based on MMPose model: First, use the Faster-RCNN network to identify the portrait in the video to obtain the target image, and then use the HRNet network to extract the skeletal joint information of the human body in the target image to generate the corresponding skeletal sequence data; S2. Falling behavior recognition: Based on the multi-stream attention enhanced adaptive graph convolutional network MS-AAGCN model, the fall behavior recognition is performed on the skeleton sequence data. The MS-AAGCN model adopts a multi-stream mechanism to divide the input data into four streams: node stream J-Stream, skeleton stream B-Stream, node-motion stream JM-Stream, and skeleton-motion stream BM-Stream. The node stream and the skeleton stream focus on the position change information of the joints and the skeletons composed of the joints, and the node-motion stream and the skeleton-motion stream focus on the movement information of the human body. S3. If abnormal behavior is identified, output the video corresponding to the abnormal behavior, where the abnormal behavior includes falling behavior.

2. The method according to claim 1, characterized in that In the fall behavior recognition, the following formula is used in the MS-AAGCN model: Where Y out and Y in They represent the feature map tensors of the output and input networks, n represents the number of layers, β represents the preset parameter to avoid empty rows, and X n represents the kernel size of the spatial dimension, Represents the weight vector of the 1×1 convolution operation, L n represents the global graph learned from the data for action recognition tasks, M n Represent a single graph and learn a unique topology for each sample; M n The vertex feature similarity is calculated by normalizing the embedded Gaussian function, and the solution formula is: In the formula, θ and represents two embedding functions with two kernel sizes of (1×1), and Denote the embedding function θ and Parameters.

3. The method according to claim 1 or 2, characterized in that: The STC attention mechanism is introduced in the node flow, skeleton flow, node-motion flow, and skeleton-motion flow of the MS-AAGCN model. A new STC attention mechanism module containing spatial attention sub-module, temporal attention sub-module and channel attention sub-module is added between the activation function layer and the transposed convolution layer corresponding to each flow.

4. The method according to claim 1, characterized in that: The Faster-RCNN network consists of three parts: the basic convolutional layer, the region proposal network RPN layer, and the RoI pooling and classification network layer; The specific steps of identifying and acquiring the target image include: S1.1: Perform image classification tasks in the base convolutional layer to output feature maps containing spatial and semantic information; S1.2: Generate candidate detection boxes through the neural network at the RPN layer; S1.3: In the RoI pooling and classification network layers, RoI pooling is used to map candidate boxes of different sizes to the feature map and extract fixed-size feature vectors. The classification network is used to process through two fully connected layers, and then the sister fully connected layer outputs coordinate fine-tuning and classification information.

5. The method according to claim 1 or 4, characterized in that: The process of using the HRNet network to extract the skeletal joint information of the human body in the target image includes the following four stages of operation: The first stage: extract high-resolution features to preserve image details; The second stage: exchange and complementation of features with different resolutions through multi-scale fusion; The third stage: using horizontal connections to fuse features of different resolutions to enhance network expression and performance; Stage 4: The fused features are used to output task results, such as image segmentation and pose estimation.

6. A fall posture detection system based on graph convolutional network, characterized in that: include: A feature extraction module based on the open source pose estimation algorithm library MMPose model, and a recognition module based on the multi-stream attention enhanced adaptive graph convolutional network MS-AAGCN; The feature extraction module is used to perform the following steps: S1. Extract human skeleton joint information based on MMPose model: First use Faster-RCNN network to identify the human figure in the video to obtain the target image, then use HRNet network to extract the human skeleton joint information in the target image and generate the corresponding skeleton sequence data; The identification module is used to perform the following steps: S2. Falling behavior recognition: Falling behavior recognition is performed on skeleton sequence data based on the multi-stream attention enhanced adaptive graph convolutional network MS-AAGCN model. The MS-AAGCN model uses a multi-stream mechanism to divide the input data into four streams: node stream J-Stream, skeleton stream B-Stream, node-motion stream JM-Stream, and skeleton-motion stream BM-Stream. The node stream and skeleton stream focus on the position change information of the joints and the skeletons composed of the joints, and the node-motion stream and skeleton-motion stream focus on the movement information of the human body. S3. If abnormal behavior is identified, output the video corresponding to the abnormal behavior, where the abnormal behavior includes falling behavior.

7. The method according to claim 6, characterized in that The feature extraction module is also used to execute the steps in the method described in claim 2 or 3 above; the recognition module is also used to execute the steps in the method described in claim 4 or 5 above.

8. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method as claimed in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method as described in any one of claims 1 to 5 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images