A panoramic cloud robot for assisting the elderly at home and its panoramic human behavior recognition method

Abstract

The present invention discloses a panoramic cloud robot for assisting the elderly at home and a method for identifying panoramic human behavior thereof, wherein the panoramic cloud robot for assisting the elderly at home includes a panoramic camera, and the panoramic camera also includes a wide-angle camera, and each wide-angle camera is equipped with a fisheye lens with a field of view greater than 180°, and the fisheye lens is fixedly installed at intervals of 180° around the vertical optical axis of the wide-angle camera. The wide-angle camera and the fisheye lens cooperate with each other to obtain a panoramic image without blind spots in real time. The method for identifying panoramic human behavior of the panoramic cloud robot for assisting the elderly at home includes a fusion of the Slowfast algorithm, the TCN algorithm, and the CNN algorithm. By using the Slowfast algorithm to analyze RGB environmental features, and using the TCN algorithm to supplement human motion features, and using the CNN algorithm to supplement human structural features, the accuracy of the behavior recognition of the elderly-assisting robot under panoramic vision can ultimately be improved.

Description

Technical Field

[0001] The present invention relates to a panoramic cloud robot for assisting the elderly at home and a panoramic human behavior recognition method thereof. Background Art

[0002] Home-assistance robots are a type of service robot that can provide assistance and support for the elderly's daily tasks and basic needs. A variety of home-assistance robots have been developed and designed to meet the diverse care needs of the elderly, both domestically and internationally. These robots can be categorized as family companions, daily care, transporters, and disability care. The technologies for these robots are gradually maturing in areas such as mobility design (walking), voice interaction (hearing), and navigation in unstructured environments (environmental perception).

[0003] Compared to conventional monocular and binocular cameras, panoramic cameras can observe all surrounding environments simultaneously, enabling robots to surpass human observation capabilities. During human-machine interaction with elderly-assisting robots, panoramic vision enables efficient human detection, facial recognition, object detection, human posture detection, and human-machine tracking, thereby enhancing the intelligence of human-machine interaction and expanding the application scenarios of home-based elderly-assistance robots.

[0004] The main methods of panoramic imaging are: monocular rotation stitching, fisheye lens and catadioptric lens. The monocular rotation stitching method fixes an ordinary camera on a gimbal, controls the rotation of the gimbal and obtains a panoramic image through stitching. However, due to the limitation of the rotation speed of the gimbal, the obtained panoramic image has a time delay, which is not suitable for the elderly-assisting robot to detect fast-moving targets in the home environment. The catadioptric lens method uses a special reflector to reflect the surrounding environmental information into a monocular camera at one time. Since catadioptric panoramic imaging follows the planar cylindrical projection method, the panoramic view of the elderly-assisting robot is limited, and there are blind spots in the field of view in 3D space, which is not conducive to human-computer interaction.

[0005] The elderly-assisting robot's posture detection and behavior recognition under panoramic vision are achieved by processing corrected panoramic images through a deep neural network. While existing panoramic image correction methods are relatively mature, the key to improving the accuracy of panoramic behavior recognition lies in the design of the behavior recognition network model. However, existing behavior recognition models fail to simultaneously address environmental, human motion, and structural characteristics, lacking comprehensive feature extraction, resulting in low behavior recognition accuracy. Summary of the Invention

[0006] In order to solve the technical problem that the panoramic images obtained by home-based elderly-assisting robots have delays or blind spots in vision, the present invention proposes a home-based elderly-assisting panoramic cloud robot. At the same time, the present invention also proposes a panoramic human behavior recognition method for a home-based elderly-assisting panoramic cloud robot that simultaneously emphasizes environmental characteristics, human movement characteristics and human structure characteristics to improve the accuracy of behavior recognition.

[0007] The technical solutions of the present invention are as follows:

[0008] A panoramic cloud robot for assisting the elderly at home comprises a panoramic camera, a 5G communication terminal, a processor and a motion mechanism. The panoramic camera also includes a wide-angle camera. Each wide-angle camera is equipped with a fisheye lens with a field of view greater than 180°. The fisheye lenses are fixedly installed at intervals of 180° around the vertical optical axis of the wide-angle camera.

[0009] Preferably, the motion mechanism serves as the chassis of the robot, and is mainly composed of a power supply, a moving mechanism, a lifting mechanism, a ranging sensor and a controller, and the panoramic camera, 5G communication terminal and processor are installed on the chassis of the robot.

[0010] Preferably, the field of view angle of the fisheye lens is 210°.

[0011] Preferably, it also includes an edge calculator and a cloud server.

[0012] Preferably, the robot is connected to the edge computer in a point-to-point communication manner through a 5G communication terminal.

[0013] A method for panoramic human behavior recognition of the above-mentioned home elderly-assisting panoramic cloud robot includes RGB features and posture skeleton features, and simultaneously processes the RGB features and posture skeleton features, including the following steps:

[0014] Step 1: Obtain raw data through a panoramic camera;

[0015] Step 2: Use the Openpose model to process the raw data, estimate the human body posture bone points and posture skeleton, and obtain the preliminarily processed data;

[0016] Step 3: Use the Slowfast algorithm to process the preliminarily processed data to obtain the last convolution layer, use the TCN algorithm to process the processed data to obtain the time domain features of the human body posture skeleton points, use the CNN algorithm to process the processed data to obtain the spatial features of the human body posture skeleton, and fuse the time domain features and the spatial features to obtain the first convolution layer;

[0017] Step 4: Make the last convolution layer and the first convolution layer pass through the fully connected layer to complete feature fusion;

[0018] Step 5: Determine the behavior category.

[0019] Preferably, the TCN algorithm includes a residual block, the TCN algorithm is connected by residual blocks, the residual block has two convolutional layers, the number of convolution kernels of the two layers is the same, batch normalization and random rectified linear unit are performed before each convolutional layer in the residual block, the random rectified linear unit is implemented using RReLU activation function, and the calculation formula of the lth residual block is Y l =Y l-1 +F(W l，1 , W l，2 , Y l-1 ), the function F of the residual block is F(W l，1 , W l，2 , Y l-1 )=W l，2 ·σ(W l，1 ·σ(Y l-1 )), where σ represents the RReLU activation function.

[0020] Preferably, the CNN algorithm is implemented through the following process: first, a spatial pose feature matrix of human skeleton points is constructed, and the feature information is down-sampled according to the low-frequency sampling rate in the Slowfast algorithm, and the key frames of human motion behaviors are selected. Then, OpenCV is used to process the skeleton point pose information in the key frames, and a background-free human skeleton point enhanced image is reconstructed. Finally, the skeleton point enhanced image is processed using the Resnet-18 model.

[0021] Preferably, the time domain features and the spatial features are weightedly fused to obtain the first convolution layer, and the weighted fusion calculation formula is V ts =avg(V t , V s )=ω t V t +ω s V s , where ω t With ω s They represent the respective weights when fusing two features.

[0022] Preferably, two cross entropy losses and one total loss are also included. The total number of video samples is M, the total number of behavior categories is N, and the cross entropy loss of RGB features is:

[0023]

[0024] The cross entropy loss of the pose skeleton feature is:

[0025]

[0026] The total loss is:

[0027]

[0028] Where y m，n is the true value of the result, and Respectively represent the prediction results when using RGB data and posture skeleton data, represents the weight decay regularization of all parameters, λ is the decay coefficient, Used to avoid overfitting.

[0029] In this invention, the panoramic camera includes a wide-angle camera and a fisheye lens with a field of view greater than 180°. The wide-angle camera and the fisheye lens work together to obtain a panoramic image with no blind spots in real time. Furthermore, by integrating the Slowfast algorithm, the TCN algorithm, and the CNN algorithm, the present invention has developed a new method that uses the Slowfast algorithm to analyze RGB environmental features, supplements human motion features with the TCN algorithm, and supplements human structural features with the CNN algorithm. Ultimately, this method can improve the accuracy of the elderly-assisting robot's behavior recognition under panoramic vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a schematic diagram of the panoramic cloud robot for assisting the elderly at home according to the present invention;

[0031] Figure 2 Schematic diagram of the panoramic camera of the family elderly-assisting panoramic cloud robot of the present invention;

[0032] Figure 3 This is a schematic diagram of the architecture of the panoramic image processing system of the panoramic cloud robot for elderly care at home according to the present invention;

[0033] Figure 4 It is the inter-frame spatial feature vector diagram of human skeleton points in the panoramic human behavior recognition method of the panoramic cloud robot for assisting the elderly at home of the present invention;

[0034] Figure 5 It is the spatial feature map of the human skeleton points in the frame in the panoramic human behavior recognition method of the panoramic cloud robot for assisting the elderly at home of the present invention;

[0035] Figure 6 1 is a schematic diagram of a model of a panoramic human behavior recognition method of a panoramic cloud robot for assisting the elderly at home according to the present invention;

[0036] Figure 7 yes Figure 6 Model diagram of the spatiotemporal posture module in ;

[0037] Figure 8 This is a residual block structure diagram of the TCN algorithm in the panoramic human behavior recognition method of the panoramic cloud robot for elderly assistance at home in the present invention;

[0038] Figure 9 This is a schematic diagram of the principle of the RReLU activation function in the TCN algorithm in the panoramic human behavior recognition method of the panoramic cloud robot for elderly assistance in the present invention;

[0039] Figure 10 It is a skeleton space pose reconstruction diagram within a key frame in the CNN algorithm in the panoramic human behavior recognition method of the panoramic cloud robot for assisting the elderly at home of the present invention. DETAILED DESCRIPTION

[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0041] like Figure 1-2 As shown, the present invention discloses a home-based elderly-assisting panoramic cloud robot, comprising a panoramic camera, a 5G communication terminal, a processor, and a motion mechanism. The panoramic camera also includes a wide-angle camera, each of which is equipped with a fisheye lens with a field of view greater than 180°, and the fisheye lenses are fixedly installed at intervals of 180° around the vertical optical axis of the wide-angle camera.

[0042] When in use, the home-based elderly-assisting panoramic cloud robot can obtain panoramic images without blind spots in real time; preferably, the robot terminal can also be equipped with a display screen, and the processor can perform operations such as splicing and fusion of dual fisheye images, and can display the panoramic image on the screen in real time, facilitating human-computer interaction; the robot terminal can also transmit the captured panoramic images to the user in real time through the 5G communication terminal, realizing real-time panoramic monitoring in the home; in addition, the robot terminal has some basic functions such as voice interaction and face detection, which can complete simple service tasks.

[0043] Preferably, to facilitate robot movement, the motion mechanism serves as the robot's chassis and primarily consists of a power supply, a moving mechanism, a lifting mechanism, a ranging sensor, and a controller. The panoramic camera, 5G communication terminal, and processor are mounted on the robot's chassis. The power supply is a rechargeable 24V lithium battery, which is stepped down to produce different voltages for the robot's sensing, communication, motion, and processor electronic modules. The moving mechanism primarily consists of a coreless DC reduction motor with an encoder, a motor driver, a power wheel, and a driven wheel. A differential wheel structure with high efficiency and low cost is employed, ensuring unconstrained movement on flat indoor surfaces within a home. The lifting mechanism primarily consists of a screw-type lifting mechanism and a stepper motor, enabling the panoramic camera to be raised and lowered within a range of 1.2 to 1.8 meters. The ranging sensor, which can be a lidar or ultrasonic sensor, serves as the distance sensor and is primarily used for obstacle avoidance and autonomous navigation. The controller utilizes an embedded single-chip microcomputer as the main control unit, controlling the movement of the moving and lifting mechanisms, the collection and processing of ranging sensor information, and the transmission of motion information via a serial bus. When in use, the robot terminal can realize autonomous navigation in the home environment by utilizing ultrasonic ranging and panoramic visual sensing fusion and configuring the navigation algorithm in the processor.

[0044] In order to facilitate the acquisition of information about the surrounding environment at one time, preferably, the field of view of the fisheye lens is 210°, and the wide-angle camera can be an ordinary USB camera. The fisheye lens is fixedly installed at intervals of 180° around its vertical optical axis. By using two fisheye cameras, information about the surrounding environment can be acquired at one time. The panoramic camera is vertically installed and fixed directly above the central axis of rotation of the lifting mechanism and the moving mechanism. The observation angle of the panoramic camera can be adjusted by the rotational movement of the moving mechanism and the up and down movement of the lifting mechanism. In a home environment, the panoramic detection target may appear at the boundary of the panoramic image acquisition or splicing due to differences in height or position, resulting in distortion or information loss, thereby affecting the detection accuracy. Through the feedback motion control of the above two degrees of freedom, the position of the target in the panorama can be adjusted in time to maximize the acquisition of its feature information in the panorama. Through multiple detections from multiple angles, the accuracy of recognition can be ensured.

[0045] Figure 3The following is a schematic diagram of the architecture of the panoramic image processing system of the present invention, which includes a service robot terminal, a point-to-point 5G communication system, an edge calculator, and a cloud server. In order to increase computing power, it is preferred that an edge calculator and a cloud server are also included. Each service robot terminal is connected to a dedicated edge calculator via 5G communication, and each edge calculator is connected to the cloud server via fiber optic Ethernet to realize intelligent processing of panoramic images. The robot can recognize human posture, object category and location and other information in the panoramic image, and can complete complex functions such as monitoring the elderly for falls, supervising the elderly to take medicine, and helping the elderly to find things, providing users with more comprehensive and intelligent services.

[0046] In order to increase the computing speed and anti-interference ability, preferably, the robot accesses the edge calculator in a point-to-point communication manner through a 5G communication terminal. The point-to-point 5G communication system includes a 5G communication terminal on the robot terminal and the edge calculator terminal, and a 5G transit router set up in the middle. The transit router can use a 5G industrial router on the market (for example, based on Huawei MH5000 or Spreadtrum chip), which has transceiver antennas, USB3.0, WAN and other interfaces. The robot sends the collected panoramic image to the transit router with a specific 5G signal through the 5G communication terminal, and the transit router forwards the information to the edge calculator, thereby realizing point-to-point 5G communication. A pair of 5G communication terminals corresponds to a specific transit router, and the data transmission is stable and has strong anti-interference ability.

[0047] The edge calculator is mainly composed of a graphics processor (for example, NVIDIA TX2). On the one hand, the edge calculator is connected to the 5G communication terminal via USB, quickly accesses the 5G local area network formed with the robot terminal, and processes the received panoramic images and detects targets in real time based on the distortion-aware spherical network, and feedback controls the movement of the robot terminal; on the other hand, the edge calculator is connected to the cloud server via fiber optic Ethernet. The distortion-aware spherical convolutional neural network is a unique processing network for dual fisheye panoramic images. The edge calculator needs to be equipped with a trained distortion-aware spherical convolutional neural network model to process dual fisheye panoramic images.

[0048] Elderly care robots use panoramic vision to detect and recognize postures and behaviors. This is achieved by processing corrected panoramic images using a deep neural network. While existing panoramic image correction methods are relatively mature, the key to improving the accuracy of panoramic behavior recognition lies in the design of the behavior recognition network model. However, existing behavior recognition models fail to simultaneously address environmental, motion, and structural features, lacking comprehensive feature extraction. Therefore, to further improve the accuracy of behavior recognition models, a more detailed and comprehensive capture of human behavior features is necessary. Therefore, this project proposes a posture skeleton-guided human behavior recognition model. First, a spatiotemporal posture model is developed to explore the feature representation and data processing methods of the human posture skeleton in both the temporal and spatial domains, capturing both temporal motion and static structural details of human behavior. Building on this model, the Slowfast model is introduced to complement the RGB environmental features required for behavior recognition. Finally, methods for integrating feature information and training a model specifically for elderly care are investigated to further improve the accuracy of elderly care robots using panoramic vision.

[0049] In order to achieve the above purpose, Figure 4-10 The present invention also discloses a panoramic human behavior recognition method adopted by the above-mentioned home-based elderly-assisting panoramic cloud robot, including RGB features and posture skeleton features, which can simultaneously process RGB features and posture skeleton features, including the following steps:

[0050] Step 1: Obtain raw data through a panoramic camera;

[0051] Step 2: Use the Openpose model to process the raw data, estimate the human body posture bone points and posture skeleton, and obtain the preliminarily processed data;

[0052] Step 3: Use the Slowfast algorithm to process the preliminarily processed data to obtain the last convolution layer, use the TCN algorithm to process the processed data to obtain the time domain features of the human body posture skeleton points, use the CNN algorithm to process the processed data to obtain the spatial features of the human body posture skeleton, and fuse the time domain features and the spatial features to obtain the first convolution layer;

[0053] Step 4: Make the last convolution layer and the first convolution layer pass through the fully connected layer to complete feature fusion;

[0054] Step 5: Determine the behavior category.

[0055] For the convenience of calculation, the Openpose model is used in step 2 to predict 15 skeletal nodes from bottom to top. Since 15 skeletal points can cover all the main joints of the human body, and the remaining skeletal points such as fingers, toes, facial organs, etc. have little effect on human behavior recognition, in order to avoid increasing unnecessary computational complexity, the present invention only takes the 15 most important skeletal points of the human body and studies the two-dimensional coordinate sequence of these 15 skeletal points.

[0056] The following describes the relevant definitions of feature extraction. After completing human posture estimation, the present invention extracts spatiotemporal feature information between sequences of human skeleton points as input data for the behavior recognition network. Compared with feature information based on distance and angle, spatiotemporal features contain richer feature information and are more sensitive to changes in human skeleton points. The human skeleton point feature representation proposed in the present invention includes: Figure 4 The spatial variation characteristics of the skeleton points in the time dimension are shown; Figure 5 The spatial pose features of the skeleton points in each video frame are shown.

[0057] In the behavior recognition model proposed in the present invention, the input data can be a two-dimensional coordinate sequence of key points of the human skeleton, and the coordinate group can be expressed as: Where N is the total number of human skeleton points, T is the total number of frames of input video data, and t is the tth frame of video data. Therefore, the coordinates of a set of human skeleton points in a single frame can be defined as the matrix M t :

[0058]

[0059] in and yes The present invention sets N to 15, and T is determined by the input data. Figure 4 As shown, the spatial variation vector feature S of the skeleton point in the time dimension t It can be expressed as:

[0060]

[0061] That is, the spatial change vector feature of the skeleton point in the time dimension is obtained by subtracting the coordinate groups of the skeleton points in the previous and next frames. This feature can clearly represent the action part of the human body, which can help the network enhance feature attention, strengthen the learning of the action part of the human body, and help improve recognition accuracy.

[0062] The spatial features of the skeleton points in the complete video sample can be expressed as a matrix M:

[0063]

[0064] In the matrix M, each row of coordinates provides a strong correlation between skeleton points in the spatial domain, while each column of coordinates represents the motion of a single skeleton point. Furthermore, using OpenCV methods, this matrix can be used to reconstruct a two-dimensional skeleton point image, excluding the surrounding background. This 2D RGB image represents the global characteristics of the skeleton point's spatial position, increasing the feature dimension and improving the model's recognition accuracy.

[0065] In step three, in order to simultaneously emphasize environmental features, human movement features, and human structure features, such as Figure 6 As shown in FIG, the present invention proposes a new model that combines the Slowfast algorithm, the TCN algorithm and the CNN algorithm, performs feature fusion in step 4, and determines the behavior category in step 5. Figure 6 In the first part, the Slowfast algorithm is used to process the raw data obtained by the panoramic camera normally to obtain the last convolution layer; Figure 7 yes Figure 6 The model diagram of the space-time posture module in the dotted part is as follows: Figure 7 As shown in the figure, the temporal features of the human body posture skeleton points are processed by the TCN algorithm to obtain a one-dimensional convolution, and then the spatial features of the human body posture skeleton are processed by the CNN algorithm to obtain a two-dimensional convolution. After that, the temporal features and the spatial features are fused to obtain the first convolution layer, and in step four, the last convolution layer and the first convolution layer are passed through the fully connected layer to complete the feature fusion.

[0066] like Figure 8-9 As shown, in order to improve the feature extraction capability of the TCN algorithm, preferably, the TCN algorithm includes a residual block, and the TCN algorithm is connected by residual blocks. The residual block has two convolution layers, and the number of convolution kernels of the two layers is the same. This can improve the feature extraction capability of the algorithm, and can effectively expand the receptive field of the network, which is conducive to learning time-dependent action behaviors. Batch normalization and random rectified linear units are performed before each convolution layer in the residual block, so that the residual block can learn more feature information. The random rectified linear unit is implemented using the RReLU activation function, and the algorithm network is connected using residual blocks, which can improve the recognition accuracy and interpretability of time series data, and can solve problems such as gradient disappearance and network degradation. The calculation formula of the lth residual block is Y l =Y l-1 +F(W l，1 , W l，2 , Y l-1 ), the function F of the residual block is F(W l，1 , W l，2 , Y l-1 )=W l，2 ·σ(W l，1 ·σ(Y l-1))), where σ represents the RReLU activation function.

[0067] As Figure 9 shown, the formula of the RReLU activation function is as follows:

[0068]

[0069] where: a ji ~U(l, u), l < u and l, u ∈ [0, 1)

[0070] In order to enhance the anti-interference ability, extract richer feature information, and improve the accuracy of behavior recognition. As Figure 10 shown, preferably, the CNN algorithm is implemented through the following process: First, construct a spatial pose feature matrix of human body bone points, downsample the feature information according to the low-frequency sampling rate in the Slowfast algorithm, select the key frames of human action behaviors, and then use OpenCV to process the bone point pose information in the key frames to reconstruct a background-free enhanced human body bone point map. Finally, use the Resnet-18 model to process the enhanced bone point map.

[0071] The behavior recognition two-stream network based on the spatio-temporal features of bone points proposed by the present invention extracts the temporal features and spatial features of the human body pose skeleton through the TCN stream and the CNN stream respectively, and finally fuses the two to obtain the spatio-temporal features of human behavior details, improving the accuracy of behavior recognition.

[0072] In real situations, the intensity of different human behaviors varies. Therefore, in the process of behavior recognition, the importance of the information contained in the temporal features and spatial features is also different. Directly using a simple average fusion method will result in a large error. To solve the above problem, preferably, the temporal features and the spatial features are fused through weighted fusion to obtain the first convolutional layer. The weighted fusion calculation formula is V ts = avg(V t , V s ) = ω t V t + ω s V s , where ω t and ω s respectively represent the respective weights when fusing the two features. The weights can be updated following the backpropagation algorithm of the network model, and thus the spatio-temporal fusion feature information of the human body pose skeleton can be obtained.

[0073] To implement a posture skeleton-guided video human behavior recognition model, step four requires integrating the spatiotemporal posture module with the Slowfast model. This method combines the final convolutional layer of Slowfast with the spatiotemporal posture module, followed by feature fusion through a fully connected layer. This directs the network model's attention to the details of human behavior, supplementing the detailed human posture skeleton information essential for elderly-assisting robots.

[0074] In addition, considering that the proposed model processes both RGB appearance features and posture skeleton features, errors may occur. To solve the above problem, preferably, two cross entropy losses and a total loss are included. The total number of video samples is M, the total number of behavior categories is N, and the cross entropy loss of RGB features is:

[0075]

[0076] The cross entropy loss of the pose skeleton feature is:

[0077]

[0078] The total loss is:

[0079]

[0080] Where y m，n is the true value of the result, and Respectively represent the prediction results when using RGB data and posture skeleton data, represents the weight decay regularization of all parameters, λ is the decay coefficient, Used to avoid overfitting.

[0081] In summary, the present invention discloses a panoramic human behavior recognition method for a panoramic cloud robot for home elderly assistance. It performs detailed feature representation and effective feature extraction on the RGB appearance environment features, temporal motion detail features, and spatial skeleton structure features of human behavior, and studies the fusion method of different features, ultimately realizing network training and iterative updating.

[0082] To ensure this invention's focus on elderly care, we also conducted research on the required datasets. Designed specifically for elderly-assistance robots, this invention not only requires extremely high recognition accuracy but also focuses on the daily behaviors of the elderly. Therefore, the training dataset must encompass common behaviors experienced by the elderly in their daily lives. Based on our research on the daily lives of the elderly, this study categorizes these behaviors as follows: Daily: sleeping / lying down, watching TV, smoking, reading, taking medicine, and waving; Exercise: walking, jogging, standing, and riding a bicycle / moped; Dangerous: uncomfortable leaning against a wall, and sudden, dangerous behaviors, primarily those threatening falls.

[0083] Based on the above requirements, we first selected the AVA (Atomic Visual Actions) dataset for model training. This dataset contains the daily behaviors of the elderly mentioned above and meets the model training requirements. It should be noted that when using the AVA dataset, the present invention needs to estimate the human posture skeleton in the original video sample through Openpose and then pass the data into the spatiotemporal posture module for training. After that, the UR Fall Detection dataset is used for special training on fall behavior. This dataset focuses on fall behavior and contains 30 types of fall behavior video samples. It uses dual cameras to capture RGB video, which can help the model improve its ability to recognize the fall behavior of the elderly.

[0084] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0085] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0086] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for panoramic human behavior recognition of a panoramic cloud robot for elderly care at home, wherein the panoramic cloud robot includes a panoramic camera, and the method includes RGB features and posture skeleton features, characterized in that: Processing RGB features and posture skeleton features simultaneously includes the following steps: Step 1: Obtain raw data through a panoramic camera; Step 2: Use the Openpose model to process the raw data, estimate the human body posture bone points and posture skeleton, and obtain the preliminarily processed data; Step 3: Use the Slowfast algorithm to process the preliminarily processed data to obtain the last convolution layer, use the TCN algorithm to process the processed data to obtain the time domain features of the human body posture skeleton points, use the CNN algorithm to process the processed data to obtain the spatial features of the human body posture skeleton, and fuse the time domain features and the spatial features to obtain the first convolution layer; Step 4: Make the last convolution layer and the first convolution layer pass through the fully connected layer to complete feature fusion; Step 5: Determine the behavior category.

2. The method for recognizing human behavior of a panoramic cloud robot for assisting the elderly at home as claimed in claim 1, characterized in that: The TCN algorithm includes a residual block, which is connected by residual blocks. The residual block has two convolutional layers with the same number of convolution kernels. Batch normalization and random rectified linear units are performed before each convolutional layer in the residual block. The random rectified linear unit is implemented using the RReLU activation function. The calculation formula of the lth residual block is Y l =Y l-1 +F(W l,1 ,W l,2 ,Y l-1 ), the function F of the residual block is F(W l,1 ,W l,2 ,Y l-1 )=W l,2 ·σ(W l,1 ·σ(Y l-1 )), where σ represents the RReLU activation function.

3. The method for recognizing human behavior of a panoramic cloud robot for assisting the elderly at home as claimed in claim 2, characterized in that: The CNN algorithm is implemented through the following process: first, a spatial pose feature matrix of human skeleton points is constructed, and the feature information is downsampled according to the low-frequency sampling rate in the Slowfast algorithm, and key frames of human motion behaviors are selected. Then, OpenCV is used to process the skeleton point pose information in the key frames, and a background-free human skeleton point enhanced image is reconstructed. Finally, the skeleton point enhanced image is processed using the Resnet-18 model.

4. The method for recognizing human behavior of a panoramic cloud robot for assisting the elderly at home as claimed in claim 3, characterized in that: The time domain features and the spatial features are weightedly fused to obtain the first convolution layer. The weighted fusion calculation formula is V ts =avg(V t ,V s )=ω t V t +ω s V s , where ω t With ω s They represent the respective weights when fusing two features.

5. The method for recognizing panoramic human behavior of a panoramic cloud robot for assisting the elderly at home as claimed in claim 4, characterized in that: It also includes two cross entropy losses and a total loss. The total number of video samples is M, the total number of behavior categories is N, and the cross entropy loss of RGB features is: The cross entropy loss of the pose skeleton feature is: The total loss is: Where y m,n is the true value of the result, and Respectively represent the prediction results when using RGB data and posture skeleton data, represents the weight decay regularization of all parameters, λ is the decay coefficient, Used to avoid overfitting.

Images