Automatic recognition method of daily behavior of the elderly based on multimodal and multi-granularity

Abstract

The present invention discloses a multimodal and multi-granular automatic recognition method for daily behaviors of the elderly, comprising the following steps: the skeletal modality and RGB modality of the original input video are respectively refined into skeletal points + skeleton and nominations + keyframes; a graph structure is established for the skeletal points and skeleton data, and features are learned using graph convolutional networks (S-GTCNs) to obtain action category scores based on the skeletal points and skeleton data; nominations are extracted from the video using a boundary-sensitive network, and the underlying features of nominations and keyframes are extracted using an I3D network. A graph structure is established based on the nominations and keyframe data, and features are learned using graph convolutional networks (R-GCNs) to obtain action category scores based on nominations and keyframe data; the recognition scores of the four granularities are fused to obtain the final recognition result. Compared with other methods, this method has a higher recognition accuracy.

Description

Technical Field

[0001] The present invention relates to behavior recognition technology in the field of computer vision, and in particular to a multimodal and multi-granularity-based automatic recognition method for the daily behavior of the elderly. Background Art

[0002] The goal of elderly behavior recognition is to automatically identify the behavioral categories of elderly individuals in videos. Research on videos of elderly individuals engaging in daily activities reveals that the semantics of elderly behavior are highly granular. This means that most elderly behaviors overlap significantly across the duration of the videos, while the semantic information required to truly distinguish between categories is relatively subtle.

[0003] Current mainstream benchmark methods for behavior recognition can be roughly divided into three categories: 1) Recurrent Neural Networks (RNNs)-based methods; 2) Convolutional Neural Networks (CNNs)-based methods; and 3) Graph Convolutional Networks (GCNs)-based methods. RNN-based methods can exploit temporal relationships to process sequential data, but they suffer from the vanishing gradient problem. CNN-based methods can process high-dimensional data and automatically extract features, but they are unable to handle non-Euclidean spatial data. GCN-based methods, on the other hand, can fully learn non-Euclidean spatial data, aggregating spatial and temporal information, offering advantages over the first two deep network models. However, existing GCN-based automatic recognition technologies for daily behaviors in the elderly are based solely on skeletal modalities and have a high data granularity. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-modal and multi-granularity based automatic recognition method for the daily behavior of the elderly.

[0005] The technical solution to achieve the purpose of the present invention is: a multimodal and multi-granular automatic recognition method for daily behavior of the elderly, comprising the following steps:

[0006] Step 1: The skeletal mode and RGB mode of the original input video are refined into skeletal points + skeleton and nominated segments + key frames respectively;

[0007] Step 2: Build a graph structure for the skeleton points and skeleton data, and use the graph convolutional network S-GTCNs to learn features and obtain action category scores based on the skeleton points and skeleton data;

[0008] Step 3: Use I3D to extract the underlying features of nominations and keyframes, build a graph structure based on nomination and keyframe data, and use graph convolutional networks (R-GCNs) to learn features and obtain action category scores based on nomination and keyframe data.

[0009] Step 4: Fusion the recognition scores of the four granularities to obtain the final recognition result.

[0010] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the above-mentioned method for automatically identifying daily behaviors of the elderly based on multimodality and multi-granularity are implemented.

[0011] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned method for automatic identification of daily behaviors of the elderly based on multimodality and multi-granularity are implemented.

[0012] Compared with the prior art, the present invention has the following significant advantages:

[0013] (1) This paper proposes a new multimodal and multi-granularity graph convolutional network, which simultaneously models data of two types of modalities and four granularities through an attention graph convolutional network to capture human behavior and solve the problem of elderly behavior recognition in actual scenarios; (2) A data representation strategy of four granularities of "point-line-surface-segment" is designed, which uses the information complementarity and integration of multi-granularity data to finely depict fine-grained human behavior in videos; (3) The proposed method is evaluated on the industry-standard elderly behavior recognition dataset ETRI-Activity3D and achieves excellent performance. Its recognition accuracy is ahead of all existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a diagram of fine-grained definition and division, where (a) is the division of skeletal modal data and (b) is the division of RGB modal data.

[0015] Figure 2 It is a structural diagram of the present invention.

[0016] Figure 3 Schematic diagram of the skeleton-based graph convolutional network S-GTCNs.

[0017] Figure 4 This is a schematic diagram of the graph convolution module R-GCNs based on the RGB modality. DETAILED DESCRIPTION

[0018] To identify the daily behaviors of the elderly, this paper proposes a multimodal multi-granularity graph convolutional network (MM-GCN). This approach uses an attention graph convolutional network to jointly model data from two modalities and four granularities, comprehensively revealing the spatiotemporal evolution of elderly behavior. Compared to other methods, this approach achieves higher recognition accuracy.

[0019] The multimodal and multi-granular automatic identification method for daily behaviors of the elderly of the present invention comprises the following steps:

[0020] Step 1: The skeletal mode and RGB mode of the original input video are refined into skeletal points + skeleton and nominated segments + key frames respectively;

[0021] Step 2: Build a graph structure for the skeleton points and skeleton data, and use the graph convolutional network S-GTCNs to learn features and obtain action category scores based on the skeleton points and skeleton data;

[0022] Step 3: Use I3D (Inflated 3D ConvNet) to extract the underlying features of the nomination and keyframes, build a graph structure based on the nomination and keyframe data, and use graph convolutional networks (R-GCNs) to learn features and obtain action category scores based on the nomination and keyframe data;

[0023] Step 4: Fusion the recognition scores of the four granularities to obtain the final recognition result.

[0024] Furthermore, the skeleton point data in step 1 is already available in the dataset, and the skeleton data is differentiated to obtain the skeleton data; a boundary-sensitive network (BSN) is used to extract multiple nominated segments of a video, and the nominated segments are sampled at equal intervals to obtain key frames.

[0025] Furthermore, step 2 specifically includes the following steps:

[0026] Step 201: For a single frame of skeleton point data, define the graph structure G S =(V S ,E S ), where V S Represents a node set with N skeleton points, E S The adjacency matrix A S ∈R N×N Defined bone edge set; if the node v i and v j There is a connecting edge, i=1,…,N,j=1,…,N,j≠i, then the initial Ai,j = 1, otherwise 0; a complete action can be represented by the graph sequence G S express, Represents G S The node feature set of x is the node v at time t n The feature vector (C dimension), T represents the total number of frames, and N represents the total number of nodes;

[0027] Step 202: For a single frame of skeleton data, define the graph structure G B =(V B ,E B ), where V B Represents a node set with N skeletons, E B The adjacency matrix A B ∈R N×N The edge set defined; if the node v i and v j There are connecting edges, i=1,2,…,N,j=1,2,…,N, then the initial A i,j =1, otherwise 0; graph sequence G B Represents a complete action, the node feature set is Where x represents the node v at time t n The feature vector (C dimension), T represents the total number of frames, and N represents the total number of nodes;

[0028] In step 203, the skeleton points and skeleton data are sent to the S-GTCNs (Skeleton-graph convolution + temporal convolution network). The S-GTCNs includes a GTCN and two A-GTCN modules. After normalization by batch normalization (BN), they are sent to two network units with the same structure and independent parameters, the GTCN (graph convolution + temporal convolution network) modules. The network structure of GTCN includes one layer of GCN and two layers of TCN. The GTCN module consists of one layer of graph convolution and two layers of temporal convolution. GCN is used to aggregate and extract the features of the skeletal modality, and TCN is used to extract temporal information. The output of the GTCN unit is sent to the A-GTCN (attention graph convolution + temporal convolution network) unit. The network structure of A-GTCN includes one layer of GCN and two layers of TCN, and the attention mechanism is used in GCN. The A-GTCN unit introduces the attention mechanism; the skeleton points and skeleton data obtain their respective outputs through the S-GTCNs module:

[0029]

[0030]

[0031] Among them, FC represents the full-connect layer (Full-Connect layer), score S and score B Represent the bone point-based and skeleton-based output scores respectively.

[0032] Furthermore, step 3 specifically includes the following steps:

[0033] Step 301: For the video nomination segment set obtained in step 1 Each p i can represent an action instance, where x i Represents the characteristics, t s Indicates the start time of the action, t e Indicates the end time of the action;

[0034] Step 302: sampling key frames in the nominated segment by sampling at equal intervals;

[0035] Step 303: Use I3D to extract I3D features x of the nominated segments and key frames. i ;

[0036] Step 304: Take the nominated segments / key frames as nodes and the relationships between the nominated segments / key frames as edges to build a graph structure. i,s ,t i,e ) and p j =(t j,s ,t j,e )'s timing overlap When it is greater than the threshold τ, a context-related edge is formed:

[0037]

[0038] The same principle is used to build a graph structure based on key frames;

[0039] In step 305, the I3D features of the nominated segment and keyframe are fed into two R-GCNs (RGB-graph convolutional network) modules with identical structures and independent parameters. R-GCNs consists of two GCN layers, where the second GCN layer introduces an attention mechanism to extract the relationship information between the nominated segment and the keyframe. The attention mechanism can focus more on useful information, making the network more efficient. The nominated segment and keyframe receive their own outputs through the R-GCNs module:

[0040]

[0041]

[0042] Among them, FC represents the fully connected layer, scoreF and score P represent the keyframe-based and nomination-based output scores, respectively.

[0043] Furthermore, step 4 specifically includes the following steps:

[0044] The final classification results are obtained by integrating the two-stream R-GCNs and the two-stream S-GTCNs:

[0045] score=α·score S +β·score B +δ·score F +σ·score P

[0046] Where α, β, δ, and σ represent the weights of each flow output.

[0047] The technical solution of the present invention is described in detail below with reference to the embodiments and drawings.

[0048] Example

[0049] Combine Figure 1 A crowd behavior recognition method based on "host-parasite" includes four processes: extracting I3D features, establishing a graph structure based on skeletal modal data and performing learning and recognition, establishing a graph structure based on RGB modal data and performing learning and recognition, and fusing the four-stream output. Figure 2 The specific structure diagram includes an S-GTCNs module and an R-GCNs module.

[0050] Extracting I3D features includes the following steps:

[0051] Step 1) Train the BSN network and use the trained BSN network to generate 100 nominated segments for each video.

[0052] Step 2) Sample the video frames in each nominated segment at equal intervals to obtain key frames.

[0053] Step 3) Train the I3D network and extract I3D features for each nominated segment and keyframe.

[0054] Establishing a graph structure based on skeletal modal data and performing learning and recognition includes the following steps:

[0055] Step 4) Subdivide the raw input into two granularities: skeleton points and skeletons. For single-frame skeleton data, the skeleton data is derived from the differences between the skeleton points. The input size is (N, M, V, C1, T), where N is the number of samples, M is the number of people in the video, C1 is the number of channels, T is the total number of frames, and V is the number of nodes.

[0056] Step 5) Perform batch normalization on the input data.

[0057] Step 6) Figure 3 The S-GTCNs in the paper are used to process skeletal modality data. They consist of a single GTCN layer and two A-GTCN layers. The GTCN layer consists of a single GCN layer and two TCN layers, while the A-GTCN layer consists of a single GCN layer with an attention mechanism and two TCN layers. Data at the two granularities are fed into two GTCN modules, each with identical structure and independent parameters. Within the GTCN module, a single graph convolution layer aggregates information from its neighborhood to enhance node features, while two layers of temporal graph convolution capture extended contextual information. The number of output channels per layer is 96, 96, and 96, respectively.

[0058] Step 7) Feed the output of step 6 into two A-GTCN units connected in series. Unlike the previous units, the A-GTCN unit introduces an attention mechanism, allowing the model to focus on key features. The number of output channels per layer is 96, 192, 192, 192, 384, and 384.

[0059] Step 8) Feed the output of step 7 into the fully connected layer to obtain the score for each action category:

[0060]

[0061]

[0062] Among them, FC represents the fully connected layer, score S and score B Represent the bone point-based and skeleton-based output scores respectively.

[0063] Building a graph structure based on RGB modality data and performing learning and recognition includes the following steps:

[0064] Step 9) Create a graph structure G based on RGB modal data P . Use the nomination collection To represent the action instance in the video, where x i Represents the characteristics, t s Indicates the start time of the action, t e Indicates the end time of the action. Nominate as node v i ,i=1,2,…,N,when two nominations p i =(t i,s ,t i,e ) and p j =(t j,s ,t j,e )'s timing overlap When it is greater than the threshold τ, a context-related edge is formed:

[0065]

[0066] Step 10) Figure 4 The R-GCNs in

[15] are designed to process RGB video data. They consist of one GCN layer and one GCN layer with attention. The I3D features of the nominated segment and keyframe are fed into two GCN modules with identical architectures and independent parameters. The GCNs consist of two layers of graph convolution, with the second layer incorporating an attention mechanism.

[0067] Step 11) The nominated segment and key frame data are outputted as follows after step 10:

[0068]

[0069]

[0070] The fusion of the four stream outputs includes the following steps:

[0071] Step 12) The two-stream R-GCNs and the two-stream S-GTCNs are integrated to obtain the final classification results, as shown below:

[0072] score=α·score S +β·score B +δ·score F +σ·score P

[0073] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

[0074] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0075] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal and multi-granular automatic recognition method for daily behavior of the elderly, characterized by: The following steps are involved: Step 1: The skeletal mode and RGB mode of the original input video are refined into skeletal points + skeleton and nominated segments + key frames respectively; Step 2: Build a graph structure for the skeleton points and skeleton data, and use the graph convolutional network S-GTCNs to learn features to obtain action category scores based on the skeleton points and skeleton data. The specific steps include the following: Step 201: For a single frame of skeleton point data, define the graph structure G S =(V S ,E S ), where V S Represents a node set with N skeleton points, E S The adjacency matrix A S ∈R N×N Defined bone edge set; if the node v i and v j There is a connecting edge, i=1,…,N,j=1,…,N,j≠i, then the initial A i,j =1, otherwise 0; a complete action can be represented by the graph sequence G S express, Represents G S The node feature set of x is the node v at time t n The C-dimensional feature vector of , T represents the total number of frames, and N represents the total number of nodes; Step 202: For a single frame of skeleton data, define the graph structure G B =(V B ,E B ), where V B Represents a node set with N skeletons, E B The adjacency matrix A B ∈R N×N The edge set defined; if the node v i and v j There are connecting edges, i=1,2,…,N,j=1,2,…,N, then the initial A i,j =1, otherwise 0; graph sequence G B Represents a complete action, the node feature set is Where x represents the node v at time t n The C-dimensional feature vector of , T represents the total number of frames, and N represents the total number of nodes; In step 203, the skeleton points and skeleton data are fed into S-GTCNs, which include one GTCN and two A-GTCN modules. After batch normalization, they are fed into two GTCN modules with identical structures and independent parameters. The GTCN network structure includes one GCN layer and two TCN layers. The GCN is used to aggregate and extract features of the skeleton modality, and the TCN is used to extract temporal information. The output of the GTCN unit is fed into the A-GTCN unit. The A-GTCN network structure includes one GCN layer and two TCN layers. The A-GTCN unit introduces an attention mechanism. The skeleton points and skeleton data are outputted by the S-GTCN modules: Among them, FC represents the fully connected layer, score S and score B Represent the bone point-based and skeleton-based output scores respectively; Step 3: Use I3D to extract the underlying features of the nominated segments and key frames, build a graph structure based on the nominated segments and key frame data, and use graph convolutional networks (R-GCNs) to learn features and obtain action category scores based on the nominated segments and key frame data. Step 4: Fusion the recognition scores of the four granularities to obtain the final recognition result.

2. The multimodal and multi-granular automatic identification method for daily behavior of the elderly according to claim 1 is characterized in that: The skeleton point data in step 1 is already available in the dataset, and the skeleton data is obtained by differential analysis. The boundary-sensitive network is used to extract multiple nominated segments of a video, and the nominated segments are sampled at equal intervals to obtain key frames.

3. The multimodal and multi-granular automatic identification method for daily behavior of the elderly according to claim 1 is characterized in that: Step 3 specifically includes the following steps: Step 301: For the video nomination segment set obtained in step 1 Each p i can represent an action instance, where x i Represents the characteristics, t i,s Indicates the start time of the action, t i,e Indicates the end time of the action; Step 302: sampling key frames in the nominated segment by sampling at equal intervals; Step 303: Use I3D to extract I3D features x of the nominated segments and key frames. i ; Step 304: Build a graph structure by using the nominated segments / keyframes as nodes and the relationships between the nominated segments / keyframes as edges; When two nominated paragraphs p i =(t i,s ,t i,e ) and p j =(t j,s ,t j,e )'s timing overlap When it is greater than the threshold τ, a context-related edge is formed: The same principle is used to build a graph structure based on key frames; In step 305, the I3D features of the nominated segment and keyframe are fed into two R-GCNs modules with identical structures and independent parameters. The R-GCNs module contains two GCN layers, where the second GCN layer introduces an attention mechanism to extract the relationship information between the nominated segment and the keyframe. The nominated segment and keyframe receive their own outputs through the R-GCNs module: Among them, score F and score P Represent the output scores based on key frames and based on nominated segments, respectively.

4. The multimodal and multi-granular automatic identification method for daily behavior of the elderly according to claim 3 is characterized in that: Step 4 is as follows: The final classification results are obtained by integrating the two-stream R-GCNs and the two-stream S-GTCNs: score=α·score S +β·score B +δ·score F +σ·score P Where α, β, δ, and σ represent the weights of each flow output.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the processor implements the steps of the method for automatically identifying daily behaviors of the elderly based on multimodality and multi-granularity as described in any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for automatically identifying daily behaviors of the elderly based on multimodality and multi-granularity as described in any one of claims 1 to 4 are implemented.

Images