Video skeleton feature extraction method and device based on hypergraph neural network

Abstract

The invention provides a video skeleton feature extraction method and device based on a hypergraph neural network, and the method comprises the steps: collecting pedestrian video data, and obtaining a target skeleton set in the pedestrian data; constructing a loss function based on the target skeleton set; constructing a skeleton adjacency tensor of multi-target interaction of the target skeleton, and extracting skeleton hidden features by using a hypergraph neural network based on the skeleton adjacency tensor; performing adaptive fusion on the skeleton hidden features to obtain video skeleton hidden features; judging whether the loss value of the loss function is converged or not or whether the number of iterations reaches an upper line or not; if yes, outputting video skeleton features; and if not, carrying out iterative optimization on the loss function by utilizing a stochastic gradient descent method. According to the invention, by introducing an adaptive fusion mechanism, the weights of different layers of features can be dynamically adjusted according to the internal attributes of the skeleton features, and the accuracy and robustness of feature extraction are further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of feature extraction, and in particular to a method and device for extracting video skeleton features based on a hypergraph neural network. Background Art

[0002] With the continuous deepening of scientific and technological revolution and industrial transformation, artificial intelligence technology has become a key core technology leading future development. my country has also positioned the development of artificial intelligence as an important strategy to enhance national competitiveness and ensure national security. In complex video surveillance scenarios, the use of artificial intelligence technology to analyze abnormal behavior of pedestrians can timely discover and warn potential abnormal behaviors, assist management departments to respond and intervene quickly, effectively prevent or mitigate the impact of public safety incidents, and ensure the safety of people's lives and property. In these scenarios, the abnormal behavior of pedestrians shows complex characteristics such as temporal sequence, heterogeneity, and interactivity. The interaction between multiple targets naturally forms a multi-target spatiotemporal skeleton graph. Therefore, graph neural network technology is used to extract the spatiotemporal pattern of the skeleton from time-varying pedestrian video data, deeply explore the potential laws in the video and the interaction pattern between multiple targets, and provide key decision support for abnormal behavior analysis.

[0003] When describing target motion behaviors, target skeletons (i.e., the topological representation of the joints and bones of pedestrian targets in video streams) can show strong robustness in complex video surveillance scenarios, even in the face of interference factors such as target scale, perspective, and speed changes. However, when describing target skeleton features, existing studies mainly focus on the independent modeling of temporal and spatial features, but fail to fully consider the overall temporal variation, high-order, and heterogeneity of target skeleton features in video streams. Although these methods can capture some of the spatiotemporal characteristics of target skeletons in video streams, they cannot avoid the breakage of global spatiotemporal correlations and the loss of core information, and also increase the complexity of the model. In addition, existing studies usually use typical graph convolutional networks to represent single target skeleton sequences. This method not only cannot handle the situation where target features are missing, but also further increases the computational burden of the model when extracting video skeleton features, resulting in low accuracy in extracting video skeleton features. Summary of the invention

[0004] The present invention aims to at least solve the technical problem of low accuracy in extracting video skeleton features in the prior art, and in particular innovatively proposes a video skeleton feature extraction method and device based on a hypergraph neural network.

[0005] In order to achieve the above-mentioned object of the present invention, the present invention provides a video skeleton feature extraction method based on a hypergraph neural network, the method comprising:

[0006] S1. Collect pedestrian video data and obtain a target skeleton set in the pedestrian data;

[0007] S2. constructing a loss function based on the target skeleton set;

[0008] S3, constructing a skeleton adjacency tensor of multi-target interaction of the target skeleton, and extracting skeleton hidden features using a hypergraph neural network based on the skeleton adjacency tensor;

[0009] S4, adaptively fusing the skeleton latent features to obtain video skeleton latent features;

[0010] S5. Determine whether the loss value of the loss function converges or whether the number of iterations reaches an upper limit;

[0011] If so, output the video skeleton features;

[0012] If not, the loss function is iteratively optimized using stochastic gradient descent.

[0013] As an optional embodiment of the present invention, optionally, the expression of the loss function in step S2 is:

[0014]

[0015] Among them, ω represents the loss value, i represents the i-th target skeleton, and Y i represents the true behavior label, log() represents the logarithmic function, represents the predicted target behavior label, λ represents the regularization factor, || || F represents the F norm, Q 0 represents the initialized skeleton latent feature tensor, M represents the time transformation matrix, and W represents the feature transformation tensor.

[0016] As an optional embodiment of the present invention, optionally, constructing the skeleton adjacency tensor of the target skeleton in step S3 includes:

[0017] S301, taking the joints of the target as nodes and the skeletons of the target as edges, constructing a skeleton topology graph based on the nodes and edges, and obtaining a skeleton topology graph sequence based on a time dimension;

[0018] S302: Standardize the skeleton topology map sequence to obtain a skeleton adjacency tensor.

[0019] As an optional embodiment of the present invention, optionally, extracting skeleton latent features using a hypergraph neural network in step S3 includes:

[0020] S303, performing a tensor multiplication conversion operation on the joint adjacency tensor, and extracting the skeleton latent features of each layer;

[0021] S304, performing a graph convolution operation on each layer of skeleton latent features to obtain the spatial dependency relationship of the joints;

[0022] S305, capturing hidden features of similar joints in each layer of the skeleton based on the spatial dependency relationship;

[0023] S306, integrating the skeleton latent features of each layer, the spatial dependency of the joints and the latent features of the adjacent joints, and performing feature transformation operations and nonlinear activation operations on the integration results to obtain skeleton latent features.

[0024] As an optional embodiment of the present invention, optionally, in step S306, the expression for integrating the skeleton latent features of each layer, the spatial dependency of the joints and the latent features of the adjacent joints is:

[0025]

[0026] Among them, Q l+1 represents the skeleton feature tensor of the l+1th layer, Z l represents the skeleton adjacency tensor used to perform the l-th layer hypergraph convolution, ×3 represents the tensor M multiplication operator, Q l represents the skeleton feature tensor of the lth layer, M represents the time transformation matrix, e represents the M transformation operation, and ○ represents the area operation.

[0027] As an optional embodiment of the present invention, optionally, in step S306, the expression for performing feature transformation operation and nonlinear activation operation on the integration result is:

[0028] Q l+1 =σ(Z l ×3Q l ×3W)

[0029] Among them, Q l+1 represents the skeleton feature tensor of the l+1th layer, σ() represents the nonlinear activation function, and Z l represents the skeleton adjacency tensor used to perform the l-th layer hypergraph convolution, ×3 represents the tensor M multiplication operator, Q l represents the skeleton feature tensor of layer l, and W represents the feature transformation tensor.

[0030] As an optional embodiment of the present invention, optionally, the expression for adaptively fusing the skeleton latent features in step S4 is:

[0031] Q=θ 0 Q 0 +θ 1 Q 1 +θ 2 Q 2 +…+θL Q L

[0032] Among them, Q represents the final extracted video skeleton features, θ 0 ,θ 1 ,θ 2 , ..., θ L Represents the weight coefficient of skeleton features at different layers, Q 0 , Q 1 , Q 2 , Q L Represent the skeleton feature tensors of layer 0, layer 1, layer 2, and layer L respectively.

[0033] On the other hand, the present invention also provides a video skeleton feature extraction device based on a hypergraph neural network, the device comprising the video skeleton feature extraction method based on a hypergraph neural network; the device also includes:

[0034] A pedestrian video data receiving module, used for receiving pedestrian video data;

[0035] A data storage module, connected to the pedestrian video data receiving module, and used to store the pedestrian video data;

[0036] A video skeleton feature extraction module, connected to the data storage module, for extracting video skeleton features using a hypergraph neural network;

[0037] The objective function construction module is connected to the video skeleton feature extraction module and is used to construct a loss function.

[0038] As another optional embodiment of the present invention, optionally, the video skeleton feature extraction module includes:

[0039] An initialization unit, used for initializing the relevant parameters for extracting the skeleton features of the video;

[0040] A skeleton adjacency tensor construction unit, used to construct the adjacency tensor of multi-objective skeleton spatiotemporal patterns;

[0041] A video skeleton latent feature extraction unit, connected to the skeleton adjacency tensor construction unit and the initialization unit, for extracting skeleton latent features of high-order spatiotemporal patterns within each skeleton in the pedestrian video data based on the adjacency tensor and the initialization parameters;

[0042] A latent feature adaptive fusion unit, connected to the video skeleton latent feature extraction unit, for fusing skeleton latent features of high-order spatiotemporal patterns to obtain the video skeleton features;

[0043] The output unit is connected to the latent feature adaptive fusion unit and is used to extract the video skeleton feature with the highest accuracy.

[0044] As another optional embodiment of the present invention, optionally, the data storage module includes:

[0045] A pedestrian video data storage unit, used to store the pedestrian video data;

[0046] A skeleton adjacency tensor storage unit for the adjacency tensor of the multi-objective skeleton spatiotemporal pattern;

[0047] The skeleton feature storage unit is used to store the video skeleton features.

[0048] The beneficial effect of the present invention is that, by introducing a hypergraph neural network, the present invention can more effectively capture the high-order spatiotemporal dependencies of skeleton features in a video. Compared with the traditional graph convolutional network-based method, the hypergraph neural network can handle more complex graph structures, thereby achieving more refined feature expression in the skeleton feature extraction process. In addition, by introducing an adaptive fusion mechanism, the present invention can dynamically adjust the weights of features at different layers according to the intrinsic properties of the skeleton features, further improving the accuracy and robustness of feature extraction.

[0049] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0051] Figure 1 It is a flow chart of a video skeleton feature extraction method based on a hypergraph neural network according to embodiment 1 of the present invention.

[0052] Figure 2 This is another flow chart of the video skeleton feature extraction method based on the hypergraph neural network in Example 1 of the present invention.

[0053] Figure 3 It is a structural schematic diagram of a video skeleton feature extraction device based on a hypergraph neural network according to embodiment 2 of the present invention. DETAILED DESCRIPTION

[0054] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0055] Example 1

[0056] like Figure 1 As shown, a video skeleton feature extraction method based on a hypergraph neural network, the method comprising:

[0057] S1. Collect pedestrian video data and obtain a target skeleton set in the pedestrian data;

[0058] It should be noted that the pedestrian video data collected in step S1 can come from a variety of scenes, such as urban surveillance cameras, traffic intersections, shopping malls, etc. These video data usually contain multiple pedestrians and are shot under different lighting, occlusion and background complexity conditions. The joints of pedestrians are used as nodes, and the skeletons of pedestrians are used as edges to generate the skeletons of each pedestrian, which is the target skeleton set.

[0059] S2. constructing a loss function based on the target skeleton set;

[0060] It should be noted that when constructing the loss function in step S2, based on the input behavior video data, this embodiment mainly focuses on the recognition accuracy of target behavior abnormalities. For a target skeleton, let X i represents the learned video skeleton features, where i refers to the i-th skeleton and its true behavior label is Y i , based on the skeleton features learned from the video skeleton features, a decoder (such as a fully connected network) can be used to obtain the predicted target behavior label Then optimize the following cross entropy loss function:

[0061]

[0062] Among them, DisFunction represents the error function that measures the prediction accuracy;

[0063] log() represents the logarithmic function.

[0064] S3, constructing a skeleton adjacency tensor of multi-target interaction of the target skeleton, and extracting skeleton hidden features using a hypergraph neural network based on the skeleton adjacency tensor;

[0065] It should be noted that when constructing the skeleton adjacency tensor in step S3, the spatial and temporal relationship between pedestrian skeletons is taken into account. By analyzing the relative position and movement trend between skeleton points, an adjacency tensor that can reflect the interaction between skeletons can be constructed. This tensor not only contains the spatial information of skeleton points, but also incorporates the changes in the temporal dimension, so as to capture the dynamic changes of skeleton features.

[0066] S4, adaptively fusing the skeleton latent features to obtain video skeleton latent features;

[0067] It should be noted that when performing adaptive fusion in step S4, the attention mechanism in deep learning is used to dynamically adjust the weights of features at different layers according to the intrinsic properties of skeleton features. This mechanism enables the model to pay more attention to features that are more important for behavior recognition, thereby improving the accuracy and robustness of feature extraction.

[0068] S5. Determine whether the loss value of the loss function converges or whether the number of iterations reaches an upper limit;

[0069] If so, output the video skeleton features;

[0070] If not, the loss function is iteratively optimized using stochastic gradient descent.

[0071] like Figure 2 As shown, it should be noted that in step S5, the difference between the predicted label and the observed true label is used as the optimization target to approximate the true label and learn better video skeleton features; and L2 regularization is used in the loss function to constrain the video skeleton latent features, time transformation matrix parameters and feature conversion matrix parameters to be trained, so as to enhance the generalization ability of the model on the test set;

[0072] Then, the basis for judging whether ω has converged on the pedestrian video dataset is to compare the value of ω before the start of this round of training with the value before the start of the previous round of training, and whether the absolute value of the difference between the two is less than the convergence judgment threshold e; if it is less than the threshold, it is judged to have converged, otherwise, it is judged to have not converged, or the number of rounds in the entire training process reaches the upper limit, otherwise it is judged to have not converged;

[0073] In the case of non-convergence, this embodiment uses a gradient descent method such as Adam to iteratively optimize ω; finally, the video skeleton feature result after adaptive fusion with the highest accuracy is obtained.

[0074] To sum up, the method of this embodiment is specifically used for pedestrian video data in complex video surveillance scenarios. It can effectively process the non-Euclidean topological structure between skeletons and joints in pedestrian videos and reduce the modeling loss of spatiotemporal information in video streams. It can then extract the video skeleton features with the strongest representation capabilities from different pedestrian video data. It overall solves the problem of discovering multi-target abnormal behaviors in massive pedestrian video data in complex video surveillance scenarios, and provides a reliable basis for the formulation of pedestrian safety regulations and abnormal behavior analysis in various scenarios.

[0075] As an optional embodiment of the present invention, optionally, the expression of the loss function in step S2 is:

[0076]

[0077] Among them, ω represents the loss value;

[0078] i represents the i-th target skeleton;

[0079] Y i Indicates the real behavior label;

[0080] log() represents the logarithmic function;

[0081] Represents the predicted target behavior label;

[0082] λ represents the regularization factor;

[0083] || || F represents the F-norm;

[0084] Q 0 Represents the initialized skeleton latent feature tensor;

[0085] M represents the time conversion matrix;

[0086] W represents the feature transformation tensor.

[0087] It should be noted that L2 regularization is used in the loss function to constrain the optimization process to avoid overfitting of the model.

[0088] As an optional embodiment of the present invention, optionally, constructing the skeleton adjacency tensor of the target skeleton in step S3 includes:

[0089] S301, taking the joints of the target as nodes and the skeletons of the target as edges, constructing a skeleton topology graph based on the nodes and edges, and obtaining a skeleton topology graph sequence based on a time dimension;

[0090] S302: Standardize the skeleton topology map sequence to obtain a skeleton adjacency tensor.

[0091] It should be noted that an adjacency tensor describing the spatiotemporal pattern of skeleton joints in pedestrian videos in complex video surveillance scenarios is constructed. The target skeleton is highly robust to changes in illumination, complex backgrounds, and body scales, and is often used as a necessary object for target action recognition. For multi-target skeletons in the video stream, a skeleton topology graph is constructed with target joints as nodes and bones as edges, and then a skeleton sequence is naturally formed in the time dimension, thereby constructing a high-order tensor describing the complex changes in the video skeleton. In order to avoid the problems of numerical instability that may arise during the training process when directly using this skeleton adjacency tensor to describe the complex non-Euclidean topological structure information of multiple targets, it is necessary to normalize the initially constructed video skeleton adjacency tensor (it can be symmetric normalization or random walk normalization). For example, the normalized adjacency tensor is used as the input of the skeleton latent feature extraction unit, and the specific operation can be as follows:

[0092]

[0093] in, represents the symmetric normalized skeleton adjacency tensor;

[0094] D represents the degree of tension according to its construction;

[0095] The importance of each joint is described;

[0096] Represents area operations;

[0097] A represents the initially constructed skeleton adjacency tensor.

[0098] As an optional embodiment of the present invention, optionally, extracting skeleton latent features using a hypergraph neural network in step S3 includes:

[0099] S303, performing a tensor multiplication conversion operation on the joint adjacency tensor, and extracting the skeleton latent features of each layer;

[0100] It should be noted that, based on the construction of the joint adjacency tensor describing the complex multi-target interaction, a new hypergraph convolution module is then implemented based on the general tensor multiplication framework. First, the skeleton adjacency tensor supplemented with time domain information is obtained by tensor multiplication, that is, the adjacency tensor of the next layer is expressed as:

[0101]

[0102] Among them, Z l+1 Represents the skeleton adjacency tensor used when performing the l+1th layer of hypergraph convolution. represents the initial sparse adjacency tensor, each layer shares the same parameters, which are initialized as random variables and then optimized during training. Specifically, the specific calculation operations of the above conversion process are:

[0103]

[0104] in, Indicates the z on the tth video slice l A single element in

[0105] T represents the number of video slices;

[0106] j represents the video slice index;

[0107] m t,j Represents a single element in M;

[0108] Indicates the z on the i-th video slice l A single element in .

[0109] S304, performing a graph convolution operation on each layer of skeleton latent features to obtain the spatial dependency relationship of the joints;

[0110] It should be noted that it is difficult to judge the target behavior through the video slices in a single frame. The above conversion operation can make the skeleton interaction in each video slice realize the intercommunication perception in the time domain, thereby enhancing the accuracy of the skeleton features. Similarly, in order to make the skeleton hidden features on each video slice intercommunication, the skeleton hidden features of each layer also perform the following tensor transformation:

[0111] Q l+1 =Q l e M

[0112] According to the above formula, the elements in each layer of video skeleton hidden features are specifically calculated as:

[0113]

[0114] in, Indicates Q on the tth video slice l A single element in

[0115] Indicates Q on the jth video slice l A single element in .

[0116] According to the above tensor calculation, we can get the skeleton adjacency tensor and skeleton hidden feature tensor after tensor multiplication and transformation, and use them as input to perform graph convolution operation to capture the spatial dependency of joints, that is, perform the following calculation:

[0117]

[0118] S305, capturing hidden features of similar joints in each layer of the skeleton based on the spatial dependency relationship;

[0119] It should be noted that, according to the above step S304, each joint can capture the hidden features of other joints in a single target or a joint of another target that is close in time and space distance. The graph convolution operation in the above formula is specifically calculated as:

[0120]

[0121] Among them, u represents the joint node directly connected to p;

[0122] N(p,l+1,t) represents the number of joints found to interact with joint p in the l+1th layer on the tth video slice after tensor multiplication transformation;

[0123] Indicates the z on the tth video slice l+1 A single element in

[0124] Indicates Q on the tth video slice l+1 A single element in

[0125] S306, integrating the skeleton latent features of each layer, the spatial dependency of the joints and the latent features of the adjacent joints, and performing feature transformation operations and nonlinear activation operations on the integration results to obtain skeleton latent features.

[0126] As an optional embodiment of the present invention, optionally, in step S306, the expression for integrating the skeleton latent features of each layer, the spatial dependency of the joints and the latent features of the adjacent joints is:

[0127]

[0128] Among them, Q l+1 Represents the skeleton feature tensor of the l+1th layer;

[0129] Z l Represents the skeleton adjacency tensor used when performing the l-th layer of hypergraph convolution;

[0130] ×3 represents the tensor M multiplication operator;

[0131] Q l Represents the skeleton feature tensor of layer l;

[0132] M represents the time conversion matrix;

[0133] e represents the M conversion operation;

[0134] Represents area operation.

[0135] As an optional embodiment of the present invention, optionally, in step S306, the expression for performing feature transformation operation and nonlinear activation operation on the integration result is:

[0136] Q l+1 =σ(Z l ×3Q l ×3W)

[0137] Among them, Q l+1 Represents the skeleton feature tensor of the l+1th layer;

[0138] σ() represents a nonlinear activation function;

[0139] Z l Represents the skeleton adjacency tensor used when performing the l-th layer of hypergraph convolution;

[0140] ×3 represents the tensor M multiplication operator;

[0141] Ql Represents the skeleton feature tensor of layer l;

[0142] W represents the feature transformation tensor.

[0143] It should be noted that the above-mentioned hypergraph convolution method extends the ordinary graph convolution on the static graph to three-dimensional space, so that it can simultaneously capture the skeleton features in pedestrian videos in complex video surveillance scenarios. Through multi-layer graph convolution operations, each joint can perceive the information of its multi-hop spatiotemporal neighbors. Different from the traditional method of simply combining the timing module and the static graph convolution network to extract the video skeleton features, the hypergraph convolution operation of this embodiment can capture richer joint interaction structure features, thereby significantly reducing information loss.

[0144] As an optional embodiment of the present invention, optionally, the expression for adaptively fusing the skeleton latent features in step S4 is:

[0145] Q=θ 0 Q 0 +θ 1 Q 1 +θ 2 Q 2 +…+θ L Q L

[0146] Among them, Q represents the video skeleton features finally extracted;

[0147] θ 0 ,θ 1 ,θ 2 , ..., θ L Represents the weight coefficients of skeleton features at different layers;

[0148] Q 0 Represents the skeleton feature tensor of layer 0;

[0149] Q 1 Represents the skeleton feature tensor of layer 1;

[0150] Q 2 Represents the skeleton feature tensor of layer 2;

[0151] Q L Represents the skeleton feature tensor of the Lth layer.

[0152] It should be noted that Q is the final extracted video skeleton latent feature, which contains the latent feature information of each joint itself and all its K-hop temporal and spatial neighbors. 0 ,θ 1 ,θ 2 , ..., θ LIt is the weight coefficient of the skeleton latent features at different layers. In this embodiment, it is adaptively trained as a parameter, and the self-attention mechanism can also be used for more refined calculations. Compared with the method of not fusing latent features or giving the same importance to the latent features of different layers of skeletons, the latent feature adaptive fusion unit of this embodiment can more accurately extract neighborhood features of different orders, so that the final skeleton latent features also have higher expression capabilities. After using the joint latent features of each skeleton, the latent features of each skeleton can be directly obtained by using pooling methods such as a fully connected network, and then a classifier can be used to obtain the predicted label of each skeleton, that is, whether the behavior is abnormal.

[0153] Example 2

[0154] like Figure 3 As shown, a video skeleton feature extraction device based on a hypergraph neural network, the device includes the video skeleton feature extraction method based on a hypergraph neural network; the device also includes:

[0155] A pedestrian video data receiving module, used for receiving pedestrian video data;

[0156] A data storage module, connected to the pedestrian video data receiving module, and used to store the pedestrian video data;

[0157] A video skeleton feature extraction module, connected to the data storage module, for extracting video skeleton features using a hypergraph neural network;

[0158] It should be noted that the video skeleton feature extraction module receives the pedestrian video data stored in the data storage module, and successively executes the latent feature and skeleton adjacency tensor initialization unit, the skeleton feature spatiotemporal pattern extraction unit and the latent feature adaptive fusion unit of the video skeleton feature extraction module, and then stores the extracted fused latent features into the skeleton feature storage unit in the data storage module as the final video skeleton features.

[0159] The objective function construction module is connected to the video skeleton feature extraction module and is used to construct a loss function.

[0160] The objective function construction module selects the most suitable distance metric function according to the data pattern of the pedestrian video to evaluate the operation and optimization process of the skeleton feature and latent feature extraction module. By selecting a suitable objective function, this module can effectively guide the training process of the skeleton feature extraction module to ensure that the extracted skeleton features have high accuracy and generalization ability. The objective function construction module continuously adjusts parameters during the training process to minimize the loss function, thereby optimizing the performance of the skeleton feature extraction module.

[0161] like Figure 3As shown, the video skeleton feature extraction module is used to execute the video skeleton feature extraction process based on the hypergraph neural network, and store the extracted video skeleton features into the skeleton feature storage unit in the data storage module; the video skeleton feature extraction module includes: an initialization unit, which initializes the relevant extraction parameters involved in the video skeleton hidden feature extraction process, such as the hidden features to be trained, etc.; a skeleton adjacency tensor construction unit, which constructs an adjacency tensor describing the multi-target skeleton spatiotemporal pattern in the video stream according to the pedestrian video network stored in the pedestrian video data storage unit in the data storage module and the relevant parameters initialized by the initialization unit; the video skeleton hidden feature extraction unit extracts the hidden features of the high-order spatiotemporal pattern contained in each skeleton contained in the pedestrian video data; the hidden feature adaptive fusion unit obtains the overall hidden features according to the different order hidden features extracted by the video skeleton hidden feature extraction unit and the initialized structural hidden features; the output unit extracts the most accurate fused hidden feature result in the hidden feature adaptive fusion unit, and stores it in the skeleton feature storage unit in the data storage module.

[0162] As another optional embodiment of the present invention, optionally, the video skeleton feature extraction module includes:

[0163] An initialization unit, used for initializing the relevant parameters for extracting the skeleton features of the video;

[0164] Initialize the relevant parameters involved in the video skeleton feature extraction process, including the dimension of the latent feature space, the number of hypergraph convolution layers, the perception window size of the time mixing matrix, the initialized skeleton latent feature tensor, the adaptive parameters of the feature fusion stage, the learning rate, the regularization factor, the training batch size, the current number of iterations, the maximum number of iterations, the termination convergence threshold, etc.; among them, the initial values ​​of the dimension of the latent feature space, the number of hypergraph convolution layers, and the perception window size of the time mixing matrix are positive integers; the initialized skeleton latent feature tensor is a tensor with p rows (number of joints), S columns (latent feature dimensions), and T time slices; the adaptive parameters of the feature fusion stage are adaptively learned, which actually represents the distribution ratio of neighbor information of different orders; the learning rate τ is initialized is a very small number, which is the initial step size of gradient descent when using the Adam optimizer during training; the regularization factor λ is a hyperparameter that controls the degree of regularization. It can prevent model overfitting and is generally set to a very small number; the training batch size b represents the batch size of each round in the small batch training process, which is generally set to an exponential multiple of 2, such as 2048; the current training round number is initialized to an integer 0, indicating that training has not yet started. During training, the value increases with the increase in the number of training rounds, indicating that the current model is in the nth training iteration process; the maximum number of iterations N is the maximum number of training rounds, and the parameter is initialized to a large positive integer, such as 500; the termination convergence threshold e represents the lower limit of the error between two training rounds. If the actual value is lower than this lower limit, training is stopped.

[0165] A skeleton adjacency tensor construction unit, used to construct the adjacency tensor of multi-objective skeleton spatiotemporal patterns;

[0166] It should be noted that the skeleton adjacency tensor construction unit constructs an adjacency tensor describing the spatiotemporal pattern of multi-target skeletons in the video stream based on the skeleton information in the pedestrian video data. The unit first analyzes the skeletons in the video data, identifies the spatiotemporal relationships between the skeletons, and then constructs the adjacency tensor based on these relationships. The construction process of the adjacency tensor involves the connection relationship between the skeleton points and the dynamic characteristics of the skeleton points over time. In this way, the skeleton adjacency tensor construction unit can capture the spatiotemporal pattern of the skeleton in the video, providing a basis for subsequent feature extraction.

[0167] A video skeleton latent feature extraction unit, connected to the skeleton adjacency tensor construction unit and the initialization unit, for extracting skeleton latent features of high-order spatiotemporal patterns within each skeleton in the pedestrian video data based on the adjacency tensor and the initialization parameters;

[0168] The video skeleton latent feature extraction unit uses a hypergraph neural network to extract the high-order spatiotemporal pattern latent features of each skeleton contained in the pedestrian video data. The unit processes the skeleton adjacency tensor through a hypergraph convolution operation to extract the spatiotemporal pattern in the skeleton features. The hypergraph convolution operation can simultaneously consider the connection relationship between skeleton points and the dynamic characteristics of skeleton points changing over time, thereby extracting richer and more accurate skeleton features.

[0169] A latent feature adaptive fusion unit, connected to the video skeleton latent feature extraction unit, for fusing skeleton latent features of high-order spatiotemporal patterns to obtain the video skeleton features;

[0170] The latent feature adaptive fusion unit obtains the overall latent feature by fusing the different order latent features extracted by the video skeleton latent feature extraction unit and the initialized structural latent features. This unit uses an adaptive mechanism to dynamically adjust the fusion weights of different order latent features according to the spatiotemporal characteristics of the skeleton features, thereby obtaining a more accurate and expressive overall latent feature. The adaptive fusion mechanism can effectively integrate the information of different order latent features and improve the expressiveness of the final skeleton features.

[0171] The output unit is connected to the latent feature adaptive fusion unit and is used to extract the video skeleton feature with the highest accuracy.

[0172] The output unit extracts the most accurate fused latent feature result from the latent feature adaptive fusion unit and stores it in the skeleton feature storage unit in the data storage module. This unit is responsible for selecting the most representative and discriminative features from the fused latent features as the final video skeleton features. The output unit selects the optimal feature result by evaluating the performance of different fused latent features for use by subsequent classifiers.

[0173] As another optional embodiment of the present invention, optionally, the data storage module includes:

[0174] A pedestrian video data storage unit, used to store the pedestrian video data;

[0175] A skeleton adjacency tensor storage unit for the adjacency tensor of the multi-objective skeleton spatiotemporal pattern;

[0176] The skeleton feature storage unit is used to store the video skeleton features.

[0177] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A video skeleton feature extraction method based on hypergraph neural network, characterized in that: The method comprises: S1. Collect pedestrian video data and obtain a target skeleton set in the pedestrian data; S2. constructing a loss function based on the target skeleton set; S3, constructing a skeleton adjacency tensor of multi-target interaction of the target skeleton, and extracting skeleton hidden features using a hypergraph neural network based on the skeleton adjacency tensor; S4, adaptively fusing the skeleton latent features to obtain video skeleton latent features; S5. Determine whether the loss value of the loss function converges or whether the number of iterations reaches an upper limit; If so, output the video skeleton features; If not, the loss function is iteratively optimized using stochastic gradient descent.

2. A video skeleton feature extraction method based on a hypergraph neural network as claimed in claim 1, characterized in that: The expression of the loss function in step S2 is: Among them, ω represents the loss value, i represents the i-th target skeleton, and Y i represents the true behavior label, log() represents the logarithmic function, represents the predicted target behavior label, λ represents the regularization factor, F represents the F norm, Q 0 represents the initialized skeleton latent feature tensor, M represents the time transformation matrix, and W represents the feature transformation tensor.

3. The video skeleton feature extraction method based on hypergraph neural network as claimed in claim 1, characterized in that: Constructing the skeleton adjacency tensor of the target skeleton in step S3 includes: S301, taking the joints of the target as nodes and the skeletons of the target as edges, constructing a skeleton topology graph based on the nodes and edges, and obtaining a skeleton topology graph sequence based on a time dimension; S302: Standardize the skeleton topology map sequence to obtain a skeleton adjacency tensor.

4. A video skeleton feature extraction method based on a hypergraph neural network as described in claim 1 or 3, characterized in that: In step S3, the skeleton latent features are extracted using the hypergraph neural network, including: S303, performing a tensor multiplication conversion operation on the joint adjacency tensor, and extracting the skeleton latent features of each layer; S304, performing a graph convolution operation on each layer of skeleton latent features to obtain the spatial dependency relationship of the joints; S305, capturing hidden features of similar joints in each layer of the skeleton based on the spatial dependency relationship; S306, integrating the skeleton latent features of each layer, the spatial dependency of the joints and the latent features of the adjacent joints, and performing feature transformation operations and nonlinear activation operations on the integration results to obtain skeleton latent features.

5. A video skeleton feature extraction method based on a hypergraph neural network as claimed in claim 4, characterized in that: In step S306, the expression for integrating the skeleton latent features of each layer, the spatial dependency of the joints and the latent features of the adjacent joints is: Among them, Q l+1 represents the skeleton feature tensor of the l+1th layer, Z l represents the skeleton adjacency tensor used to perform the l-th layer hypergraph convolution, ×3 represents the tensor M multiplication operator, Q l represents the skeleton feature tensor of the lth layer, M represents the time transformation matrix, e represents the M transformation operation, Represents area operation.

6. A video skeleton feature extraction method based on a hypergraph neural network as claimed in claim 4, characterized in that: In step S306, the expression for performing feature transformation operation and nonlinear activation operation on the integration result is: Ql+1=σ(Zl×3Ql×3W) Among them, Q l+1 represents the skeleton feature tensor of the l+1th layer, σ() represents the nonlinear activation function, and Z l represents the skeleton adjacency tensor used to perform the l-th layer hypergraph convolution, ×3 represents the tensor M multiplication operator, Q l represents the skeleton feature tensor of layer l, and W represents the feature transformation tensor.

7. The video skeleton feature extraction method based on hypergraph neural network as claimed in claim 1, characterized in that: The expression for adaptively fusing the skeleton latent features in step S4 is: Q=θ0·Q0+θ1·Q1+θ2·Q2+…+θL·QL Among them, Q represents the final extracted video skeleton features, θ 0 ,θ 1 ,θ 2 , ..., θ L Represents the weight coefficient of skeleton features at different layers, Q 0 , Q 1 , Q 2 , Q L Represent the skeleton feature tensors of layer 0, layer 1, layer 2, and layer L respectively.

8. A video skeleton feature extraction device based on a hypergraph neural network, characterized in that: The device comprises a video skeleton feature extraction method based on a hypergraph neural network as claimed in any one of claims 1 to 7; The device also includes: A pedestrian video data receiving module, used for receiving pedestrian video data; A data storage module, connected to the pedestrian video data receiving module, and used to store the pedestrian video data; A video skeleton feature extraction module, connected to the data storage module, for extracting video skeleton features using a hypergraph neural network; The objective function construction module is connected to the video skeleton feature extraction module and is used to construct a loss function.

9. The video skeleton feature extraction device based on hypergraph neural network as claimed in claim 8, characterized in that: The video skeleton feature extraction module comprises: An initialization unit, used for initializing the relevant parameters for extracting the skeleton features of the video; A skeleton adjacency tensor construction unit, used to construct the adjacency tensor of multi-objective skeleton spatiotemporal patterns; A video skeleton latent feature extraction unit, connected to the skeleton adjacency tensor construction unit and the initialization unit, for extracting skeleton latent features of high-order spatiotemporal patterns within each skeleton in the pedestrian video data based on the adjacency tensor and the initialization parameters; A latent feature adaptive fusion unit, connected to the video skeleton latent feature extraction unit, for fusing skeleton latent features of high-order spatiotemporal patterns to obtain the video skeleton features; The output unit is connected to the latent feature adaptive fusion unit and is used to extract the video skeleton feature with the highest accuracy.

10. The video skeleton feature extraction device based on hypergraph neural network according to claim 9, characterized in that: The data storage module comprises: A pedestrian video data storage unit, used for storing the pedestrian video data; A skeleton adjacency tensor storage unit for the adjacency tensor of the multi-objective skeleton spatiotemporal pattern; The skeleton feature storage unit is used to store the video skeleton features.

Images